For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

Peter_Reilly's avatar
Peter_Reilly
Icon for Nimbostratus rankNimbostratus
Feb 28, 2008

Disallow Robots

Will the following iRule work for disallowing robots?     Thanks,     when HTTP_REQUEST {   if {[HTTP::path] eq "/robots.txt"}   {   HTTP::respond 200 content "   User-agent...
application delivery
dev
devops
iRules
hoolio's avatar
hoolio
Icon for Cirrostratus rankCirrostratus
Jan 23, 2012
If you want to check for any path ending in /robots.txt, you could use: 


when HTTP_REQUEST {
   if {[string tolower [HTTP::path]] ends_with "/robots.txt"}{
      HTTP::respond 200 content "User-agent: *\r\nDisallow: /"
   }
}

But the common use is for a robot to only request /robots.txt.

http://www.robotstxt.org/robotstxt.html

Aaron