Forum Discussion
Peter_Reilly
Nimbostratus
Feb 28, 2008Disallow Robots
Will the following iRule work for disallowing robots?
Thanks,
when HTTP_REQUEST {
if {[HTTP::path] eq "/robots.txt"}
{
HTTP::respond 200 content "
User-agent...
hoolio
Cirrostratus
Feb 29, 2008That is close. robots.txt is a text file, so you don't need to send HTML tags in the response. You can use \r\n to insert a new line:
when HTTP_REQUEST {
if {[string tolower [HTTP::path]] eq "/robots.txt"}{
HTTP::respond 200 content "User-agent: *\r\nDisallow: /"
}
}It looks like Google has a robots.txt analyzer you might be able to use to verify your rule:
How do I check that my robots.txt file is working as expected? (Click here)
Aaron
Help guide the future of your DevCentral Community!
What tools do you use to collaborate? (1min - anonymous)Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects
