Forum Discussion
W__Ho_172333
Nimbostratus
Aug 17, 2015Remove server banner when block the robots
Hi all,
I have a requirement to block the robots. I have the following code that works just fine:
when HTTP_REQUEST {
if {
....
}
some actions....
elseif {
[string ...
Michael__
Nimbostratus
Aug 18, 2015ok i placed the noserver option before the content ... now it is working
when HTTP_REQUEST {
if { .... }{
some actions....
} elseif {[string tolower [HTTP::uri]] equals "/robots.txt"}{
HTTP::respond 200 content "User-agent: *\r\nDisallow: /" noserver "Content-Type" "text/plain"
log local0. "Attempt by [IP::client_addr] crawling our site"
}
}
a quick test:
[root@f5vm01:] ~ openssl s_client -quiet -connect f5vmvs01.local:443
...
GET /robots.txt HTTP/1.1
Host: f5vmvs01.local
HTTP/1.0 200 OK
Content-Type: text/plain
Connection: Keep-Alive
Content-Length: 26
User-agent: *
Disallow: /
- W__Ho_172333Aug 19, 2015
Nimbostratus
Hi Michael, It works! Thank you so much!
Help guide the future of your DevCentral Community!
What tools do you use to collaborate? (1min - anonymous)Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects