Forum Discussion
Bob_10976
Feb 18, 2011Nimbostratus
iRule to Block Google and other Search Engines
Hello all,
We would like to use an iRule to block google and other search engines from crawling our sites and was hopeing someone could point me in the right direction. In the past we would use robot.txt files, however with some major chances to our envirnoment it seems it would be simply to apply an iRule.
We did some research and found the below code, however I'm not sure how old it is or if it will work with our LTM, ver 10.2.x, The site is: http://blog.regisdonovan.org/2010/1...ic-by.html
when HTTP_REQUEST {
if { [HTTP::header "User-Agent"] contains "AnnoyingRobot" }
{
drop
return }
}
Thanks,
Bob
- Steve_Brown_882Historic F5 AccountWhat you have will work for a single user agent, but you might want to add a number of them. You might try something like this and add all the user agents you want to block. I found a good list of these user agents here.
- Bob_10976NimbostratusThanks!! stjBrown.. I'll give that a shot...
- hooleylistCirrostratusYou could also add the user-agent strings to a datagroup and then use the class (v10) or matchclass (v9) commands to look up the user-agent header value against the datagroup.
- Steve_Brown_882Historic F5 AccountI agree with Aaron that a datagroup is another good way to do this and it would allow you to keep a much larger txt list that you simply upload and apply.
- Bob_10976NimbostratusThanks again for the input.. I like the idea of the datagroup, I think it would be much easier to create and maintain. However the question I have is the list I create, where do I store it? Do i simply upload the txt file to the LTMs and if so would I need to put it in a specfic directory..
- hooleylistCirrostratusHi Bob,
class namevalue { "name1" := "value", "name2" := "value", }
class namevalue { { "name1" { "value" } "name2" { "value" } } }
Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects