Forum Discussion
Fletcher_Cocquy
Nimbostratus
May 14, 2012iRule to redirect crawlers and bots
Hi - by policy we are open to crawlers and bots to get our sites indexed.
Sometimes they take up all our apache threads (despite robots.txt settings)
Is there an iRule to key off the us...
nathe
Cirrocumulus
May 14, 2012Fletcher
Check out this Codeshare link using the switch command: https://devcentral.f5.com/wiki/iRules.ControllingBots.ashx
You can amend accordingly, of course, to control which brand of bots.
Rgds
N
Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects