Fletcher_Cocquy
May 14, 2012Nimbostratus
iRule to redirect crawlers and bots
Hi - by policy we are open to crawlers and bots to get our sites indexed.
Sometimes they take up all our apache threads (despite robots.txt settings)
Is there an iRule to key off the user agent (eg bingbot)
User agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
to segment off this traffic to a separate pool from user traffic
Thereby isolating the users from bots taking up all the threads?
thanks!