Forum Discussion
MocoSpace_63233
Nimbostratus
NimbostratusRedirecting User-Agent Googlebot to new pool or server
Ok - simple problem but I am a newbie. We are getting killed with Googlebot-Mobile spiders. Chewing up all ouer connections. We don't want to stop it but ultimately we may want to rate limit it or something similar. For now we what to redirect the spider to a non-production server so we can better evaluate why it's behaving the way it is. Wrote a quick and dirtyt iRule but does not seem to be working. Ideas on what to look for.
when HTTP_REQUEST {
if {[HTTP::header "User-Agent"] contains "*Googlebot-Mobile.*"} {
pool HTTP-SearchEngines
}
}
Also any hints on running the traffic generator to test this iRule would also be useful.
THanks!
Cheers
Mark
- The "contains" operator will interpret the * as a literal character.
But the wildcarding is implied with "contains", so if you remove the *, I think your iRule will work as expected.
You can insert headers with the iRuler traffic generation dialog by populating the Headers/Name and Headers/Value fields & popping them into the list on the right.
HTH
/deb 
Blake_Traister2So I already have a similar iRule. We parse against a large datalist of "unapproved" users and redirect them to a different set of servers. How would I go about rewriting the User-agent as well to something like "x-bot"?
THanks!- Blake_Traister2So I already have a similar iRule. We parse against a large datalist of "unapproved" users and redirect them to a different set of servers. How would I go about rewriting the User-agent as well to something like "x-bot"?
THanks! 
hoolioCirrostratus
There is a codeshare entry which gives two options for selecting a pool based on the user-agent header:
Controlling bots (Click here)
Aaron- hoolioActually, I think I misread your request. You want to rewrite the user-agent header to a static value? If so, you can use something like this:
when HTTP_REQUEST { if {$something == 1}{ Replace the User-Agent header (inserting it if it doesn't exist already) HTTP::header replace "User-Agent" "x-bot" } }
Aaron
