Forum Discussion
Eduardo_Saito_1
Nimbostratus
Nov 06, 2006Limit number of connections for googlebot
Hello!
I'm trying to write an irule that limits the number of connections that Google Bot can use. I found some good scripts that help me acomplish this (one made by Joe and the other one from tips).
Here go the script. I didn't put it on production yet since I don't know the impact it will have on cpu utilization.
Hope that you guys can help me.
Here go what I was thinking. I don't know if there's a more efficient way to do this.
when RULE_INIT {
array set ::active_clients { }
}
when CLIENT_ACCEPTED {
switch -glob [string tolower [HTTP::header "User-Agent"]] {
"*googlebot*" {
set client_ip [IP::remote_addr]
if { [info exists ::active_clients($client_ip)] } {
if {$::active_clients($client_ip) > 10 } {
reject
log local0. "Reject GOOGLEBOT IP $client_ip ($::active_clients($client_ip))"
return
} else {
incr ::active_clients($client_ip)
}
} else {
set ::active_clients($client_ip) 1
}
}
}
}
when CLIENT_CLOSED {
switch -glob [string tolower [HTTP::header "User-Agent"]] {
"*googlebot*" {
set client_ip [IP::remote_addr]
if { [info exists ::active_clients($client_ip)] } {
incr ::active_clients($client_ip) -1
if { $::active_clients($client_ip) <= 0 } {
unset ::active_clients($client_ip)
}
}
}
}
}
- Colin_Walker_12Historic F5 AccountIt looks like the logic is pretty decent, after taking a quick peek. You shouldn't need those two switch statements, though, if there is only one case in each of them. Replacing them each with a single if structure would probably net you a few cycles.
- Eduardo_Saito_1
Nimbostratus
Thanks Colin. - John_Pope_42012
Nimbostratus
any follow up on this? - Deb_Allen_18Historic F5 AccountThis codeshare contribution will allow you to limit by rateclass any UserAgent on the list: http://devcentral.f5.com/wiki/default.aspx/iRules/ControllingBots.html (Click here)
- Deb_Allen_18Historic F5 AccountYeah, just remove the switch statement & you'd have something like this, which won't discriminate against just googlebot:
when RULE_INIT { array set ::active_clients { } } when CLIENT_ACCEPTED { set client_ip [IP::remote_addr] if { [info exists ::active_clients($client_ip)] } { if {$::active_clients($client_ip) > 10 } { reject log local0. "Reject GOOGLEBOT IP $client_ip ($::active_clients($client_ip))" return } else { incr ::active_clients($client_ip) } } else { set ::active_clients($client_ip) 1 } } when CLIENT_CLOSED { set client_ip [IP::remote_addr] if { [info exists ::active_clients($client_ip)] } { incr ::active_clients($client_ip) -1 if { $::active_clients($client_ip) <= 0 } { unset ::active_clients($client_ip) } } }
- f5now_28704
Nimbostratus
right on, thanks deb now I cant wait till those bastards try and shoot all there connection at me again!!!!
Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects