Forum Discussion

Mike_62629's avatar
Mike_62629
Icon for Nimbostratus rankNimbostratus
Jul 16, 2008

Rate limiting Search Spiders

We're currently having some problems with some web spiders beating up our webservers sucking up available sessions in our application and slurping up a whole bunch of our bandwidth. We're interested i...
application delivery
dev
devops
iRules
Mike_62629's avatar
Mike_62629
Icon for Nimbostratus rankNimbostratus
Jul 16, 2008
Here's the rate limiting approach: 

when RULE_INIT { 
 array set ::active_crawlers { } 
 set ::min_interval 1 
 } 
  
 when HTTP_REQUEST { 
 set user_agent [string tolower [HTTP::header "User-Agent"]] 
  Logic only relevant for crawler user agents 
 if { [matchclass $user_agent contains $::Crawlers] } { 
  Throttle crawlers. 
 set curr_time [clock seconds] 
 if { [info exists ::active_crawlers($user_agent)] } { 
 if { [ $::active_crawlers($user_agent) < $curr_time ] } { 
 set ::active_crawlers($user_agent) [expr {$curr_time + $::min_interval}] 
 } else { 
 reject 
 } 
 } else { 
 set ::active_crawlers($user_agent) [expr {$curr_time + $::min_interval}] 
 } 
 } 
 }