Forum Discussion

Eduardo_Saito_1's avatar
Eduardo_Saito_1
Icon for Nimbostratus rankNimbostratus
Nov 06, 2006

Limit number of connections for googlebot

Hello!

I'm trying to write an irule that limits the number of connections that Google Bot can use. I found some good scripts that help me acomplish this (one made by Joe and the other one from tips).

Here go the script. I didn't put it on production yet since I don't know the impact it will have on cpu utilization.

Hope that you guys can help me.

Here go what I was thinking. I don't know if there's a more efficient way to do this.


when RULE_INIT {
  array set ::active_clients { }
}
when CLIENT_ACCEPTED {
switch -glob [string tolower [HTTP::header "User-Agent"]] {
"*googlebot*" {
set client_ip [IP::remote_addr]
if { [info exists ::active_clients($client_ip)]  } {
if {$::active_clients($client_ip) > 10 } {
reject
log local0. "Reject GOOGLEBOT IP $client_ip ($::active_clients($client_ip))"
return
} else {
incr ::active_clients($client_ip)
}
} else {
set ::active_clients($client_ip) 1
}
}
}
}
when CLIENT_CLOSED {
switch -glob [string tolower [HTTP::header "User-Agent"]] {
"*googlebot*" {
set client_ip [IP::remote_addr]
if { [info exists ::active_clients($client_ip)] } {
incr ::active_clients($client_ip) -1
if { $::active_clients($client_ip) <= 0 } {
unset ::active_clients($client_ip)
}
}
}
}
}
  • Colin_Walker_12's avatar
    Colin_Walker_12
    Historic F5 Account
    It looks like the logic is pretty decent, after taking a quick peek. You shouldn't need those two switch statements, though, if there is only one case in each of them. Replacing them each with a single if structure would probably net you a few cycles.

     

     

    Colin
  • Deb_Allen_18's avatar
    Deb_Allen_18
    Historic F5 Account
    This codeshare contribution will allow you to limit by rateclass any UserAgent on the list: http://devcentral.f5.com/wiki/default.aspx/iRules/ControllingBots.html (Click here)

     

     

    You could use a custom log message and custom trap to generate email on any condition observed within your iRule as noted in this post (Click here)

     

     

    HTH

     

    /deb
  • Deb_Allen_18's avatar
    Deb_Allen_18
    Historic F5 Account
    Yeah, just remove the switch statement & you'd have something like this, which won't discriminate against just googlebot:
    when RULE_INIT {
      array set ::active_clients { }
    }
    when CLIENT_ACCEPTED {
      set client_ip [IP::remote_addr]
      if { [info exists ::active_clients($client_ip)]  } {
        if {$::active_clients($client_ip) > 10 } {
          reject
          log local0. "Reject GOOGLEBOT IP $client_ip ($::active_clients($client_ip))"
          return
        } else {
          incr ::active_clients($client_ip)
        }
      } else {
        set ::active_clients($client_ip) 1
      }
    }
    when CLIENT_CLOSED {
      set client_ip [IP::remote_addr]
      if { [info exists ::active_clients($client_ip)] } {
        incr ::active_clients($client_ip) -1
        if { $::active_clients($client_ip) <= 0 } {
          unset ::active_clients($client_ip)
        }
      }
    }
  • right on, thanks deb now I cant wait till those bastards try and shoot all there connection at me again!!!!