Limiting connections base on user-agent and URI

Just to clarify the purpose here. I don't control the third party me.you.com servers and I want to add some control in case the third party servers go 'wonky'. This way if their servers are bad, we don't get hit with a ton of connections that may be hanging around on the BigIP.

 

 

Thanks!

 

 

Byzandula

You can get at the User-Agent with the HTTP::header "User-Agent" command and the URI with the HTTP::uri command. You can then issue a redirect with the "HTTP::redirect" command (I'm assuming you are doing a HTTP redirect since that's what you stated. I'm not sure with the RewriteRule does with Apache. At first glance it would seem to rewrite the URI as it sends it to the backend server. If this is the case, then you'll want to use the HTTP::uri command to modify the URI. If you do want a true redirect sent do the browser you'll want the HTTP::redirect command

Here's a first stab.

when HTTP_REQUEST {
   step 1
   RewriteRule ^/(.*\.shtml)$ http://me.you.com/$1 [P]
  switch -glob [HTTP::uri] {
    "*.shtml" {
       For a full HTTP redirect
      HTTP::redirect "http://me.you.com/"
       -or- if you just want to modify the request to the server
       HTTP::header replace "Host" "me.you.com"
       HTTP::uri "/"
    }
  }
   step 2
   RewriteCond User-Agent: .*(?:Googlebot|msnbot|Slurp|Ask\sJeeves|Inktomi|Yahoo).* 
   RewriteRule ^/$ http://me.you.com/ [P] 
   RewriteCond User-Agent: .*(?:Googlebot|msnbot|Slurp|Ask\sJeeves|Inktomi|Yahoo).* 
   RewriteRule ^/index.jsp.* http://me.you.com/ [P]
  switch -glob [string tolower [HTTP::header "User-Agent"]] {
    "*googlebot*" -
    "*msnbot*" -
    "*slurp*" -
    "*ask jeeves*" -
    "*inktomi*" -
    "*yahoo*" {
      switch -glob [string tolower [HTTP::uri]] {
        "/" -
        "/index.jsp*" {
          HTTP::redirect "http://me.you.com/"
        }
      }
    }
  }
}

You could have just have easily used if/elseif's but I tend to favor switch statements for their readability.

Hope this helps...

-Joe

from the irule runner up winner Eduardo Saito

 

 

irule_limit_num_connections_googlebot

 

 

http://devcentral.f5.com/Default.aspx?tabid=109

 

 

 

when RULE_INIT {

 

array set ::active_clients { }

 

}

 

 

when CLIENT_ACCEPTED {

 

switch -glob [string tolower [HTTP::header "User-Agent"]] {

 

"*googlebot*" {

 

set client_ip [IP::remote_addr]

 

if { [info exists ::active_clients($client_ip)] } {

 

if {$::active_clients($client_ip) > 10 } {

 

reject

 

log local0. "Reject GOOGLEBOT IP $client_ip ($::active_clients($client_ip))"

 

return

 

} else {

 

incr ::active_clients($client_ip)

 

}

 

} else {

 

set ::active_clients($client_ip) 1

 

}

 

}

 

}

 

}

 

 

when CLIENT_CLOSED {

 

switch -glob [string tolower [HTTP::header "User-Agent"]] {

 

"*googlebot*" {

 

set client_ip [IP::remote_addr]

 

if { [info exists ::active_clients($client_ip)] } {

 

incr ::active_clients($client_ip) -1

 

if { $::active_clients($client_ip) <= 0 } {

 

unset ::active_clients($client_ip)

 

}

 

}

 

}

 

}

 

}

 

 

After copying and pasting directly into irules editor I get the two errors at both Client_accepted & CLIENT_CLOSED

 

command is not valid in current event context ((Client_accepted)) [HTTP:header "User-agent"]

 

 

Do I need to upgrade bigip version or..... ?

 

 

I'm not sure which versions ever allowed you to access HTTP headers from the CLIENT_ACCEPTED event. In current versions, the HTTP headers haven't been parsed in the CLIENT_ACCEPTED event, so you get an error stating that the HTTP::header command is invalid in that context. If you changed the CLIENT_ACCEPTED event to HTTP_REQUEST and CLIENT_CLOSED to HTTP_RESPONSE, you'd get per HTTP request throttling, instead of per-TCP connection throttling.

 

 

Can you give that a shot?

 

 

Aaron