Controlling Bots

[bwolmarans_1284]

Problem this snippet solves:

Webbots, you can't live with them, you can't live without them... This iRule determines if a webbot is accessing your systems and assigns them to a lower priority resource. The first example includes the bot list inside the rule and uses the switch statement to find a match.

Code :

when HTTP_REQUEST {
   switch -glob [string tolower [HTTP::header User-Agent]] {
      "*scooter*" -
      "*slurp*" -
      "*msnbot*" -
      "*fast-*" -
      "*teoma*" -
      "*googlebot*" {
        # Send bots to the bot pool
         pool slow_webbot_pool
      }
      default {
         # Send all other requests to a default pool
         pool default_pool
      }
   }
}

### or if you prefer data groups ###

---- String Class ----

class bots {
  "scooter"
  "slurp"
  "msnbot"
  "fast-"
  "teoma"
  "googlebot"
}

---- iRule ----

when HTTP_REQUEST {
  if { [matchclass [string tolower [HTTP::header User-Agent]] contains $::bots] } {
    pool slow_webbot_pool
  } else {
    pool default_pool
  }
}

Tested this on version:

10.0

[Oeg87_310913]

By default the bots can access or I have to create this small snippet in order to allow the bots access??