Forum Discussion

adalan_1411's avatar
adalan_1411
Icon for Nimbostratus rankNimbostratus
Aug 15, 2008

Bot Redirections

Here's the iRule:

 
 class bots  { 
    "archiver" 
    "argus" 
    "arks" 
    "bot" 
    "core" 
    "crawl" 
    "cursor" 
    "custo" 
    "exalaed" 
    "fast-" 
    "favorg" 
    "googlebot" 
    "jakarta" 
    "java" 
    "larbin" 
    "libwww" 
    "msnbot" 
    "muncher" 
    "nutch" 
    "pioneer" 
    "powermarks" 
    "python" 
    "redcarpet" 
    "sbider" 
    "scanalert" 
    "scooter" 
    "shopwiki" 
    "slurp" 
    "spider" 
    "sunrise" 
    "susie" 
    "t-h-u-n-d-e-r-s-t-o-n-e" 
    "teoma" 
    "traveler" 
    "trivial" 
    "walhello" 
    "webcollage" 
    "zyborg" 
 } 
  
 when RULE_INIT { 
    set ::vip_https_ports [list \ 
 443 \ 
    ] 
    set ::vip_http_ports [list \ 
 80 \ 
    ] 
    set ::special_bots [list \ 
 "googlebot" \ 
 ] 
 } 
 when HTTP_REQUEST { 
 set ::is_bot 0 
 if { [matchclass [string tolower [HTTP::header User-Agent]] contains $::special_bots] } { 
 HTTP::header insert Is_Bot "Yes" 
 persist none 
 set ::is_special 1 
 set ::is_bot 1 
 } elseif { [matchclass [string tolower [HTTP::header User-Agent]] contains $::bots] } { 
 HTTP::header insert Is_Bot "Yes" 
 persist none 
 set ::is_special 0 
 set ::is_bot 1 
 } elseif { [matchclass [HTTP::method] equals $::sec_http_methods] } { 
 reject 
     } 
 if {$::is_bot} { 
 if { ([matchclass [TCP::local_port] equals $::vip_https_ports]) }{ 
 if {$::is_special}{pool pool_bot_1a_https} 
 if {!$::is_special}{pool pool_bot_2a_https} 
 } elseif {([matchclass [TCP::local_port] equals $::vip_http_ports]) }{ 
 if {$::is_special}{pool pool_bot_1a_http} 
 if {!$::is_special}{pool pool_bot_2a_http} 
 }  
 } 
 }

I am trying move Googlebot into it's own 2 servers and all other bots onto their own 2 servers. It seems though, that bots are still managing to get by this rule. Any ideas?
  •  

    Do you need the "googlebot" entry in both lists? Seems like it would be easier to code without the duplicate entry.