Forum Discussion

royceking_18057's avatar
royceking_18057
Icon for Cirrus rankCirrus
May 29, 2015

iRule to block multiple user agents.

I have this clunky iRule to block unwanted user agent bots that I have identified hit our site. Unfortunately it is a pain to manage. Is there a better way to write this and not have to use a big if statement with a lot of "or" operators? 

when HTTP_REQUEST {
    if {[HTTP::header "User-Agent"] starts_with "AndroidDownloadManager" or [HTTP::header "User-Agent"] contains "Baiduspider" or [HTTP::header "User-Agent"] contains "GrapeshotCrawler" or [HTTP::header "User-Agent"] contains "ScreenerBot" or [HTTP::header "User-Agent"] contains "YandexBot" or [HTTP::header "User-Agent"] contains "SISTRIX" or [HTTP::header "User-Agent"] contains "SPUTNIK"} {
        HTTP::respond 503 content "UnavailableRequest denied."
    }
}

I really wish ASM had a way to block unwanted user agents. I have bot detection enabled but a lot of the time these bots come in at a slow enough rate that they are undetected by bot detection.

Any feedback is appreciated.

Thanks

application delivery
ASM Advanced WAF
devops
iRules
security
  • Tim_K_92675's avatar
    Tim_K_92675
    Icon for Cirrostratus rankCirrostratus
    May 29, 2015

    Is using a string type data group an option for you? Something like this:

        ltm data-group internal blocked_user_agents {
    records {
        androiddownloadmanager { }
        baiduspider { }
        grapeshotvrawler { }
    }
    type string

    when HTTP_REQUEST {
        if { [class [string tolower [HTTP::header "User-Agent"]] contains $blocked_user_agents ] } {
            drop
            return 
        }
    }
  • royceking_18057's avatar
    royceking_18057
    Icon for Cirrus rankCirrus
    May 29, 2015

    Tim,

     

    I was actually just researching data groups. So do I just add the data group right into the iRule?

     

  • Tim_K_92675's avatar
    Tim_K_92675
    Icon for Cirrostratus rankCirrostratus
    May 29, 2015

    Go to Local Traffic ›› iRules : Data Group List and then hit Create...

     

    From there you type in the data group name (this will be the variable name you insert into the irule) and select 'String' for the type.

     

    Enter in the String Records String field the names of the user agents. I would recommend all lower case. This way the irule logic 'string tolower' will be able to compare apples-to-apples.

     

  • royceking_18057's avatar
    royceking_18057
    Icon for Cirrus rankCirrus
    May 29, 2015

    Do I need to do something special to reference the data group? Right now my testing is blocking all requests.

    iRule:

    when HTTP_REQUEST {
  if { [class [string tolower [HTTP::header "User-Agent"]] contains user_agent_blacklist ] } {
    log local0. "User_agent [HTTP::header "User-Agent"] is blacklisted. from: [IP::client_addr] "
    drop
    return 
  }
}

    Data Group:

  • royceking_18057's avatar
    royceking_18057
    Icon for Cirrus rankCirrus
    May 29, 2015
    I think I figured it out. I needed to add the match after the class. when HTTP_REQUEST { if { [class match [string tolower [HTTP::header "User-Agent"]] contains user_agent_blacklist ] } { log local0. "User_agent [HTTP::header "User-Agent"] is blacklisted. from: [IP::client_addr] " drop return } } Seems to be working now. Thanks for the help.