cancel
Showing results for 
Search instead for 
Did you mean: 
Kirk_Bauer_1018
Nimbostratus
Nimbostratus

Problem this snippet solves:

iRule to limit the number of requests clients can make within a certain amount of time. Units used are requests/minute but this could be changed to requests/sec pretty easily. Once users hit a predefined limit of requests per minute they are throttled. Clients can also be blacklisted (automatically restricted to a very low rate) or whitelisted. This rule as it is shown here has a very low limit as it only applies to certain URLs. If you remove those 'if' statements then you will need to allow many more requests as most web pages have a number of objects per page. This has not been tested against production traffic so the impact on the BIG-IP is not known.

Code :

# This is a complete rewrite that is CMP-friendly, see older TMOS v9 code below.

## HTTP Request Throttling
##
##  This I-Rule allows only "maxReqs" HTTP requests within "timeout" interval.
##  This version throttles by URI and allows IP address whitelists. IP address can be in 
##  the IP header or the X-Forwarded-For HTTP header.
##
##  CMP compatible:  Yes
##
##  This rule requires:
##  A default pool so that the session table can be used and to forward requests.
##
## 09/14/2014, Irule revised to use CMP compatible commands.
##    - "static" is added to global variable names.
##    - arrays replaced with subtables.
##
## This rule tested on:
##    TMOS v11.6.0, should work on an 11.x version
##    LTM
##



when RULE_INIT {
    # This defines the maximum requests to be served within the timing interval defined by the static::timeout variable below. 
    set static::maxReqs 4;

    # Timer Interval in seconds within which only static::maxReqs Requests are allowed.  
    # (i.e: 10 req per 2 sec == 5 req per sec) 
    # If this timer expires, it means that the limit was not reached for this interval and the request 
    # counting starts over.
    # Making this timeout large increases memory usage.  Making it too small negatively affects performance.  
    set static::timeout 2;
}

when HTTP_REQUEST {
# The iRule allows throttling for only sepecific URIs.  You list the URIs_to_throttle
# in a datagroup.  URIs_to_throttle or Methods_to_throttle.
# if you need to throttle by Method use an statement like this:
#                               if { [class match [HTTP::uri] equals URIs_to_throttle] }
# Note: a URI is everything after the hostname: e.g. /path1/login.aspx?name=user1
#  

   if { [class match [HTTP::uri] equals URIs_to_throttle] } {

        # The following expects the IP addresses in multiple X-forwarded-for headers.  It picks the first one.
        if { [HTTP::header exists X-forwarded-for] } {
            set client_IP_addr [getfield [lindex  [HTTP::header values X-Forwarded-For]  0] "," 1]
        } else {
            set client_IP_addr [IP::client_addr]
        }
        # The matching below expects a datagroup named: Throttling_Whitelist_IPs
        # The Condition of the if statement is true if the IP address is NOT in the whitelist.
        if { not ([class match $client_IP_addr equals Throttling_Whitelist_IPs ] )} {
               set getcount [table lookup -notouch $client_IP_addr]
               if { $getcount equals "" } {
                   table set $client_IP_addr "1" $static::timeout $static::timeout
                   # record of this session does not exist, starting new record, request is allowed.
                } else {
                      if { $getcount < $static::maxReqs } {
                          # log local0. "Request Count for $client_IP_addr is $getcount"  
                          table incr -notouch $client_IP_addr
                          # record of this session exists but request is allowed.
                      } else {
                           HTTP::respond 403 content {
                                
                                HTTP Request denied
                                Your HTTP requests are being throttled.
                                
                           }
                     }
               }
        }
   }
}
Comments
oedo808_68685
Altostratus
Altostratus
I made a variation of this rule for rate limiting individual IP addresses and UserAgents, not to specific URIs but to all virtual servers covered by the rule, testing for x-forwarded-for, etc.. My tests performed very well so we tried it during a load test. The static variables needed to be tuned quite a bit, I don't recall what the final values were and I no longer have access to the production rule but I can get it if anyone is interested. We used it last night in production and it was very effective against rule breaking bots that are causing performance impact.
MrVJTod_64267
Nimbostratus
Nimbostratus
@oedo808 : interested in your modified code.
Sumanta_88744
Cirrus
Cirrus

Hi oedo808

 

Thanks for the update. Can I have the code you mentioned? I have to apply in production with high traffic.

 

John_H
Nimbostratus
Nimbostratus

Hi oedo808,

 

I would also like the code you mentioned, specifically to address "The static variables needed to be tuned quite a bit, I don't recall what the final values were and I no longer have access to the production rule but I can get it if anyone is interested."

 

Thanks!

 

AlexDeMarco
Nimbostratus
Nimbostratus

I would be interested in the code mentioned as well. Thank you in advance.

 

oedo808_68685
Altostratus
Altostratus

Oh hey, sorry I missed the other notifications. I will post the modified version and link back here.

 

Once again, sorry I missed these requests, here is a link to it: https://devcentral.f5.com/s/articles/http-request-throttle-by-ip-and-useragent-1032

 

Mohanad
Cirrostratus
Cirrostratus

Thanks for the irule, i managed to limit 1 request per minute for specific URL

Version history
Last update:
‎18-Mar-2015 11:37
Updated by:
Contributors