Forum Discussion

Eblakely_225577's avatar
Eblakely_225577
Icon for Nimbostratus rankNimbostratus
Sep 30, 2016

iRule for rate limiting.

We are experiencing an odd issue where an iRule works in our staging environment and only works intermittently in production. The iRule (see below) filters based on the URI and sets a max rate limit of 30 HTTP requests per second (this is for an API). I've built a test script and run it against the virtual server with the iRule and the rate limiting is intermittently triggered.

Is there something I am missing ? Apologies if this is thin on details but any assistance and pointers would be appreciated.

Thanks.

iRule is below:

POLLING API RATE LIMIT RULE

when RULE_INIT { set static::maxRate 120 set static::timeout 1 } when HTTP_REQUEST { if { ([HTTP::uri] starts_with "/v3") } { set methodCount [table key -count -subtable [IP::client_addr]] log local0. "[IP::client_addr]: methodCount=$methodCount" if { $methodCount < $static::maxRate } { incr methodCount 1 log local0. "Adding entry for [IP::client_addr]"

        table set -subtable [IP::client_addr] $methodCount "ignore" $static::timeout
        } else {
         log local0. "[IP::client_addr] exceeded max HTTP requests per second"
         HTTP::respond 429 content "Request blockedExceeded requests/sec limit."
         return

    }
}

}

iRules
security
  • Eblakely_225577's avatar
    Eblakely_225577
    Icon for Nimbostratus rankNimbostratus
    Sep 30, 2016

    Correction, it is not 30 requests per second. It is 120 requests per second.

     

    I've done further testing and I noticed in the logs the following behaviour when the iRule does not trigger:

     

    Sep 30 08:47:21 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:21 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:21 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.67 pool_v3 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.67: methodCount=9 Sep 30 08:47:22 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : Adding entry for 172.20.0.67

     

    The methodCount=9 sticks at 9 on and on until the script times out (it is set to send 10000 connections).

     

    When the rate limiting is triggered I see this in the logs:

     

    Sep 30 08:51:23 bigip01-game-stg info tmm[12864]: Rule /Common/v20-api-redirect : 172.20.0.69 pool_v3 Sep 30 08:51:23 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.69: methodCount=30 Sep 30 08:51:23 bigip01-game-stg info tmm[12864]: Rule /Common/v20-pollingapi-ratelimit : 172.20.0.69 exceeded max HTTP requests per second

     

    (I have our staging environment set to 30 requests per second for testing).

     

  • Kai_Wilke's avatar
    Kai_Wilke
    Icon for MVP rankMVP
    Sep 30, 2016

    Hi Eblakely,

    the iRule includes a logicical flaw in its Sliding-Window mechanic, that may cause the iRule to allow more requests then expected.

    Let me short explain how 

    $methodCount
    and 
    [table set]
    are causing havoc if table entries are beginning to expire...

    Request Rate = 10 Requests / Second

0,0sec |  1st request > $methodCount =  0 > creating subtable entry  1 with duration of 1 sec
0,1sec |  2nd request > $methodCount =  1 > creating subtable entry  2 with duration of 1 sec
0,2sec |  3rd request > $methodCount =  2 > creating subtable entry  3 with duration of 1 sec
0,3sec |  4th request > $methodCount =  3 > creating subtable entry  4 with duration of 1 sec
0,4sec |  5th request > $methodCount =  4 > creating subtable entry  5 with duration of 1 sec
0,5sec |  6th request > $methodCount =  5 > creating subtable entry  6 with duration of 1 sec
0,6sec |  7th request > $methodCount =  6 > creating subtable entry  7 with duration of 1 sec
0,7sec |  8th request > $methodCount =  7 > creating subtable entry  8 with duration of 1 sec
0,8sec |  9th request > $methodCount =  8 > creating subtable entry  9 with duration of 1 sec
0,9sec | 10th request > $methodCount =  9 > creating subtable entry 10 with duration of 1 sec
1,0sec > subtable entry 1 expires (created at 0,0 sec)
1,0sec | 11th request > $methodCount =  9 > overwriting subtable entry 10 with duration of 1 sec
1,1sec > subtable entry 2 expires (created at 0,1 sec)
1,1sec | 13th request > $methodCount =  8 > overwriting subtable entry  9 with duration of 1 sec
1,2sec > subtable entry 3 expires (created at 0,2 sec)
1,2sec | 14th request > $methodCount =  7 > overwriting subtable entry  8 with duration of 1 sec
1,3sec > subtable entry 4 expires (created at 0,3 sec)
1,3sec | 15th request > $methodCount =  6 > overwriting subtable entry  7 with duration of 1 sec
1,4sec > subtable entry 5 expires (created at 0,4 sec)
1,4sec | 16th request > $methodCount =  5 > overwriting subtable entry  6 with duration of 1 sec
1,5sec | 17th request > $methodCount =  5 > overwriting subtable entry  6 with duration of 1 sec
1,6sec | 18th request > $methodCount =  5 > overwriting subtable entry  6 with duration of 1 sec
1,7sec | 19th request > $methodCount =  5 > overwriting subtable entry  6 with duration of 1 sec
1,8sec | 20th request > $methodCount =  5 > overwriting subtable entry  6 with duration of 1 sec
1,9sec | 21th request > $methodCount =  5 > overwriting subtable entry  6 with duration of 1 sec
2,0sec > subtable entry 10 expires (last overwrite at 1,0 sec)
2,0sec | 22th request > $methodCount =  4 > creating subtable entry 5 with duration of 1 sec
2,1sec > subtable entry  9 expires (last overwrite at 1,1 sec)
2,1sec | 23th request > $methodCount =  4 > overwriting subtable entry  5 with duration of 1 sec
... etc ...

    As you can see, your iRule will start to overwrite 

    [table]
    records in the case that records are starting to expire and then result in slightly inaccurate enforcements.

    The problem in your iRule can be easily solved by using either a unique identifier to build the Sliding-Window queue, or by using a rather simple Request-Per-Timeframe mechanic (less CPU/RAM ressources required).

    Fixed Sliding-Window-Counting mechanic using 

    [clock clicks]
    as 
    [table]
    identifier to avoid queue colisions

    when RULE_INIT { 
    set static::maxRate 120 
    set static::timeout 1 
} 
when HTTP_REQUEST { 
    if { [HTTP::uri] starts_with "/v3" } then { 
        set methodCount [table key -count -subtable [IP::client_addr]] 
        log local0. "[IP::client_addr]: methodCount=$methodCount" 
        if { $methodCount < $static::maxRate } then { 
            incr methodCount 1 log local0. "Adding entry for [IP::client_addr]"
            table set -subtable [IP::client_addr] [clock clicks] "1" indef $static::timeout
        } else {
            log local0. "[IP::client_addr] exceeded max HTTP requests per second"
            HTTP::respond 429 content "Request blockedExceeded requests/sec limit."
            return
        }
    }
}

    Request-Per-Timeframe Mechanic

    when RULE_INIT { 
    set static::maxRate 120 
    set static::timeout 1 
} 
when HTTP_REQUEST {
    if { [HTTP::uri] starts_with "/v3" } then { 
        if { [set methodCount [table incr -mustexist "Count_[IP::client_addr]"]] ne "" } then {
            if { $methodCount > $static::maxRate } then {
                log local0. "[IP::client_addr] exceeded max HTTP requests per second"
                HTTP::respond 429 content "Request blockedExceeded requests/sec limit."
                return
            }   
        } else {
            table set "Count_[IP::client_addr]" 1 indef $static::timeout
        }
        log local0. "[IP::client_addr]: methodCount=$methodCount"
    }
}

    Note: In my opinion the Sliding-Window mechanic is way to accurate and complex for a request limiter that has to deal with 120 request per seconds. I would recommend to try the Request-Per-Timeframe mechanic, since its almost as accurate when dealing with 120 RPS but its using much less ressources to maintain the counters.

    Cheers, Kai