iRule to limit the number of requests clients can make within a certain amount of time. Units used are requests/minute but this could be changed to requests/sec pretty easily. Once users hit a predefined limit of requests per minute they are throttled. Clients can also be blacklisted (automatically restricted to a very low rate) or whitelisted. This rule as it is shown here has a very low limit as it only applies to certain URLs. If you remove those 'if' statements then you will need to allow many more requests as most web pages have a number of objects per page. This has not been tested against production traffic so the impact on the BIG-IP is not known.
Code :
# This is a complete rewrite that is CMP-friendly, see older TMOS v9 code below.
## HTTP Request Throttling
##
## This I-Rule allows only "maxReqs" HTTP requests within "timeout" interval.
## This version throttles by URI and allows IP address whitelists. IP address can be in
## the IP header or the X-Forwarded-For HTTP header.
##
## CMP compatible: Yes
##
## This rule requires:
## A default pool so that the session table can be used and to forward requests.
##
## 09/14/2014, Irule revised to use CMP compatible commands.
## - "static" is added to global variable names.
## - arrays replaced with subtables.
##
## This rule tested on:
## TMOS v11.6.0, should work on an 11.x version
## LTM
##
when RULE_INIT {
# This defines the maximum requests to be served within the timing interval defined by the static::timeout variable below.
set static::maxReqs 4;
# Timer Interval in seconds within which only static::maxReqs Requests are allowed.
# (i.e: 10 req per 2 sec == 5 req per sec)
# If this timer expires, it means that the limit was not reached for this interval and the request
# counting starts over.
# Making this timeout large increases memory usage. Making it too small negatively affects performance.
set static::timeout 2;
}
when HTTP_REQUEST {
# The iRule allows throttling for only sepecific URIs. You list the URIs_to_throttle
# in a datagroup. URIs_to_throttle or Methods_to_throttle.
# if you need to throttle by Method use an statement like this:
# if { [class match [HTTP::uri] equals URIs_to_throttle] }
# Note: a URI is everything after the hostname: e.g. /path1/login.aspx?name=user1
#
if { [class match [HTTP::uri] equals URIs_to_throttle] } {
# The following expects the IP addresses in multiple X-forwarded-for headers. It picks the first one.
if { [HTTP::header exists X-forwarded-for] } {
set client_IP_addr [getfield [lindex [HTTP::header values X-Forwarded-For] 0] "," 1]
} else {
set client_IP_addr [IP::client_addr]
}
# The matching below expects a datagroup named: Throttling_Whitelist_IPs
# The Condition of the if statement is true if the IP address is NOT in the whitelist.
if { not ([class match $client_IP_addr equals Throttling_Whitelist_IPs ] )} {
set getcount [table lookup -notouch $client_IP_addr]
if { $getcount equals "" } {
table set $client_IP_addr "1" $static::timeout $static::timeout
# record of this session does not exist, starting new record, request is allowed.
} else {
if { $getcount < $static::maxReqs } {
# log local0. "Request Count for $client_IP_addr is $getcount"
table incr -notouch $client_IP_addr
# record of this session exists but request is allowed.
} else {
HTTP::respond 403 content {
HTTP Request denied
Your HTTP requests are being throttled.
}
}
}
}
}
}
I made a variation of this rule for rate limiting individual IP addresses and UserAgents, not to specific URIs but to all virtual servers covered by the rule, testing for x-forwarded-for, etc.. My tests performed very well so we tried it during a load test. The static variables needed to be tuned quite a bit, I don't recall what the final values were and I no longer have access to the production rule but I can get it if anyone is interested. We used it last night in production and it was very effective against rule breaking bots that are causing performance impact.
I would also like the code you mentioned, specifically to address "The static variables needed to be tuned quite a bit, I don't recall what the final values were and I no longer have access to the production rule but I can get it if anyone is interested."
