ASM/WAF rate limit and block clients by source ip (or device_id fingerprint) if there are too many violations
Problem this snippet solves:
For attackers that cause too many violations it is better to block them at the start of the HTTP_REQUEST or CLIENT_ACCEPTED events as the ASM/WAF processing causes too much CPU and the CLIENT_ACCEPTED and HTTP_REQUEST events are before the ASM and the ASM irule events and this may help the CPU a little.
In additon many services offer generation of device id/fingerprint like apple for ios or google for android or TMIXID for web or phone with SDK(https://money.tmx.com/en/quote/ID) , that is saved in HTTP header (https://clouddocs.f5.com/api/irules/HTTP__header.html) and you can make irule to use the HTTP header as a fingerprint value and just modify the irule code below!
How to use this snippet:
You can use this code with the variables set static::maxReqs 3 and set static::timeout 60 also in the code also the CLIENT_ACCEPTED event can be uncommented and the HTTP_REQUEST event coommented but if the clients come by using a NAT device before the F5 better use the X-Forwarded-For header to get the client ip address and rate limit by it.
Also the data group ip_whitelist needs to be created maybe for not blocking vunrability tests and etc.
Also the event ASM_REQUEST_DONE and [ASM::status] equals "blocked" can be replaced with ASM_REQUEST_BLOCKING but when ASM_REQUEST_BLOCKING happens after them, so this is why I use ASM_REQUEST_DONE and [ASM::status] equals "blocked"
I got the idea from:
https://clouddocs.f5.com/training/community/irules/html/class2/module1/lab2.html
https://support.f5.com/csp/article/K07150625
Don't forget to enable the asm irule events under the security policy that Virtual server where the irule is attached uses (I forgot and it took me 30 minutes of testing to figure it out 😞 😞
Also the irule can be modified to use a subtable "table set -subtable asm_violations" for example for the RAM memory limits and also the table ans subtable are shared between VIP servers, so another VIP even without ASM policy can check the same table or subtable and block the user. If you add the between line 15 and 32 to another VS even without ASM the VS will check the RAM table and block users by source ip that tried to attack the VS with the ASM policy.
https://clouddocs.f5.com/api/irules/table.html
https://devcentral.f5.com/s/articles/v101-the-table-command-subtables
This way you can use the primary VS as a honeypot that allows all the other servers to learn the bad client's activities. For how to redirect to a honeypot VS with the ASM policy you can check:
For blocking with the device_id fingerprint see the comment section. The fingerprint device id lookup may return "0" and then the lookup will failover to using source ip addresses. Also the new example uses 2 different subtables for device id or client ip addresses.
In additon many services offer generation of device id/fingerprint like apple for ios or google for android or TMIXID for web or phone with SDK(https://money.tmx.com/en/quote/ID) , that is saved in HTTP header and you can make irule to use the HTTP header (https://clouddocs.f5.com/api/irules/HTTP__header.html) as a fingerprint value and just modify the irule code below!
Code :
when RULE_INIT {
# The max requests served within the timing interval per the static::timeout variable
set static::maxReqs 3
# Timer Interval in seconds within which only static::maxReqs Requests are allowed.
# (i.e: 10 req per 2 sec == 5 req per sec)
# If this timer expires, it means that the limit was not reached for this interval and
# the request counting starts over. Making this timeout large increases memory usage.
# Making it too small negatively affects performance.
set static::timeout 60
}
#when CLIENT_ACCEPTED {
#set cIP_addr [IP::client_addr]
# set getcount [table lookup -notouch $cIP_addr]
# if { ! ( [class match $cIP_addr equals ip_whitelist] ) } {
# The following expects the IP addresses in multiple X-forwarded-for headers.
# It picks the first one. If XFF isn’t defined it can grab the true source IP.
# if { $getcount > $static::maxReqs } {
# log local0. "Request Count for $cIP_addr is $getcount"
# drop
# }
# }
#}
when HTTP_REQUEST {
# Allows throttling for only specific URIs. List the URIs_to_throttle in a data group.
# Note: a URI is everything after the hostname: e.g. /path1/login.aspx?name=user
if { [HTTP::header exists X-forwarded-for] } {
set cIP_addr [getfield [lindex [HTTP::header values X-Forwarded-For] 0] "," 1]
} else {
set cIP_addr [IP::client_addr]
}
set getcount [table lookup -notouch $cIP_addr]
if { ! ( [class match $cIP_addr equals ip_whitelist] ) && ( [class match [HTTP::uri] equals URIs_to_throttle] )} {
# The following expects the IP addresses in multiple X-forwarded-for headers.
# It picks the first one. If XFF isn’t defined it can grab the true source IP.
if { $getcount > $static::maxReqs } {
log local0. "Request Count for $cIP_addr is $getcount"
# drop
HTTP::respond 403 content {
Your HTTP requests are being blocked because of too many violations.
}
}
}
}
#when ASM_REQUEST_BLOCKING {
when ASM_REQUEST_DONE {
#log local0.debug "\[ASM::status\] = [ASM::status]"
if { [ASM::status] equals "blocked" } {
# Allows throttling for only specific URIs. List the URIs_to_throttle in a data group.
# Note: a URI is everything after the hostname: e.g. /path1/login.aspx?name=user1
# if { [class match [HTTP::uri] equals URIs_to_throttle] } {
# The following expects the IP addresses in multiple X-forwarded-for headers.
# It picks the first one. If XFF isn’t defined it can grab the true source IP.
# set getcount [table lookup -notouch $cIP_addr]
if { $getcount equals "" } {
table set $cIP_addr "1" $static::timeout $static::timeout
# Record of this session does not exist, starting new record
# Request is allowed.
} else {
log local0. "Request Count for $cIP_addr is $getcount"
table incr -notouch $cIP_addr
# record of this session exists but request is allowed.
}
# }
}
}
Tested this on version:
15.1
Updated Jan 31, 2023
Version 3.0
- Nikoolayy1
MVP
For you to share ASM violation data between F5 devices you will need Splunk or other SIEM that will make a list that can be used as AFM custom feed list or exter data group. For more about that see my comments under:
https://devcentral.f5.com/s/question/0D51T00006aFjFF/managing-datagroups-from-bigiq
https://support.f5.com/csp/article/K17523
As external data group enties don't time out like the table command you can feed the extarnal list using sideband function to load the external cvs list by HTTP command in the F5 device table:
https://devcentral.f5.com/s/articles/populating-tables-with-csv-data-via-sideband-connections
Another solution is to add the data group entries like:
1.1.1.1 2016-05-30
2.2.2.2 2016-06-01
3.3.3.3 2016-07-20
And use a bash script that may be triggered with cronjob on the server that feeds the F5 to check every day if there is an old entry:
cutoff=$(date -d 'now - 30 days' '+%Y-%m-%d')
awk -v cutoff="$cutoff" '$2 >= cutoff { print }' <in.txt >out.txt && mv out.txt in.txt
- Daniel_Wolf
Hi ,
nice iRule. How would you compare this solution with Session Awareness as described in https://support.f5.com/csp/article/K02212345? Session Awareness gives additional categories for blocking based on Device IP or Session.
KR
Daniel
- Nikoolayy1
Hello the F5 Session Awareness is a great feature but it is CPU intensive and this solution if for blocking the bad actors after the tcp_handshake (CLIENT_ACCEPTED) when the system does not havevery much free CPU or after the first HTTP checks at the HTTP profile (HTTP_REQUEST). What could I have done is to make the irule with ASM::fingerprint event (https://clouddocs.f5.com/api/irules/ASM__fingerprint.html) but then the code will look like the one below as the ASM::fingerprint command can only be used under ASM events and because the processing will be all at the ASM event better then to use the Session Awareness as the CPU difference may not be much. The only point is if the client starts attacking to be blocked after 3 or 4 detected attempts as there could an attack that even the F5 ASM/WAF may not catch but as you mentioned Session Awareness is better in this case if we want to use the device ID or we have enough CPU for to also use session awareness for the client address or user session if we have APM + ASM.
The example is like the one in https://clouddocs.f5.com/training/community/irules/html/class2/module1/lab2.html but modified for ASM with fingerprint
when RULE_INIT {
# The max requests served within the timing interval per the static::timeout variable
set static::maxReqs 4
# Timer Interval in seconds within which only static::maxReqs Requests are allowed.
# (i.e: 10 req per 2 sec == 5 req per sec)
# If this timer expires, it means that the limit was not reached for this interval and
# the request counting starts over. Making this timeout large increases memory usage.
# Making it too small negatively affects performance.
set static::timeout 2
}
when ASM_REQUEST_DONE {
#log local0.debug "\[ASM::status\] = [ASM::status]"
if { [ASM::status] equals "blocked" } {
# Allows throttling for only specific URIs. List the URIs_to_throttle in a data group.
# Note: a URI is everything after the hostname: e.g. /path1/login.aspx?name=user1
if { ( [class match [HTTP::uri] equals URIs_to_throttle] ) and not ( [class match [IP::client_addr] equals whitelist] ) } {
# The following expects the IP addresses in multiple X-forwarded-for headers.
# It picks the first one. If XFF isn’t defined it can grab the true source IP.
set cIP_finger [ASM::fingerprint]
set getcount [table lookup -notouch $cIP_finger]
if { $getcount equals "" } {
table set $cIP_finger "1" $static::timeout $static::timeout
# Record of this session does not exist, starting new record
# Request is allowed.
} else {
if { $getcount < $static::maxReqs } {
log local0. "Request Count for $cIP_finger is $getcount"
table incr -notouch $cIP_finger
# record of this session exists but request is allowed.
} else {
#drop
HTTP::respond 403 content {
<html>
<head><title>HTTP Request denied</title></head>
<body>Your HTTP requests are being throttled.</body>
</html>
}
}
}
}
}
}
- Nikoolayy1
Here is an example with failover to using the client ip address if ASM fingerprint if it is equal to "0" (no device ID fingerprint available). As I mentioned better use the main irule for blocking by source IP address or for device fingerprint the session tracking and awareness (https://support.f5.com/csp/article/K02212345).
I just played with this as the ASM fingerprint is not aways generated for a web request and you need to first enable a feature like web scraping or in the newer versions the unified bot defense (see https://devcentral.f5.com/s/question/0D51T00006kGZft/asm-fingerprinting-with-irule and the general articles https://support.f5.com/csp/article/K19556739 ), so this irule can be helpfull if the default F5 features that use the fingerprint don't failover to client source IP address if there is no available fingerprint.
I also created 2 subtables for the ip addresses and fingerprints for optimizing the lookups.
when RULE_INIT {
# The max requests served within the timing interval per the static::timeout variable
set static::maxReqs 2
# Timer Interval in seconds within which only static::maxReqs Requests are allowed.
# (i.e: 10 req per 2 sec == 5 req per sec)
# If this timer expires, it means that the limit was not reached for this interval and
# the request counting starts over. Making this timeout large increases memory usage.
# Making it too small negatively affects performance.
set static::timeout 20
}
#when CLIENT_ACCEPTED {
#set cIP_addr [IP::client_addr]
# set getcount [table lookup -subtable "bad_ip" -notouch $cIP_addr]
# if { ! ( [class match $cIP_addr equals ip_whitelist] ) } {
# The following expects the IP addresses in multiple X-forwarded-for headers.
# It picks the first one. If XFF isn’t defined it can grab the true source IP.
# if { $getcount > $static::maxReqs } {
# log local0. "Request Count for $cIP_addr is $getcount"
# drop
# }
# }
#}
when HTTP_REQUEST {
# Allows throttling for only specific URIs. List the URIs_to_throttle in a data group.
# Note: a URI is everything after the hostname: e.g. /path1/login.aspx?name=user
if { [HTTP::header exists X-forwarded-for] } {
set cIP_addr [getfield [lindex [HTTP::header values X-Forwarded-For] 0] "," 1]
} else {
set cIP_addr [IP::client_addr]
}
set getcount1 [table lookup -subtable "bad_ip" -notouch $cIP_addr]
if { ! ( [class match $cIP_addr equals ip_whitelist] ) } {
# The following expects the IP addresses in multiple X-forwarded-for headers.
# It picks the first one. If XFF isn’t defined it can grab the true source IP.
if { $getcount1 >= $static::maxReqs } {
log local0. "Request Count for $cIP_addr is $getcount1"
# drop
HTTP::respond 403 content {
<html>
<head><title>HTTP Request denied</title></head>
<body>Your HTTP requests are being blocked because of too many violations.</body>
</html>
}
}
}
}
when ASM_REQUEST_DONE {
log local0.debug "\[ASM::status\] = [ASM::status] and ASM_Device_id = [ASM::fingerprint] "
if { [ASM::status] equals "blocked" } {
# Allows throttling for only specific URIs. List the URIs_to_throttle in a data group.
# Note: a URI is everything after the hostname: e.g. /path1/login.aspx?name=user1
if { not ( [class match [IP::client_addr] equals whitelist] ) } {
# The following expects the IP addresses in multiple X-forwarded-for headers.
# It picks the first one. If XFF isn’t defined it can grab the true source IP.
set cIP_finger [ASM::fingerprint]
if { $cIP_finger == 0 } {
if { $getcount1 equals "" } {
table set -subtable "bad_ip" $cIP_addr "1" $static::timeout $static::timeout
# Record of this session does not exist, starting new record
# Request is allowed.
} else {
log local0. "Request Count for $cIP_addr is $getcount1"
table incr -subtable "bad_ip" -notouch $cIP_addr
# record of this session exists but request is allowed.
}
} else {
set getcount2 [table lookup -subtable "bad_id" -notouch $cIP_finger]
if { $getcount2 equals "" } {
table set -subtable "bad_id" $cIP_finger "1" $static::timeout $static::timeout
# Record of this session does not exist, starting new record
# Request is allowed.
} else {
if { $getcount2 >= $static::maxReqs } {
log local0. "Request Count for $cIP_finger is $getcount"
table incr -subtable "bad_id" -notouch $cIP_finger
# record of this session exists but request is allowed.
} else {
#drop
HTTP::respond 403 content {
<html>
<head><title>HTTP Request denied</title></head>
<body>Your HTTP requests are being throttled.</body>
</html>
}
}
}
}
}
}
}
