The BIG-IP Application Security Manager Part 6: IP Address Intelligence and Whitelisting
This is the sixth article in a 10-part series on the BIG-IP Application Security Manager (ASM). The first five articles in this series are: What is the BIG-IP ASM? Policy Building The Importance of File Types, Parameters, and URLs Attack Signatures XML Security This article will discuss some really cool ASM features: IP address intelligence and whitelisting. It's hard to defend against all the crazy cyber threats out there today, so wouldn't it be nice to know if the IP address requesting access to your application is trusted or not? And, wouldn't it be convenient to tell certain IP addresses that you explicitly trust them? Well the ASM allows you to do all that! So turn on that ASM and get ready to configure some awesomeness... IP Address Intelligence In·tel·li·gence noun \in-ˈte-lə-jən(t)s\: information concerning an enemy or possible enemy or an area Imagine this...you just launched a fantastic web application, and you want as many visitors as you can possibly get. But, you also want to make sure those visitors are not harmful. These days it's hard to know if the user accessing your application is fraudulent or not. There are so many botnets, proxies, scanners, infected sources, etc running rampant today that it becomes a very daunting task to figure out which ones are good and which ones are bad. The IP Address Intelligence feature on the BIG-IP ASM identifies IP addresses that are associated with high risk activity. When a client connection is initialized, the ASM monitors information from Layer 3 and determines if a client is already known to have a high risk profile. It's the application-equivalent of the FBI's most wanted list! The system uses an automated algorithm to gather evidence of threats based on observation, context, and statistical modeling. The bad IP addresses are catalogued and tracked indefinitely. If one of these bad IP addresses attempts to access your application...guess what? Sorry, no dice for the bad IP. The ASM also enables the use of the HTTP X-Forwarded-For (XFF) header as the source of the client IP identification instead of the Layer 3 address header. If you allow the XFF to be trusted, then this header's inner-most value is used, but if the XFF is not trusted, the source address from the IP header is used. The Database I'm sure by now you are curious about the size and function of that IP Address Intelligence database. The IP Address Intelligence feature uses the online IP address reputation service that is maintained by Webroot security services. As you can imagine, the list of bad IP addresses grows every day. Currently, the database contains well over 230 million IP addresses...and counting! The IP Address Intelligence feature uses a BIG-IP shared daemon called "iprepd" and a matching database file. The iprepd daemon updates the database file every 5 minutes...that's almost real-time updates! It does this automatically (there's no manual update option), so you can have the peace of mind that comes with knowing your application is protected from the most up-to-date list of known bad IP addresses. When the database is updated, it only downloads the changes from your current database, so the downloads go pretty quick (except the very first one). Because the IP Address Intelligence feature uses an external service for database maintenance and functionality, it requires a separate add-on license. The database file is not included with the ASM bundled software, but once you activate the license, the BIG-IP will contact the provider site and download the database. Here's another really cool thing about this feature...once it's enabled, you can use it with all your BIG-IP modules...not just the ASM! Also, if your license ever expires (we all know you would never let this happen, but just play along for a second), the local database will still be queried and used...it just won't get the every-5-minute updates any more. If you ever want to check on the status of the database (how many IPs were added/deleted/changed during updates), you can use the following command (the last row will show you the total number of IP addresses in the database): tail /var/log/iprepd/iprepd.log One last thing you should know before we dive into the BIG-IP configuration details...IP Address Intelligence is available with BIG-IP version 11.2 and newer. So, get off that 10.x (or 9.x) box and upgrade to these features that are not only really cool but are also extremely important for the protection of your valuable business assets. BIG-IP Configuration You can find all the IP Address Intelligence goodness by navigating to Security >> Application Security >> IP Addresses >> IP Address Intelligence. The following screenshot provides the details of the configuration options for this feature. You may have noticed that in my lab version of the BIG-IP ASM I don't have IP Address Intelligence licensed, but also notice that if it were licensed, the blue text on the right side of the screen toward the top of the page would show the last time/day the database was updated. This is an easy way to check on the database updates from the GUI rather than the command line if you lean that way (don't worry, we don't judge). As you can see, there are several IP Address Intelligence Categories, and each bad IP address will fall into one (or more) of these categories. You have the option of Alarming or Blocking (or both) each category. Here's a quick list of what each category includes: Windows Exploits - includes active IP address offering or distributing malware, shell code, rootkits, worms, and viruses Web Attacks - includes cross site scripting, iFrame injection, SQL injection, cross domain injection, and domain password brute force BotNets - includes Botnet Command and Control channels and an infected zombie machine controlled by a Bot master Scanners - includes all reconnaissance, such as probes, host scan, domain scan, and password brute force Denial of Service - includes DoS, DDoS, anomalous syn flood, and anomalous traffic detection Infected Sources - includes IP addresses currently known to be infected with malware and IP addresses with an average "low" Reputation Index score. Phishing Proxies - includes IP addresses hosting phishing sites and other kind of fraud activities such as Ad Click Fraud and Gaming fraud Anonymous Proxy - includes IP addresses that provide proxy and anonymizing services and IP addresses registered with the Tor anonymity network The last thing I'll mention on the IP Address Intelligence Categories is that you can look up a specific IP address and see what category(ies) it falls into. Type this little beauty into the command line and you'll see the categories (if any) for the given IP address: iprep_lookup x.x.x.x (where x.x.x.x is the IP address) Don't Forget The Other Block... After you select the blocking settings for the categories listed above, you need to make one more stop before the ASM will block these bad boys. Head over to Security >> Application Security >> Blocking >> Settings and make sure you check the "Block" setting for the "Access from malicious IP address" setting. The screenshot below gives you all the details... Is My Name On That List? Now that we have figured out all the ways to block these bad IP addresses, let's turn our focus on how to let the good guys in. The BIG-IP ASM includes a feature called "IP Address Exceptions" that gives you the ability to explicitly allow certain IP addresses. You can navigate to Security >> Application Security >> IP Addresses >> IP Address Exceptions and you will see the following screen: As you can see, this one is pretty simple and straightforward. You simply add an IP Address and optional Netmask (255.255.255.255 will be the Netmask if you don't add one), and then you select one or more of the options listed. When the Policy Builder trusted IP option is enabled, the Policy Builder will consider traffic from this specified IP address as being safe. The Policy Builder will automatically add to the security policy any data logged from traffic sent from this IP address. Selecting this option also automatically adds this IP address to the Trusted IP Addresses setting on the Policy Building Configuration screen. If you don't enable this option, the Policy Builder will not consider traffic from this IP address as being any different than traffic from any other IP address. When the Ignore in Anomaly Detection option is enabled, the ASM will consider this IP address as legitimate and will not consider it when performing brute force prevention and web scraping detection. Once you enable this option, the system automatically adds this IP address to the IP Address Whitelist setting for Anomaly Detection. If you don't enable this option, the ASM will not consider traffic from this IP address as being any safer than traffic from any other IP address. When the Ignore in Learning Suggestions option is enabled, the ASM will not generate learning suggestions from traffic sent from this IP address. If you don't enable this option, the ASM will generate learning suggestions from the IP's traffic. When the Never block this IP Address option is enabled, guess what? The ASM will not block requests sent from this IP address...even if your security policy is configured to block all traffic. If you don't enable this option, the ASM will treat this IP address the same as all others. When the Never log traffic from this IP Address option is enabled, the system will not log requests or responses sent from this IP address...even if the traffic is illegal and even if your security policy is configured to log all traffic. If you don't enable this option, the ASM will simply continue to log traffic as specified in the settings of your security policy’s Logging Profile. On a related note, this option can be quite helpful when you have approved scanners testing your application on a regular basis. You may want to keep the scanner traffic out of your logs so that you can more easily focus on the user traffic. When the Ignore IP Address Intelligence option is enabled, the ASM will consider this IP address as legitimate even if it is found in the IP Address Intelligence database (you know...the one we just talked about). Once you enable this option, the system automatically adds this IP address to the IP Address Whitelist setting for IP Addresses Intelligence (you can check out the screenshot in the section above and see where these IP addresses would be listed). If you don't enable this option, the ASM will not consider traffic from this IP address as being any safer than traffic from any other IP address. Well, that's it for this edition of the BIG-IP ASM series...be sure to check back next time when we dive into some more really cool features! Update: Now that the article series is complete, I wanted to share the links to each article. If I add any more in the future, I'll update this list. What is the BIG-IP ASM? Policy Building The Importance of File Types, Parameters, and URLs Attack Signatures XML Security IP Address Intelligence and Whitelisting Geolocation Data Guard Username and Session Awareness Tracking Event Logging
Add or delete a parameter from multiple ASM policy or modify multiple ASM policy via API (iControlREST)
Problem this snippet solves: Sometimes it is necessary to add a parameter into multiple policy or all policies or to delete a parameter from multiple policies. If you have hundreds of asm polices and you try to do it via GUI, It takes long time and It is boring. For example, you have a new vulnerability scanner and you want to add all policies, or your contract with a security analysis company and you want to delete their IP address from all asm policies. If you have lots of policy, this gets big issue. How to use this snippet: I wrote a sample bash script, It adds an IP into the trusted IP list of multiple asm policy or deletes an IP from the trusted IP list of all asm policies. Firstly, you must choose which asm polices you want to change. Use this command to get list of the asm policies and write it into a file(asmPolicies.txt😞 curl -k -u <admin>:<password> -H "Content-Type: application/json" -X GET https://<F5 IP Address>/mgmt/tm/asm/policies?$select=id,name,fullPath | jq -r '.items[] | "\(.id) \(.name) \(.fullPath)"' > asmPolicies.txt This is the sample content of an asmPolicies.txt [root@f5 asmPolicies]# cat asmPolicies.txt x3yyOJTe3CvcWJDMqpnrgQ First /Common/First RqXf73h6qZY94EFGVDSlbg SecPolManual_First /Common/SecPolManual_First d928o8by0WBrWdW7oadMQg SecPol-Lab14 /Common/SecPol-Lab14 i4LnoF4GwMKRhTZ81RCeSQ SecPol-Lab14.2 /Common/SecPol-Lab14.2 kLoqhuDoa6bEeBjcrFo4VA SecPol-Lab15.1 /Common/SecPol-Lab15.1 DvE_fPp2tLUZvJi8cb8Rpg SecPol-Lab15.2 /Common/SecPol-Lab15.2 52dxLNxjExt6QRNvbg7fHA SecPol-Lab15.3 /Common/SecPol-Lab15.3 DcSvljkbLZQD19adkVdV3A SecPol-Lab16.2 /Common/SecPol-Lab16.2 rJ6Mt9sPxzgLu6WHyyifLg SecPol-Lab16.4 /Common/SecPol-Lab16.4 Sy_0vNh-5VXal-xDlMXMqw Single_URI /Common/Single_URI Hzyj8pZF6flV3VhTkCFkig SecPol-Lab22.2 /Common/SecPol-Lab22.2 sPR5LNQrrf29I1xZ8MtcRA SecPol-Lab16.4_2 /Common/SecPol-Lab16.4_2 Secondly, check the asmPolicies.txt, and erase the lines which policies you dont want to change Last, copy updateAsmPolicies.sh(attached) in a directory, then run updateAsmPolicies.sh with an appropriate command and parameter Usage: updateAsmPolicies.sh command parameter Commands: -a, -add add an IP address into the trusted IP list -d, -delete delete an IP address from the trusted IP list -c, -change <orgIP-newIP> delete the orgIP from the trusted IP list, then add the newIP into the trusted IP list updateAsmPolicies.sh -a 1.1.1.1 -> adds 1.1.1.1 into the trusted IP list updateAsmPolicies.sh -d 1.1.1.1 -> delete 1.1.1.1 from the trusted IP list that is it. This is just a sample. Code : #!/bin/bash #### #### AUTHOR: FARUK AYDIN --- farukaydin at yahoo.com #### DATE: 2018.01.25 #### This script adds or deletes or changes the trusted IP addresses in the asm policies #### #### Prerequest commands: ####echo ####curl ####jq ####shift ####cut ####cat function usage { echo "Usage: $0 command parameter" echo "Commands:" echo "-a, -add add an IP address into the trusted IP lists" echo "-d, -delete delete an IP address from the trusted IP lists" echo "-c, -change delete the orgIP from trusted IP lists, then add the newIP into the trusted IP lists" exit 0 } if [ ${#@} == 0 ]; then usage fi addingIP() { echo adding $2 into $1 policy; curl -sk -u ${f5user}:${f5pass} -H "Content-Type: application/json" -X POST -d '{"ipAddress":"'"$2"'","ipMask":"255.255.255.255","trustedByPolicyBuilder":true}' https://${f5host}/mgmt/tm/asm/policies/$1/whitelist-ips } deleteIP() { md5IP=$(curl -sk -u ${f5user}:${f5pass} -H "Content-Type: application/json" -X GET https://${f5host}/mgmt/tm/asm/policies/$1/whitelist-ips | jq -r '.items[] | select(.ipAddress=="'"$2"'") |"\(.id)"') if [ -z "$md5IP" ]; then echo $2 is not found in $1 policy; else echo deleting $1 from $1 policy; curl -sk -u ${f5user}:${f5pass} -H "Content-Type: application/json" -X DELETE https://${f5host}/mgmt/tm/asm/policies/$1/whitelist-ips/${md5IP} fi } UNKNOWN=() param=0 whatTodo="nothing" whatToDoN=0 f5user="admin" f5pass="password" f5host="192.168.1.245" while [[ $# -gt 0 ]] do key="$1" case $key in -a|--add) ((param++)) addIP="$2" whatToDo="adding a new trusted IP(${addIP}) to all asm policies" whatToDoN=1 shift # past argument shift # past value ;; -d|--delete) ((param++)) delIP="$2" whatToDo="deleting the trusted IP(${delIP}) from all asm policies" whatToDoN=2 shift # past argument shift # past value ;; -c|--change) ((param++)) changeIP="$2" orgIP=$(echo $changeIP | cut -f1 -d-) newIP=$(echo $changeIP | cut -f2 -d-) if [ "${orgIP}" == "${newIP}" ] ; then orgIP=$(echo $changeIP | cut -f1 -d:) newIP=$(echo $changeIP | cut -f2 -d:) fi whatToDo="changing the trusted IP(${orgIP}) with the new IP(${newIP}) in all asm policies" whatToDoN=3 shift # past argument shift # past value ;; --default) DEFAULT=YES ((param++)) shift # past argument ;; *) # unknown option UNKNOWN+=("$1") # save it in an array for later shift # past argument ;; esac done if [ "${param}" -gt 1 ] ; then echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" echo "!!!!!!!! you used ${param} commands !!!!!!!!" echo "!!! you must use only one command !!!" echo "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" usage fi echo "${whatToDo}", Option: "${whatToDoN}" for i in $(cat asmPolicies.txt | cut -d " " -f 1); do case $whatToDoN in 1) addingIP $i $addIP ;; 2) deleteIP $i $delIP ;; 3) deleteIP $i $orgIP addingIP $i $newIP ;; esac done Tested this on version: 12.1
iRule for IP restriction with multiple virt servers and multiple DGL of allowed IPs.
I have read through a multitude of threads, but my scenario seems a little unique. A little background so it all makes sense. We serve multiple customers with their own site, each site is a virt server and arte using the header to match rather than a single IP per. Each customer has a unique data group list of allowed IP's. We did not want a single list of allowed IP's in case a customer was emailed an incorrect URL by mistake, or just started browsing other dns records for the domain etc. We are changing our monitoring company and I would like to have a second data group list of IP's that are allowed so that any time there is a change for a source IP of monitoring, one of our offices etc, we don't have to touch 100 lists. The current iRule we are using is: when HTTP_REQUEST priority 100 { # This iRule will check if the client request is SITE.DOMAIN.COM and the client source IP is NOT a member of the datagroup specified which is a list of allowed IPs # If the client ip address is matched to the list of allowed IPs then it will bring up the web page, if it isnt, then it will bring up the COMPANY IP Forbidden Page. if { ( [string tolower [HTTP::host]] equals "1000-t01.DOMAIN.COM" ) and not ( [class match [IP::client_addr] equals COMPANY-1000-CUSTOMER-DG-Allow ] ) } { # log local0."Invalid CUSTOMER client IP: [IP::client_addr] - Blocking traffic" HTTP::respond 200 content [ifile get COMPANY_ip_forbidden] after 50 drop event disable } } How do I add the second data group, and allow if the source IP is in either of the two data groups?
DDoS IPI List - Whitelist NTP Servers
Problem this snippet solves: Legitimate IP address ranges of valid NTP Servers. Additional info can be found: https://github.com/c2theg/DDoS_lists How to use this snippet: Add to IPI feeds. Security >> Network Firewall >> IP Intelligence : Feed Lists Create new list: DDoS_Feeds Add rule. Give a good name, IE: whitelist_ntp_servers List Type: whitelist Update frequency: 3600 Default Blacklist Category: (Create new one) Whitelisted_Source Admin / Password: <Leave Blank> Tested this on version: 13.0DDoS IPI List - Whitelist DNS Servers
Problem this snippet solves: Legitimate IP address ranges of valid DNS Servers Additional info can be found: https://github.com/c2theg/DDoS_lists How to use this snippet: Add to IPI feeds. Security >> Network Firewall >> IP Intelligence : Feed Lists Create new list: DDoS_Feeds Add rule. Give a good name, IE: whitelist_dns_servers List Type: whitelist Update frequency: 3600 Default Blacklist Category: (Create new one) Whitelisted_Source Admin / Password: <Leave Blank> Tested this on version: 13.0DDoS IPI List - Bogons
Problem this snippet solves: Bogon IP address ranges to block traffic from Additional info can be found: https://github.com/c2theg/DDoS_lists How to use this snippet: Add to IPI feeds. Security >> Network Firewall >> IP Intelligence : Feed Lists Create new list: DDoS_Feeds Add rule. Give a good name, IE: blacklist_bogon List Type: blacklist Update frequency: 432000 Default Blacklist Category: (Create new one) Blacklisted_Source Admin / Password: <Leave Blank> Tested this on version: 13.0DDoS IPI List - Whitelist Update Domains
Problem this snippet solves: Legitimate IP address ranges and Domain Names of valid update servers. Additional info can be found: https://github.com/c2theg/DDoS_lists How to use this snippet: Add to IPI feeds. Security >> Network Firewall >> IP Intelligence : Feed Lists Create new list: DDoS_Feeds Add rule. Give a good name, IE: whitelist_update_servers List Type: whitelist Update frequency: 3600 Default Blacklist Category: (Create new one) Whitelisted_Source Admin / Password: <Leave Blank> Tested this on version: 13.0
