Trigger js challenge/Captcha for ip reputation/ip intelligence categories
Problem solved by this Code Snippet Because some ISP or cloud providers do not monitor their users a lot of times client ip addresses are marked as "spam sources" or "windows exploits" and as the ip addresses are dynamic and after time a legitimate user can use this ip addresses the categories are often stopped in the IP intelligence profile or under the ASM/AWAF policy. To still make use of this categories the users coming from those ip addresses can be forced to solve captcha checks or at least to be checked for javascript support! How to use this Code Snippet Have AWAF/ASM and ip intelligence licensed Add AWAF/ASM policy with irule support option (by default not enabled under the policy) or/and Bot profile under the Virtual server Optionally add IP intelligence profile or enable the Ip intelligence under the WAF policy without the categories that cause a lot of false positives, Add the irule and if needed modify the categories for which it triggers Do not forget to first create the data group, used in the code or delete that part of the code and to uncomment the Bot part of the code, if you plan to do js check and not captcha and maybe comment the captcha part ! Code Snippet Meta Information Version: 17.1.3 Coding Language: TCL Code You can find the code and further documentation in my GitHub repository: reputation-javascript-captcha-challlenge/ at main · Nikoolayy1/reputation-javascript-captcha-challlenge when HTTP_REQUEST { # Take the ip address for ip reputation/intelligence check from the XFF header if it comes from the whitelisted source ip addresses in data group "client_ip_class" if { [HTTP::header exists "X-Forwarded-For"] && [class match [IP::client_addr] equals "/Common/client_ip_class"] } { set trueIP [HTTP::header "X-Forwarded-For"] } else { set trueIP [IP::client_addr] } # Check if IP reputation is triggered and it is containing "Spam Sources" if { ([llength [IP::reputation $trueIP]] != 0) && ([IP::reputation $trueIP] contains "Spam Sources") }{ log local0. "The category is [IP::reputation $trueIP] from [IP::client_addr]" # Set the variable 1 or bulean true as to trigger ASM captcha or bot defense javascript set js_ch 1 } else { set js_ch 0 } # Custom response page just for testing if there is no real backend origin server for testing if {!$js_ch} { HTTP::respond 200 content { <html> <head> <title>Apology Page</title> </head> <body> We are sorry, but the site you are looking for is temporarily out of service<br> If you feel you have reached this page in error, please try again. </body> </html> } } } # when BOTDEFENSE_ACTION { # Trigger bot defense action javascript check for Spam Sources # if {$js_ch && (not ([BOTDEFENSE::reason] starts_with "passed browser challenge")) && ([BOTDEFENSE::action] eq "allow") }{ # BOTDEFENSE::action browser_challenge # } # } when ASM_REQUEST_DONE { # Trigger ASM captcha check only for users comming from Spam sources that have not already passed the captcha check (don't have the captcha cookie) if {$js_ch && [ASM::captcha_status] ne "correct"} { set res [ASM::captcha] if {$res ne "ok"} { log local0. "Cannot send captcha_challenge: \"$res\"" } } } Extra References: BOTDEFENSE::action ASM::captcha ASM::captcha_status
F5 AFM IP intelligence whitelist content
We have F5 AFM IP intelligence and we dont use the AFM license so no external feed list but only the local categories on the system. it is licensed with IP intelligence, now the question. How can we see the whitelist category content from CLI or GUI? Security ›› Network Firewall : IP Intelligence : Blacklist Categories whitelist Here in the menu you have the option Add to gategory and delete from category but how can we see the content of the whitelist category? Hence then we know if this list is still accurate....
traffic flow between IPI, application security policy, bot detection, DoS protection, irule, and geolocation
I want to know how the traffic flow between IPI, application security policy, bot detection, DoS protection, irule, and Geolocation (using irule for Geolocation). I am using Global IPI (mean IPI does not attached to any VS) and have an irule for Geolocation and only have module ASM and LTM (No APM and AFM). I understand that irule can be arranged by the order. The application security policy, bot detection, DoS protection, irule are attached to VS. Here is what I understand the traffic flow. The traffic hits Global IPI -> reached VS for irules in order (including Geolocation, I always put Geolocation at first place) -> Application security policy -> DoS -> Bot detection. Is this correct? Or will application security policy , Dos, Bot detection happen at the same time? What is the best practice for Geolocation? Using an irule for Geolocation or using Geolocation in application security policy?
IP-Intelligence and IP-Shunning
Dear Reader, With this article we have arrived at the “last” layer of protection, which is actually the first one from the hud-chain point of view: Blocking on the IP layer. The idea is rather simple: When you know that an IP address (or multiple IP addresses) is doing bad things, you block them as early as possible at the speed of line. IP-Intelligence Policy You can create IP-Intelligence (IPI) policies, which describe what should happen (Drop or Allow) with IPs (source or destination) belonging to a specific category. BIG-IP already has a bunch of pre-defined categories, but you can also add your own. Figure 1: Example on an IPI-Policy Important: A great benefit of using this, is that the “Drop” action is done on the FPGA level, which means in hardware. Basically, as soon as a packet hits the BIG-IP. But this is only possible when you set the logging to “No” (no logging, only dropping) or “Limited”. “Limited” means that some packets get leaked into software (CPU) for dropping them there and then they can be logged here as well. By default, every 256th packet goes into software when logging is set to “limited”. This is controlled by the sys db variable “dos.blleaklimit”. You can modify it to values like 1, 3, 7, 15, 31, 63, 127, 255, 511, 1023, and so on. The default value is 255. When you set the logging to “Yes”, then all packets will get logged, because the action (Drop) is done always in software. My recommendation is to use “Limited” logging to enable FPGA support for this feature. This is also what we call “shunning” when we drop on FPGA level. You can create multiple policies and assign them to different contexts: VS/PO or Route Domains to have individual configurations for your protected objects (services, networks, etc.) and you can also assign an IPI policy to the global level of BIG-IP, which means this will be executed first and for the whole box. Even when you want to block via IPI only on VS/PO level, my recommendation is to have a global IPI policy attached. It can be empty. Figure 2: Global Policy assignment Figure 3: Per VS/PO Policy assignment Figure 4: Per Route Domain Policy assignment Order of Mitigation Mitigation using the hardware (FPGA) is always done first. Next, traffic goes into software and can be dropped here as well. For IPI the following order takes place. Note that IPI mitigation done on Route Domain level is always in software. Figure 5: Mitigation Order How to get information about “bad” IPs? There are multiple options available. One is to use an external service (3 rd party), which does the categorization of IPs based on their reputation. If you have the IP Intelligence feature licensed, the information is being updated every 5 minutes. Another one is to use BIG-IP to automatically identify “bad actors” via DoS vectors. In my article Increasing accuracy using Bad Actor and Attacked Destination detection, I described how that works and for what it is useful for. But when BIG-IP identifies “bad actors” and this source IP(s) are behaving aggressive for a configurable amount of time, then it may make sense to drop everything coming from this IP(s) for a specific time interval. Figure 6: Bad Actor Detection In this example, the IP(s) which are identified as bad actors and behaving “bad” for 30 seconds get added to the category “denial_of_service” for 600 seconds. Once the “duration time” is over, they get released from the category and the process starts from the beginning, if they are still “bad”. Drop Attacked Destination Another use case to carefully consider is to drop based on attacked destination. Here you can drop all traffic going to a specific destination which is under attack. True, you stop the attack hitting the target, but you also drop all legitimate traffic going there, which would be a great success for the attacker. But, of course, if for whatever reason the attack would potential bring the BIG-IP to its knees this is an option you can choose. You would loose the destination that is under brutal attack but at least you safe all other services on the BIG-IP from being impacted. Figure 7: Attacked Destination Detection Use your up-stream router Think of the following scenario: An attack is about to saturate your Internet-pipe. You mitigate the bad actors on BIG-IP, but your pipe is still full. Why not “signal” the information of bad actors to your upstream router and drop the attackers already there, even before your pipe gets saturated? This is what you can do with using Remote Triggered Black Hole routing (RTBH) or Flowspec. By enabling “Allow External Advertisement” within your “Bad Actor Detection” or “Attacked Destination Detection” you can enable that functionality. Figure 8: External Advertisement I will cover the topic on how to use the Blacklist Publisher to “signal” bad actors including an action (Drop, Rate-limit, DSCP) to an upstream router via RTBH or Flowspec in a dedicated article. I will also explain in the same article how to use the “Scrubbing profile” to execute an action (redirect, rate-limit, DSCP, drop) on networks, PO/VS when these objects reach a certain bandwidth utilization. More options to add bad actors into categories Returning to how you can add IP(s) into categories. In addition to the already discussed options of using F5 subscription service and DoS vectors you can also add manually IPs. You may have your own sources and want to add this information into the categories. Here you have multiple options again. GUI On the Blacklist Categories page, you can manually add/delete IPs for a selected category. Figure 9: Add IP/Geo/FQDN to categories Feed List You can create your own Feed List using HTTP(s) or FTP to regularly download a file and update your categories. Figure 10: External Feed List Don´t forget to then enable your Feed List within your IPI-Policy! Figure 11: Enable Feed Lists TMSH Using tmsh you can also add IPs to categories. Here is an example: $ tmsh run security ip-intelligence category name denial_of_service ip-ttl add { 123.123.123.123,300 } Rest-API Another way is to use Rest-API. Here is an example: $ curl -sk -u sVen:mySecrectPW -H "Content-Type: application/json" -X POST -d '{"command":"run","utilCmdArgs":"name denial_of_service ip-ttl add {123.123.123.123,300}"}' https://mgmnt-address-big-ip/mgmt/tm/security/ip-intelligence/category/ | python -m json.tool Whitelist IPs for IPI/shunning If you want to whitelist IP(s) from not being shunned on IPI level, you need to add this IP(s) to the “whitelist” category. Please we aware that this only works when you use an external feed to add the IP(s). Figure 12: Feed List Properties More useful CLI commands To get an idea on hits per category, you can use a “tmctl” command: # tmctl -w 180 ip_intelligence_stat It gives you an overview of hits per context, category and drop list type. Figure 13: Stats for categories Here you can also see that you can not only drop based on IP. You can also use GEO information and FQDN. If you want to verify the status of an IP, you can use this command: # tmsh show security ip-intelligence info address <ip-address> Figure 13: Example of status of a GEOIP Conclusion IPI/shunning is a very effective and hardware driven way of blocking traffic based on IP information. Especially on amplification and reflection attacks it is a very useful because you are mostly able to identify IP addresses causing the high load. Often operators do also block traffic from countries on which they are sure they do not communicate with for specific services. After finishing the explanation of this last (first) mitigation layer, you hopefully have a better understanding on how F5 L3/4/DNS DDoS mitigation works. Hardware (FPGA) support is an essential element of successful blocking high packet rates. These high-performance chips get dynamically programmed via the “smart” component, the software which runs on CPU, and which also needs to be protected. Therefore, you can configure the DoS mitigation mechanism for protecting your services (server, network, …) behind the BIG-IP, but also BIG-IP itself! Figure 14: Multi-layer DDoS mitigation architecture F5 also provides the ability to protect with hardware (FPGA) support in virtualized environments. Since more and more data center go virtual, the requirement of running DDoS mitigation in virtualized environments has increased. But, because FPGA support is needed for this type of DDoS mitigation, since CPU alone is usually not powerful enough, it can present a problem. Together with Intel F5 provides a strong solution. This solution uses SmartNICs (network card with FPGA on board) within your Hypervisor. DHD (DDoS Hybrid Defender) and AFM (Advanced Firewall Manager) can program these FPGAs in basically the same way as it can be done on dedicated Hardware appliances. This again gives you the power needed for DDoS mitigation, plus the flexibility of virtualized environments. If you are interested in learning more about this, I would like to recommend the article of my colleague Ryan Howard: https://devcentral.f5.com/s/articles/Mitigating-40Gbps-DDoS-Attacks-with-the-new-BIG-IP-VE-for-SmartNICs-Solution My next article will focus on logging and reporting DDoS events. Together, my colleague Mohamed Shaath and I created a DDoS Dashboard including DDoS statistics visibility on all vectors based on an ELK stack. We will share in that article how to install and use it. Thank you, Sven MuellerDDoS IPI List - Whitelist Update Domains
Problem this snippet solves: Legitimate IP address ranges and Domain Names of valid update servers. Additional info can be found: https://github.com/c2theg/DDoS_lists How to use this snippet: Add to IPI feeds. Security >> Network Firewall >> IP Intelligence : Feed Lists Create new list: DDoS_Feeds Add rule. Give a good name, IE: whitelist_update_servers List Type: whitelist Update frequency: 3600 Default Blacklist Category: (Create new one) Whitelisted_Source Admin / Password: <Leave Blank> Tested this on version: 13.0DDoS IPI List - Whitelist NTP Servers
Problem this snippet solves: Legitimate IP address ranges of valid NTP Servers. Additional info can be found: https://github.com/c2theg/DDoS_lists How to use this snippet: Add to IPI feeds. Security >> Network Firewall >> IP Intelligence : Feed Lists Create new list: DDoS_Feeds Add rule. Give a good name, IE: whitelist_ntp_servers List Type: whitelist Update frequency: 3600 Default Blacklist Category: (Create new one) Whitelisted_Source Admin / Password: <Leave Blank> Tested this on version: 13.0DDoS IPI List - Whitelist DNS Servers
Problem this snippet solves: Legitimate IP address ranges of valid DNS Servers Additional info can be found: https://github.com/c2theg/DDoS_lists How to use this snippet: Add to IPI feeds. Security >> Network Firewall >> IP Intelligence : Feed Lists Create new list: DDoS_Feeds Add rule. Give a good name, IE: whitelist_dns_servers List Type: whitelist Update frequency: 3600 Default Blacklist Category: (Create new one) Whitelisted_Source Admin / Password: <Leave Blank> Tested this on version: 13.0DDoS IPI List - Bogons
Problem this snippet solves: Bogon IP address ranges to block traffic from Additional info can be found: https://github.com/c2theg/DDoS_lists How to use this snippet: Add to IPI feeds. Security >> Network Firewall >> IP Intelligence : Feed Lists Create new list: DDoS_Feeds Add rule. Give a good name, IE: blacklist_bogon List Type: blacklist Update frequency: 432000 Default Blacklist Category: (Create new one) Blacklisted_Source Admin / Password: <Leave Blank> Tested this on version: 13.0
best way to reject SSL Connections
We use IPI and we drop the requests via iRule because we cannot use ASM at every VS. today we reject the connect in then CLIENT_ACCEPTED but the result is a SSL Handshake failed for TCP xxx.xxx.xxx.xxx:nnn -> xxx.xxx.xxx.xxx:nnn in ltm log. do we have to accept that or is there a better way to reject connections like that? let the connect go on until HTTP_REQUEST is not option because we have the same problem when we use a required Client Certificate where we check for example the UPN and we like to drop the connection if the UPN is invalid or missing.
