ASM/WAF Management Automation - TMOS
This post is to go over some of my thoughts on ASM/WAF management, and some custom solutions I've made to make it easier and more accurate. This will be highly technical and will apply to most use-cases. Written based on TMOS v15 software, but will be mostly applicable to all recent TMOS versions. Automatic Learning: Informational Automatic Learning will automatically accept most Learning Suggestions that hit 100% confidence Fully Automatic Learning will automatically accept all Learning Suggestions that hit 100% confidence If your application open to the public, it will likely get scanned a lot. Some of the Learning Suggestions will be to loosen your policies based on scanner traffic it sees, like this one that would've accepted a SQL injection attack suggestion from traffic that's clearly from a Veracode scanner Tips Reduce scanner traffic that hits ASM policy: Apply an iRule to your VIPs that have ASM policies that drops external IP-based requests. A lot of scanner traffic hits your VIPs by IP address instead of by hostname, and iRules process before ASM policies. So dropping IP-based traffic via iRule will drastically reduce the amount of bad ASM Learning Suggestions you get I use the attached iRule "irule_all_asm_VIPs" on all ASM VIPs. It has a line to increment the "triggered" field in a Statistics profile named "asm_irule_triggered," so you'll have to create the Statistics profile with the "triggered" field and attach it to the VIP before it will let you attach the iRule to the VIP. This makes sure you don't lose track of how many bad requests are hitting the device If ASM VIP has an existing iRule, irule_all_asm_VIPs must be attached first in order. See below for an automated way to attach this iRule to all ASM VIPs. Ongoing Maintenance Tasks: Enforce Ready Signatures ASM policies have a configured staging period (AKA Enforcement Readiness Period) for all new and modified Attack Signatures. Once this staging period is complete, they must be enforced manually Under Security > Application Security > Policy Building > Traffic Learning, there's an Enforcement Readiness Summary section where you can see ASM signatures that are Ready To Be Enforced. You can also Enforce Ready Entities on this page See here for a script I made that can be pushed from the Big-IQ to Enforce Ready Entities on all ASM Policies: https://community.f5.com/discussions/technicalforum/automate-asm-ready-to-be-enforced-attack-signatures/245055 If using Automatic Learning (not Fully Automatic Learning), periodically review Learning Suggestions at 100% confidence Manual Learning: Informational Manual Learning doesn’t accept any Learning Suggestions automatically Without the automation explained in this post or a full WAF team, it can be hard to manage many policies manually Tips Reduce scanner traffic that hits ASM policy. <see above> Ongoing Maintenance Enforce Ready Signatures <see above> Periodically Review Learning Suggestions at 100% confidence <see below for automation> Big-IQ Script Automation: Enforce Ready Entities <see above> https://community.f5.com/discussions/technicalforum/automate-asm-ready-to-be-enforced-attack-signatures/245055 Review Learning Suggestions Periodic reviews of Learning Suggestions can be tedious because there isn't a main overview page that tells you how many Learning Suggestions each policy has at 100% confidence. So without automation, you have to click through each policy one-by-one to see which policies need to be reviewed. I made the attached "Email Ready Learning Suggestions" script, that builds an HTML table line-by-line showing how many Learning Suggestions each policy has at 100% confidence, then uses the sendmail command to email it. There are 2 initial one-time setup commands at the top of the script you'll have to check first to make sure your F5 can send emails, and you'll have to replace email@example.com with your own email address in at the bottom. If it doesn't work after that, it's likely that your F5 SMTP traffic isn't allowed through a firewall Attach iRule and Statistics Profile to all ASM VIPs I made the attached "Attach iRule and Statistics Profile to all ASM VIPs" script that can be pushed from the Big-IQ to make a list of all VIPs with ASM policies, then attach the iRule if the VIP doesn't already have it. It doesn't remove existing iRules, and it puts this one first in the list if there are existing iRules Obviously for this to work, your F5 needs to already have the "irule_all_asm_VIPs" iRule and "asm_irule_triggered" Statistics profile created. The Statistics profile needs a field called "triggered" Cron Job Automation: Reset Log Statistics Profile Count, and Reset I use these 3 commands in a daily cron job that runs at midnight to send a syslog entry with the number of times an IP-Based request got dropped by the iRule, then reset the counter EXECUTIONS=$(tmsh show ltm profile statistics asm_irule_triggered | grep triggered | tail -n1 | sed s/"triggered"//) logger -p local0.notice "Daily number of IP-based requests that hit ASM iRule: $EXECUTIONS" tmsh reset-stats ltm profile statistics asm_irule_triggeredSMTP Smugglers Blues
The SMTP protocol has been vulnerable to email smuggling for decades. Many of the mail servers out there have mitigations in place to handle this vulnerability but not all of them, especially the quick libraries and add-ons you can find on web sites. Protecting your server from these attacks is simple with F5 BIG-IP Advanced WAF and our SMTP Protocol Security profiles. Read to learn how to give those bad actors the “Smugglers Blues”
F5 powered API security and management
Editor's Note: The F5 Beacon capabilities referenced in this article hosted on F5 Cloud Services are planning a migration to a new SaaS Platform - Check out the latest here. Introduction Application Programming Interfaces (APIs) enable application delivery systems to communicate with each other. According to a survey conducted by IDC, security is the main impediment to delivery of API-based services. Research conducted by F5 Labs shows that APIs are highly susceptible to cyber-attacks. Access or injection attacks against the authentication surface of the API are launched first, followed by exploitation of excessive permissions to steal or alter data that is reachable via the API. Agile development practices, highly modular application architectures, and business pressures for rapid development contribute to security holes in both APIs exposed to the public and those used internally. API delivery programs must include the following elements : (1) Automated Publishing of APIs using Swagger files or OpenAPI files, (2) Authentication and Authorization of API calls, (3) Routing and rate limiting of API calls, (4) Security of API calls and finally (5) Metric collection and visualization of API calls. The reference architecture shown below offers a streamlined way of achieving each element of an API delivery program. F5 solution works with modern automation and orchestration tools, equipping developers with the ability to implement and verify security at strategic points within the API development pipeline. Security gets inserted into the CI/CD pipeline where it can be tested and attached to the runtime build, helping to reduce the attack surface of vulnerable APIs. Common Patterns Enterprises need to maintain and evolve their traditional APIs, while simultaneously developing new ones using modern architectures. These can be delivered with on-premises servers, from the cloud, or hybrid environments. APIs are difficult to categorize as they are used in delivering a variety of user experiences, each one potentially requiring a different set of security and compliance controls. In all of the patterns outlined below, NGINX Controller is used for API Management functions such as publishing the APIs, setting up authentication and authorization, and NGINX API Gateway forms the data path. Security controls are addressed based on the security requirements of the data and API delivery platform. 1. APIs for highly regulated business Business APIs that involve the exchange of sensitive or regulated information may require additional security controls to be in compliance with local regulations or industry mandates. Some examples are apps that deliver protected health information or sensitive financial information. Deep payload inspection at scale, and custom WAF rules become an important mechanism for protecting this type of API. F5 Advanced WAF is recommended for providing security in this scenario. 2. Multi-cloud distributed API Mobile App users who are dispersed around the world need to get a response from the API backend with low latency. This requires that the API endpoints be delivered from multiple geographies to optimize response time. F5 DNS Load Balancer Cloud Service (global server load balancing) is used to connect API clients to the endpoints closest to them. In this case, F5 Cloud Services Essential App protect is recommended to offer baseline security, and NGINX APP protect deployed closer to the API workload, should be used for granular security controls. Best practices for this pattern are described here. 3. API workload in Kubernetes F5 service mesh technology helps API delivery teams deal with the challenges of visibility and security when API endpoints are deployed in Kubernetes environment. NGINX Ingress Controller, running NGINX App Protect, offers seamless North-South connectivity for API calls. F5 Aspen Mesh is used to provide East-West visibility and mTLS-based security for workloads. The Kubernetes cluster can be on-premises or deployed in any of the major cloud provider infrastructures including Google’s GKE, Amazon’s EKS/Fargate, and Microsoft’s AKS. An example for implementing this pattern with NGINX per pod proxy is described here, and more examples are forthcoming in the API Security series. 4. API as Serverless Functions F5 cloud services Essential App Protect offering SaaS-based security or NGINX App Protect deployed in AWS Fargate can be used to inject protection in front of serverless API endpoints. Summary F5 solutions can be leveraged regardless of the architecture used to deliver APIs or infrastructure used to host them. In all patterns described above, metrics and logs are sent to one or many of the following: (1) F5 Beacon (2) SIEM of choice (3) ELK stack. Best practices for customizing API related views via any of these visibility solutions will be published in the following DevCentral series. DevOps can automate F5 products for integration into the API CI/CD pipeline. As a result, security is no longer a roadblock to delivering APIs at the speed of business. F5 solutions are future-proof, enabling development teams to confidently pivot from one architecture to another. To complement and extend the security of above solutions, organizations can leverage the power of F5 Silverline Managed Services to protect their infrastructure against volumetric, DNS, and higher-level denial of service attacks. The Shape bot protection solutions can also be coupled to detect and thwart bots, including securing mobile access with its mobile SDK.
Trigger js challenge/Captcha for ip reputation/ip intelligence categories
Problem solved by this Code Snippet Because some ISP or cloud providers do not monitor their users a lot of times client ip addresses are marked as "spam sources" or "windows exploits" and as the ip addresses are dynamic and after time a legitimate user can use this ip addresses the categories are often stopped in the IP intelligence profile or under the ASM/AWAF policy. This usually happens in Public Clouds that do not monitor what their users do and the IP gets marked as bad then another good user after a day or two has this ip address and this causes the issue. For many of my clients I had to stop the ip reputation/ip intelligence category "spam sources" and in some cases "windows exploits" so having a javascript/captcha checks seems a nice compromise 😎 To still make use of this categories the users coming from those ip addresses can be forced to solve captcha checks or at least to be checked for javascript support! How to use this Code Snippet Have AWAF/ASM and ip intelligence licensed Add AWAF/ASM policy with irule support option (by default not enabled under the policy) or/and Bot profile under the Virtual server Optionally add IP intelligence profile or enable the Ip intelligence under the WAF policy without the categories that cause a lot of false positives, Add the irule and if needed modify the categories for which it triggers Do not forget to first create the data group, used in the code or delete that part of the code and to uncomment the Bot part of the code, if you plan to do js check and not captcha and maybe comment the captcha part ! Code Snippet Meta Information Version: 17.1.3 Coding Language: TCL Code You can find the code and further documentation in my GitHub repository: reputation-javascript-captcha-challlenge/ at main · Nikoolayy1/reputation-javascript-captcha-challlenge when HTTP_REQUEST { # Take the ip address for ip reputation/intelligence check from the XFF header if it comes from the whitelisted source ip addresses in data group "client_ip_class" if { [HTTP::header exists "X-Forwarded-For"] && [class match [IP::client_addr] equals "/Common/client_ip_class"] } { set trueIP [HTTP::header "X-Forwarded-For"] } else { set trueIP [IP::client_addr] } # Check if IP reputation is triggered and it is containing "Spam Sources" if { ([llength [IP::reputation $trueIP]] != 0) && ([IP::reputation $trueIP] contains "Spam Sources") }{ log local0. "The category is [IP::reputation $trueIP] from [IP::client_addr]" # Set the variable 1 or bulean true as to trigger ASM captcha or bot defense javascript set js_ch 1 } else { set js_ch 0 } # Custom response page just for testing if there is no real backend origin server for testing if {!$js_ch} { HTTP::respond 200 content { <html> <head> <title>Apology Page</title> </head> <body> We are sorry, but the site you are looking for is temporarily out of service<br> If you feel you have reached this page in error, please try again. </body> </html> } } } # when BOTDEFENSE_ACTION { # Trigger bot defense action javascript check for Spam Sources # if {$js_ch && (not ([BOTDEFENSE::reason] starts_with "passed browser challenge")) && ([BOTDEFENSE::action] eq "allow") }{ # BOTDEFENSE::action browser_challenge # } # } when ASM_REQUEST_DONE { # Trigger ASM captcha check only for users comming from Spam sources that have not already passed the captcha check (don't have the captcha cookie) if {$js_ch && [ASM::captcha_status] ne "correct"} { set res [ASM::captcha] if {$res ne "ok"} { log local0. "Cannot send captcha_challenge: \"$res\"" } } } Extra References: BOTDEFENSE::action ASM::captcha ASM::captcha_statusJavascript injecting systems effect on web application end users - a scenario review
Hello! ArvinF is back to share a scenario review where Javascript-injecting systems affected web application end users - web and mobile application. Problem Users are failing to login to a web application protected by BIG-IP ASM/Adv WAF and Shape Security Defense. The site owner notes that the authentication was failing for an unknown reason. There were ASM Support ID noted and an error informing to enable Javascript. Please enable JavaScript to view the page’s content. Your support ID is: xxxxxxxxxxxx Troubleshooting To understand the cause of the authentication failure, we gathered HTTP traffic through a HTTP sniffer. We used httpwatch and gathered HAR (HTTP Archive) files. The site was protected with both on-premise BIG-IP ASM/Adv WAF bot defense and back then, Shape Security Defense (now F5 Distributed Cloud Bot Defense). After the review of the HAR file in httpwatch, the following were noted: ASM blocks a request in a URL related to authentication with a Support ID in the response. There was also javascript code included and it references https[:]//s[.]go-mpulse[.]net/boomerang/. The authentication attempt failed with an error in the HTTP response: ...unable to process your request. Please try again later... BIG-IP ASM/Adv WAF related HTTP cookies from its various features such as Bot Defense Client Side challenges as TSPD_101* cookie was present and other TS cookies, which could also come from Bot defense and DoS profile and security policy configurations. There were also HTTP cookies coming from BIG-IP AVR - f5_cspm cookie was present. Application Visibility and Reporting (AVR) module provides detailed charts and graphs to give you more insight into the performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.). https://clouddocs.f5.com/training/community/analytics/html/index.html https://clouddocs.f5.com/api/irules/AVR_CSPM_INJECTION.html Seeing the javascript code referencing "/boomerang/" included in the ASM blocking response was interesting. Reviewing the HAR file, there were several instances of this "/boomerang/". This finding was inquired with the site owner and they noted that there is another system that is in the path between the end users and their web application - a CDN. The traffic flow is as follows: End user web browser / mobile application >>> CDN >>> FW >>> BIG-IP >>> web application On the BIG-IP Virtual Server that fronts the web application, F5 AVR profile, ASM/Adv WAF Bot defense, and security policy and Shape Security defense iRule are configured. From the F5 side, these were the products with features that may insert Javascript in the client-side response. As part of troubleshooting, to isolate the feature that might be causing the failing authentication for the web application, the bot defense profile was removed from the site's Virtual Server and the Shape Security iRule and AVR profile were left untouched. Site owner noted that the authentication works after this change. Shape Security Defense was implemented using an iRule to protect specific URIs. When the iRule was removed from the Virtual Server and the Bot defense and AVR profile were left on, the VS, Site owner noted that the authentication works after this change. But if both ASM/Adv WAF Bot defense and Shape Security Defense iRule is configured on the VS, the site's authentication fails. Per the site owner, there were no changes in the Bot Defense or Shape Security Defense iRule configurations prior to the incident and that these configurations were in place way before the incident. Site owners shared the findings with their respective internal teams for their review. Resolution Afterwards, Site owner shared that their site now works as expected and authentication works for the web application with no changes done on both ASM/Adv WAF Bot defense and Shape Security Defense iRule on the site's VS. The cause of the authentication failure was undetermined. A theory on the possible cause of the issue was perhaps, there was another system inserting Javascript code in the responses and it might have affected the authentication process of the web application by prevented that portion of the site from loading. Additional Troubleshooting Notes The data gathered during the troubleshooting were the qkview and HTTPWatch capture - HAR files. It would help if a packet capture was taken along with the HTTPWatch capture while the issue was happening to have a full view of the issue. Decrypt the packet capture to observe HTTP exchanges and to correlate it with HTTPWatch capture events. The corresponding BIG-IP ASM/Adv WAF application event logs, Bot Defense or DoS protection logs will also be helpful in the correlation. Having a visual idea on how the Security Policy, Bot Defense or DoS protection profile are configured is also helpful - so its good to have a screenshot of these. It helps in analysis when there is complete data. Gathering the asmqkview with report and traffic data and corresponding ASM and AVR db dumps helps in the analysis. asmqkview -s0 --add-request-log --include-traffic-data -f /var/tmp/`/bin/hostname`_asmqkview_`date +%Y%m%d%H%M%S`.tgz #mysqldump -uroot -p`perl -I/ts/packages -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw` DCC | gzip -9 > /shared/tmp/dcc.dump.gz # mysqldump -uroot -p`perl -I/ts/packages -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw` PLC | gzip -9 > /shared/tmp/plc.dump.gz # mysqldump -uroot -p`perl -I/ts/packages -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw` PRX | gzip -9 > /shared/tmp/prx.dump.gz # mysqldump -uroot -p`perl -I/ts/packages -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw` logdb | gzip -9 > /shared/tmp/logdb.dump.gz It would also help if the systems in the path of the web application are known and whether it has features that may interfere with the features of BIG-IP ASM/Adv WAF or Shape Security Defense. Per the findings, there was a CDN that was injecting javascript code in the HTTP response and it may have contributed to the authentication failure for the end users. Isolate potentially conflicting features by removing one of them one at a time and observe the HTTP responses. Per the reference configuration, BIG-IP ASM/Adv WAF, Shape Security Defense, and BIG-IP AVR worked well prior to the incident. boomerang The injected javascript code noted in the ASM blocking page response was loaded from https[:]//s[.]go-mpulse[.]net/boomerang/. Checking this reference, it was related to https://github.com/akamai/boomerang. boomerang is a JavaScript library that measures the page load time experienced by real users, commonly called RUM (Real User Measurement). It has the ability to send this data back to your server for further analysis. With boomerang, you find out exactly how fast your users think your site is. In BIG-IP, the similar product we have is BIG-IP AVR - Application Visibility and Reporting (AVR) - where it collects "performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.)." Organizations may have specific needs on data that they need to collect from their site/web application and using a customizable solution such as boomerang can help. That's It For Now I hope this scenario review on Javascript-injecting systems effect on web application end users will be helpful on your next troubleshooting and hopefully gives you guidance on what data to gather and look for and troubleshooting options. The F5 SIRT creates security-related content posted here in DevCentral, sharing the team’s security mindset and knowledge. Feel free to view the articles that are tagged with the following: F5 SIRT series-F5SIRT-this-week-in-security TWIS
Big-IP ASM automatically removes my hostname
, but I don't see the violation reaching the threshold of 100. Hello everyone, Recently, my service has encountered an issue. In the evening, while everything was running normally, I received a block warning from ASM. Upon checking, I found that my hostname was automatically removed from the policy by ASM. I am using fully automatic as per this link: https://my.f5.com/manage/s/article/K000134503. However, the problem is that when I checked for violations, I did not see any violations related to violations="Illegal host name." So, why did it reach the threshold of 100 and remove my hostname? Could this be a bug? I checked that there were no accept suggestions at that time, only violations="Illegal repeated parameter name," which I do not think is the issue. Thank you.
ASM::unblock not working for specific violation
Hello there, I set an iRule for ASM that should unblock a specific violation: when ASM_REQUEST_DONE { set asm_support_id [ASM::support_id] if { $uri starts_with "/my/uri" and [ASM::violation names] contains "VIOL_ENCODING" } then { ASM::unblock } } This rule is not triggered for that specific violation. What am I missing? I would also like to point out that if I remove the second part of the IF and leave just the uri match, the rule triggers just fine. I am 100% sure the violation I am targeting is correct. Any clues on what is wrong with it? Thank you!
