From ASM to Advanced WAF: Advancing your Application Security
TL;DR: As of April 01, 2021, F5 has officially placed Application Security Manager (ASM) into End of Sale (EoS) status, signifying the eventual retirement of the product. (F5 Support Announcement - K72212499 ) Existing ASM,or BEST bundle customers, under a valid support contract running BIG-IP version 14.1 or greater can simply reactivate their licenses to instantly upgrade to Advanced WAF (AdvWAF) completely free of charge. Introduction Protecting your applications is becoming more challenging every day; applications are getting more complex, and attackers are getting more advanced. Over the years we have heard your feedback that managing a Web Application Firewall (WAF) can be cumbersome and you needed new solutions to protect against the latest generation of attacks. Advanced Web Application Firewall, or AdvWAF, is an enhanced version of the Application Security Manager (ASM) product that introduces new attack mitigation techniques and many quality-of-life features designed to reduce operational overhead. On April 01, 2021 – F5 started providing free upgrades for existing Application Security Manager customers to the Advanced WAF license. Keep on reading for: A brief history of ASM and AdvWAF How the AdvWAF license differs from ASM (ASM vs AdvWAF How to determine if your BIG-IPs are eligible for this free upgrade Performing the license upgrade How did we get here? For many years, ASM has been the gold standard Web Application Firewall (WAF) used by thousands of organizations to help secure their most mission-critical web applications from would-be attackers. F5 acquired the technology behind ASM in 2004 and subsequently ‘baked’ it into the BIG-IP product, immediately becoming the leading WAF product on the market. In 2018, after nearly 14 years of ASM development, F5 released the new, Advanced WAF license to address the latest threats. Since that release, both ASM and AdvWAF have coexisted, granting customers the flexibility to choose between the traditional or enhanced versions of the BIG-IP WAF product.As new features were released, they were almost always unique to AdvWAF, creating further divergence as time went on, and often sparking a few common questions (all of which we will inevitably answer in this very article) such as: Is ASM going away? What is the difference between ASM and AdvWAF? Will feature X come to ASM too? I need it! How do I upgrade from ASM to AdvWAF? Is the BEST bundle no longer really the BEST? To simplify things for our customers (and us too!), we decided to announce ASM as End of Sale (EoS), starting on April 01, 2021. This milestone, for those unfamiliar, means that the ASM product can no longer be purchased after April 01 of this year – it is in the first of 4 stages of product retirement. An important note is that no new features will be added to ASM going forward. So, what’s the difference? A common question we get often is “How do I migrate my policy from ASM to AdvWAF?” The good news is that the policies are functionally identical, running on BIG-IP, with the same web interface, and have the same learning engine and underlying behavior. In fact, our base policies can be shared across ASM, AdvWAF, and NGINX App Protect (NAP). The AdvWAF license simply unlocks additional features beyond what ASM has, that is it – all the core behaviors of the two products are identical otherwise. So, if an engineer is certified in ASM and has managed ASM security policies previously, they will be delighted to find that nothing has changed except for the addition of new features. This article does not aim to provide an exhaustive list of every feature difference between ASM and AdvWAF. Instead, below is a list of the most popular features introduced in the AdvWAF license that we hope you can take advantage of. At the end of the article, we provide more details on some of these features: Secure Guided Configurations Unlimited L7 Behavioral DoS DataSafe (Client-side encryption) OWASP Compliance Dashboard Threat Campaigns (includes Bot Signature updates) Additional ADC Functionality Micro-services protection Declarative WAF Automation I’m interested, what’s the catch? There is none! F5 is a security company first and foremost, with a mission to provide the technology necessary to secure our digital world. By providing important useability enhancements like Secure Guided Config and OWASP Compliance Dashboard for free to existing ASM customers, we aim to reduce the operational overhead associated with managing a WAF and help make applications safer than they were yesterday - it’s a win-win. If you currently own a STANDALONE, ADD-ON or BEST Bundle ASM product running version 14.1 or later with an active support contract, you are eligible to take advantage of this free upgrade. This upgrade does not apply to customers running ELA licensing or standalone ASM subscription licenses at this time. If you are running a BIG-IP Virtual Edition you must be running at least a V13 license. To perform the upgrade, all you need to do is simply REACTIVATE your license, THAT IS IT! There is no time limit to perform the license reactivation and this free upgrade offer does not expire. *Please keep in mind that re-activating your license does trigger a configuration load event which will cause a brief interruption in traffic processing; thus, it is always recommended to perform this in a maintenance window. Step 1: Step 2: Choose “Automatic” if your BIG-IP can communicate outbound to the Internet and talk to the F5 Licensing Server. Choose Manual if your BIG-IP cannot reach the F5 Licensing Server directly through the Internet. Click Next and the system will re-activate your license. After you’ve completed the license reactivation, the quickest way to know if you now have AdvWAF is by looking under the Security menu. If you see "Guided Configuration”, the license upgrade was completed successfully. You can also login to the console and look for the following feature flags in the /config/bigip.license file to confirm it was completed successfully by running: grep -e waf_gc -e mod_waf -e mod_datasafe bigip.license You should see the following flags set to enabled: Waf_gc: enabled Mod_waf: enabled Mod_datasafe: enabled *Please note that the GUI will still reference ASM in certain locations such as on the resource provisioning page; this is not an indication of any failure to upgrade to the AdvWAF license. *Under Resource Provisioning you should now see that FPS is licensed. This will need to be provisioned if you plan on utilizing the new AdvWAF DataSafe feature explained in more detail in the Appendix below. For customers with a large install base, you can perform license reactivation through the CLI. Please refer to the following article for instructions: https://support.f5.com/csp/article/K2595 Conclusion F5 Advanced WAF is an enhanced WAF license now available for free to all existing ASM customers running BIG-IP version 14.1 or greater, only requiring a simple license reactivation. The AdvWAF license will provide immediate value to your organization by delivering visibility into the OWASP Top 10 compliance of your applications, configuration wizards designed to build robust security policies quickly, enhanced automation capabilities, and more. If you are running ASM with BIG-IP version 14.1 or greater, what are you waiting for? (Please DO wait for your change window though 😊) Acknowledgments Thanks to Brad Scherer , John Marecki , Michael Everett , and Peter Scheffler for contributing to this article! Appendix: More details on select AdvWAF features Guided Configurations One of the most common requests we hear is, “can you make WAF easier?” If there was such a thing as an easy button for WAF configurations, Guided Configs are that button. Guided Configurations easily take you through complex configurations for various use-cases such as Web Apps, OWASP top 10, API Protection, DoS, and Bot Protection. L7DoS – Behavioral DoS Unlimited Behavioral DoS - (BaDoS) provides automatic protection against DoS attacks by analyzing traffic behavior using machine learning and data analysis. With ASM you were limited to applying this type of DoS profile to a maximum of 2 Virtual Servers. The AdvWAF license completely unlocks this capability, removing the 2 virtual server limitation from ASM. Working together with other BIG-IP DoS protections, Behavioral DoS examines traffic flowing between clients and application servers in data centers, and automatically establishes the baseline traffic/flow profiles for Layer 7 (HTTP) and Layers 3 and 4. DataSafe *FPS must be provisioned DataSafe is best explained as real-time L7 Data Encryption. Designed to protect websites from Trojan attacks by encrypting data at the application layer on the client side. Encryption is performed on the client-side using a public key generated by the BIG-IP system and provided uniquely per session. When the encrypted information is received by the BIG-IP system, it is decrypted using a private key that is kept on the server-side. Intended to protect, passwords, pins, PII, and PHI so that if any information is compromised via MITB or MITM it is useless to the attacker. DataSafe is included with the AdvWAF license, but the Fraud Protection Service (FPS) must be provisioned by going to System > Resource Provisioning: OWASP Compliance Dashboard Think your policy is air-tight? The OWASP Compliance Dashboard details the coverage of each security policy for the top 10 most critical web application security risks as well as the changes needed to meet OWASP compliance. Using the dashboard, you can quickly improve security risk coverage and perform security policy configuration changes. Threat Campaigns (includes Bot Signature updates) Threat campaigns allow you to do more with fewer resources. This feature is unlocked with the AdvWAF license, it, however, does require an additional paid subscription above and beyond that. This paid subscription does NOT come with the free AdvWAF license upgrade. F5’s Security Research Team (SRT) discovers attacks with honeypots – performs analysis and creates attack signatures you can use with your security policies. These signatures come with an extremely low false-positive rate, as they are strictly based on REAL attacks observed in the wild. The Threat Campaign subscription also adds bot signature updates as part of the solution. Additional ADC Functionality The AdvWAF license comes with all of the Application Delivery Controller (ADC) functionality required to both deliver and protect a web application. An ASM standalone license came with only a very limited subset of ADC functionality – a limit to the number of pool members, zero persistence profiles, and very few load balancing methods, just to name a few. This meant that you almost certainly required a Local Traffic Manager (LTM) license in addition to ASM, to successfully deliver an application. The AdvWAF license removes many of those limitations; Unlimited pool members, all HTTP/web pertinent persistence profiles, and most load balancing methods, for example.Automation of F5 Distributed Cloud Platform Client-Side Defense feature - Part I
Objective: The purpose of this article is to automate F5 Distributed Cloud Platform Client-Side Defense feature (F5 XC CSD) detection of malicious 3rd party domains and integrating code in GitHub. This article shows how we can use the Github available Actions workflow to provide the flexibility of updating existing infrastructure after every change using CI/CD event triggers. In this article we showed a small use case of CI/CD deployment using GitHub Actions, Terraform and Python developed in a generic way where users can bring up the complete setup within a few clicks. For more details about this feature please refer: https://community.f5.com/t5/technical-articles/javascript-supply-chains-magecart-and-f5-xc-client-side-defense/ta-p/296612 Overview: Client-Side Defense (CSD) feature provides a web application protection solution against Magecart style and similar malicious JavaScript attacks. This solution supports below features: Detection: A continuously evolving signal set allows CSD to understand when scripts on web pages exhibit signs of exfiltration. CSD detects network requests made by malicious scripts that attempt to exfiltrate PII data. Alerting: CSD generates timely alerts on the behavior of malicious scripts, provided by a continuously improving Analysis Engine. Mitigation: CSD detects threats in real-time and provides enforcement with one-click mitigation. Automation Design: As part of this automation, we are deploying a demo application in AWS and NGINX web service which hosts a simple web login page. The demo application has a malicious 3rd party Java script which captures the provided username and passwords during the login and sends these details to a malicious control server which keeps recording these credentials. Once the demo app is deployed, we are then configuring the origin pool and load balancer in F5 XC and generating web login traffic using Selenium script. Once traffic is logged in F5 XC platform, CSD feature will detect malicious domain network and will display domain in client-Side defense dashboard. After researching the 3rd party domain details customers can either approve or mitigate these network requests. Above workflow is integrated using GitHub Actions file which ensures dynamic deployment of the demo app and F5 XC load balancer which can be exposed using public domain name. Note: Currently this repo code covers automation till CSD malicious domain detection only and will cover mitigation part in the upcoming article of this series. Code is available here. Deployment steps: Security users can simply clone repo, update variables.tf as per their infra and run workflow which will bring entire infrastructure in few mins. They can login to the F5 XC console and explore the functionality of Client-Side Defense in an interactive way. Users can use this as a plugin to demonstrate CSD feature. Conclusion: This article demonstrated how we can leverage the power of CI/CD to create or upgrade our existing infrastructure and maintain the testing scope of Client Side Defense feature. For further information check the links below: F5 Distributed Cloud Platform (Link) F5 Distributed Cloud Client-Side Defense Overview (Link) F5 Distributed Cloud Client-Side Defense Docs (Link)VIPTest: Rapid Application Testing for F5 Environments
VIPTest is a Python-based tool for efficiently testing multiple URLs in F5 environments, allowing quick assessment of application behavior before and after configuration changes. It supports concurrent processing, handles various URL formats, and provides detailed reports on HTTP responses, TLS versions, and connectivity status, making it useful for migrations and routine maintenance.Automation of Malicious User detection/mitigation using F5 Distributed Cloud Platform
Introduction: In today’s modern world where attackers are leveraging the benefits of readily available automated attack tools it is highly recommended to go with security driven framework which helps in negating the impact of possible attack vectors used by these attackers. F5 Distributed Cloud (F5 XC) platform comes with a wide range of signature set for known attacks and has a machine learning capability to mitigate possible threats from malicious users by analyzing the user behavior. In this article, we have discussed the demo scenarios we are covering as part of the automation using GitHub Actions, Terraform and Python. For more information, please have a look at these articles: AI/ML detection of Malicious Users using F5 Distributed Cloud WAAP – Part I AI/ML detection of Malicious Users using F5 Distributed Cloud WAAP – Part II AI/ML detection of Malicious Users using F5 Distributed Cloud WAAP – Part III Demonstration: We’ve come up with automation scripts to build up the infrastructure for malicious user detection and mitigation in XC console and malicious events triggering script. You can use the github workflows to see how effectively XC WAAP can detect and mitigate malicious user events The repository consists of two workflows covering demo scenarios for malicious user detection and mitigation using XC WAAP: Single LB malicious user detection and default mitigation of high-risk IPs: In this scenario we are bringing up a HTTP load balancer and configure it to detect and mitigate malicious user events using default mitigation rule. In the second part of this demo, we will generate tor requests and fetch the logs from XC console to validate the detection and mitigation action Multi LB malicious user detection and custom mitigation of WAF security events: In this scenario we are bringing up a HTTPS load balancer with an app type enabling detection with custom app settings object, app firewall in blocking mode and custom malicious user mitigation policy. In the second part of this scenario, we are generating XSS attacks and validate the logs fetched from XC console. Repository Link: Automation demonstration for malicious user detection and mitigation feature of F5 XC Platform Conclusion: By using the repository, you will see F5 Distributed Cloud platform’s efficiency to detect users exhibiting suspicious behavior and perform mitigation actions on them safeguarding end application from possible attacks Hope you liked the demonstration. Please feel free to raise your concerns via GitHub. Thank you! For further information click the links below: F5 Distributed Cloud Platform F5 Distributed Cloud Services F5 Distributed Cloud WAAPNGINX Virtual Machine Building with cloud-init
Traditionally, building new servers was a manual process. A system administrator had a run book with all the steps required and would perform each task one by one. If the admin had multiple servers to build the same steps were repeated over and over. All public cloud compute platforms provide an automation tool called cloud-init that makes it easy to automate configuration tasks while a new VM instance is being launched. In this article, you will learn how to automate the process of building out a new NGINX Plus server usingcloud-init.
BIG-IP Next – Introduction to the Blueprints API
If you have ever attempted to automate the BIG-IP configuration, you are probably familiar with F5’s AS3 extension. Although AS3 is supported in BIG-IP Next, there is another API that might be the better option if you haven’t started your migration journey up until now. This is called the Blueprints API. In this article, I want to show you how to use it to automate your applications with AS3. Overview When you use the BIG-IP Next GUI, you instantly see the benefits of having a centrally managed configuration across all your BIG-IP instances. The steps to create an application service in the GUI now have a siloed setup where you define 4 main sections separately: Application Properties Virtual Server Properties Pool Properties Deployment Properties Each one of these sections allows you to adjust areas of your application service while still having a way to manage configurations across multiple BIG-IP instances. In other words, you can define one pool under the pool properties, but still have different pool members under the deployment properties for each of your BIG-IP instances. This creates a centrally managed application service that does not require the exact same configuration in each environment. When you perform these tasks in the GUI, BIG-IP Next is generating its own API calls internally. It takes each of your configuration items outlined in the 4 sections above and defines the application service as a Blueprint. This Blueprint is then used to modify anything about the configuration/deployment moving forward. If you aren’t a fan of using a GUI and you are trying to automate, this same exact API is exposed to you as well. This means we get to use the same centrally managed configuration in our API calls. It also means that we can easily automate existing application services by simply using the API to manage them moving forward. So what does this Blueprint API look like? Below is sample JSON used to create a Blueprint called “bobs-blueprint”: { "name":"bobs-blueprint", "set_name": "Examples", "template_name": "http", "parameters": { "application_description": "", "application_name": "bobs-blueprint", "enable_Global_Resiliency": false, "pools": [ { "loadBalancingMode": "round-robin", "monitorType": [ "http" ], "poolName": "juice", "servicePort": 8080 } ], "virtuals": [ { "FastL4_TOS": 0, "FastL4_idleTimeout": 600, "FastL4_looseClose": true, "FastL4_looseInitialization": true, "FastL4_pvaAcceleration": "full", "FastL4_pvaDynamicClientPackets": 1, "FastL4_pvaDynamicServerPackets": 0, "FastL4_resetOnTimeout": true, "FastL4_tcpCloseTimeout": 43200, "FastL4_tcpHandshakeTimeout": 43200, "InspectionServicesEnum": [], "TCP_idle_timeout": 60, "UDP_idle_timeout": 60, "ciphers": "DEFAULT", "ciphers_server": "DEFAULT", "enable_Access": false, "enable_FastL4": false, "enable_FastL4_DSR": false, "enable_HTTP2_Profile": false, "enable_HTTP_Profile": false, "enable_InspectionServices": false, "enable_SsloPolicy": false, "enable_TCP_Profile": false, "enable_TLS_Client": false, "enable_TLS_Server": false, "enable_UDP_Profile": false, "enable_WAF": false, "enable_iRules": false, "enable_mirroring": true, "enable_snat": true, "iRulesEnum": [], "multiCertificatesEnum": [], "perRequestAccessPolicyEnum": "", "pool": "juice", "snat_addresses": [], "snat_automap": true, "tls_c_1_1": false, "tls_c_1_2": true, "tls_c_1_3": false, "tls_s_1_2": true, "tls_s_1_3": false, "trustCACertificate": "", "virtualName": "bobs-vs", "virtualPort": 80 } ] } } As you can see, the structure of this JSON is siloed in a very similar way to the GUI: Note: For those readers who are wondering where the Deployment section is, that is handled in a separate API call after the blueprint has been created. I’ll discuss that in more detail later. In the sections below, I’ll review a few of the API endpoints you can use with some steps on how to perform the following common tasks: Viewing an existing Blueprint Creating a new Blueprint Deploying a Blueprint Viewing an Existing Blueprint Before we start creating a new Blueprint from scratch, it is probably a good idea to explain how we can view a list of our current Blueprints. To do so, we simply make a GET request to the following API endpoint: GET https://{{bigip_next_cm_mgmt_ip}}/api/v1/spaces/default/appsvcs/blueprints This will return a list of every Blueprint created by the GUI and/or API. Below is an example output: { "_embedded": { "appsvcs": [ { "_links": { "self": { "href": "/api/v1/spaces/default/appsvcs/blueprints//3f2ef264-cf09-45c8-a925-f2c8fccf09f6" } }, "created": "2024-06-25T13:32:22.160399Z", "deployments": [ { "id": "1e5f9c06-8800-4ab7-ad5e-648d55b83b68", "instance_id": "ce179e66-b075-4068-bc4e-8e212954da49", "target": { "address": "10.2.1.3" }, "parameters": { "pools": [ { "isServicePool": false, "poolMembers": [ { "address": "10.1.3.100", "name": "old" }, { "address": "10.2.3.100", "name": "new" } ], "poolName": "juice" } ], "virtuals": [ { "allow_networks": [], "enable_allow_networks": false, "virtualAddress": "10.2.2.11", "virtualName": "juice-shop" } ] }, "last_successful_deploy_time": "2024-06-25T17:46:00.193649Z", "modified": "2024-06-25T17:46:00.193649Z", "last_record": { "id": "cb6a06a1-c66d-41c3-a747-9a27b101a0f1", "task_id": "6a65642d-810b-4194-9693-91a15f6d1ef0", "created_application_path": "/applications/tenantSrLEVevFQnWwXT90590F3USQ/juice-shop", "start_time": "2024-06-25T17:45:55.392732Z", "end_time": "2024-06-25T17:46:00.193649Z", "status": "completed" } }, { "id": "001e14e8-7900-482a-bdd4-aca35967a5cc", "instance_id": "0546acf5-3b88-422d-a948-28bbf0973212", "target": { "address": "10.2.1.4" }, "parameters": { "pools": [ { "isServicePool": false, "poolMembers": [ { "address": "10.1.3.100", "name": "old" }, { "address": "10.2.3.100", "name": "new" } ], "poolName": "juice" } ], "virtuals": [ { "allow_networks": [], "enable_allow_networks": false, "virtualAddress": "10.2.2.12", "virtualName": "juice-shop" } ] }, "last_successful_deploy_time": "2024-06-25T17:46:07.94836Z", "modified": "2024-06-25T17:46:07.94836Z", "last_record": { "id": "8a8a29be-e56b-4ef1-bf6a-92f7ccc9e9b7", "task_id": "0632cc18-9f0f-4a5e-875a-0d769b02e19b", "created_application_path": "/applications/tenantSrLEVevFQnWwXT90590F3USQ/juice-shop", "start_time": "2024-06-25T17:45:55.406377Z", "end_time": "2024-06-25T17:46:07.94836Z", "status": "completed" } } ], "deployments_count": { "total": 2, "completed": 2 }, "description": "", "fqdn": "", "gslb_enabled": false, "id": "3f2ef264-cf09-45c8-a925-f2c8fccf09f6", "modified": "2024-06-25T17:32:09.174243Z", "name": "juice-shop", "set_name": "Examples", "successful_instances": 2, "template_name": "http", "tenant_name": "tenantSrLEVevFQnWwXT90590F3USQ", "type": "FAST" } ] }, "_links": { "self": { "href": "/api/v1/spaces/default/appsvcs/blueprints/" } }, "count": 1, "total": 1 } In the example above, I only have one Blueprint in my BIG-IP Next CM instance. If we look deeper into the output, we can start to see some detail around the configuration parameters as well as the deployment parameters for each BIG-IP. There is also an ID field in the JSON that we can use to reference a specific Blueprint. In the example above, we have: "id": "3f2ef264-cf09-45c8-a925-f2c8fccf09f6" This is important because as we start to modify, deploy, or delete our existing blueprints, we will need this ID to be able to make changes. We can also use this ID to view more detail on a specific Blueprint rather than an entire list of all Blueprints. To do so, we simply follow make a request to the endpoint below: GET https://{{bigip_next_cm_mgmt_ip}}/api/v1/spaces/default/appsvcs/blueprints/{{Blueprint_id}} In our example, this would be: GET https://{{bigip_next_cm_mgmt_ip}}/api/v1/spaces/default/appsvcs/blueprints/3f2ef264-cf09-45c8-a925-f2c8fccf09f6 The output from this API call provides robust detail on the Blueprint. It is probably too much detail to paste in an article like this, but there are some examples here if interested: https://clouddocs.f5.com/products/bigip-next/mgmt-api/latest/ApiReferences/bigip_public_api_ref/r_openapi-next.html#tag/Application-Services/operation/GetApplicationByID Viewing a Blueprint like this can provide us with the latest configuration of our application service so that we can ensure we are using the most up-to-date files. It also can provide us with some templates/example configurations that we can use to create new application services moving forward. Creating a new Blueprint Now that we have a pretty good understanding of the JSON structure and we know how to view some examples of Blueprints that have already been created, we can simply use them as a reference and create our own Blueprint from scratch. The basic format for creating a new Blueprint is below: { "name": <blueprint_name>, "set_name": <template_set> "template_name": <template_name>, "parameters": { "application_description": <simple_description>, "application_name": <blueprint_name>, "enable_Global_Resiliency": false, "pools": [ { <pool_configuration_parameters> } ], "virtuals": [ { <virtual_server_configuration parameters> } ] } } More detail on each of the variable values below: <blueprint_name> - This is the name you choose for your blueprint. I generally recommend this name be the same in both the “name” field and the “application_name” field which is why in the JSON above you’ll see this in both. <template_set> - This is going to be the template set containing your FAST template. If you are using the default templates provided to you, this value would be “Examples” <template_name> - This is the specific FAST template you are going to use for the configuration. If you are using the default template provided to you, this value would be “http” <simple_description> - This can be any short description you would like to use for your application service. <pool_configuration_parameters> - This will be the list of parameters that you are going to define for your pool. You do not have to fill in every single value if the FAST template contains values for that field. <virtual_server_configuration parameters> - This will be the list of parameters that you are going to define for your virtual server. You do not have to fill in every single value if the FAST template contains values for that field. Important Note: When creating your JSON, you will be defining a FAST template to use along with your application service (just like you would in the GUI). This means that you do not have to fill out every value under your pool and virtual server configuration. It will take in the values provided from your FAST template. With our guide above, we can now make a new Blueprint for the “bobs-blueprint” example I referenced earlier: { "name":"bobs-blueprint", "set_name": "Examples", "template_name": "http", "parameters": { "application_description":"This is a test of the blueprints api", "application_name": "bobs-blueprint", "pools": [ { "loadBalancingMode": "round-robin", "monitorType": [ "http" ], "poolName": "juice_pool", "servicePort": 3000 } ], "virtuals": [ { "pool": "juice_pool", "virtualName": "blueprint_vs", "virtualPort": 80 } ] } } As you can see, this is a much more condensed version of the original JSON I had for the example shown in the beginning of this article. Again, this is because we are referencing the FAST template “Examples/http” and taking in those values to configure the rest of the application service. With our newly created JSON, the last step to creating this Blueprint is to send this in a POST request to the following API endpoint: POST https://{{bigip_next_cm_mgmt_ip}}/api/v1/spaces/default/appsvcs/blueprints After sending our POST, you'll notice we are given the "id" of the Blueprint in the response. As mentioned above, we can use this ID to modify, deploy, etc. { "_links": { "self": { "href": "/api/v1/spaces/default/appsvcs/blueprints/9c35a614-65ac-4d18-8082-589ea9bc78d9/deployments" } }, "deployments": [ { "deploymentID": "1a153f44-acaf-4487-81c7-61b8b5498627", "instanceID": "4ef739d1-9ef1-4eb7-a5bb-36c6d1334b16", "status": "pending", "taskID": "518534c8-9368-48dd-b399-94f55e72d5a7", "task_type": "CREATE" }, { "deploymentID": "82ca8d6d-dbfd-43a8-a604-43f170d9f190", "instanceID": "0546acf5-3b88-422d-a948-28bbf0973212", "status": "pending", "taskID": "7ea24157-1412-4963-9560-56fb0ab78d8c", "task_type": "CREATE" } ], "id": "9c35a614-65ac-4d18-8082-589ea9bc78d9" } Now that the Blueprint has been created, we can go into our BIG-IP Next CM GUI and see our newly created application service: You’ll notice that our newly created application service is in Draft mode. This is because we have not deployed the service yet. We’ll discuss that in the next section. Deploying a Blueprint Once we have our Blueprint created, the final step is to configure the deployment. As discussed above, this is done through a separate API Call. The format for a deployment is as follows: { "deployments": [ { "parameters": { "pools": [ { "poolName": <pool_name> "poolMembers": [ { "name": <node1_name>, "address": <node1_ip_address> }, { "name": <node2_name>, "address": <node2_ip_address> } ] } ], "virtuals": [ { "virtualName": <virtual_server_name>, "virtualAddress": <virtual_server_ip_address> } ] }, "target": { "address": <bigip_instance_ip_address> }, "allow_overwrite": true } ] } Keep in mind that some of these values above are referencing names from your Blueprint configuration. These names need to be exactly the same. See below for more detail on each of these values: <pool_name> - This references the pool from your Blueprint. This value must match what was used for “poolName” in the pool configuration of the Blueprint. <node1_name> - This is any name you choose to describe your node in the pool <node1_ip_address> - This is the specific IP address for your node <node2_name> - If you are using more than one node, this would be the name you choose to represent your second node in the pool. This format can repeat for 3 nodes, 4 nodes, etc. <node2_ip_address> - If you are using more than one node, this would be the IP address of your second node. This format can repeat for 3 nodes, 4 nodes, etc. <virtual_server_name> - This references the virtual server from your Blueprint. This value must match what was used for “virtualName” in the virtuals configuration of the Blueprint. <virtual_server_ip_address> - This would be the IP address of the Virtual Server being deployed on the BIG-IP <bigip_instance_ip_address> - This is the IP address of the BIG-IP instance we are deploying to Important Note: If you are deploying the same application to more than one BIG-IP instance, you can include multiple deployment blocks in your API call. Using the format above, we can now create our deployment for “bobs-blueprint”: { "deployments": [ { "parameters": { "pools": [ { "poolName": "juice_pool", "poolMembers": [ { "name": "node1", "address": "10.1.3.100" } ] } ], "virtuals": [ { "virtualName": "blueprint_vs", "virtualAddress": "10.2.2.15" } ] }, "target": { "address": "10.2.1.3" }, "allow_overwrite": true }, { "parameters": { "pools": [ { "poolName": "juice_pool", "poolMembers": [ { "name": "node1", "address": "10.2.3.100" } ] } ], "virtuals": [ { "virtualName": "blueprint_vs", "virtualAddress": "10.2.2.20" } ] }, "target": { "address": "10.2.1.4" }, "allow_overwrite": true } ] } In this example deployment, I am deploying to two separate BIG-IP instances (10.2.1.3 and 10.2.1.4). This is where we can really start to see the value of the Blueprints API. With this structure, I can use the same pool and virtual server setup from our Blueprint, while still using different Virtual Server and Node IP addresses for the deployment at each instance. All of this is done with one API call. The final step is to send this in a POST request to our deployments endpoint. The endpoint is very similar to viewing a blueprint as it uses our same Blueprint ID. See the format below: POST https://{{bigip_next_cm_mgmt_ip}}/api/v1/spaces/default/appsvcs/blueprints/{{Blueprint_id}}/deployments Using the ID from the response we received after creating our example blueprint, our endpoint would be: POST https://{{bigip_next_cm_mgmt_ip}}/api/v1/spaces/default/appsvcs/blueprints/9c35a614-65ac-4d18-8082-589ea9bc78d9/deployments After sending our POST, we can go back to the BIG-IP Next CM GUI and see our application service is no longer considered a Draft. If we click into the application service, we’ll see our two deployments are up and healthy: Conclusion Hopefully after reading this article, you can see the value of using the Blueprints API for your automation. I think as an alternative to other automation methods, this can provide benefits such as: Same centrally managed format/structure as GUI created applications Since the BIG-IP Next CM GUI is already creating these JSON files under the hood, we can easily automate existing applications by using the Blueprints API for them moving forward Deployments are handled separately from your application configuration You can deploy your application service to multiple BIG-IP instances at once Combining the Blueprints API with FAST templates allows you make application on-boarding much more streamlined. If you liked this article and are looking for more information our our Blueprints API, please visit the API documentation here: https://clouddocs.f5.com/products/bigip-next/mgmt-api/latest/ApiReferences/bigip_public_api_ref/r_openapi-next.html#tag/Application-Services/operation/GetAllApplicationsAutomation of OWASP Injection mitigation using F5 Distributed Cloud Platform
Objective: The purpose of this article is to automate F5 Distributed Cloud Platform (F5 XC) detection and mitigation of OWASP TOP 10 Injection attacks and integrating code in GitHub. This article shows how we can use Terraform, Python and Github workflow to provide the flexibility of updating existing infrastructure after every change using CI/CD event triggers. For more details about this feature please refer: Injection Attack Mitgation Article Introduction to Injection: An application is vulnerable to attack when: The data provided is not validated by the application User requested schema is not being analyzed before processing Data is used within search parameters to extract additional and sensitive records If a user tries to use Cross-site Scripting to get some unauthorized data Some of the common injections are SQL, NoSQL, OS command, Object Relational Mapping (ORM), Etc. In this automation article we are trying to bypass password validation in a demo application using SQL Injection code. Design: For the purpose of reusability, I have separated demo application and F5 XC resources deployments in 2 different flows as below. 1. First, we are deploying a demo application (Juice Shop) as a docker container in AWS EC2 machine (if customer already has their application running, they can skip this and use their application public IP directly in below flow) 2. Once the demo app is deployed, we are using application public IP to configure the origin pool, WAF and load balancer in F5 XC Once demo application and F5 XC resources are deployed successfully, python script is generating login request consisting of malicious SQL Injection. Once traffic is generated, F5 XC platform WAF will detect and block the malicious request. Finally, we are destroying the above resources using terraform Above workflow is integrated using GitHub Actions file which ensures dynamic deployment of the demo app and F5 XC load balancer which can be exposed using public domain name. Repo code URL:https://github.com/f5devcentral/owasp-injection-mitigation Conclusion: In this article we have showed how we can leverage power of CI/CD deployment to automate end to end verification of injection attacks mitigationusing GitHub Actions, Terraform and Python developed in a generic way where users can bring up the complete setup within a few clicks. For further information check the links below: F5 Distributed Cloud Platform (Link) F5 Distributed Cloud WAF Overview (Link) OWASP Injection Attacks (Link)
Using CryptoNice as a Sanity-Checking Tool for Automated Application Deployments with Ansible
Any good automation pipeline should have some validation built in to perform sanity checks on what it should have done. This article describes how you can use CryptoNice as a simple and easy way of sanity checking the SSL/TLS configuration of your automated application deployments, by integrating CryptoNice into your pipeline directly after your automation solution has deployed the application. I am going to show a simple Ansible playbook that illustrates this point. The Ansible playbook deploys an application to a BIG-IP that resides in AWS, but it could be any BIG-IP on premises or in any cloud. After the deployment is complete, I use CryptoNice to validate that the application is reachable via TLS and that the deployed VIP is using best practices for SSL deployments. What Is CryptoNice? CryptoNice is both a command line tool and Python library that is developed by F5 Labs and is publicly available; it provides the ability to scan and report on the configuration of SSL/TLS for your internet or internal-facing web services. Built using the sslyze API and SSL, http-client, and DNS libraries, CryptoNice collects data on a given domain and performs a series of tests to check TLS configuration. You can get CryptoNice here:https://github.com/F5-Labs/cryptonice What Is Ansible? Ansible is an open-source software-provisioning, configuration-management, and application-deployment tool enabling infrastructure as code. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows and also F5 BIG-IPs. You can learn how to install Ansible here:https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html F5 publishes instructions on how to integrate the Ansible plugins here:https://clouddocs.f5.com/products/orchestration/ansible/devel/ In this article I use the published version 1 F5 Ansible plugin that uses the iControl REST API. If we take a look briefly under the hood here, this plugin is using an imperative API. As of today, F5 has a version 2 plugin in preview that focuses on managing F5 BIG-IP/BIG-IQ through declarative APIs such asAS3, DO, TS, and CFE. You can also check the version 2 plugin preview out on F5 Cloud Docs. I used a stock Ubuntu image as a basis for my Ansible server and followed the instructions for installing Ansible on Ubuntu, then followed the instructions for installing the F5 plugin for Ansible referenced above. You can take a look at the F5/Ansible 1.0 plugin that is published on the Ansible Galaxy Hub here:https://galaxy.ansible.com/f5networks/f5_modules For my demonstration, I also installed CryptoNice on the same server where Ansible and the F5 Ansible plugin are installed; at that point you are off to the races and you can build a simple Ansible script and begin to automate a BIG-IP. The F5 Ansible Module provides you with a great deal of programmability for the BIG-IP basic onboarding, WAF, APM, and LTM. The following is a great reference to give you an idea of the scope of capabilities with Ansible examples that this module provides. You can also automate the infrastructure creation if you so choose, meaning you could stand up a BIG-IP in AWS and then configure everything required to get the device onboarded, and then after that, configure traffic-management objects like VIPs/pools, etc. https://clouddocs.f5.com/products/orchestration/ansible/devel/modules/module_index.html For my test, I created a simple Ansible script that automates the creation of a VIP with an iRule to respond to http get requests. The example also associates a pool and pool members to the VIP to give you an idea of what a simple application deployment may look like. After that, I run a CryptoNice to test the quality of the SSL/TLS. My simple Ansible playbook looks like this: --- - name: Create a VIP, pool and pool members hosts: f5 connection: local vars: provider: password: notmypassword server: f5cove.me user: auser validate_certs: no server_port: 8443 tasks: - name: Create a pool bigip_pool: provider: "{{ provider }}" lb_method: ratio-member name: examplepool slow_ramp_time: 120 delegate_to: localhost - name: Add members to pool bigip_pool_member: provider: "{{ provider }}" description: "webserver {{ item.name }}" host: "{{ item.host }}" name: "{{ item.name }}" pool: examplepool port: '80' with_items: - host: 10.0.0.68 name: web01 - host: 10.0.0.67 name: web02 delegate_to: localhost - name: Create a VIP bigip_virtual_server: provider: "{{ provider }}" description: avip destination: 10.0.0.66 name: vip-1 irules: - responder pool: examplepool port: '443' snat: Automap profiles: - http - f5cove delegate_to: localhost - name: Create Ridirect bigip_virtual_server: provider: "{{ provider }}" description: avipredirect destination: 10.0.0.66 name: vip-1-redirect irules: - _sys_https_redirect port: '80' snat: None profiles: - http delegate_to: localhost - name: play cryptonice nicely hosts: 127.0.0.1 connection: local tasks: - pause: seconds: 30 - name: run cryptonice shell: "cryptonice f5cove.me" register: output - debug: var=output.stdout_lines My output from running the playbook looks like this: PLAY [Create a VIP, pool and pool members] ************************************************************************************************************************************************************************************************* TASK [Gathering Facts] ********************************************************************************************************************************************************************************************************************* ok: [1.2.3.4] TASK [Create a pool] *********************************************************************************************************************************************************************************************************************** ok: [1.2.3.4 -> localhost] TASK [Add members to pool] ***************************************************************************************************************************************************************************************************************** ok: [1.2.3.4 -> localhost] => (item={'host': '10.0.0.68', 'name': 'web01'}) ok: [1.2.3.4 -> localhost] => (item={'host': '10.0.0.67', 'name': 'web02'}) TASK [Create a VIP] ************************************************************************************************************************************************************************************************************************ changed: [1.2.3.4 -> localhost] TASK [Create Ridirect] ********************************************************************************************************************************************************************************************************************* ok: [1.2.3.4 -> localhost] PLAY [play cryptonice nicely] ************************************************************************************************************************************************************************************************************** TASK [Gathering Facts] ********************************************************************************************************************************************************************************************************************* ok: [localhost] TASK [pause] ******************************************************************************************************************************************************************************************************************************* Pausing for 30 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [localhost] TASK [run cryptonice] ********************************************************************************************************************************************************************************************************************** changed: [localhost] TASK [debug] ******************************************************************************************************************************************************************************************************************************* ok: [localhost] => { "output.stdout_lines": [ "Pre-scan checks", "-------------------------------------", "Scanning f5cove.me on port 443...", "Analyzing DNS data for f5cove.me", "Fetching additional records for f5cove.me", "f5cove.me resolves to 34.217.132.104", "34.217.132.104:443: OPEN", "TLS is available: True", "Connecting to port 443 using HTTPS", "Queueing TLS scans (this might take a little while...)", "Looking for HTTP/2", "", "", "RESULTS", "-------------------------------------", "Hostname:\t\t\t f5cove.me", "", "Selected Cipher Suite:\t\t ECDHE-RSA-AES128-GCM-SHA256", "Selected TLS Version:\t\t TLS_1_2", "", "Supported protocols:", "TLS 1.2:\t\t\t Yes", "TLS 1.1:\t\t\t Yes", "TLS 1.0:\t\t\t Yes", "", "TLS fingerprint:\t\t 29d29d15d29d29d21c29d29d29d29d930c599f185259cdd20fafb488f63f34", "", "", "", "CERTIFICATE", "Common Name:\t\t\t f5cove.me", "Issuer Name:\t\t\t R3", "Public Key Algorithm:\t\t RSA", "Public Key Size:\t\t 2048", "Signature Algorithm:\t\t sha256", "", "Certificate is trusted:\t\t False (Mozilla not trusted)", "Hostname Validation:\t\t OK - Certificate matches server hostname", "Extended Validation:\t\t False", "Certificate is in date:\t\t True", "Days until expiry:\t\t 29", "Valid From:\t\t\t 2021-02-12 23:28:14", "Valid Until:\t\t\t 2021-05-13 23:28:14", "", "OCSP Response:\t\t\t Successful", "Must Staple Extension:\t\t False", "", "Subject Alternative Names:", "\t f5cove.me", "", "Vulnerability Tests:", "No vulnerability tests were run", "", "HTTP to HTTPS redirect:\t\t False", "None", "", "RECOMMENDATIONS", "-------------------------------------", "HIGH - TLSv1.0 Major browsers are disabling TLS 1.0 imminently. Carefully monitor if clients still use this protocol. ", "HIGH - TLSv1.1 Major browsers are disabling this TLS 1.1 immenently. Carefully monitor if clients still use this protocol. ", "Low - CAA Consider creating DNS CAA records to prevent accidental or malicious certificate issuance.", "", "Scans complete", "-------------------------------------", "Total run time: 0:00:05.688562" ] } Note that after the playbook is complete, CryptoNice is telling me that I need to improve the quality of my SSL profile on my BIG-IP. As a result, I create a second SSL profile and Disable TLS 1.0 and 1.1. Upgrade the certificate from a 2048 bit key to a 4096 bit key. Make sure that the certificate bundle is configured correctly to ensure that the certificate is properly trusted. Create a cipher rule and group and use the following cipher string to improve the quality of the SSL connections: !EXPORT:!DHE+AES-GCM:!DHE+AES:ECDHE+AES-GCM:ECDHE+AES:RSA+AES-GCM:RSA+AES:-MD5:-SSLv3:-RC4:!3DES I then alter the playbook to reference the new SSL profile, and then re-run the Ansible playbook. Here is the relevant playbook snippet: - name: Create a VIP bigip_virtual_server: provider: "{{ provider }}" description: avip destination: 10.0.0.66 name: vip-1 irules: - responder pool: examplepool port: '443' snat: Automap profiles: - http - f5cove <change this to the new SSL Profile> delegate_to: localhost The resulting output from CryptoNice after running the playbook again to switch the SSL profile addresses all of the HIGH recommendations. TASK [debug] ********************************************************************************************************************************************************************** ok: [localhost] => { "output.stdout_lines": [ "Pre-scan checks", "-------------------------------------", "Scanning f5cove.me on port 443...", "Analyzing DNS data for f5cove.me", "Fetching additional records for f5cove.me", "f5cove.me resolves to 34.217.132.104", "34.217.132.104:443: OPEN", "TLS is available: True", "Connecting to port 443 using HTTPS", "Queueing TLS scans (this might take a little while...)", "Looking for HTTP/2", "", "", "RESULTS", "-------------------------------------", "Hostname:\t\t\t f5cove.me", "", "Selected Cipher Suite:\t\t ECDHE-RSA-AES256-GCM-SHA384", "Selected TLS Version:\t\t TLS_1_2", "", "Supported protocols:", "TLS 1.2:\t\t\t Yes", "", "TLS fingerprint:\t\t 2ad2ad0002ad2ad0002ad2ad2ad2adcb09dd549309271837f87ac5dad15fa7", "", "", "HTTP/2 supported:\t\t False", "", "", "CERTIFICATE", "Common Name:\t\t\t f5cove.me", "Issuer Name:\t\t\t R3", "Public Key Algorithm:\t\t RSA", "Public Key Size:\t\t 4096", "Signature Algorithm:\t\t sha256", "", "Certificate is trusted:\t\t True (No errors)", "Hostname Validation:\t\t OK - Certificate matches server hostname", "Extended Validation:\t\t False", "Certificate is in date:\t\t True", "Days until expiry:\t\t 88", "Valid From:\t\t\t 2021-04-13 00:55:09", "Valid Until:\t\t\t 2021-07-12 00:55:09", "", "OCSP Response:\t\t\t Successful", "Must Staple Extension:\t\t False", "", "Subject Alternative Names:", "\t f5cove.me", "", "Vulnerability Tests:", "No vulnerability tests were run", "", "HTTP to HTTPS redirect:\t\t False", "None", "", "RECOMMENDATIONS", "-------------------------------------", "Low - CAA Consider creating DNS CAA records to prevent accidental or malicious certificate issuance.", "", "Scans complete", "-------------------------------------", "Total run time: 0:00:05.638186" ] } Now you can see in the CryptoNice output that There are no trust issues with the certificate/certificate bundle. I have upgraded from 2048 bits to 4096 bit key length. I have disabled TLS1.0 and TLS1.1. The connection is handshaking with a more secure cryptographic algorithm. I no longer have any HIGH recommendations for improvement to the quality of TLS connection. Conclusion It is always best practice to perform a sanity check on the services that you create in your automation pipelines. This article describes using CryptoNice as part of a simple Ansible playbook to run an SSL/TLS sanity check on the application in order to improve the security of your application-delivery automation. Ultimately you should not just be checking the SSL/TLS; another good idea would be to introduce application-vulnerability scanners too as part of the automation pipeline.F5 Automation with Ansible Tips and Tricks
Getting Started with Ansible and F5 In this article we are going to provide you with a simple set of videos that demonstrate step by step how to implement automation with Ansible. In the last video, however we will demonstrate how telemetry and automation may be used in combination to address potential performance bottlenecks and ensure application availability. To start, we will provide you with details on how to get started with Ansible automation using the Ansible Automation Platform®: Backing up your F5 device Once a user has installed and configured Ansible Automation Platform, we will now transition to a basic maintenance function – an automated backup of a BIG-IP hardware device or Virtual Edition (VE). This is always recommended before major changes are made to our BIG-IP devices Configuring a Virtual Server Next, we will use Ansible to configure a Virtual Server, a task that is most frequently performed via manual functions via the BIG-IP. When changes to a BIG-IP are infrequent, manual intervention may not be so cumbersome. However large enterprise customers may need to perform these tasks hundreds of times: Replace an SSL Certificate The next video will demonstrate how to use Ansible to replace an SSL certificate on a BIG-IP. It is important to note that this video will show the certificate being applied on a BIG-IP and then validated by browsing to the application website: Configure and Deploy an iRule The next administrative function will demonstrate how to configure and push an iRule using the Ansible Automation Platform® onto a BIG-IP device. Again this is a standard administrative task that can be simply automated via Ansible: Delete the Existing Virtual Server Ok so now we have to delete the above configuration to roll back to a steady state. This is a common administrative task when an application is retired. We again demonstrate how Ansible automation may be used to perform these simple administrative tasks: Telemetry and Automation: Using Threshold Triggers to Automate Tasks and Fix Performance Bottlenecks Now you have a clear demonstration as to how to utilize Ansible automation to perform routine tasks on a BIG-IP platform. Once you have become proficient with more routine Ansible tasks, we can explore more high-level, sophisticated automation tasks. In the below demonstration we show how BIG-IP administrators using SSL Orchestrator® (SSLO) can combine telemetry with automation to address performance bottlenecks in an application environment: Resources: So that is a short series of tutorials on how to perform routine tasks using automation plus a preview of a more sophisticated use of automation based upon telemetry and automatic thresholds. For more detail on our partnership, please visit our F5/Ansible page or visit the Red Hat Automation Hub for information on the F5 Ansible certified collections. https://www.f5.com/ansible https://www.ansible.com/products/automation-hub https://galaxy.ansible.com/f5networks/f5_modulesThese Are Not The Scrapes You're Looking For - Session Anomalies
In my first article in this series, I discussed web scraping -- what it is, why people do it, and why it could be harmful. My second article outlined the details of bot detection and how the ASM blocks against these pesky little creatures. This last article in the series of web scraping will focus on the final part of the ASM defense against web scraping: session opening anomalies and session transaction anomalies. These two detection modes are new in v11.3, so if you're using v11.2 or earlier, then you should upgrade and take advantage of these great new features! ASM Configuration In case you missed it in the bot detection article, here's a quick screenshot that shows the location and settings of the Session Opening and Session Transactions Anomaly in the ASM. You'll find all the fun when you navigate to Security > Application Security > Anomaly Detection > Web Scraping. There are three different settings in the ASM for Session Anomaly: Off, Alarm, and Alarm and Block. (Note: these settings are configured independently...they don't have to be set at the same value) Obviously, if Session Anomaly is set to "Off" then the ASM does not check for anomalies at all. The "Alarm" setting will detect anomalies and record attack data, but it will allow the client to continue accessing the website. The "Alarm and Block" setting will detect anomalies, record the attack data, and block the suspicious requests. Session Opening Anomaly The first detection and prevention mode we'll discuss is Session Opening Anomaly. But before we get too deep into this, let's review what a session is. From a simple perspective, a session begins when a client visits a website, and it ends when the client leaves the site (or the client exceeds the session timeout value). Most clients will visit a website, surf around some links on the site, find the information they need, and then leave. When clients don't follow a typical browsing pattern, it makes you wonder what they are up to and if they are one of the bad guys trying to scrape your site. That's where Session Opening Anomaly defense comes in! Session Opening Anomaly defense checks for lots of abnormal activities like clients that don't accept cookies or process JavaScript, clients that don't scrape by surfing internal links in the application, and clients that create a one-time session for each resource they consume. These one-time sessions lead scrapers to open a large number of new sessions in order to complete their job quickly. What's Considered A New Session? Since we are discussing session anomalies, I figured we should spend a few sentences on describing how the ASM differentiates between a new or ongoing session for each client request. Each new client is assigned a "TS cookie" and this cookie is used by the ASM to identify future requests from the client with a known, ongoing session. If the ASM receives a client request and the request does not contain a TS cookie, then the ASM knows the request is for a new session. This will prove very important when calculating the values needed to determine whether or not a client is scraping your site. Detection There are two different methods used by the ASM to detect these anomalies. The first method compares a calculated value to a predetermined ceiling value for newly opened sessions. The second method considers the rate of increase of newly opened sessions. We'll dig into all that in just a minute. But first, let's look at the criteria used for detecting these anomalies. As you can see from the screenshot above, there are three detection criteria the ASM uses...they are: Sessions opened per second increased by: This specifies that the ASM considers client traffic to be an attack if the number of sessions opened per second increases by a given percentage. The default setting is 500 percent. Sessions opened per second reached: This specifies that the ASM considers client traffic to be an attack if the number of sessions opened per second is greater than or equal to this number. The default value is 400 sessions opened per second. Minimum sessions opened per second threshold for detection: This specifies that the ASM considers traffic to be an attack if the number of sessions opened per second is greater than or equal to the number specified. In addition, at least one of the "Sessions opened per second increased by" or "Sessions opened per second reached" numbers must also be reached. If the number of sessions opened per second is lower than the specified number, the ASM does not consider this traffic to be an attack even if one of the "Sessions opened per second increased by" or "Sessions opened per second" reached numbers was reached. The default value for this setting is 200 sessions opened per second. In addition, the ASM maintains two variables for each client IP address: a one-minute running average of new session opening rate, and a one-hour running average of new session opening rate. Both of these variables are recalculated every second. Now that we have all the basic building blocks. let's look at how the ASM determines if a client is scraping your site. First Method: Predefined Ceiling Value This method uses the user-defined "minimum sessions opened per second threshold for detection" value and compares it to the one-minute running average. If the one-minute average is less than this number, then nothing else happens because the minimum threshold has not been met. But, if the one-minute average is higher than this number, the ASM goes on to compare the one-minute average to the user-defined "sessions opened per second reached" value. If the one-minute average is less than this value, nothing happens. But, if the one-minute average is higher than this value, the ASM will declare the client a web scraper. The following flowchart provides a pictorial representation of this process. Second Method: Rate of Increase The second detection method uses several variables to compare the rate of increase of newly opened sessions against user-defined variables. Like the first method, this method first checks to make sure the minimum sessions opened per second threshold is met before doing anything else. If the minimum threshold has been met, the ASM will perform a few more calculations to determine if the client is a web scraper or not. The "sessions opened per second increased by" value (percentage) is multiplied by the one-hour running average and this value is compared to the one-minute running average. If the one-minute average is greater, then the ASM declares the client a web scraper. If the one-minute average is lower, then nothing happens. The following matrix shows a few examples of this detection method. Keep in mind that the one-minute and one-hour averages are recalculated every second, so these values will be very dynamic. Prevention The ASM provides several policies to prevent session opening anomalies. It begins with the first method that you enable in this list. If the system finds this method not effective enough to stop the attack, it uses the next method that you enable in this list. The following screenshots show the different options available for prevention. The "Drop IP Addresses with bad reputation" is tied to Rate Limiting, so it will not appear as an option unless you enable Rate Limiting. Note that IP Address Intelligence must be licensed and enabled. This feature is licensed separately from the other ASM web scraping options. Here's a quick breakdown of what each of these prevention policies do for you: Client Side Integrity Defense: The system determines whether the client is a legal browser or an illegal script by sending a JavaScript challenge to each new session request from the detected IP address, and waiting for a response. The JavaScript challenge will typically involve some sort of computational challenge. Legal browsers will respond with a TS cookie while illegal scripts will not. The default for this feature is disabled. Rate Limiting: The goal of Rate Limiting is to keep the volume of new sessions at a "non-attack" level. The system will drop sessions from suspicious IP addresses after the system determines that the client is an illegal script. The default for this feature is also disabled. Drop IP Addresses with bad reputation: The system drops requests from IP addresses that have a bad reputation according to the system’s IP Address Intelligence database (shown above). The ASM will drop all request from any "bad" IP addresses even if they respond with a TS cookie. IP addresses that do not have a bad reputation also undergo rate limiting. The default for this option is disabled. Keep in mind that this option is available only after Rate Limiting is enabled. In addition, this option is only enforced if at least one of the IP Address Intelligence Categories is set to Alarm mode. Prevention Duration Now that we have detected session opening anomalies and mitigated them using our prevention options, we must figure out how long to apply the prevention measures. This is where the Prevention Duration comes in. This setting specifies the length of time that the system will prevent an attack. The system prevents attacks by rejecting requests from the attacking IP address. There are two settings for Prevention Duration: Unlimited: This specifies that after the system detects and stops an attack, it performs attack prevention until it detects the end of the attack. This is the default setting. Maximum <number of> seconds: This specifies that after the system detects and stops an attack, it performs attack prevention for the amount of time indicated unless the system detects the end of the attack earlier. So, to finish up our Session Opening Anomaly part of this article, I wanted to share a quick scenario. I was recently reading several articles from some of the web scrapers around the block, and I found one guy's solution to work around web scraping defense. Here's what he said: "Since the service conducted rate-limiting based on IP address, my solution was to put the code that hit their service into some client-side JavaScript, and then send the results back to my server from each of the clients. This way, the requests would appear to come from thousands of different places, since each client would presumably have their own unique IP address, and none of them would individually be going over the rate limit." This guy is really smart! And, this would work great against a web scraping defense that only offered a Rate Limiting feature. Here's the pop quiz question: If a user were to deploy this same tactic against the ASM, what would you do to catch this guy? I'm thinking you would need to set your minimum threshold at an appropriate level (this will ensure the ASM kicks into gear when all these sessions are opened) and then the "sessions opened per second" or the "sessions opened per second increased by" should take care of the rest for you. As always, it's important to learn what each setting does and then test it on your own environment for a period of time to ensure you have everything tuned correctly. And, don't forget to revisit your settings from time to time...you will probably need to change them as your network environment changes. Session Transactions Anomaly The second detection and prevention mode is Session Transactions Anomaly. This mode specifies how the ASM reacts when it detects a large number of transactions per session as well as a large increase of session transactions. Keep in mind that web scrapers are designed to extract content from your website as quickly and efficiently as possible. So, web scrapers normally perform many more transactions than a typical application client. Even if a web scraper found a way around all the other defenses we've discussed, the Session Transaction Anomaly defense should be able to catch it based on the sheer number of transactions it performs during a given session. The ASM detects this activity by counting the number of transactions per session and comparing that number to a total average of transactions from all sessions. The following screenshot shows the detection and prevention criteria for Session Transactions Anomaly. Detection How does the ASM detect all this bad behavior? Well, since it's trying to find clients that surf your site much more than other clients, it tracks the number of transactions per client session (note: the ASM will drop a session from the table if no transactions are performed for 15 minutes). It also tracks the average number of transactions for all current sessions (note: the ASM calculates the average transaction value every minute). It can use these two figures to compare a specific client session to a reasonable baseline and figure out if the client is performing too many transactions. The ASM can automatically figure out the number of transactions per client, but it needs some user-defined thresholds to conduct the appropriate comparisons. These thresholds are as follows: Session transactions increased by: This specifies that the system considers traffic to be an attack if the number of transactions per session increased by the percentage listed. The default setting is 500 percent. Session transactions reached: This specifies that the system considers traffic to be an attack if the number of transactions per session is equal to or greater than this number. The default value is 400 transactions. Minimum session transactions threshold for detection: This specifies that the system considers traffic to be an attack if the number of transactions per session is equal to or greater than this number, and at least one of the "Sessions transactions increased by" or "Session transactions reached" numbers was reached. If the number of transactions per session is lower than this number, the system does not consider this traffic to be an attack even if one of the "Session transactions increased by" or "Session transaction reached" numbers was reached. The default value is 200 transactions. The following table shows an example of how the ASM calculates transaction values (averages and individual sessions). We would expect that a given client session would perform about the same number of transactions as the overall average number of transactions per session. But, if one of the sessions is performing a significantly higher number of transactions than the average, then we start to get suspicious. You can see that session 1 and session 3 have transaction values higher than the average, but that only tells part of the story. We need to consider a few more things before we decide if this client is a web scraper or not. By the way, if the ASM knows that a given session is malicious, it does not use that session's transaction numbers when it calculates the average. Now, let's roll in the threshold values that we discussed above. If the ASM is going to declare a client as a web scraper using the session transaction anomaly defense, the session transactions must first reach the minimum threshold. Using our default minimum threshold value of 200, the only session that exceeded the minimum threshold is session 3 (250 > 200). All other sessions look good so far...keep in mind that these numbers will change as the client performs additional transactions during the session, so more sessions may be considered as their transaction numbers increase. Since we have our eye on session 3 at this point, it's time to look at our two methods of detecting an attack. The first detection method is a simple comparison of the total session transaction value to our user-defined "session transactions reached" threshold. If the total session transactions is larger than the threshold, the ASM will declare the client a web scraper. Our example would look like this: Is session 3 transaction value > threshold value (250 > 400)? No, so the ASM does not declare this client as a web scraper. The second detection method uses the "transactions increased by" value along with the average transaction value for all sessions. The ASM multiplies the average transaction value with the "transactions increased by" percentage to calculate the value needed for comparison. Our example would look like this: 90 * 500% = 450 transactions Is session 3 transaction value > result (250 > 450)? No, so the ASM does not declare this client as a web scraper. By the way, only one of these detection methods needs to be met for the ASM to declare the client as a web scraper. You should be able to see how the user-defined thresholds are used in these calculations and comparisons. So, it's important to raise or lower these values as you need for your environment. Prevention Duration In order to save you a bunch of time reading about prevention duration, I'll just say that the Session Transactions Anomaly prevention duration works the same as the Session Opening Anomaly prevention duration (Unlimited vs Maximum <number of> seconds). See, that was easy! Conclusion Thanks for spending some time reading about session anomalies and web scraping defense. The ASM does a great job of detecting and preventing web scrapers from taking your valuable information. One more thing...for an informative anomaly discussion on the DevCentral Security Forum, check out this conversation. If you have any questions about web scraping or ASM configurations, let me know...you can fill out the comment section below or you can contact the DevCentral team at https://devcentral.f5.com/s/community/contact-us.
