A Guide to F5 Volumetric (Routed) DDoS Protection in F5 Distributed Cloud
Introduction F5 Volumetric (Routed) DDoS protection is a service in F5 Distributed Cloud (F5 XC) available for standard deployment and emergency use. F5 has over 100 engineers in its incident response team and 24/7 dedicated SOC analysts in 3 security operations centers around the world. This means F5 can help with the quick detection, mitigation, and resolution of Layer3-4 routed DDoS attacks. F5 Volumetric DDoS Protection stands out for several key reasons, especially for enterprises needing fully managed, hybrid, and multicloud-based DDoS mitigation with human-led and AI-assisted support. Here’s some of the ways Volumetric DDoS protection with F5 stands out: Fully Managed 24/7 Security Operations Center (SOC) F5’s SOC continuously monitors traffic for DDoS attacks in real time. Unlike purely automated solutions, human analysts intervene to fine-tune attack mitigation. The SOC provides expert-led response to mitigate complex or evolving threats. Hybrid Deployment Flexibility Cloud-based, always-on, or on-demand models for different use cases. Integrates with on-prem F5 BIG-IP solutions for a hybrid defense strategy. Helps reduce false positives by fine-tuning security policies. Advanced Attack Detection & AI-driven Mitigation Uses behavioral analytics to differentiate between legitimate traffic and attacks. Mitigates volumetric, application-layer, and multi-vector attacks. AI-assisted rules dynamically adapt to new attack patterns. Large-Scale Scrubbing Capacity Global scrubbing centers prevent volumetric DDoS attacks from overwhelming networks. Reduces the risk of downtime by filtering malicious traffic before it reaches critical infrastructure. F5 blocks volumetric DDoS attacks by denying offending /24 prefixes (via BGP) the ability to route to the Distributed Cloud scrubbing centers. (reference DevCentral) API-Driven and Customizable Security Policies Offers API integration for automated DDoS mitigation and security orchestration. Supports custom policies to protect specific applications from targeted attacks. Enterprise-Grade Support & Compliance Designed for large enterprises, financial institutions, and high-security industries. Meets compliance standards such as PCI DSS, GDPR, and SOC 2. Backed by F5’s global threat intelligence network. Logging & Observability Recently introduced is the capability to observe security events using external handlers via the Global Log Receiver (GLR) service. Organizations can use AWS S3 buckets, HTTP(s) servers, Datadog, Splunk, AWS CloudWatch, Azure Event Hubs and Blog Storage, Google Cloud Platform (GCP), Kafka Receiver, NewRelic, IBM QRadar, and SumoLogic, to store Distributed Cloud events. Then, they can use any platform to watch DDoS and other security events. If you’re curious how Distributed Cloud events look using ELK (Elasticsearch, Logstash, and Kibana), including how to set it up, see this related article in DevCentral. To configure Distributed Cloud to send events from Global Log Receiver, log in to the Distributed Cloud console and navigate to Shared Configuration > Manage > Global Log Receiver. Add a new item, and ensure the following: Log Type: Security Events Log Message Selection: Select logs from all namespaces For this example, I use Distributed Cloud App Connect to securely deliver events to an instance of ELK Stack running on AWS. To deliver the events locally with internal networking between Distributed Cloud and ELK Stack, I use a Customer Edge (CE) appliance, also in AWS. Having the CE deployed locally provides a secure endpoint with only local routing in the AWS VPC. ➡️ See the following documentation for how to deploy a CE in AWS. Next is to use App Connect with an HTTP Load Balancer. In this case, the origin pool is my ELK Stack receiver, and I’ve configured ELK to receive events over HTTP. Because I’ve configured the HTTP Load Balancer to be publicly available on the Internet to accept traffic from the Global Log Receiver, a Service Policy has been configured to restrict access to specific IP ranges. Although not shown, only traffic from the F5 Global Log Receiver designated IP ranges is allowed to access this load balancer. ➡️ See the following Allowlist reference documentation to learn which IP addresses to allow. To receive and process events in ELK, I’ve configured the following for logstash: root@3c99db3fa334:/etc/logstash/conf.d# cat 50-f5xc-logs.conf input { http { port => 8080 } } filter { json { source => "message" } } output { elasticsearch { hosts => ["localhost"] index => "f5xc-logs-%{+YYY.MM.dd}" } } In the ELK console, new messages are visible under Analytics > Discover. With messages arriving from GLR, we can now see many of the fields becoming searchable in the “message_parsed” hierarchy. Volumetric (Routed) DDoS events appear in the field “sec_event_type” with value “routed_ddos_sec_event”. The following alert and mitigation messages may be classified and searched as follows: New ongoing alert msg = “alert created” no “alert_ended_at” field present New and already completed alert msg = “alert created” alert_ended_at field present Completed ongoing alert msg = “alert completed” alert_started_at field present alert_ended_at field present New ongoing mitigation msg = “mitigation created” mitigation_ongoing = true no “mitigation_stop_time” field present New and already-completed mitigation msg = “mitigation created and completed” mitigation_ongoing = false migitation_stop_time field present Completed mitigation msg = “mitigation completed” mitigation_ongoing = false “mitigation_stop_time” field present Putting it all together in ELK, it’s easy to visualize each routed_ddos_sec_event with a filtered dashboard. Using the pie visual below allows security admins to decide what type of attacks have happened and whether any are still occurring. The dashboard visual can be added to other existing security dashboards in Kibana to provide a complete and robust overview of your security posture. Demo The following video further illustrates the capabilities of Volumetric (Routed) DDoS protection in Distributed Cloud. In it, I walk through the different ways protection can be activated and what some of the mitigation events and alerts look like in the console. 🎥 YouTube: https://youtu.be/jYiqog_tz2I Conclusion F5 Volumteric (Routed) DDoS protection combines integrated services to provide core-protect, auto-mitigation, security-analyst-initiated mitigations, and advanced deep packet inspection and filtering to provide the best protection available for Layer-3 and Layer-4 routed networking. Adding routed DDoS to networks is a simple onboarding process. F5 also provides emergency DDoS mitigation for customers who are actively being attacked. Observing DDoS attacks is not only available in the Distributed Cloud console but is also available directly in your monitoring platform of choice when using Global Log Receiver. Additional Resources 🎥 YouTube: Tour of Routed (Layer3 & Layer4) DDoS Protection in F5 Distributed Cloud How I did it - "Remote Logging with the F5 XC Global Log Receiver and Elastic" Deploy Secure Mesh Site v2 in AWS (ClickOps) Firewall and Proxy Server Allowlist Reference How To: Configure Global Log Receiver
F5 BIG-IP Advanced WAF – DOS profile configuration options.
F5 BIG IP Advanced WAF is the perfect tool for detection and prevention of application Distributed Denial-of-Service (DDoS) attacks against a web application. This article will review the possible configurations of the dos profile also known as Adv WAF anti DDoS feature to stop those attacks.DevCentral Top 5: Sep 8, 2014
But soft! What light through yonder window breaks? It is the east, and this week's edition of the DevCentral Top 5 is the sun. Yep, you guessed it. The top 5 is back...but unlike Shakespeare's Romeo and Juliet, this is no tragedy. Rather, it's a celebration of the most awesome articles you'll read anywhere on the Internet. Our DevCentral authors have been writing with freakish speed and determination, and they have turned out quality articles that are simply second to none. Choosing only five articles was a tough task given all the great content out there, but here's my take on the top articles since our last posting. F5 SOC Malware Summary Report: Neverquest I literally could have chosen five Lori MacVittie articles for this "top 5" but I resisted the urge and only chose one. In this article, Lori explains the details of a Trojan known as "Neverquest" that has been active since July 2013. Most of us get that warm, fuzzy, secure feeling when using 2-factor authentication because, you know, it's got 2 factors! Maybe automated malware has a shot at cracking one factor, but two? No way. Well, apparently Neverquest has found a way to automate the demise of our beloved 2FA. Lori does a magnificent job of explaining how Neverquest works, and then she discusses the amazing work that was completed by our F5 Security Operations Center in their analysis of this malware (in case you didn't know, F5 has a Security Operations Center that analyzes malware like this and provides amazing reports that are free for anyone to read). Lori provides links to the downloads of the executive summary as well as the full technical analysis of Neverquest. This one is not optional...if you care about anything at all, you gotta read this one. Leveraging BIG-IP APM for seamless client NTLM Authentication Michael Koyfman reminds us why we love the BIG-IP APM...transparent seamless authentication for users. In this article, Michael specifically discusses how to configure the APM to perform client NTLM authentication and use it in the context of sending a SAML assertion to the Office 365 service. This is a step-by-step masterpiece that shows you exactly what to do at every turn. In the end, you point your browser to the FQDN of the APM virtual server and you will be silently authenticated (let's be honest...silent authentication is a bucket-list item for each and every one of us). Michael also reminds us of the SSO options at the end of his article. Webshells Nir Zigler introduces us to Webshells (web scripts that act as a control panel for the server running them), and talks about some of the common uses for these scripts. But you know the story...scripts that were created for good can also be used for evil. After Nir explains all the valid uses for legitimate webshells, he takes us to a place where mere mortals dare not tread...through a webshell attack. He gives us an overview of how a webshell attack works, and then he explains some of the specific tools that are used for these nefarious actions. After walking through the power and functionality of an open source webshell called b374k, Nir shows how this tool can be used to attack an unsuspecting user. But have no fear! Nir finishes up the article by discussing the power of the BIG-IP ASM and how it will detect and prevent webshell attacks. Continuing the DDoS Arms Race How long have DDoS attacks been around, and why are they still news today? Because they are consistently one of the top attack vectors that companies face today. Shauntine'z discusses the DDoS arms race and provides some poignant statistics that remind us of the very real and credible DDoS threat. But the article doesn't stop there...it goes on to provide some excellent tips on what to do to strengthen your DDoS defense posture (it even has a well-placed picture of Professor John Frink...you gotta check this one out). Last, Shauntine'z reveals new features that are loaded in the latest release of the BIG-IP...version 11.6. The AFM and ASM have some new and exciting capabilities that are "must haves" for any company that is serious about securing their applications and critical business functions. (Editors note: the LineRate product has been discontinued for several years. 09/2023) Why ECC and PFS Matter: SSL offloading with LineRate We all know that sensitive data traverses our networks every day. We also know it's critically important to secure this information. We also know that SSL/TLS is the primary method used to secure said information. Andrew Ragone discusses SSL offloading and tells us why Elliptical Curve Cryptography (ECC) and Perfect Forward Secrecy (PFS) are great candidates for securing your information. He highlights the advantages of the software based LineRate solution, and gives great examples of why LineRate is the clear-cut winner over any existing software-based or hardware-based SSL/TLS offload solutions. Andrew also published another series of articles related to this very topic, and in these articles he walks you through the exact steps needed to configure SSL certificates and offload SSL on LineRate. On that subject...if you haven't had a chance to check out LineRate and learn all about the awesomeness that it is, do yourself a favor and visit
Protect your applications using F5’s Distributed Cloud and Fast ACL’s
Introduction In this article I will show you how to easily create Fast ACL’s to protect your applications from DDoS attacks. Layer 3-4 DDoS Mitigation is included with the F5 Distributed Cloud service. When planning your DDoS strategy, you must plan at many layers. This means organizations need multiple tools and capabilities to protect themselves and keep their infrastructure and applications running. This layered approach uses network firewalls for Layer 3-4 DDoS protection, Web Application Firewalls (WAF) for Layer 7 protections, and as I’ll cover here Access Control Lists (ACLs) or as we call them Fast ACLs that can include rate-limiting. These ACL rules are applied at very early stages in datapath ingress processing and form a first line of defense against attack. Typical Use case(s) are: Rate-limiting traffic to destination Accepting traffic from certain source IPs to destination Rate-limiting or dropping traffic from source IPs to destination These rules are evaluated for each packet coming into the system (ingress), unlike session-based ACL’s where action is calculated only on first packet in the session. It is specified in terms of five tuple of the packet {destination ip, destination port, source ip, source port, protocol}. This gives you the ability to fine tune your DDoS strategy based on your network infrastructure and application performance. Getting Started Log in to your F5 Distributed Cloud Service. Select the Cloud and Edge Sites Tile. Navigate to Manage >> Firewall >> Fast ACL’s. We will also be discussing and using Policers and Protocol Policers. They can be added from this screen or as we build out our DDoS protection. We will show how during the build out. Click Add Fast ACL Give your Fast ACL a Name, add a Label and Description to help identify later. Next is what sets F5 XC Services apart. Under Fast ACL Type you have 2 options. This can be at the F5 XC Services regional edge (RE) or your own customer edge (CE) for apps deployed locally with F5 Distributed Cloud Service nodes. For this article I will cover the Customer Edge (CE). Select Configure. Here you have the option to select which network to apply this to at the CE, Inside or Outside, I will use the Outside Network. Next select the Destination IP, where you have three options to protect. I will use All Interface IP(s) as VIP. Finally, under Source, we will configure the Rules we wish to apply. Click Configure Under the Rules Section. Click Add Item Give your Rule a Name and a Description. Under Action you have 3 options, Simple Action, Policer Action and Protocol Policer Action. First, we will cover the Simple Action. You have two options, allow or deny. Under Source Port, click Add Item. You have the option to select All Ports, A User Defined Port or DNS. I will select User defined and add the value of 443. Under Source I'll allow all from 0.0.0.0./0 and click Add Item. Now you can go back in Rules and any additional rules that reflect your architecture. Click Add Item. This time I'll select Deny as the Action and ALL as Source Ports and Source Prefix as 0.0.0.0/0 When complete click Apply, this takes you back a Screen, Click Apply again. Protocol Policer Finally, we will configure a FAST ACL Protocol Policer. Give your Protocol Policer a Name, Labels and Description. Select a pre-configured Protocol Policer if one is already configured or you have system wide one you wish to apply. For this demonstration we will click Create new Protocol Policer. Click Add Item This will give you the option of Packet Type. The options are TCP, ICMP, UDP and DNS. For this we will select TCP. Then you select the appropriate TCP Flags, we will select SYN. Policer Dropping to the Policer section, we either need to select a preconfigured policer that might be used system wide or Create a new one. We will select Create New Policer. Creating a Policer is straightforward. Give it a Name, Labels and Description. Select If the Policer is to be Shared or Not Shared System wide. Here is where creating Fast ACLs helps you fine tune your DDoS protection for your application. You will enter both a Committed Information Rate in pps and a Burst Size in pps. Click Continue Add item and then Continue. Finally Save and Exit. These are all the steps necessary to get started using Fast ACL's. Two additional steps are needed and beyond the scope of this article. Most builds will already have the necessary configurations required. You need to have a Network Firewall designated and what F5 Distributed Cloud calls a Fleet. The Firewall will reference the ACL and the Firewall and Fleet Tag will be asisgned to your Customer Edge (CE). Conclusion In this article you learned how to configure Fast ACLs DDoS protection quickly and easily with the F5's Distributed Cloud. We included Rate Limiting as a viable option to tune your DDoS settings. In a few short minutes you would be able to react to an attack on your network by going into the F5 Distributed Cloud Console and adjusting or adding DDoS protections. "Nature is a mutable cloud, which is always and never the same." - Ralph Waldo Emerson We might not wax that philosophically around here, but our heads are in the cloud nonetheless! Join the F5 Distributed Cloud user group today and learn more with your peers and other F5 experts. For further information or to get started: F5 Distributed Cloud WAAP YouTube series (Link) F5 Distributed Cloud WAAP Services (Link) F5 Distributed Cloud WAAP Get Started (Link)
Explanation of F5 DDoS threshold modes
Der Reader, In my article “Concept of Device DOS and DOS profile”, I recommended to use the “Fully Automatic” or “Multiplier” based configuration option for some DOS vectors. In this article I would like to explain how these threshold modes work and what is happening behind the scene. When you configure a DOS vector you have the option to choose between different threshold modes: “Fully Automatic”, “Auto Detection / Multiplier Based”, “Manual Detection / Auto Mitigation” and “Fully Manual”. Figure 1: Threshold Modes The two options I normally use on many vectors are “Fully Automatic” and “Auto Detection / Multiplier Based”. But what are these two options do for me? To manually set thresholds is for some vectors not an easy task. I mean who really knows how many PUSH/ACK packets/sec for example are usually hitting the device or a specific service? And when I have an idea about a value, should this be a static value? Or should I better take the maximum value I have seen so far? And how many packets per second should I put on top to make sure the system is not kicking in too early? When should I adjust it? Do I have increasing traffic? Fully Automatic In reality, the rate changes constantly and most likely during the day I will have more PUSH/ACK packets/sec then during the night. What happens when there is a campaign or an event like “Black Friday” and way more users are visiting the webpage then usually? During these high traffic events, my suggested thresholds might be no longer correct which could lead to “good” traffic getting dropped. All this should be taken into consideration when setting a threshold and it ends up being very difficult to do manually. It´s better to make the machine doing it for you and this is what “Fully Automatic” is about. Figure 2: Expected EPS As soon as you use this option, it leverages from the learning it has done since traffic is passing through the BIG-IP, or since you have enforced the relearning, which resets everything learned so far and starts from new. The system continuously calculates the expected rates for all the vectors based on the historic rates of traffic. It takes the information up to one year and calculates them with different weights in order to know which packets rate should be expected at that time and day for that specific vector in the specific context (Device, Virtual Server/Protected Object). The system then calculates a padding on top of this expected rate. This rate is called Detection Rate and is dependent on the “threshold sensitivity” you have configured: Low Sensitivity means 66% padding Medium Sensitivity means 40% padding High Sensitivity means 0% padding Figure 3: Detection EPS As soon as the current rate is above the detection value, the BIG-IP will show the message “Attack detected”, which actually means anomaly detected, because it sees more packets of that specific vector then expected + the padding (detection_rate). But DoS mitigation will not start at that point! Figure 4: Current EPS Keep in mind, when you run the BIG-IP in stateful mode it will drop 'out of state' packets anyway. This has nothing to do with DoS functionalities. But what happens when there is a serious flood and the BIG-IP CPU gets high because of the massive number of packets it has to deal with? This is when the second part of the “Fully Automatic” approach comes into the game. Again, depending on your threshold sensitivity the DOS mitigation starts as soon as a certain level of stress is detected on the CPU of the BIG-IP. Figure 5: Mitigation Threshold Low Sensitivity means 78,3% TMM load Medium Sensitivity means 68,3% TMM load High Sensitivity means 51,6% TMM load Note, that the mitigation is per TMM and therefore the stress and rate per TMM is relevant. When the traffic rate for that vector is above the detection rate and the CPU of the BIG-IP (Device DOS) is “too” busy, the mitigation kicks in and will rate limit on that specific vector. When a DOS vector is hardware supported, FPGAs drop the packets at the switch level of the BIG-IP. If that DOS vector is not hardware supported, then the packet is dropped at a very early stage of the life cycle of a packet inside a BIG-IP. The rate at which are packets dropped is dynamic (mitigation rate), depending on the incoming number of packets for that vector and the CPU (TMM) stress of the BIG-IP. This allows the stress of the CPU to go down as it has to deal with less packets. Once the incoming rate is again below the detection rate, the system declares the attack as ended. Note: When an attack is detected, the packet rate during that time will not go into the calculation of expected rates for the future. This ensures that the BIG-IP will not learn from attack traffic skewing the automatic thresholds. All traffic rates below the detection (or below the floor value, when configured) rate modify the expected rate for the future and the BIG-IP will adjust the detection rate automatically. For most of the vectors you can configure a floor and ceiling value. Floor means that as long as the traffic value is below that threshold, the mitigation for that vector will never kick in. Even when the CPU is at 100%. Ceiling means that mitigation always kicks in at that rate, even when the CPU is idle. With these two values the dynamic and automatic process is done between floor and ceiling. Mitigation only gets executed when the rate is above the rate of the Floor EPS and Detection EPS AND stress on the particular context is measured. Figure 6: Floor and Ceiling EPS What is the difference when you use “Fully Automatic” on Device level compared to VS/PO (DOS profile) level? Everything is the same, except that on VS or Protected Object (PO) level the relevant stress is NOT the BIG-IP device stress, it is the stress of the service you are protecting (web-, DNS, IMAP-server, network, ...). BIG-IP can measure the stress of the service by measuring TCP attributes like retransmission, window size, congestion, etc. This gives a good indication on how busy a service is. This works very well for request/response protocols like TCP, HTTP, DNS. I recommend using this, when the Protected Object is a single service and not a “wildcard” Protected Object covering for example a network or service range. When the Protected Object is a “wildcard” service and/or a UDP service (except DNS), I recommend using “Auto Detection / Multiplier Option”. It works in the same way as the “Fully Automatic” from the learning perspective, but the mitigation condition is not stress, it is the multiplication of the detection rate. For example, the detection rate for a specific vector is calculated to be 100k packets/sec. By default, the multiplication rate is “500”, which means 5x. Therefore, the mitigation rate is calculated to 500k packets/sec. If that particular vector has more than 500k packets/sec those packets would be dropped. The multiplication rate can also be individually configured. Like in the screenshot, where it is set to 8x (800). Figure 7: Auto Detection / Multiplier Based Mitigation The benefit of this mode is that the BIG-IP will automatically learn the baseline for that vector and will only start to mitigate based on a large spike. The mitigation rate is always a multiplication of the detection rate, which is 5x by default but is configurable. When should I use “Fully Manual”? When you want to rate-limit a specific vector to a certain number of packets/sec, then “Fully Manual” is the right choice. Very good examples for that type of vector are the “Bad Header” vector types. These type of packets will never get forwarded by the BIG-IP so dropping them by a DoS vector saves the CPU, which is beneficial under DoS conditions. In the screenshot below is a vector configured as “Fully Manual”. Next I’ll describe what each of the options means. Figure 8: Fully Manual Detection Threshold EPS configures the packet rate/sec (pps) when you will get a log messages (NO mitigation!). Detection Threshold % compares the current pps rate for that vector with the multiplication of the configured percentage (in this example 5 for 500%) with the 1-minute average rate. If the current rate is higher, then you will get a log message. Mitigation Threshold EPS rate limits to that configured value (mitigation). I recommend setting the threshold (Mitigation Threshold EPS) to something relatively low like ‘10’ or ‘100’ on ‘Bad Header’ type of vectors. You can also set it also to ‘0’, which means all packets hitting this vector will get dropped by the DoS function which usually is done in hardware (FPGA). With the ‘Detection Threshold EPS’ you set the rate at which you want to get a log messages for that vector. If you do it this way, then you get a warning message like this one to inform you about the logging behavior: Warning generated: DOS attack data (bad-tcp-flags-all-set): Since drop limit is less than detection limit, packets dropped below the detection limit rate will not be logged. Another use-case for “Fully Manual” is when you know the maximum number of these packets the service can handle. But here my recommendation is to still use “Fully Automatic” and set the maximum rate with the Ceiling threshold, because then the protected service will benefit from both threshold options. Important: Please keep in mind, when you set manual thresholds for Device DoS the thresholds are to protect each TMM. Therefore the value you set is per TMM! An exception to this is the Sweep and Flood vector where the threshold is per BIG-IP/service and not per TMM like on DoS profiles. When using manual thresholds for a DOS profile of a Protected Object the threshold configuration is per service (all packets targeted to the protected service) NOT per TMM like on the Device level. Here the goal is to set how many packets are allowed to pass the BIG-IP and reach the service. The distribution of these thresholds to the TMMs is done in a dynamic way: Every TMM gets a percentage of the configured threshold, based on the EPS (Events Per Second, which is in this context Packets Per Second) for the specific vector the system has seen in the second before on this TMM. This mechanism protects against hash type of attacks. Ok, I hope this article gives you a better understanding on how ‘Fully Manual’, ‘Fully Automatic’ and ‘Auto Detection / Multiplier Based Mitigation’ works. They are important concepts to understand, especially when they work in conjunction with stress measurement. This means the BIG-IP will only kick in with the DoS mitigation when the protected object (BIG-IP or the service behind the BIG-IP) is under stress. -Why risk false positives, when not necessary? With my next article I will demonstrate you how Device DoS and the DoS profiles work together and how the stateful concept cooperates with the DoS mitigation. I will show you some DoS commands to test it and also commands to get details from the BIG-IP via CLI. Thank you, sVen MuellerDemonstration of Device DoS and Per-Service DoS protection
Dear Reader, This article is intended to show what effect the different threshold modes have on the Device and Per-Service (VS/PO) context. I will be using practical examples to demonstrate those effects. You will get to review a couple of scripts which will help you to do DoS flood tests and “visualizing” results on the CLI. In my first article (https://devcentral.f5.com/s/articles/Concept-of-F5-Device-DoS-and-DoS-profiles), I talked about the concept of F5 Device DoS and Per-Service DoS protection (DoS profiles). I also covered the physical and logical data path, which explains the order of Device DoS and Per-Service DoS using the DoS profiles. In the second article (https://devcentral.f5.com/s/articles/Explanation-of-F5-DDoS-threshold-modes), I explained how the different threshold modes are working. In this third article, I would like to show you what it means when the different modes work together. But, before I start doing some tests to show the behavior, I would like to give you a quick introduction into the toolset I´m using for these tests. Toolset First of all, how to do some floods? Different tools that can be found on the Internet are available for use. Whatever tools you might prefer, just download the tool and run it against your Device Under Test (DUT). If you would like to use my script you can get it from GitHub: https://github.com/sv3n-mu3ll3r/DDoS-Scripts With this script - it uses hping - you can run different type of attacks. Simply start it with: $ ./attack_menu.sh <IP> <PORT> A menu of different attacks will be presented which you can launch against the used IP and port as a parameter. Figure 1: Attack Menu To see what L3/4 DoS events are currently ongoing on your BIG-IP, simply go to the DoS Overview page. Figure 2: DoS Overview page I personally prefer to use the CLI to get the details I´m interested in. This way I don´t need to switch between CLI to launch my attacks and GUI to see the results. For that reason, I created a script which shows me what I am most interested in. Figure 3: DoS stats via CLI You can download that script here: https://github.com/sv3n-mu3ll3r/F5_show_DoS_stats_scripts Simply run it with the “watch” command and the parameter “-c” to get a colored output (-c is only available starting with TMOS version 14.0): What is this script showing you? context_name: This is the context, either PO/VS or the Device in which the vector is running vector_name: This is the name of the DoS vector attack_detected: When it shows “1”, then an attack has been detected, which means the ‘stats_rate’ is above the ‘detection-rate'. stats_rate: Shows you the current incoming pps rate for that vector in this context drops_rate: Shows you the number dropped pps rate in software (not FPGA) for that vector in this context int_drops_rate: Shows you the number dropped pps rate in hardware (FPGA) for that vector in this context ba_stats_rate: Shows you the pps rate for bad actors ba_drops_rate: Shows you the pps rate of dropped ‘bad actors’ in HW and SW bd_stats_rate: Shows you the pps rate for attacked destination bd_drop_rate: Shows you the pps rate for dropped ‘attacked destination’ mitigation_curr: Shows the current mitigation rate (per tmm) for that vector in that context detection: Shows you the current detection rate (per tmm) for that vector in that context wl_count: Shows you the number of whitelist hits for that vector in that context hw_offload: When it shows ‘1’ it means that FPGAs are involved in the mitigation int_dropped_bytes_rate: Gives you the rate of in HW dropped bytes for that vector in that context dropped_bytes_rate: Gives you the rate of in SW dropped bytes for that vector in that context When a line is displayed in green, it means packets hitting that vector. However, no anomaly is detected or anything is mitigated (dropped) via DoS. If a line turns yellow, then an attack - anomaly – has been detected but no packets are dropped via DoS functionalities. When the color turns red, then the system is actually mitigating and dropping packets via DoS functionalities on that vector in that context. Start Testing Before we start doing some tests, let me provide you with a quick overview of my own lab setup. I´m using a DHD (DDoS Hybrid Defender) running on a i5800 box with TMOS version 15.1 My traffic generator sends around 5-6 Gbps legitimate (HTTP and DNS) traffic through the box which is connected in L2 mode (v-wire) to my network. On the “client” side, where my clean traffic generator is located, my attacking clients are located as well by use of my DoS script. On the “server” side, I run a web service and DNS service, which I´m going to attack. Ok, now let’s do some test so see the behavior of the box and double check that we really understand the DDoS mitigation concept of BIG-IP. Y-MAS flood against a protected server Let’s start with a simple Y-MAS (all TCP flags cleared) flood. You can only configure this vector on the device context and only in manual mode. Which is ok, because this TCP packet is not valid and would get drop by the operating system (TMOS) anyway. But, because I want this type of packet get dropped in hardware (FPGA) very early, when they hit the box, mostly without touching the CPU, I set the thresholds to ‘10’ on the Mitigation Threshold EPS and to ‘10’ on Detection Threshold EPS. That means as soon as a TMM sees more then 10 pps for that vector it will give me a log message and also rate-limit this type of packets per TMM to 10 packets per second. That means that everything below that threshold will get to the operating system (TMOS) and get dropped there. Figure 4: Bad TCP Flags vector As soon as I start the attack, which targets the web service (10.103.1.50, port 80) behind the DHD with randomized source IPs. $ /usr/sbin/hping3 --ymas -p 80 10.103.1.50 --flood --rand-source I do get a log messages in /var/log/ltm: Feb 5 10:57:52 lon-i5800-1 err tmm3[15367]: 01010252:3: A Enforced Device DOS attack start was detected for vector Bad TCP flags (all cleared), Attack ID 546994598. And, my script shows me the details on that attack in real time (the line is in ‘red’, indicating we are mitigating): Currently 437569 pps are hitting the device. 382 pps are blocked by DDoS in SW (CPU) and 437187 are blocked in HW (FPGA). Figure 5: Mitigating Y-Flood Great, that was easy. :-) Now, let’s do another TCP flood against my web server. RST-flood against a protected server with Fully manual Threshold mode: For this test I´m using the “Fully Manual” mode, which configures the thresholds for the whole service we are protecting with the DoS profile, which is attached to my VS/PO. Figure 6: RST flood with manual configuration My Detection Threshold and my Mitigation Threshold EPS is set to ‘100’. That means as soon as we see more then 100 RST packets hitting the configured VS/PO on my BIG-IP for this web server, the system will start to rate-limit and send a log message. Figure 7: Mitigating RST flood on PO level Perfect. We see the vector in the context of my web server (/Common/PO_10.103.1.50_HTTP) is blocking (rate-limiting) as expected from the configuration. Please ignore the 'IP bad src' which is in detected mode. This is because 'hping' creates randomized IPs and not all of them are valid. RST-flood against a protected server with Fully Automatic Threshold mode: In this test I set the Threshold Mode for the RST vector on the DoS profile which is attached to my web server to ‘Fully Automatic’ and this is what you would most likely do in the real world as well. Figure 8: RST vector configuration with Fully Automatic But, what does this mean for the test now? I run the same flood against the same destination and my script shows me the anomaly on the VS/PO (and on the device context), but it does not mitigate! Why would this happen? Figure 9: RST flood with Fully Automatic configuration When we take a closer look at the screenshot we see that the ‘stats_rate’ shows 730969 pps. The detection rate shows 25. From where is this 25 coming? As we know, when ‘Fully Automatic’ is enabled then the system learns from history. In this case, the history was even lower than 25, but because I set the floor value to 100, the detection rate per TMM is 25 (floor_value / number of TMMs), which in my case is 100/4 = 25 So, we need to recognize, that the ‘stats_rate’ value represents all packets for that vector in that context and the detection value is showing the per TMM value. This value explains us why the system has detected an attack, but why is it not dropping via DoS functionalities? To understand this, we need to remember that the mitigation in ‘Fully Automatic’ mode will ONLY kick in if the incoming rate for that vector is above the detection rate (condition is here now true) AND the stress on the service is too high. But, because BIG-IP is configures as a stateful device, this randomized RST packets will never reach the web service, because they get all dropped latest by the state engine of the BIG-IP. Therefor the service will never have stress caused by this flood.This is one of the nice benefits of having a stateful DoS device. So, the vector on the web server context will not mitigate here, because the web server will not be stressed by this type of TCP attack. This does also explains the Server Stress visualization in the GUI, which didn´t change before and during the attack. Figure 10: DoS Overview in the GUI But, what happens if the attack gets stronger and stronger or the BIG-IP is too busy dealing with all this RST packets? This is when the Device DOS kicks in but only if you have configured it in ‘Fully Automatic’ mode as well. As soon as the BIG-IP receives more RST packets then usually (detection rate) AND the stress (CPU load) on the BIG-IP gets too high, it starts to mitigate on the device context. This is what you can see here: Figure 11: 'massive' RST flood with Fully Automatic configuration The flood still goes against the same web server, but the mitigation is done on the device context, because the CPU utilization on the BIG-IP is too high. In the screenshot below you can see that the value for the mitigation (mitigation_curr) is set to 5000 on the device context, which is the same as the detection value. This value results from the 'floor' value as well. It is the smallest possible value, because the detection and mitigation rate will never be below the 'floor' value. The mitigation rate is calculated dynamically based on the stress of the BIG-IP. In this test I artificially increased the stress on the BIG-IP and therefor the mitigation rate was calculated to the lowest possible number, which is the same as the detection rate. I will provide an explanation of how I did that later. Figure 13: Device context configuration for RST Flood Because this is the device config, the value you enter in the GUI is per TMM and this is reflected on the script output as well. What does this mean for the false-positive rate? First of all, all RST packets not belonging to an existing flow will kicked out by the state engine. At this point we don´t have any false positives. If the attack increases and the CPU can´t handle the number of packets anymore, then the DOS protection on the device context kicks in. With the configuration we have done so far, it will do the rate-limiting on all RST packets hitting the BIG-IP. There is no differentiation anymore between good and bad RST, or if the RST has the destination of server 1 or server 2, and so on. This means that at this point we can face false positives with this configuration. Is this bad? Well, false-positives are always bad, but at this point you can say it´s better to have few false-positives then a service going down or, even more critical, when your DoS device goes down. What can you do to only have false positives on the destination which is under attack? You probably have recognized that you can also enable “Attacked Destination Detection” on the DoS vector, which makes sense on the device context and on a DoS profile which is used on protected object (VS), that covers a whole network. Figure 14: Device context configuration for RST Flood with 'Attacked Destination Detection' enabled If the attack now hits one or multiple IPs in that network, BIG-IP will identify them and will do the rate-limiting only on the destination(s) under attack. Which still means that you could face false positives, but they will be at least only on the IPs under attack. This is what we see here: Figure 15: Device context mitigation on attacked destination(s) The majority of the RST packet drops are done on the “bad destination” (bd_drops_rate), which is the IP under attack. The other option you can also set is “Bad Actor Detection”. When this is enabled the system identifies the source(s) which causes the load and will do the rate limiting for that IP address(es). This usually works very well for amplification attacks, where the attack packets coming from ‘specific’ hosts and are not randomized sources. Figure 16: Device context mitigation on attacked destination(s) and bad actor(s) Here you can see the majority of the mitigation is done on ‘bad actors’. This reduces the chance of false positives massively. Figure 17: Device context configuration for RST Flood with 'Attacked Destination Detection' and 'Bad Actor Detection' enabled You also have multiple additional options to deal with ‘attacked destination’ and ‘bad actors’, but this is something I will cover in another article. Artificial increase BIG-IPs CPU load Before I finish this article, I would like to give you an idea on how you could increase the CPU load of the BIG-IP for your own testing. Because, as we know, with “Fully Automatic” on the device context, the mitigation kicks only in if the packet rate is above the detection rate AND the CPU utilization on the BIG-IP is “too” high. This is sometimes difficult to archive in a lab because you may not be able to send enough packets to stress the CPUs of a HW BIG-IP. In this use-case I use a simple iRule, which I attach to the VS/PO that is under attack. Figure 18: Stress iRule When I start my attack, I send the traffic with a specific TTL for identification. This TTL is configured in my iRule in order to get a CPU intensive string compare function to work on every attack packet. Here is an example for 'hping': $ /usr/sbin/hping3 --rst -p 80 10.103.1.50 --ttl 5 --flood --rand-source This can easily drive the CPU very high and the DDoS rate-limiting kicks in very aggressive. Summary I hope that this article provides you with an even better understanding on what effect the different threshold modes have on the attack mitigation. Of course, keep in mind this are just the ‘static DoS’ vectors. In later articles I will explain also the 'Behavioral DoS' and the Cookie based mitigation, which helps to massively reduce the chance of a false positives. But, also keep in mind, the DoS vectors start to act in microseconds and are very effective for protecting. Thank you, sVen Mueller
6. SYN Cookie: Hardware vs Software
Introduction Currently, you know the differences between LTM and AFM when talking about SYN Cookie capabilities and configuration. In this article I describe how SYN Cookie can perform its tasks using two different ways, in software or offloaded in hardware. The most important difference between these functioning modes is clearly the resource consuming. In hardware based platforms SYN Cookie is a countermeasure that will not consume extra memory nor CPU, while in software this task must be handled by TMM, so some load could be added comparing to normal operation. Hardware SYN Cookie If you have a hardware offloading capable platform and you configure SYN Cookie to work in hardware, the creation and validation of SYN Cookie challenges will be managed by dedicated FPGA (Programmable Field Array) or NSP (Neuron Search Processor) In a few words this means that SYN Cookie running in hardware basically allows you to forget about TCP SYN flood attacks. You do not have to care about it, if it is correctly configured, since BIG-IP will not only mitigate attack without impacting legitimate users, who will still access to application during the attack, but also no extra resources will be needed. The only handicap you have to take into account currently when hardware SYN Cookie is activated is that you cannot keep TCP options information. In order to keep this information you need to enable SYN Cookie in software and client connection need to activate TCP TS option (this is enabled by default in BIG-IP). See second article in this series for more details (SYN Cookie Operation). Neuron Platforms shipped with Neuron chip and running a TMOS version greater than 14, improve performance because this chip, which is directly connected to HSBe2, provide with extra functionalities to SYN Cookie. Neuron improves performance and solves limitations that you could find in HSBe2 only platforms when SYN Cookie is activated for wildcard virtual servers. A brief description of this limitation can be found in K50955355. Note that Neuron does not only offloads SYN Cookie tasks from software, if you have an AFM provisioned device you can take advantage of other features like extended whitelists and blacklists in hardware. However these article series are only focused on SYN Cookie. With this in mind the functioning of Neuron is simple, HSBe2 makes requests to Neuron when a TCP SYN reaches BIG-IP, and depending of the type of request Neuron then will create a SYN Cookie for the connection or it will validate SYN Cookie response presented by client. For example, as I showed in previous articles, when an ACK comes from a client then HSBe2 will ask Neuron if SYN Cookie is correct, then Neuron will response with the requested information, so TMM will know if it must drop or allow the connection through the datapath. SYN Cookie Neuron does not offload AFM Global SYN Cookie, it works for: LTM Global SYN Cookie Per VLAN SYN Cookie LTM Per Virtual SYN Cookie AFM Per Virtual SYN Cookie In order to enable SYN Cookie Neuron in your platform you need to activate the Turboflex profile for Security/Securityv1 (AFM) or ADC (LTM). There is many literature in AskF5 about Turboflex, but in a few words it allows you to group several related features to be accelerated in hardware. I will not give technical details about Neuron since it is not the intention of these article series. For troubleshooting issues with Neuron please raise a case to F5 Networks, so expert engineer can investigate it. Making internal changes to Neuron could lead to a worst result. Disabling SYN Cookie SYN Cookie is enabled by default in software and hardware, if we disable hardware SYN Cookie then software SYN Cookie comes into play. There is a DB key for disabling Hardware SYN Cookie in all contexts: tmsh list sys db pvasyncookies.enabled sys db pvasyncookies.enabled { value "false" } Of course, if the specific platform cannot offload SYN Cookie into hardware then above DB key has no effect. In this case software SYN Cookie is enabled by default. There is not such DB key for enabling/disabling software SYN Cookie as we have for hardware, instead, if you want to disable software SYN Cookie then you have different options depending on context that you can consult in table below. Remember that AFM SYN Cookie has precedence over LTM SYN Cookie, this means that if AFM SYN Cookie is configured and you want to disable completely SYN Cookie you need to disable both, AFM and LTM SYN Cookie. The simplest path to disable completely SYN Cookie assuming default config, that is, hardware SYN Cookie is enabled, it would be: Fig14. Disabling SYN Cookie In table below I summarize how to disable hardware and software SYN Cookie in each context for the different scopes. For AFM disabling SYN Cookie is quite easier, you just need to change DoS Device and DoS profiles TCP Half Open vector to 0/Infinite. Warnings It can be possible that SYN Cookie works in software in your device whilst you expected SYN Cookie working in hardware. Sometimes you end up in this situation due to a wrong configuration. Below I make a list of possible reasons that you can check: You have disabled autolasthop. This option must be enabled if you want to offload SYN Cookie into hardware (K16887). You have changed DB key connection.syncookies.algorithm manually. This key defines which algorithm is used for generating hash that is part of the SYN Cookie challenge (as I described in the first article of this series). When set a value of Hardware to this DB key the error detecting code used for validating the hash will be understood by both, software and hardware. If set to Software then it will only be understood by software. If your version also has the option ‘both’, this means that BIG-IP dynamically discovers which algorithm should be used rather than forcing to Software or Hardware. In summary, if virtual is configured for hardware SYN Cookie but algorithm is configured in software then hardware will inspect SYN Cookies, it will confirm it cannot validate them and then it will send it to TMM for validation. Until v14.1 SYN cookie hardware it is not recommended for protecting networks (wildcard virtual servers), this is because SYN Cookie will only protect a single flood network destination when flooding towards multiple network destination at a time. The other networks will be protected by software SYN cookie. This could cause unexpected extra CPU utilization. This is greatly improved in Neuron capable platforms. If there are collisions when SYN Cookie is configured in hardware you will still see non zero stats for software SYN Cookie. Also, these stats can increase due to validation of first challenges, when SYN Cookie is activated, and TMMs handles TCP SYN until it enables hardware to do it. This is expected and it does not mean that hardware SYN Cookie is not working. I will show examples in article dedicated to stats. In order to end up this section with clear ideas, remember that hardware SYN Cookie and hardware flow acceleration are two different concepts. Conclusion In this article you learnt how behaves SYN Cookie when working in software and hardware and the advantages or disadvantages. In next two articles I will show you how to troubleshoot SYN Cookie issues.4. SYN Cookie: LTM Configuration
Introduction Since you already know how SYN Cookie works now it is time to start configuring BIG-IP devices. In this article I explain how to configure BIG-IP LTM devices for protecting against TCP SYN flood attack at different contexts. Configuration BIG-IP is an application focused device, so from security perspective is in charge of protecting these applications, but it is also important to protect BIG-IP itself since compromising it means that all applications could be affected. This is why SYN Cookie can be configured not only for protecting applications but also for protecting whole BIG-IP, and even for protecting specific VLANs connected to BIG-IP. In this section I will explain how to configure SYN Cookie for each context. Device/Global At this context we protect BIG-IP itself, so the idea is defining how many embryonic connections are allowed to be handled by BIG-IP TCP stack, regardless virtual server destination. This means that SYN Cache counter will increase for each embryonic connection that device is handling globally. Since TMM is the core of the BIG-IP, the element that manipulate traffic, it makes sense to protect each TMM specifically. This is why SYN Cache at global context is defined per TMM. So for example, configuring a value of 1000 in a 4 TMM device means that the Big-IP will potentially (if connections are evenly balanced among TMMs) handle 4000 embryonic connections before activating SYN Cookie. Fig9. Device context SYN cookie will take into account all TCP SYN packets sent to any listener exposed, this includes also selfIPs and virtual servers, but note that it MUST be enabled in the protocol profile applied to virtual server (which is in fact the default value) in order to Global SYN cookie take the specific virtual server into account when counting embryonic connections. Configuration example for a device with one virtual server using a tcp profile: VLAN context The threshold is defined per TMM as in device context . This is only available for SYN Cookie hardware offloading platforms. See related AskF5 article for more details. With this feature you can protect a group of virtual servers listening in the same VLAN. Reasons why you would want to do this can vary, from architecture perspective you could have a dedicated VLAN serving applications on old servers and therefore expecting lower number of connections for example. Or you could want to configure SYN Cookie in a more granular way instead configuring a global value for whole device. In older versions also there could be a risk of collisions when multiple virtual servers were under attack at the same time, so using this method instead SYN Cookie per virtual could help in these cases. If SYN Cookie is enabled at Global context the SYN Cookie Per-VLAN is disabled because Device protection is ON at all-VLAN basis and it would interfere with Per VLAN SYN cookie. Fig10. VLAN context At VLAN context you can configure not only SYN Cookie but also TCP SYN flood DDoS vector, even with only LTM license. It is important to note that setting a threshold for this vector could have consequences, unlike SYN Cookie, this vector will start to drop TCP SYN packets once the configured number of TCP SYN packets per second is reached. This is because TCP SYN flood vector just counts TCP SYN packets reaching the VLAN, instead embryonic connections, and if a TCP SYN packet exceeds the threshold then it is just dropped. TCP SYN flood and TCP SYN Cookie are totally different and independent countermeasures. Virtual server context In this case the threshold is defined globally. This means that the value is defined for all TMMs, unlike SYN Cookie at Device/VLAN context. If you want to know how many embryonic connections will be handled by each TMM prior the TMM activates SYN Cookie then you must divide configured threshold among the number of existing TMMs. When configuring SYN Cookie in a LTM device you only can define a general threshold that it will be common for all virtual servers (in next article I will show differences with AFM SYN Cookie). So all virtual servers will have the same threshold. Fig11. Virtual Server context Configuring SYN Cookie at this context requires setting a common threshold for all virtual servers but also you MUST enable SYN Cookie in specific protocol profile that is applied to the virtual server in order to be able to enable the countermeasure for that virtual server. This will allow you to enable SYN Cookie in some virtual servers and disable it in others. Although you cannot have specific SYN Cache threshold for a virtual server, you can have different SYN Cookie behaviours depending on virtual server, like for example using whitelist or not (see second table below). Example for fastL4 configuration: Note that in protocol profile you could still have see twp options in TMSH that are deprecated from version 13 and have no effect anymore. They will be removed in a future release: tmsh modify ltm profile fastl4 fastL4 hardware-syn-cookie <value> tmsh modify ltm profile fastl4 fastL4 software-syn-cookie <value> As you can see in table above, as part of DSR configuration you can decide the device that will send the RST to the TCP connection: tmsh modify ltm profile fastl4 <name> syn-cookie-dsr-flow-reset-by <bigip | client | none> Sometimes it could be better choosing ‘client’ since some applications do not like Big-IP sending the reset. In order to get this, Big-IP sends the SYN Cookie in the ACK instead than in the sequence number, in this way client will RST the connection for this specific ACK, so when Big-IP gets the RST knows about the referred connection and it adds it into DSR whitelist. SYN Cookie can work in two different modes at this context. Example for tcp protocol profile: In case you have wildcard virtual servers and you want to protect them with SYN Cookie at virtual server context, then you will have to configure Software SYN Cookie unless you have a Neuron capable platform. I will give more details about this in next articles. Default values In TMOS versions higher than v12 we have the below default values: Three comments related to above values. DB Key pvasyncookies.virtual.maxsyncache is deprecated in version >v12. This means that you should ignore the value of this DB Key whatever it is, since it is not used anymore by the system. Reason why we disable SYN Cookie at virtual server context when device is installed from scratch, is that we do not know the capacity of the pool members attached to virtual servers, so we let that value be disabled for customers to decide this threshold. However after upgrade from v12 we do not want to modify whatever behaviour security administrator considered correct in v12, so we just keep the value. At virtual server context SYN Cookie threshold in TMOS ≤v12 was set per TMM. In TMOS >v12 we define this value globally, as explained above in this same article. This is why threshold after upgrade is calculated as the existing value configured in v12 multiplied by the number of TMMs. In this way we keep the same value, so customer will not notice any change in the behaviour of their device. But note that there are also a couple of things to take into account when upgrading from v12.x currently: If virtual server threshold in v12 (DB key pvasyncookies.virtual.maxsyncache) was < 4095 then value will be the same in >v12 after upgrade. If virtual server threshold in v12 (DB key pvasyncookies.virtual.maxsyncache) was ≥ 4095 then value after upgrade to ≥v12 will be 8192. This behaviour it is being improved for new versions, so threshold will be kept regardless its original value, as I have commented. Conclusion You know how to configure SYN Cookie in LTM module based system, in next article I will explain differences between LTM and AFM SYN Cookie, so you will be aware of the limitations of each one and have a global perspective of the feature.
