Concept of F5 Device DoS and DoS profiles
Dear Reader, I´m about to write a series of DDoS articles, which will hopefully help you to better understand how F5 DDoS works and how to configure it. I plan to release them in a more or less regular frequency. Feel free to send me your feedback and let me know which topics are relevant for you and should get covered. This first article is intended to give an explanation of the BIG-IP Device DOS and Per Service-DOS protection (DOS profile). It covers the concepts of both approaches and explains in high level the threshold modes “Fully manual”, “Fully Automatic” and “Multiplier Based Mitigation” including the principles of stress measurement. In later articles I will describe this threshold modes in more detail. In the end of the article you will also find an explanation of the physical and logical data path of the BIG-IP. Device DOS with static DoS vectors The primary goal of “Device DOS” is to protect the BIG-IP itself. BIG-IP has to deal with all packets arriving on the device, regardless of an existing listener (Virtual Server (VS)/ Protected Object (PO)), connection entry, NAT entry or whatever. As soon as a packet hits the BIG-IP it´s CPU has basically to deal with it. Picking up each and every packet consumes CPU cycles, especially under heavy DOS conditions. The more operations each packet needs, depending on configurations, the higher the CPU load gets, because it consumes more CPU cycles. The earlier BIG-IP can drop a packet (if the packet needs to get dropped) the better it is for the CPU load. If the drop process is done in hardware (FPGA), the packet gets dropped before it consumes CPU cycles. Using static DOS vectors helps to lower the number of packets hitting the CPU when under DoS. If the number of packets is above a certain threshold (usually when a flood happens), BIG-IP rate-limits for that specific vector, which keeps the CPU available, because it sees less packets. Figure 1: Principle of Attack mitigation with static DoS vectors The downside with this approach is that the BIG-IP cannot differentiate between "legitimate" and "attacking" packets, when using static DOS rate-limit. BIG-IP (FPGA) just drops on the predicates it gets from the static DOS vector. Predicates are attributes of a packet like protocol “TCP” or “flag” set “ACK”. -Keep in mind, when BIG-IP runs stateful, "bad" packets will get dropped in software anyway by the operating system (TMOS), when it identifies them as not belonging to an existing connection or as invalid/broken (for example bad CRC checksum). This is of course different for SYN packets, because they create connection entries. This is where SYN-Cookies play an important role under DoS conditions and will be explained in a later article. I usually recommend running Behavioral DoS mitigation in conjunction with static DoS vectors. It is way more precise and able to filter out only the attack traffic, but I will discuss this in more detail also in one of the following articles. Manual threshold vs. Fully Automatic Before I start to explain the “per service DoS protection” (DoS profiles), I would like to give you a brief overview of some Threshold modes you can use per DoS vector. There are different ways to set thresholds to activate the detection and rate-limiting on a DOS vector. The operator can do it manual by using the option “Fully Manual” and then fine tune the pre-configured values on the DOS vectors, which can be challenging, because beside doing it all manually it´s mostly difficult to know the thresholds, especially because they are usually related to the day and time of the week. Figure 2: Example of a manual DoS vector configuration That’s why most of the vectors have the option "Fully Automatic" available. It means BIG-IP will learn from the history and “knows” how many packets it usually “sees” at that specific time for the specific vector. This is called the baselining and calculates the detection threshold. Figure 3: Threshold Modes As soon as a flood hits the BIG-IP and crosses the detection threshold for a vector, BIG-IP detects it as an attack for that vector, which basically means it identifies it as an anomaly. However, it does not start to mitigate (drop). This will only happen as soon as a TMM load (CPU load) is also above a certain utilization (Mitigation sensitivity: Low: 78%, Medium 68%, High: 51%). If both conditions (packet rate and TMM/CPU load is too high) are true, the mitigation starts and lowers (rate-limit) the number of packets for this vector going into the BIG-IP for the specific TMM. That means dropping packets by the DOS feature will only happen if necessary, in order to protect the BIG-IP CPU. This is a dynamic process which drops more packets when the attack becomes more aggressive and less when the attack becomes less aggressive. For TCP traffic keep in mind the “invalid” traffic will not get forwarded anyway, which is a strong benefit of running the device stateful. When a DOS vector is hardware supported then FPGAs drop the packets basically at the switch level of the BIG-IP. If it’s not hardware supported, then the packet is dropped at a very early stage of the life cycle of a packet “inside” a BIG-IP. Device DOS counts ALL packets going to the BIG-IP. It doesn´t matter if they have a destination behind the BIG-IP, or when it is the BIG-IP itself. BecauseDevice DOSis to protect the BIG-IP,the thresholdsyou can configure in the “manual configuration” mode on the DOS vectorsare per TMM! This means you configure the maximum number of packets one TMM can handle. Very often operators want to set an overall threshold of let’s say 10k, then they need to divide this limit by the number of TMMs.An Exception is the Sweep and Flood vector. Here the threshold is per BIG-IP and not per TMM. DOS profile for protected Objects Now let’s talk about the “per Service DoS Protection (DoS profile)”. The goal is to protect the service which runs behind the BIG-IP. Important to know, when BIG-IP runs stateful, the service is already protected by the state of BIG-IP (true for TCP traffic). That means a randomized ACK flood for example will never hit the service. On stateless traffic (UDP/ICMP) it is different, here the BIG-IP simply forward packets.Fully Automatic used on a DoS profile discovers thehealth of the service, which works well if its TCP or DNS traffic. (This is done by evaluating the TCP behavior like Window Size, Retransmission, Congestions, or counting DNS requests vs. responses, ....). If the detection rate for that service is crossed (anomaly detected) and stress on the service is identified, the mitigation starts. Again, keep in mind for TCP this will only happen for “legitimate” traffic, because out of the state traffic will never reach the service. It is already protected by the BIG-IP being configured state-full. An example could be a HTTP flood.My recommendation:If you have the option to configureL7 BaDOS(Behavioral DOS running on DHD/A-WAF), then go with that one. It gives much better results and better mitigation options on HTTP floods then just L3/4 DOS, because it does also operate on L7. Please also do thennot configureTCP DOS for “Push Flood”, "bad ACK Flood" and “SYN Flood”, on that DOS profile, except you use a TMOS version newer than 14.1. For other TCP services like POP3, SMTP, LDAP … you can always go with the L3/4 DOS vectors. Since TMOS version 15.0 L7 BaDOS and L3/4 DOS vectors work in conjunction. Stateless traffic (UDP/ICMP) For UDP traffic I would like to separate UDP into DNS traffic and non-DNS-UDP traffic. DNS is a very good example where the health detection mechanism works great. This is done by measuring the ratio of requests vs. responses. For example, if BIG-IP sees 100k queries/sec going to the DNS server and the server sends back 100k answers/sec, the server shows it can handle the load. If it sees 200k queries/sec going to the server and the server sends back only 150k queries/sec, then it is a good indication of that the server cannot handle the load. In that case the BIG-IP would start to rate-limit, when the current rate is also above the detection rate (The rate BIG-IP expects based on the history rates). The 'Device UDP' vector gives you the option to exclude UDP traffic on specific ports from the UDP vector (packet counting). For example, when you exclude port 53 and UDP traffic hits that port, then it will not count into the UDP counter. In that case you would handle the DNS traffic with the DNS vectors. Figure 4 : UDP Port exclusion Here you see an example for port 53 (DNS), 5060 (SIP), 123 (NTP) Auto Detection / Multiplier Based Mitigation If the traffic is non-DNS-UDP traffic or ICMP traffic the stress measurement does not work very accurate, so I recommend going with the“multiplier option” on the DOS profile. Here BIG-IP does the baselining (calculating the detection rate) similar to “fully automatic” mode, except it will kick in (rate-limit) if it sees more than the defined multiplication for the specific vector, regardless of the CPU load. For example, when the calculated detection rate is 250k packets/sec and the multiplication is set to 500 (which means 5 times), then the mitigation rate would be 250k x 5 = 1.250.000 packets/sec. The multiplier feature gives the nice option to configure a threshold based on the multiplication of an expected rate. Figure 5 : Multiplier based mitigation On aDOS profilethethreshold configuration is per service (all packets targeted to the protected service), which actually means on that BIG-IP and NOT per TMM like on the Device level. Here the goal is to set how many packets are allowed to pass the BIG-IP and reach the service. The distribution of these thresholds to the TMMs is done in a dynamic way: Every TMM gets a percentage of the configured threshold, based on the EPS (Events Per Second, which is in this context Packets Per Second) for the specific vector the system has seen in the second before on this TMM. This mechanism protects against hash type of attacks. Physical and logical data path There is aphysical and logical traffic pathin BIG-IP. When a packet gets dropped via a DOS profile on a VS/PO, then BIG-IP will not see it on the device level counter anymore. If the threshold on the device level is reached, the packet gets dropped by the device level and will not get to the DOS profile. So, the physical path is device first and then comes the VS/PO, but the logical path is VS/PO first and then device. That means, when a packet arrives on the device level (physical first), that counter gets incremented. If the threshold is not reached, the packet goes to the VS/PO (DOS profile) level. If the threshold is reached here, then the packet gets dropped and the counter for the device DOS vector decremented. If the packet does not get dropped, then the packet counts into both counters (Device/VS). This means that the VS/PO thresholds should always set lower than device thresholds (remember on Device level you set the thresholds per TMM!). The device threshold should be the maximum the BIG-IP (TMM) can handle overall. It is important to note that mitigation in manual mode are “hard” numbers and do not take into account the stress on the service. In “Fully Automatic” mode on the VS/PO level, mitigation kicks in when the detection threshold is reached AND the stress on the service is too high. Now let’s assume a protected TCP service behind the BIG-IP gets attacked by a TCP attack like a randomized Push/ACK food. Because of the high packet rate hitting that vector on the attached DOS profile for that PO, it will go into 'detect mode'. But because it is a TCP flood and BIG-IP is configured to run state-full, the attack packets will never reach the service behind the BIG-IP and therefore the stress on that service will never go high in this case. The flood is then handled by the session state (CPU) of the BIG-IP until this gets too much under pressure and then the Device DOS will kick in “upfront” and mitigate the flood. If you use the “multiplication” option, then the mitigation kicks in when the packet detection rate + multiplication is reached. This can happen on the VS/PO level and/or on the Device level, because it is independent of stress. “Fully automatic” will not work properly with asymmetric traffic for VS/PO, because here BIG-IP cannot identify if the service is under stress due to the fact that BIG-IP will only see half of the traffic. -Either request or response depending on where it is initiated. For deployments with asymmetric traffic I recommend using the manual or “multiplier option”, on VS/PO configurations. The “Multiplier option” keeps it dynamic in regard to the calculated detection and mitigation rate based on the history. “Fully Automatic” works with Asymmetric traffic on Device DOS, because it measures the CPU (TMM) stress of the BIG-IP itself. Of course, when you run asymmetric traffic, BIG-IP can´t be state-full. I recommend to always start first with the Device DOS configuration to protect the BIG-IP itself against DOS floods and then to focus on the DOS protection for the services behind the BIG-IP. On Device DOS you will probably mostly use “Fully Manual” and “Fully Automatic” threshold modes. To protect services, you will mostly go with “Fully Manual” and “Fully Automatic” or “Auto Detection/ Multiplier based mitigation”. In one of the next articles I will describe in more detail when to use what and why. As mentioned before, additional to the static DoS vectors I recommend enabling Behavioral DOS, which is way smarter than the static DOS vectors and can filter out specifically only the attack traffic which tremendously decrease the chance of false positives. You can enable it like the static DoS vectors on Device- an on VS/PO level. But this topic will be covered in another article as well. With that said I would like to finish my first article. Let me know your feedback. Thank you, sVen MuellerExplanation of F5 DDoS threshold modes
Der Reader, In my article “Concept of Device DOS and DOS profile”, I recommended to use the “Fully Automatic” or “Multiplier” based configuration option for some DOS vectors. In this article I would like to explain how these threshold modes work and what is happening behind the scene. When you configure a DOS vector you have the option to choose between different threshold modes: “Fully Automatic”, “Auto Detection / Multiplier Based”, “Manual Detection / Auto Mitigation” and “Fully Manual”. Figure 1: Threshold Modes The two options I normally use on many vectors are “Fully Automatic” and “Auto Detection / Multiplier Based”. But what are these two options do for me? To manually set thresholds is for some vectors not an easy task. I mean who really knows how many PUSH/ACK packets/sec for example are usually hitting the device or a specific service? And when I have an idea about a value, should this be a static value? Or should I better take the maximum value I have seen so far? And how many packets per second should I put on top to make sure the system is not kicking in too early? When should I adjust it? Do I have increasing traffic? Fully Automatic In reality, the rate changes constantly and most likely during the day I will have more PUSH/ACK packets/sec then during the night. What happens when there is a campaign or an event like “Black Friday” and way more users are visiting the webpage then usually? During these high traffic events, my suggested thresholds might be no longer correct which could lead to “good” traffic getting dropped. All this should be taken into consideration when setting a threshold and it ends up being very difficult to do manually. It´s better to make the machine doing it for you and this is what “Fully Automatic” is about. Figure 2: Expected EPS As soon as you use this option, it leverages from the learning it has done since traffic is passing through the BIG-IP, or since you have enforced the relearning, which resets everything learned so far and starts from new. The system continuously calculates the expected rates for all the vectors based on the historic rates of traffic. It takes the information up to one year and calculates them with different weights in order to know which packets rate should be expected at that time and day for that specific vector in the specific context (Device, Virtual Server/Protected Object). The system then calculates a padding on top of this expected rate. This rate is called Detection Rate and is dependent on the “threshold sensitivity” you have configured: Low Sensitivity means 66% padding Medium Sensitivity means 40% padding High Sensitivity means 0% padding Figure 3: Detection EPS As soon as the current rate is above the detection value, the BIG-IP will show the message “Attack detected”, which actually means anomaly detected, because it sees more packets of that specific vector then expected + the padding (detection_rate). But DoS mitigation will not start at that point! Figure 4: Current EPS Keep in mind, when you run the BIG-IP in stateful mode it will drop 'out of state' packets anyway. This has nothing to do with DoS functionalities. But what happens when there is a serious flood and the BIG-IP CPU gets high because of the massive number of packets it has to deal with? This is when the second part of the “Fully Automatic” approach comes into the game. Again, depending on your threshold sensitivity the DOS mitigation starts as soon as a certain level of stress is detected on the CPU of the BIG-IP. Figure 5: Mitigation Threshold Low Sensitivity means 78,3% TMM load Medium Sensitivity means 68,3% TMM load High Sensitivity means 51,6% TMM load Note, that the mitigation is per TMM and therefore the stress and rate per TMM is relevant. When the traffic rate for that vector is above the detection rate and the CPU of the BIG-IP (Device DOS) is “too” busy, the mitigation kicks in and will rate limit on that specific vector. When a DOS vector is hardware supported, FPGAs drop the packets at the switch level of the BIG-IP. If that DOS vector is not hardware supported, then the packet is dropped at a very early stage of the life cycle of a packet inside a BIG-IP. The rate at which are packets dropped is dynamic (mitigation rate), depending on the incoming number of packets for that vector and the CPU (TMM) stress of the BIG-IP. This allows the stress of the CPU to go down as it has to deal with less packets. Once the incoming rate is again below the detection rate, the system declares the attack as ended. Note: When an attack is detected, the packet rate during that time will not go into the calculation of expected rates for the future. This ensures that the BIG-IP will not learn from attack traffic skewing the automatic thresholds. All traffic rates below the detection (or below the floor value, when configured) rate modify the expected rate for the future and the BIG-IP will adjust the detection rate automatically. For most of the vectors you can configure a floor and ceiling value. Floor means that as long as the traffic value is below that threshold, the mitigation for that vector will never kick in. Even when the CPU is at 100%. Ceiling means that mitigation always kicks in at that rate, even when the CPU is idle. With these two values the dynamic and automatic process is done between floor and ceiling. Mitigation only gets executed when the rate is above the rate of the Floor EPS and Detection EPS AND stress on the particular context is measured. Figure 6: Floor and Ceiling EPS What is the difference when you use “Fully Automatic” on Device level compared to VS/PO (DOS profile) level? Everything is the same, except that on VS or Protected Object (PO) level the relevant stress is NOT the BIG-IP device stress, it is the stress of the service you are protecting (web-, DNS, IMAP-server, network, ...). BIG-IP can measure the stress of the service by measuring TCP attributes like retransmission, window size, congestion, etc. This gives a good indication on how busy a service is. This works very well for request/response protocols like TCP, HTTP, DNS. I recommend using this, when the Protected Object is a single service and not a “wildcard” Protected Object covering for example a network or service range. When the Protected Object is a “wildcard” service and/or a UDP service (except DNS), I recommend using “Auto Detection / Multiplier Option”. It works in the same way as the “Fully Automatic” from the learning perspective, but the mitigation condition is not stress, it is the multiplication of the detection rate. For example, the detection rate for a specific vector is calculated to be 100k packets/sec. By default, the multiplication rate is “500”, which means 5x. Therefore, the mitigation rate is calculated to 500k packets/sec. If that particular vector has more than 500k packets/sec those packets would be dropped. The multiplication rate can also be individually configured. Like in the screenshot, where it is set to 8x (800). Figure 7: Auto Detection / Multiplier Based Mitigation The benefit of this mode is that the BIG-IP will automatically learn the baseline for that vector and will only start to mitigate based on a large spike. The mitigation rate is always a multiplication of the detection rate, which is 5x by default but is configurable. When should I use “Fully Manual”? When you want to rate-limit a specific vector to a certain number of packets/sec, then “Fully Manual” is the right choice. Very good examples for that type of vector are the “Bad Header” vector types. These type of packets will never get forwarded by the BIG-IP so dropping them by a DoS vector saves the CPU, which is beneficial under DoS conditions. In the screenshot below is a vector configured as “Fully Manual”. Next I’ll describe what each of the options means. Figure 8: Fully Manual Detection Threshold EPS configures the packet rate/sec (pps) when you will get a log messages (NO mitigation!). Detection Threshold % compares the current pps rate for that vector with the multiplication of the configured percentage (in this example 5 for 500%) with the 1-minute average rate. If the current rate is higher, then you will get a log message. Mitigation Threshold EPS rate limits to that configured value (mitigation). I recommend setting the threshold (Mitigation Threshold EPS) to something relatively low like ‘10’ or ‘100’ on ‘Bad Header’ type of vectors. You can also set it also to ‘0’, which means all packets hitting this vector will get dropped by the DoS function which usually is done in hardware (FPGA). With the ‘Detection Threshold EPS’ you set the rate at which you want to get a log messages for that vector. If you do it this way, then you get a warning message like this one to inform you about the logging behavior: Warning generated: DOS attack data (bad-tcp-flags-all-set): Since drop limit is less than detection limit, packets dropped below the detection limit rate will not be logged. Another use-case for “Fully Manual” is when you know the maximum number of these packets the service can handle. But here my recommendation is to still use “Fully Automatic” and set the maximum rate with the Ceiling threshold, because then the protected service will benefit from both threshold options. Important: Please keep in mind, when you set manual thresholds for Device DoS the thresholds are to protect each TMM. Therefore the value you set is per TMM! An exception to this is the Sweep and Flood vector where the threshold is per BIG-IP/service and not per TMM like on DoS profiles. When using manual thresholds for aDOS profileof a Protected Object thethreshold configuration is per service (all packets targeted to the protected service) NOT per TMM like on the Device level. Here the goal is to set how many packets are allowed to pass the BIG-IP and reach the service. The distribution of these thresholds to the TMMs is done in a dynamic way: Every TMM gets a percentage of the configured threshold, based on the EPS (Events Per Second, which is in this context Packets Per Second) for the specific vector the system has seen in the second before on this TMM. This mechanism protects against hash type of attacks. Ok, I hope this article gives you a better understanding on how ‘Fully Manual’, ‘Fully Automatic’ and ‘Auto Detection / Multiplier Based Mitigation’ works. They are important concepts to understand, especially when they work in conjunction with stress measurement. This means the BIG-IP will only kick in with the DoS mitigation when the protected object (BIG-IP or the service behind the BIG-IP) is under stress. -Why risk false positives, when not necessary? With my next article I will demonstrate you how Device DoS and the DoS profiles work together andhow the stateful concept cooperates with the DoS mitigation. I will show you some DoS commands to test it and also commands to get details from the BIG-IP via CLI. Thank you, sVen MuellerDemonstration of Device DoS and Per-Service DoS protection
Dear Reader, This article is intended to show what effect the different threshold modes have on the Device and Per-Service (VS/PO) context. I will be using practical examples to demonstrate those effects. You will get to review a couple of scripts which will help you to do DoS flood tests and “visualizing” results on the CLI. In my first article (https://devcentral.f5.com/s/articles/Concept-of-F5-Device-DoS-and-DoS-profiles), I talked about the concept of F5 Device DoS and Per-Service DoS protection (DoS profiles). I also covered the physical and logical data path, which explains the order of Device DoS and Per-Service DoS using the DoS profiles. In the second article (https://devcentral.f5.com/s/articles/Explanation-of-F5-DDoS-threshold-modes), I explained how the different threshold modes are working. In this third article, I would like to show you what it means when the different modes work together. But, before I start doing some tests to show the behavior, I would like to give you a quick introduction into the toolset I´m using for these tests. Toolset First of all, how to do some floods? Different tools that can be found on the Internet are available for use. Whatever tools you might prefer, just download the tool and run it against your Device Under Test (DUT). If you would like to use my script you can get it from GitHub: https://github.com/sv3n-mu3ll3r/DDoS-Scripts With this script - it uses hping - you can run different type of attacks. Simply start it with: $ ./attack_menu.sh <IP> <PORT> A menu of different attacks will be presented which you can launch against the used IP and port as a parameter. Figure 1:Attack Menu To see what L3/4 DoS events are currently ongoing on your BIG-IP, simply go to the DoS Overview page. Figure 2:DoS Overview page I personally prefer to use the CLI to get the details I´m interested in. This way I don´t need to switch between CLI to launch my attacks and GUI to see the results. For that reason, I created a script which shows me what I am most interested in. Figure 3:DoS stats via CLI You can download that script here: https://github.com/sv3n-mu3ll3r/F5_show_DoS_stats_scripts Simply run it with the “watch” command and the parameter “-c” to get a colored output (-c is only available starting with TMOS version 14.0): What is this script showing you? context_name: This is the context, either PO/VS or the Device in which the vector is running vector_name: This is the name of the DoS vector attack_detected:When it shows “1”, then an attack has been detected, which means the ‘stats_rate’ is above the ‘detection-rate'. stats_rate: Shows you the current incoming pps rate for that vector in this context drops_rate: Shows you the number dropped pps rate in software (not FPGA) for that vector in this context int_drops_rate: Shows you the number dropped pps rate in hardware (FPGA) for that vector in this context ba_stats_rate: Shows you the pps rate for bad actors ba_drops_rate: Shows you the pps rate of dropped ‘bad actors’ in HW and SW bd_stats_rate:Shows you the pps rate for attacked destination bd_drop_rate: Shows you the pps rate for dropped ‘attacked destination’ mitigation_curr: Shows the current mitigation rate (per tmm) for that vector in that context detection: Shows you the current detection rate (per tmm) for that vector in that context wl_count: Shows you the number of whitelist hits for that vector in that context hw_offload: When it shows ‘1’ it means that FPGAs are involved in the mitigation int_dropped_bytes_rate: Gives you the rate of in HW dropped bytes for that vector in that context dropped_bytes_rate: Gives you the rate of in SW dropped bytes for that vector in that context When a line is displayed in green, it means packets hitting that vector. However, no anomaly is detected or anything is mitigated (dropped) via DoS. If a line turns yellow, then an attack - anomaly – has been detected but no packets are dropped via DoS functionalities. When the color turns red, then the system is actually mitigating and dropping packets via DoS functionalities on that vector in that context. Start Testing Before we start doing some tests, let me provide you with a quick overview of my own lab setup. I´m using a DHD (DDoS Hybrid Defender) running on a i5800 box with TMOS version 15.1 My traffic generator sends around 5-6 Gbps legitimate (HTTP and DNS) traffic through the box which is connected in L2 mode (v-wire) to my network. On the “client” side, where my clean traffic generator is located, my attacking clients are located as well by use of my DoS script. On the “server” side, I run a web service and DNS service, which I´m going to attack. Ok, now let’s do some test so see the behavior of the box and double check that we really understand the DDoS mitigation concept of BIG-IP. Y-MAS flood against a protected server Let’s start with a simple Y-MAS (all TCP flags cleared) flood. You can only configure this vector on the device context and only in manual mode. Which is ok, because this TCP packet is not valid and would get drop by the operating system (TMOS) anyway. But, because I want this type of packet get dropped in hardware (FPGA) very early, when they hit the box, mostly without touching the CPU, I set the thresholds to ‘10’ on the Mitigation Threshold EPS and to ‘10’ on Detection Threshold EPS. That means as soon as a TMM sees more then 10 pps for that vector it will give me a log message and also rate-limit this type of packets per TMM to 10 packets per second. That means that everything below that threshold will get to the operating system (TMOS) and get dropped there. Figure 4:Bad TCP Flags vector As soon as I start the attack, which targets the web service (10.103.1.50, port 80) behind the DHD with randomized source IPs. $ /usr/sbin/hping3 --ymas -p 80 10.103.1.50 --flood --rand-source I do get a log messages in /var/log/ltm: Feb5 10:57:52 lon-i5800-1 err tmm3[15367]: 01010252:3: A Enforced Device DOS attack start was detected for vector Bad TCP flags (all cleared), Attack ID 546994598. And, my script shows me the details on that attack in real time (the line is in ‘red’, indicating we are mitigating): Currently 437569 pps are hitting the device. 382 pps are blocked by DDoS in SW (CPU) and 437187 are blocked in HW (FPGA). Figure 5:Mitigating Y-Flood Great, that was easy. :-) Now, let’s do another TCP flood against my web server. RST-flood against a protected server with Fully manual Threshold mode: For this test I´m using the “Fully Manual” mode, which configures the thresholds for the whole service we are protecting with the DoS profile, which is attached to my VS/PO. Figure 6:RST flood with manual configuration My Detection Threshold and my Mitigation Threshold EPS is set to ‘100’. That means as soon as we see more then 100 RST packets hitting the configured VS/PO on my BIG-IP for this web server, the system will start to rate-limit and send a log message. Figure 7:Mitigating RST flood on PO level Perfect. We see the vector in the context of my web server (/Common/PO_10.103.1.50_HTTP) is blocking (rate-limiting) as expected from the configuration. Please ignore the 'IP bad src' which is in detected mode. This is because 'hping' creates randomized IPs and not all of them are valid. RST-flood against a protected server with Fully Automatic Threshold mode: In this test I set the Threshold Mode for the RST vector on the DoS profile which is attached to my web server to ‘Fully Automatic’ and this is what you would most likely do in the real world as well. Figure 8:RST vector configuration with Fully Automatic But, what does this mean for the test now? I run the same flood against the same destination and my script shows me the anomaly on the VS/PO (and on the device context), but it does not mitigate! Why would this happen? Figure 9:RST flood with Fully Automatic configuration When we take a closer look at the screenshot we see that the ‘stats_rate’ shows 730969 pps. The detection rate shows 25. From where is this 25 coming? As we know, when ‘Fully Automatic’ is enabled then the system learns from history. In this case, the history was even lower than 25, but because I set the floor value to 100, the detection rate per TMM is 25 (floor_value / number of TMMs), which in my case is 100/4 = 25 So, we need to recognize, that the ‘stats_rate’ value represents all packets for that vector in that context and the detection value is showing the per TMM value. This value explains us why the system has detected an attack, but why is it not dropping via DoS functionalities? To understand this, we need to remember that the mitigation in ‘Fully Automatic’ mode will ONLY kick in if the incoming rate for that vector is above the detection rate (condition is here now true) AND the stress on the service is too high. But, because BIG-IP is configures as a stateful device, this randomized RST packets will never reach the web service, because they get all dropped latest by the state engine of the BIG-IP. Therefor the service will never have stress caused by this flood.This is one of the nice benefits of having a stateful DoS device. So, the vector on the web server context will not mitigate here, because the web server will not be stressed by this type of TCP attack. This does also explains the Server Stress visualization in the GUI, which didn´t change before and during the attack. Figure 10:DoS Overview in the GUI But, what happens if the attack gets stronger and stronger or the BIG-IP is too busy dealing with all this RST packets? This is when the Device DOS kicks in but only if you have configured it in ‘Fully Automatic’ mode as well. As soon as the BIG-IP receives more RST packets then usually (detection rate) AND the stress (CPU load) on the BIG-IP gets too high, it starts to mitigate on the device context. This is what you can see here: Figure 11:'massive' RST flood with Fully Automatic configuration The flood still goes against the same web server, but the mitigation is done on the device context, because the CPU utilization on the BIG-IP is too high. In the screenshot below you can see that the value for the mitigation (mitigation_curr) is set to 5000 on the device context, which is the same as the detection value. This value resultsfrom the 'floor' value as well. It is the smallest possible value, because the detection and mitigation rate will never be below the 'floor' value. The mitigation rate is calculated dynamically based on the stress of the BIG-IP. In this test I artificially increased the stress on the BIG-IP and therefor the mitigation rate was calculated to the lowest possible number, which is the same as the detection rate. I will provide an explanation of how I did that later. Figure 13:Device context configuration for RST Flood Because this is the device config, the value you enter in the GUI is per TMM and this is reflected on the script output as well. What does this mean for the false-positive rate? First of all, all RST packets not belonging to an existing flow will kicked out by the state engine. At this point we don´t have any false positives. If the attack increases and the CPU can´t handle the number of packets anymore, then the DOS protection on the device context kicks in. With the configuration we have done so far, it will do the rate-limiting on all RST packets hitting the BIG-IP. There is no differentiation anymore between good and bad RST, or if the RST has the destination of server 1 or server 2, and so on. This means that at this point we can face false positives with this configuration. Is this bad? Well, false-positives are always bad, but at this point you can say it´s better to have few false-positives then a service going down or, even more critical, when your DoS device goes down. What can you do to only have false positives on the destination which is under attack? You probably have recognized that you can also enable “Attacked Destination Detection” on the DoS vector, which makes sense on the device context and on a DoS profile which is used on protected object (VS), that covers a whole network. Figure 14:Device context configuration for RST Flood with 'Attacked Destination Detection' enabled If the attack now hits one or multiple IPs in that network, BIG-IP will identify them and will do the rate-limiting only on the destination(s) under attack. Which still means that you could face false positives, but they will be at least only on the IPs under attack. This is what we see here: Figure 15:Device context mitigation on attacked destination(s) The majority of the RST packet drops are done on the “bad destination” (bd_drops_rate), which is the IP under attack. The other option you can also set is “Bad Actor Detection”. When this is enabled the system identifies the source(s) which causes the load and will do the rate limiting for that IP address(es). This usually works very well for amplification attacks, where the attack packets coming from ‘specific’ hosts and are not randomized sources. Figure 16:Device context mitigation on attacked destination(s) and bad actor(s) Here you can see the majority of the mitigation is done on ‘bad actors’. This reduces the chance of false positives massively. Figure 17:Device context configuration for RST Flood with 'Attacked Destination Detection' and 'Bad Actor Detection' enabled You also have multiple additional options to deal with ‘attacked destination’ and ‘bad actors’, but this is something I will cover in another article. Artificial increase BIG-IPs CPU load Before I finish this article, I would like to give you an idea on how you could increase the CPU load of the BIG-IP for your own testing. Because, as we know, with “Fully Automatic” on the device context, the mitigation kicks only in if the packet rate is above the detection rate AND the CPU utilization on the BIG-IP is “too” high. This is sometimes difficult to archive in a lab because you may not be able to send enough packets to stress the CPUs of a HW BIG-IP. In this use-case I use a simple iRule, which I attach to the VS/PO that is under attack. Figure 18:Stress iRule When I start my attack, I send the traffic with a specific TTL for identification. This TTL is configured in my iRule in order to get a CPU intensive string compare function to work on every attack packet. Here is an example for 'hping': $ /usr/sbin/hping3 --rst -p 80 10.103.1.50 --ttl 5 --flood --rand-source This can easily drive the CPU very high and the DDoS rate-limiting kicks in very aggressive. Summary I hope that this article provides you with an even better understanding on what effect the different threshold modes have on the attack mitigation. Of course, keep in mind this are just the ‘static DoS’ vectors. In later articles I will explain also the 'Behavioral DoS' and the Cookie based mitigation, which helps to massively reduce the chance of a false positives. But, also keep in mind, the DoS vectors start to act in microseconds and are very effective for protecting. Thank you, sVen MuellerUnderstanding L3/4 and DNS Behavioral DoS Mitigation (BDoS)
Dear Reader, In this article I am writing about another mitigation technology which runs in parallel to the other mitigation methods you can use on a BIG-IP: Dynamic Signatures. These Dynamic Signatures are created by the Behavioral DoS (BDoS) engine. This is a very smart and powerful mechanism to only drop attack traffic without touching legitimate traffic. Introduction We know that the CPU can do smart decisions and is usually able to differentiate between good and bad traffic. Frequently, this is simply accomplished by using the flow table. But, the CPU is limited in performance and this is when the FPGAs (Field Programmable Gate Arrays) come into play. This special purpose chips can handle massive numbers of packets but they are “stupid”. Therefore, the CPU needs to program them with detailed information about which packets should get dropped. The static DoS vectors are relatively simple because they are based on limited attack information such as the protocol, flags, source IPs, destination IPs, and others. But they kick in very fast! This means they (vectors) can program the FPGAs within microseconds and protect the BIG-IP or service. The downside is that the FPGA cannot differentiate if for example a PUSH/ACK packet is valid or one of the attackers because there is not enough information provided. This is where Behavioral DoS can help. How does BDoS work? BDoS is about collecting and working with large amounts of statistical and traffic information. It collects the details that can be obtained from L3/4 and DNS. All this information is called “predicates” and the value of a predicate is called a “bin”. For example, the TTL of a packet is the predicate and it can have a value (bin) from 0-255. By collecting this information all the time, the BDoS engine knows how packets usually look like. It knows the predicates and the bin values together with its rates. This provides a pretty good view on how traffic usually looks like and what the rates are for all this information. When an attack happens, the BDoS engine will be able to identify the ‘abnormal’ bins of the predicates and will create a dynamic signature with all these suspicious predicates. This will give a more precise description of the attack traffic then just saying rate limit “TCP PSH/ACK” packets. Figure 1: Statistical side model A simple example This example is just to explain the principle of BDoS and it uses a very limited set of predicates. Imagine, based on historical information, the system knows that right at this time we usually see 50k PSH/ACK packets on the BIG-IP and all these packets have a TTL between 50 and 60. Further, the load on the BIG-IP is low and the numbers or packets as well as the associated TTL is within the expected norm. Suddenly we see on top of this “normal” traffic 10Mpps of PSH/ACK packets with TTL of 30 and the load of the BIG-IP gets too high. BDoS needs to create a signature describing just the attack traffic to program it into the FPGA and lower the rate of packets going into the CPU. Which PSH/ACK packets should be eliminated? It’s easy: the ones which are not fitting into the “normal”. Which in this case are the PSH/ACK packets with a TTL of 30. Again, this is a very simplified and limited example because it only looks at the predicates protocol (TCP), flags (PSH/ACK), and TTL. In reality, it would of course require way more predicates to create a more specific signature. Example of dynamic signatures Figure 2: Example of a DNS dynamic signature Figure 3: Example of a L3 dynamic signature Behavioral DoS can also be used on HTTP and TLS traffic. But this is done via another Behavioral DoS Engine (BaDOS) which comes from Advanced WAF (A-WAF) and is also included in DDoS Hybrid Defender (DHD). I will discuss BaDOS in another article. How to configure BDoS BDoS can be used on the Device level of BIG-IP to protect the resources of the BIG-IP itself and also on a per Service level via a DoS profile attached to a PO/VS. You can enable BDoS by configuring it on the Network and DNS layer. You simply need to click on the small pencil right of the relevant layer. Figure 4: Network family layers For BDoS you can choose between different Mitigation Modes: “Stress Based Mitigation” and “Manual Multiplier”. These modes are discussed in my article Explanation of F5 DDoS threshold modes. Figure 5: Network properties If you want to run BDoS in “detect only mode”, then you need to set the Mitigation Sensitivity to “None”. This is different than on the static vectors, where you set the state of the vector to “Detect Only”. The same is true for the DoS profile where you can also enable BDoS for the Network and DNS layer and BaDOS for HTTP(S). Here is a useful hint! When you are under attack and the BIG-IP has generate a dynamic attack signature and mitigates based on stress, then the mitigation threshold gets continuously recalculated based on the health of the protected context (device or service). It will always let “some” traffic go through based on the sensitivity level you have configured. But, you have the ability to change the threshold mode from “Fully Automatic” to “Fully Manual” on the specific signatures and set the Mitigation threshold for example to “0”, which will then block the whole attack traffic via the BDOS signature. Figure 6: Dynamic signature with manual threshold Custom Signatures If you are aware of specific attack patterns or you simply want to have HW supported FW rules, then you can create your own signatures. Figure 7: Add manual a signature Under DoS Configuration > Signatures you can choose “Add Signature” in the “Persistent” tab. Figure 8: add predicates to a manual signature A simple example could be an NTP flood signatures, which kicks in if too many NTP packets hit the box. Figure 9: manual NTP predicates After you have created your custom signature you can configure the thresholds and mitigation state in the DoS profiles or on the Device DoS configuration. Figure 10: manual threshold configuration of a manual NTP signature To exclusively handle NTP traffic via this signature and not by the UDP vector, I recommend excluding port 123 from the UDP vector, like you should do with DNS traffic. Figure 12: Exclude port 123 from UDP vector Multilayer DDoS protection Beginning with TMOS version 15, BDoS runs in the Hud-chain before the static DoS vectors. That means that when there is an attack, the static DoS vectors will kick in immediately and protect the service/network behind the BIG-IP and also the BIG-IP itself. Within the next seconds BDoS analyses the traffic and will generate a specific dynamic signature, which describes the attack traffic and will then rate-limit it. Below you can see that F5 DDoS uses a multilayer DDoS approach. When we just look into L3/4 and what I have so far discussed in my articles, then we have covered the stateful engine with its PVA accelerated connection handling, protected by static DoS vectors which kick in in microseconds and then taken over by the granular behavioral DDoS (BDOS) within seconds. Figure 12: Multi-layer DDoS mitigation architecture From the figure you can see that there are even more protection mechanisms, which I will explain in my next articles. Thank you, Sven MuellerCookie-based DDoS protection
Dear Reader, DDoS protection starts with being able to identify that you are under attack. Next, you will need to be able to differentiate between good and bad traffic. This will be particularly challenging if the attack (bad) traffic looks like your good traffic. So, how can you ideally filter out only the attack traffic without affecting the good traffic? Well, and finally, you need to have the capacity and performance to deal with the high number of flood packets. In this article, I want to explain to you another layer of F5’s network DDoS protection, which perfectly meets these requirements: cookie-based DDoS protection. Figure 1: Multi-layer DDoS mitigation architecture Cookie-based protection No, I´m not talking about HTTP, we are still within the Network protection. Even on the network layer, BIG-IP can deal with cookies and can use it in a very effective way to mitigate without false/positives. The overall idea is to store connection- related information, encrypted within a packet, which is being forwarded. The response packet will have this cookie included as well and can be used to identify that the packet belongs to an existing connection. Therefore the packet is valid and not just a randomized packet to drive up CPU utilization. When the packet is “wrong” it can be dropped at an early stage by a dedicated chip (FPGA) and doesn´t consume CPU resources. But let´s start talking about attack vectors. SYN flood SYN flood is a very old, but still very effective way for DDoS attacks. The idea is to overwhelm stateful devices with too many half-open connections, which mostly results in (too) high CPU usage and/or fully utilized memory consumption. Attackers just send SYN packets towards the target (server, firewall, etc.). The first packet during a 3-way handshake of a connection setup will let the target immediately allocate memory for the TCP socket data structure holding the tuple, sequence number, and “state” of the session. The target then replies to the client (attacker) with an “SYN-ACK” packet. Under normal circumstances, packet loss or slow communication can cause the SYN-ACK packet to be delayed or lost altogether. The server must decide how long to hold the socket in a half-open state before transmitting a reset and releasing the memory for other tasks. This means the higher this timer is, the faster memory can get filled up under SYN flood conditions. But, on the other hand, the lower the value is, the higher the risk that you drop a valid connection request (for example, because of latency). SYN flood attackers usually spoof the source IP, so the SYN-ACK does not hit the initial source (attacker). This way the SYN flood can also result in an SYN-ACK flood done by the target (server, firewall, etc.), which will be especially painful because the spoof involves only one or a small number of IP addresses. This can be easily identified on the mitigation side and can get blocked or rate-limited. Keep in mind, however, that this could also be a strategy of the attacker… Some interesting side effects are also, that depending on the TCP stack, servers send a RST packet when they release a timed-out connection. Some TCP stacks do also send a RST packet when they receive an SYN-ACK packet for which they don´t have a connection entry. Therefore, often SYN floods do also end in SYN-ACK and RST floods as well. Figure 2: SYN flood But, how can you protect against this type of attack? SYN rate-limiting via static DoS vectors is one option, but it can potentially lead to false/positives. This is true in particular when you cannot limit the mitigation to “attacked destination” and/or “bad actors” (source IPs causing the stress). For further information, please read my article “Increasing accuracy using Bad Actor and Attacked Destination detection“. Behavioral DoS (BDoS) is a very effective way but usually takes a few seconds to kick in. Another strong option is to use F5’s SYN-Cookie mitigation. SYN-Cookie mitigation SYN-Cookie mitigation is an effective way to resist SYN floods. It gets activated when the threshold of the configured number of half-open connections is reached. On AFM (Advanced Firewall Manager) or DHD (DDoS Hybrid Defender), the threshold can be configured via the “tcp-half-open” vector on the device level and also on the per VS/PO level via a DoS profile attached to it. Figure 3: TCP Half Open vector configuration Mitigation threshold means, activate the SYN-cookie process after (in this example) 10.000 half-open connections. Keep in mind, on the Device level you configure the threshold per TMM and within a DoS profile it is per PO/VS to which it is attached. SYN-Cookie mitigation gets deployed by choosing initial TCP sequence numbers in the SYN-ACK response according to the properties of the originating TCP SYN. It encrypts connection information within the sequence number of the responding SYN-ACK packet. When the SYN was sent by a valid client, BIG-IP will receive a well-formed ACK response from the client including the cookie stored within the "ack" field and incremented by one. BIG-IP can now decrypt and reconstruct the connection details and establish the connection by performing the necessary SYN establishment handshakes with the server on behalf of the client. The beauty of this technique is that it smartly protects the memory, but the process of “encrypt” and “decrypt” the cookies consumes CPU, which can be potentially dangerous. To enhance the capability of the BIG-IP systems, SYN cookie functionality is extended to the FPGA, which is radically more effective than software mitigation alone. In this way, BIG-IP can differentiate between “valid” and “bad” SYN packets without negatively impacting memory or CPU performance! Here is a useful hint! As mentioned above, you can configure the activation of the SYN cookie process by defining a threshold of the number of half-open connections. This can be done via the “tcp-half-open” vector on the Device level and also via the DoS profile for a VS/PO. This is my recommended approach. Keep the DoS config in one place and therefore disable the second option (via traditional TMOS configuration). This can be done under System/Configuration: Local Traffic: General, by setting the “Default Per Virtual Server SYN Check Threshold” and “Global SYN Check Threshold” to “0”. Also, disable “Hardware VLAN SYN Cookie Protection”. Figure 4: LTM SYN-Cookie configuration Now it’s all controlled by the “tcp-half-open” vector, which makes it less confusing. Please also keep in mind, SYN Cookie does only work in FPGA when “Auto Last Hop” is enabled! What do the stats of "tcp-half open" mean in the context of SYN-Cookie protection? When we take a look into the stats via the script I provided in my article “Demonstration of Device DoS and Per-Service DoS protection” during an SYN flood attack, we see a rate for int_drops. This only appears when the mitigation threshold (number of half-open connections) is reached. Figure 5: TCP Half Open vector shown in dos_stats output The “stats_rate” shows the number of half-open connections in the last second and the “drop_rate” or “int_drop_rate” = (the number of cookies sent) – (number of valid ACKs received). SYN-ACK flood Let’s talk about another attack vector: SYN-ACK flood. Again, this attack can be done in different ways. The attackers simply send a high rate of randomized SYN-ACK packets towards the victim. Or it is a result of an SYN flood with responding SYN-ACK packets towards a spoofed source IP of the SYN packet. Anyhow, every high packet rate can cause massive damage to the performance of Firewalls, Servers, or other devices and this is what DoS attackers usually try to archive with this type of attack. And again, the question is: how to differentiate between “good” and “bad” SYN/ACK packets? Asking the connection table would increase the CPU and is therefore not the best choice. So, ideally, it is done upfront (CPU) within the hardware chip (FPGA) to be high performant. SYN-ACK flood mitigation By enabling “Only Count Suspicious Events” on the “TCP SYN ACK Flood” vector on the Device level you do enable a cookie mechanism that allows the BIG-IP to differentiate if an SYN-ACK packet belongs to an SYN which has passed the BIG-IP before. Figure 6: TCP SYN ACK Flood vector configuration This mechanism is processed within the FPGA and does therefore not consume CPU. This enables the BIG-IP to be extremely resistant against this type of flood. Please keep in mind, this cookie function was added to the product with version 15.1. Before 15.1, you could also enable “Only Count Suspicious Events”, but it puts the vector into software mode. That means it can differentiate between “good” and “bad” SYN-ACK packets, but the mitigation is done within the CPU. How does SYN-ACK cookie mitigation work When the mechanism is enabled and an SYN packet received on a BIG-IP and is being forwarded, then the initial SNI gets replaced by an own SNI which includes properties of the originating TCP SYN in an encrypted format. When BIG-IP now receives SYN-ACK packets, it expects the initial SNI (cookie) with the ACK field to be incremented by 1. BIG-IP can now decrement and decrypt that value to check if the “cookie properties” fit the properties of the initial SYN packet. If that is the case, then the "ack" field gets replaced with the incremented SNI of the initial SYN packet and it will then be returned to the original client. If the cookie check fails, then it will be counted as a bad SYN-ACK and when the mitigation threshold on the vector is reached the rate-limiting for bad SYN-ACK packets gets activates in the hardware (FPGA). Figure 7: TCP SYN ACK cookie mitigation ACK flood and mitigation Let’s get to a third attack vector: ACK flood. This one is also widely used by attackers and can be very damaging. We already know it’s about the differentiation between valid and bad ACK packets plus being able to use the hardware (FPGA) for mitigation. Here again, the overall idea is to encrypt connection properties within a cookie and store them in the TCP packet. Valid response packets do have this cookie included and can be validated. When the number of invalid ACK packets reaches the mitigation threshold, then the rate-limiting will start. This cookie protection mechanism can be enabled per VLAN on the “bad_ack” vector on the Device configuration. Figure 8: TCP BADACK flood vector (cookie) configuration Simply move the VLANs you want to have this feature being enabled to the “Selected” area. Conclusion TCP floods can be easily used to bring down even strong firewalls, servers, or other security devices. Generally, it is not the volume, in terms of bandwidth, that is at issue here. It is the high packet rate, which in particular puts stateful devices under pressure if they don´t have dedicated hardware (FPGA) supported defense mechanism. With the cookie-based detection and mitigation defense technic, F5 delivers another strong layer within the layered network DDoS architecture. Even the whole cookie process is done within FPGAs, these chips can differentiate between valid and bad TCP packets without checking the connection state table. SYN-ACK and ACK cookies, which got introduced into the product with version 15.1, are being processed even before the Behavioral DDoS layer. SYN cookies are done after the static vectors. One of my next articles will provide an overview of best practices and an explanation on how to integrate DHD (DDoS Hybrid Defender) or AFM (Advanced Firewall Manager) into the Network for DDoS protection. But, next, I will explain the latest part of the Network protection layers, which is the first layer from the data path order: IP-Intelligence/ IP shunning and also RTBH (Remote triggered Black Hole Routing) and Flowspec. I hope this article was useful to you and keep in touch! Thank you, Sven MuellerIP-Intelligence and IP-Shunning
Dear Reader, With this article we have arrived at the “last” layer of protection, which is actually the first one from the hud-chain point of view: Blocking on the IP layer. The idea is rather simple: When you know that an IP address (or multiple IP addresses) is doing bad things, you block them as early as possible at the speed of line. IP-Intelligence Policy You can create IP-Intelligence (IPI) policies, which describe what should happen (Drop or Allow) with IPs (source or destination) belonging to a specific category. BIG-IP already has a bunch of pre-defined categories, but you can also add your own. Figure 1: Example on an IPI-Policy Important: A great benefit of using this, is that the “Drop” action is done on the FPGA level, which means in hardware. Basically, as soon as a packet hits the BIG-IP. But this is only possible when you set the logging to “No” (no logging, only dropping) or “Limited”. “Limited” means that some packets get leaked into software (CPU) for dropping them there and then they can be logged here as well. By default, every 256th packet goes into software when logging is set to “limited”. This is controlled by the sys db variable “dos.blleaklimit”. You can modify it to values like 1, 3, 7, 15, 31, 63, 127, 255, 511, 1023, and so on. The default value is 255. When you set the logging to “Yes”, then all packets will get logged, because the action (Drop) is done always in software. My recommendation is to use “Limited” logging to enable FPGA support for this feature. This is also what we call “shunning” when we drop on FPGA level. You can create multiple policies and assign them to different contexts: VS/PO or Route Domains to have individual configurations for your protected objects (services, networks, etc.) and you can also assign an IPI policy to the global level of BIG-IP, which means this will be executed first and for the whole box. Even when you want to block via IPI only on VS/PO level, my recommendation is to have a global IPI policy attached. It can be empty. Figure 2: Global Policy assignment Figure 3: Per VS/PO Policy assignment Figure 4: Per Route Domain Policy assignment Order of Mitigation Mitigation using the hardware (FPGA) is always done first. Next, traffic goes into software and can be dropped here as well. For IPI the following order takes place. Note that IPI mitigation done on Route Domain level is always in software. Figure 5: Mitigation Order How to get information about “bad” IPs? There are multiple options available. One is to use an external service (3 rd party), which does the categorization of IPs based on their reputation. If you have the IP Intelligence feature licensed, the information is being updated every 5 minutes. Another one is to use BIG-IP to automatically identify “bad actors” via DoS vectors. In my article Increasing accuracy using Bad Actor and Attacked Destination detection, I described how that works and for what it is useful for. But when BIG-IP identifies “bad actors” and this source IP(s) are behaving aggressive for a configurable amount of time, then it may make sense to drop everything coming from this IP(s) for a specific time interval. Figure 6: Bad Actor Detection In this example, the IP(s) which are identified as bad actors and behaving “bad” for 30 seconds get added to the category “denial_of_service” for 600 seconds. Once the “duration time” is over, they get released from the category and the process starts from the beginning, if they are still “bad”. Drop Attacked Destination Another use case to carefully consider is to drop based on attacked destination. Here you can drop all traffic going to a specific destination which is under attack. True, you stop the attack hitting the target, but you also drop all legitimate traffic going there, which would be a great success for the attacker. But, of course, if for whatever reason the attack would potential bring the BIG-IP to its knees this is an option you can choose. You would loose the destination that is under brutal attack but at least you safe all other services on the BIG-IP from being impacted. Figure 7: Attacked Destination Detection Use your up-stream router Think of the following scenario: An attack is about to saturate your Internet-pipe. You mitigate the bad actors on BIG-IP, but your pipe is still full. Why not “signal” the information of bad actors to your upstream router and drop the attackers already there, even before your pipe gets saturated? This is what you can do with using Remote Triggered Black Hole routing (RTBH) or Flowspec. By enabling “Allow External Advertisement” within your “Bad Actor Detection” or “Attacked Destination Detection” you can enable that functionality. Figure 8: External Advertisement I will cover the topic on how to use the Blacklist Publisher to “signal” bad actors including an action (Drop, Rate-limit, DSCP) to an upstream router via RTBH or Flowspec in a dedicated article. I will also explain in the same article how to use the “Scrubbing profile” to execute an action (redirect, rate-limit, DSCP, drop) on networks, PO/VS when these objects reach a certain bandwidth utilization. More options to add bad actors into categories Returning to how you can add IP(s) into categories. In addition to the already discussed options of using F5 subscription service and DoS vectors you can also add manually IPs. You may have your own sources and want to add this information into the categories. Here you have multiple options again. GUI On the Blacklist Categories page, you can manually add/delete IPs for a selected category. Figure 9: Add IP/Geo/FQDN to categories Feed List You can create your own Feed List using HTTP(s) or FTP to regularly download a file and update your categories. Figure 10: External Feed List Don´t forget to then enable your Feed List within your IPI-Policy! Figure 11: Enable Feed Lists TMSH Using tmsh you can also add IPs to categories. Here is an example: $ tmsh run security ip-intelligence category name denial_of_service ip-ttl add { 123.123.123.123,300 } Rest-API Another way is to use Rest-API. Here is an example: $ curl -sk -u sVen:mySecrectPW -H "Content-Type: application/json" -X POST -d '{"command":"run","utilCmdArgs":"name denial_of_service ip-ttl add {123.123.123.123,300}"}' https://mgmnt-address-big-ip/mgmt/tm/security/ip-intelligence/category/ | python -m json.tool Whitelist IPs for IPI/shunning If you want to whitelist IP(s) from not being shunned on IPI level, you need to add this IP(s) to the “whitelist” category. Please we aware that this only works when you use an external feed to add the IP(s). Figure 12: Feed List Properties More useful CLI commands To get an idea on hits per category, you can use a “tmctl” command: # tmctl -w 180 ip_intelligence_stat It gives you an overview of hits per context, category and drop list type. Figure 13: Stats for categories Here you can also see that you can not only drop based on IP. You can also use GEO information and FQDN. If you want to verify the status of an IP, you can use this command: # tmsh show security ip-intelligence info address <ip-address> Figure 13: Example of status of a GEOIP Conclusion IPI/shunning is a very effective and hardware driven way of blocking traffic based on IP information. Especially on amplification and reflection attacks it is a very useful because you are mostly able to identify IP addresses causing the high load. Often operators do also block traffic from countries on which they are sure they do not communicate with for specific services. After finishing the explanation of this last (first) mitigation layer, you hopefully have a better understanding on how F5 L3/4/DNS DDoS mitigation works. Hardware (FPGA) support is an essential element of successful blocking high packet rates. These high-performance chips get dynamically programmed via the “smart” component, the software which runs on CPU, and which also needs to be protected. Therefore, you can configure the DoS mitigation mechanism for protecting your services (server, network, …) behind the BIG-IP, but also BIG-IP itself! Figure 14: Multi-layer DDoS mitigation architecture F5 also provides the ability to protect with hardware (FPGA) support in virtualized environments. Since more and more data center go virtual, the requirement of running DDoS mitigation in virtualized environments has increased. But, because FPGA support is needed for this type of DDoS mitigation, since CPU alone is usually not powerful enough, it can present a problem. Together with Intel F5 provides a strong solution. This solution uses SmartNICs (network card with FPGA on board) within your Hypervisor. DHD (DDoS Hybrid Defender) and AFM (Advanced Firewall Manager) can program these FPGAs in basically the same way as it can be done on dedicated Hardware appliances. This again gives you the power needed for DDoS mitigation, plus the flexibility of virtualized environments. If you are interested in learning more about this, I would like to recommend the article of my colleague Ryan Howard: https://devcentral.f5.com/s/articles/Mitigating-40Gbps-DDoS-Attacks-with-the-new-BIG-IP-VE-for-SmartNICs-Solution My next article will focus on logging and reporting DDoS events. Together, my colleague Mohamed Shaath and I created a DDoS Dashboard including DDoS statistics visibility on all vectors based on an ELK stack. We will share in that article how to install and use it. Thank you, Sven Mueller
2. SYN Cookie: Operation
Introduction As concluded in the last article, in order to avoid allocating space for TCB, the attacked device needs to reject TCP SYN packets sent by clients. In this article I will explain how a system can do this without causing service disruption, and more specifically I will explain how this work in BIG-IP. Implementation Since attacked device need to alter default TCP 3WHS behaviour the best option is implementing SYN Cookie countermeasure in an intermediate device, so you offload your servers from this task. In this way if connection is not legitimate then it is just never forwarded to the backend server and therefore it will not waste any kind of extra resourcess. Since BIG-IP is already in charge of handling application traffic directed towards servers it is the best place to implement SYN Cookie. What BIG-IP does is adding an extra layer, which we can call SYN Cookie Agent, that basically implement a stateless buffer between client and BIG-IP TCP stack. This agent is in charge of handling client’s TCP SYN packets by modifying slightly the standard behaviour of TCP 3WHS, this modification comes in two flavours depending on what type of routing role BIG-IP is playing between client and server. Symmetric routing This is the typical situation. In this environment BIG-IP is sited in the middle of the TCP conversation and all the traffic from/to server pass through it. The SYN Cookie operation in this case is briefly described below: Client sends a TCP SYN packet to BIG-IP. BIG-IP uses a stateless buffer for answering each client SYN with a SYN/ACK. BIG-IP generates the 32 bits sequence number which will be included in SYN/ACK packet sent to the client. BIG-IPencodes essential and mandatory information of the connection in 24 bits. BIG-IP hashes the previous encoded information. BIG-IP also encodes other values like MSS in the remaining bits. BIG-IP generates the SYN/ACK response packet and includes the hash and other encoded information as sequence number for the packet. This is the so calledSYN Cookie. BIG-IP sends SYN/ACK to client. BIG-IP discards the SYN from the stateless buffer, in other words, BIG-IP removes all information related to this TCP connection. At this point no memory has been allocated for TCP connection (TCB). Client sends correct ACK to BIG-IP acknowledging BIG-IP’s sequence number. BIG-IP validates ACK. BIG-IP subtracts one to this ACK. BIG-IP runs the hash function using connection information as input (see point 3-b above). Then it compares it with the hash provided in the ACK, if they match then it means client sent a correct SYN Cookie response and that client is legitimate. BIG-IP uses connection information in the ACK TCP/IP headers to create an entry in connflow. At this time is when BIG-IP builds TCB entry, so it's the first time BIG-IP uses memory to save connection information. BIG-IP increase related SYN Cookie stats. BIG-IP starts and complete a TCP 3WHS with backend server on behalf of the client since we are sure that client is legitimate. Now traffic for the connection is proxied by BIG-IP as usual attending to configured L4 profile in the virtual server. *If ACK packet received by BIG-IP is a spurious ACK then BIG-IP will discard it and no entry will be created in connection table. Since attackers will never send a correct response to a SYN/ACK then you can be sure that TCB entries are never created for them. Only legitimate clients will use BIG-IP resources as shown in diagram: Fig4. TCP SYN Flood attack with SYN Cookie countermeasure Asymmetric routing In an asymmetric environment, also called nPath or DSR, you face a different problem because Big-IP cannot establish a direct TCP 3WHS with the server (point 11 in last section). As you can see in below diagram SYN/ACK packet sent from server to client would not traverse Big-IP, so method used for symmetric routing cannot be used in this case. You need another way to trust in clients and discard them as attackers, so clients then can complete TCP 3WHS directly with the server. Fig5. TCP state diagram section for asymmetric routing + FastL4 In order to circumvent this problem BIG-IP takes the advantage of the fact that applications will try to start a second TCP connection if a first TCP 3WHS is RST by the server. What BIG-IP does in asymmetric routing environments is completing the TCP three way handshake with client, issuing SYN Cookie as I described for symmetric routing, and once BIG-IP confirms that client is trustworthy it will add its IP to SYN Cookie Whitelist, it will send a RST to the client and it will close the TCP connection. At this point client will try to start a new connection and this time BIG-IP will let the client talk directly to the server as it would do under normal circumstances (see diagram above). The process is briefly described below: Client sends a TCP SYN packet to BIG-IP. BIG-IP generates the sequence number as explained for symmetric routing. BIG-IP generates the SYN/ACK response packet and includes the calculated sequence number for the packet. BIG-IP sends SYN/ACK to client and then remove all information related to this TCP connection. At this point no memory has been allocated for TCP connection (TCB). Client sends correct ACK to BIG-IP acknowledging sequence number. BIG-IP validates ACK. BIG-IP substract one to this ACK so it can decode needed information and check hash. BIG-IP increase related SYN Cookie stats. BIG-IP sends a RST to client and discards TCP connection. Big-IP adds client’s IP to Whitelist. Clients starts a new TCP connection. BIG-IP lets client to start this new TCP connection directly to the server since it knows that server is legitimate. *If ACK packet received by BIG-IP is a spurious ACK then BIG-IP will discard it and no entry will be created in connection table. Fig6. TCP 3WHS flow in asymmetric routing DSR whitelist has some important characteristics you need to take into account: By default IP is added 30 seconds to the whitelist (DB Key tm.flowstate.timeout). Client’s IP is added to whitelist for 30 seconds but only if there is no traffic from this client, if BIG-IP have already seen ACK from this client then its IP is removed from whitelist since connection has been already established between client and server. Whitelist is common to all virtual servers, so if a client is whitelisted it will be for all applications. Whitelist it is not mirrored. Whitelist it is not shared among blades. SYN Cookie Challenge When a TCP connection is initiated the TCP SYN packet sent by the client specify certain values that define the connection, some of these values are mandatory, like source and destination IP and port, otherwise you would not be able to identify the correct connection when packets arrive. Some other values are optional and they are used to improve performance, like TCP options or WAN optimizations. Under normal circumstances, upon TCP SYN reception the system creates a TCB entry where all this information is saved. BIG-IP will use this information to set up the TCP connection with the backend server. The problem we face is that when SYN Cookie is in play device does not create the TCB entry, only a limited piece of connection information is collected, so the information that BIG-IP has about the TCP connection is limited. Remember that SYN Cookie is not handled by TCP stack but by the stateless SYN Cookie Agent, so we cannot save the connection details. What BIG-IP does instead is encoding key information in 24 bits, this left only some bits for the rest of data. While there is no room for values like Window Scale information, however we have space for other values, for example 3 bits reserved for MSS value. This limits MSS possible values to 8, the most commonly used values, and the one chosen will the nearest to the original MSS value. However, note that MSS limitation has a workaround through configuration in FastL4 profile that it can help in some cases. The option SYN Cookie MSS in this profile specifies a value that overrides the SYN cookie maximum segment size (MSS) value in the SYN-ACK packet that is returned to the client. Valid values are 0, and values from 256 through 9162. The default is 0, which means no override. You might use this option if backend servers use a different MSS value for SYN cookies than the BIG-IP system does. If this is not enough for some customers, BIG-IP overcomes this problem by using TCP timestamp space to save extra information about TCP connection, in other words we create a second extra SYN Cookie. This space is used to record the client and server side Window Scale values, or the SACK info which are then made available to the TCP stack via this cookie if the connection is accepted. Note that TCP connection will only be accepted if both SYN Cookies (standard and Timestamp) are correct. So all this means that if the client starts a connection with TCP TS set then Big-IP will have more space to encode information and hence the performance will be improved when under TCP SYN flood attack. NOTE: SYN Cookie Timestamp extension (software or hardware) only work for standard virtual servers currently. FastL4 virtual servers only supports MMS option in SYN Cookie mode. Conclusion Now you know how SYN Cookie works under the hoods. In next article I will describe when SYN Cookie is activated and I will give specific details of BIG-IP SYN Cookie operation. Note that limitations related to SYN Cookie commented in this article are not caused specifically by BIG-IP SYN Cookie implementation but by SYN Cookie standard itself. In fact, F5 Networks implements some improvements on SYN Cookie to get a better performance and provide some extra features.
Securing APIs with BigIP
Introduction API servers respond to requests using the HTTP protocol, much like Web Servers. Therefore, API servers are susceptible to HTTP attacks in ways similar to Web Servers. Previous articles covered how to publish an API using the NGINX platform as an API management gateway. These APIs are still exposed to web attacks and defensive mechanisms are needed to defend the API against web attacks, denial of service, and Bots.The diagram below shows all the layers needed to deliver and defend APIs. BIGIP provides the protection and NGINX Plus provides API management. Picture 1. This article covers Advanced Web Application Firewall (AWAF) to protect against HTTP vulnerabilities Unified Bot Defense to protect against bots Behavioral Anomaly DoS Defense to prevent DoS attacks As shown in Picture 1, above BIG-IP goes in front of the API management gateway as an additional security gateway. The beauty of this approach is that BIG-IP can be initially deployed on the side while the API is being delivered to users through the NGINX Plus gateway directly. Once BIG-IP is configured to forward good requests to NGINX Plus and security policies are in place, BIG-IP can be brought into the traffic flow by simply changing the DNS records for "prod.httpbin.internet.lab" to point to BIG-IP instead of NGINX Plus. From this point on all calls will automatically arrive at BIG-IP for inspection and only those that pass all verifications will be forwarded to the next layer. Configuring Data Path Data path configuration for this use case is pretty common for BIG-IP which is historically a load balancer. It includes: Virtual Server (listens for API calls) SSL profile (defines SSL settings) SSL certificate and key (cryptographically identifies virtual server) Pool (destination for passed calls) Picture below shows how all of the configuration pieces work together. Picture 3. At first upload server certificate and key Setup SSL profile to use a certificate from the previous step Finally, create a virtual server and a pool to accept API calls and forward them to the backend From this point IG-IP accepts all requests which go to "prod.httpbin.secured-internet.lab" hostname and forwards them to API management gateway powered by NGINX Plus. Setting up WAF policy As you may already know every API starts from the OpenAPI file which describes all available endpoints, parameters, authentication methods, etc. This file contains all details related to API definition and it is widely used by most tools including F5's WAF for self-configuration. Imported OpenAPI file automatically configures policy with all API specific parameters as a list of allowed URLs, parameters, methods, and so on. Therefore WAF configuration narrows down to importing OpenAPI file and using policy template for API security. Create a policy Specify policy name, template, swagger file, virtual server and logging profile. API security template pre-configures WAF policy with all necessary violations and signatures to protect API backend. OpenAPI file introduces application-specific configuration to a policy as a list of allowed URLs, parameters, and methods. That is it. WAF policy is configured and assigned to the virtual server. Now we can test that only legitimate requests to allowed resources go through. For example request to URL which does not exist in the policy will be blocked: ubuntu@ip-10-1-1-7:~$ http -v https://prod.httpbin.secured-internet.lab/urldoesntexist GET /urldoesntexist HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Connection: keep-alive Host: prod.httpbin.secured-internet.lab User-Agent: HTTPie/0.9.2 HTTP/1.1 403 Forbidden Cache-Control: no-cache Connection: close Content-Length: 38 Content-Type: application/json; charset=utf-8 Pragma: no-cache { "supportID": "1656927099224588298" } Request with SQL injection also blocked: Configuring Bot Defense Starting from BIG-IP release 14.1 proactive bot defense, web scraping, and bot detection features are combined under Bot Defense profile. Therefore current bot defense forms a unified tool to prevent all types of bots from accessing your web asset. Bot detection and mitigation mechanisms heavily rely on signatures and javascript (JS) based challenges. JS challenges run in a client browser and help to identify client type/malicious activities or apply mitigation by injecting a CAPTCHA or slowing down a client by making a browser perform a heavy calculation. Since this article is focused on protecting APIs it is important to note that JS challenges need to be used with caution in this case. Keep in mind that robots might be legitimate users for an API. However, robots similar to bots can not execute javascript. So when a robot receives JS it considers JS as API response. Such response does not align with what a robot expects and the application may break. If you know an API is serving automated processes avoid the use of JS-based challenges or test every JS-based feature in the staging environment first. Configuration to detect and handle many different types of Bots can be simplified by using any of the three pre-configured security modes: Relaxed (Challenge free, mitigates only 100% bad bots based on signatures ) Balanced (Let suspicious clients prove good behavior by executing JS challenges or solving CAPTCHA) Strict (Blocks all kinds of bots, verifies browsers, and collects device id from all clients) It is best to start with the relaxed template and tighten up the configuration as familiarity grows with the traffic that the API endpoints see. Once the profile is created assign it to the same virtual server at "Local Traffic ›› Virtual Servers : Virtual Server List ›› prod.httpbin.secured-internet.lab ›› Security" page. Such configuration performs bot detection based on data that is available in requests such as URL, user-agent, or header order. This mode is safe for all kinds of API users (browser-based or code-based robots) and you can see transaction outcome on "Security ›› Event Logs : Bot Defense : Bot Requests" page. If there are false positives you can adjust bot status or create a new trusted one for your robots through "Security ›› Bot Defense : Bot Signatures : Bot Signatures List". Setting Up DDoS Defense WAF and BOT defenses can detect requests with attack signatures or requests that are generated by malicious clients. However, attackers can send attacks composed of legitimate requests at a high scale, that can bring down an API endpoint. The following features present in the BIG-IP can be used to defeat Denial of Service attacks against API endpoints. Transaction per second (TPS) Based DoS Defense Stress Based DoS Defense Behavioral Anomaly Based DoS Defense Eviction Policies TPS-based DoS defense is the most straightforward protection mechanism. In this mode, BigIP measures requests rate for parameters such as Source IP, URL, Site, etc. In case the per minute rate becomes higher than the configured threshold then the attack gets triggered, and selected mitigation modes are applied to ‘all’ requests identified by the parameter. The stress-based mode works similarly to TPS, but instead of applying mitigation right after the threshold is crossed it only mitigates when the protected asset is under stress. This approach significantly reduces false positives. Behavioral anomaly detection (BADoS) mode offers the most advanced security and accuracy. This mode does not require the administrator to perform any configuration, other than turning the feature on. A machine learning algorithm is used to detect whether the protected asset is under attack or not. Another machine learning algorithm is used to baseline the traffic in peacetime. When the ‘attack detector algorithm’ identifies that the protected asset is under stress and non-responsive, then the second algorithm stops learning and looks for anomalies. Signatures matching these anomalies are automatically created. Anomalies discovered during attack time are likely nefarious and are eliminated from the traffic by application of dynamically discovered signatures. BADoS will automatically build a good traffic baseline, detect anomalies and stop them if the API endpoint is under stress. Conclusion F5 offers a multi-layered solution for protecting APIs, which is easy to configure.Please connect with me via comments and keep an eye on more articles in this series. Good luck!Increasing accuracy using Bad Actor and Attacked Destination detection
Dear Reader, In this article, I will return to the mitigation layer of static DoS vectors once again. Before I want to go to the next layers within subsequent articles, I would like to discuss some options I haven´t touched on regarding DoS vectors so far. It is a functionality that helps to massively lower the chance of a false/positive when using static vectors. It also is the base of another mitigation technology we can use as the very first mitigation withing the hud-chain. This can be done by either using RTBH (Remote Triggered Black Hole) routing, Flowspec before the BIG-IP, or IP shunning (dropping on IP level at the speed of line) on the BIG-IP. I am talking about “Bad Actor Detection” and “Attacked Destination Detection”. Bad Actor Detection In my previous article “Explanation of F5 DDoS threshold modes”, I outlined the different Threshold modes: Fully Automatic, Auto Detection/Multiplier Based Mitigation, and Fully Manual, but I didn´t talk about the other options you can configure per vector. The two I want to explain now are the “Bad Actor Detection” and the “Attacked Destination Detection” options. Back in my first article “Concept of F5 Device DoS and DoS profiles”, I described that the idea of static DoS vectors is to lower the number of attack packets going into the BIG-IP (CPU) and to the services which need to be protected by BIG-IP. This is done by dropping these packets in a very early stage, ideally via a dedicated hardware chip (FPGA). It is like having the F5 TMOS operating system with all its smartness and feature-reach capabilities, but in front of it, there is a DDoS device protecting it. But, the beauty here is, that it is all in a single device and interacting with each other. Figure 1: DDoS Mitigation Order Anyhow, when the CPU detects the attack via a static vector and the mitigation threshold is reached, it programs that vector into the FPGA for mitigation. When you run DDoS protection on a BIG-IP VE, then of course the mitigation will be still done in software (CPU), but on a very early stage of the hud-chain as to not “waste” CPU cycles for additional functionalities. At this point, I would like to recommend the article of my colleague Ryan Howard, “Mitigating 40Gbps DDoS Attacks with the new BIG-IP VE for SmartNICs Solution”, who posted on the option to offload DDoS mitigation from a BIG-IP virtual edition into the FPGAs sitting on a network card (smartNIC). This works especially well in the “new” modern world of highly virtualized environments, with the capability to do DDoS mitigation in hardware without having a dedicated device. Using static DoS vectors has the strong advantage that they kick in within microseconds and protect the BIG-IP itself (Device DoS) or the service/network behind the BIG-IP via DoS profiles attached to protected object/virtual server configurations. When the attacker is using invalid packets like a bad header, then the FPGA has enough information to block exactly those bad packets. But when it is for example a TCP PUSH/ACK flood, then there is a chance of false/positives, because you do have many, many TCP PUSH/ACK packets within your legitimate traffic and the FPGA cannot differentiate with only the information of protocol “TCP” and flag set “PUSH/ACK”. This is why BDoS is a powerful functionality running in parallel, but executed before the static vectors from the hud-chain perspective. But, if you can add information about the attacking source IPs (bad actors) and/or the information about the IPs being under attack (attacked destination), then you can potentially lower the chance of a false/positive. Or, at least you lower them to the IPs being involved in the attack rather than having to mitigate on a global level. When you enable “Bad Actor Detection” on a vector and you use the “Fully Automatic” threshold mode, then the BIG-IP will automatically identify the “heavy” source-IP(s) causing the high load and will do the mitigation for that vector only on that source IP(s). Figure 2: Fully Automatic Bad Actor Detection Of course, if the attacker uses randomized source IPs and sends a different source IP with every packet, then it will not work. But it does work very well for example on Reflection and Amplification attacks, where specific DNS, LDAP, or NTP servers are being used for sending the attack traffic. These IP addresses can get easily identified and get used to driving a higher accuracy on the static DoS vector mitigation. In one of my next articles, I will also explain what else you can do with this bad actor IPs by using the “Add Source Address to Category” function. If you choose the manual threshold configuration, then you can define your own thresholds for detecting bad actors and at which level you want to mitigate them. Figure 3: Manual Bad Actor Detection Attacked Destination Detection The next option that you can configure on the static vector is the “Attacked Destination Detection”. Here it works in the same way as on the “Bad Actor Detection”. But, using “Fully Automatic” mode, the system will automatically identify which IP(s) are under attack and will add them to the attack traffic definition. Figure 4: Fully Automatic Attacked Destination This is very helpful on the Device-level protection of the BIG-IP, because as you know here the BIG-IP counts all packets into the specific vector which “hits” the device. It doesn´t matter if it is on a self-IP, a service behind the BIG-IP, or something else. If the BIG-IP must deal with it in any way, it will get counted. Now let us assume an attacker starts a strong TCP PUSH/ACK packet flood against a service located behind the BIG-IP. As discussed in my article “Demonstration of Device DoS and Per-Service DoS protection” we know this flood will never hit the service, because the BIG-IP is a stateful-device (at least this is the recommended configuration, whenever possible). Therefore, all TCP packets not belonging to an existing connection will get dropped at the latest by the state engine. We also know, when the vector is configured in “Fully Automatic” mode, the mitigation will start when we see more packets/sec as the detection rate has calculated and the stress on the BIG-IP (CPU) gets too high. Let us imagine now that the flood is so strong that the mitigation starts and would be executed on all PUSH/ACK packets hitting the BIG-IP, it could lead to false/positives. But when “Attacked Destination Detection” is enabled, then the mitigation will only be done on PUSH/ACK packets targeted for that IP address (or IP addresses) being under attack. Therefore, we will only see false/positives for the destination(s) being under attack. If bad actors are also involved, then the attack mitigation will only be done on TCP PUSH/ACK packets coming from the bad actor(s) and targeting the attacked destination(s). Of course, there is still a chance of a false/positive, but keep in mind this is the first mitigation that kicks in extremely fast and then the granular Behavioral DoS mitigation will take over. I usually recommend enabling “Attacked Destination Detection” on the Device Protection for all vectors which do not cover “bad header” packets, because they are broken anyway and will not get forwarded. But, on all others, I want to differentiate if a packet belongs to a destination that is under attack or not. On the PO/VS DoS profiles, I usually only enable it on network wildcard PO/VS, because on these objects we have multiple destinations. Not like on PO/VS which covers one IP. I hope that this article gives you a little better inside into the two options available for configuring a DoS vector. My next article will cover another layer of Network DoS mitigation: Cookie-based DoS protection, which gives you HW accelerated and granular DoS protection on different attack vectors. Thank you, sVen MuellerHTTP Brute Force Mitigation Playbook: Slow Brute Force Protection Using Behavioural DOS - Chapter 6
Brute Force attack is where attacker tries to find the password of users quickly, there are times when attacker is not in hurry and do make his attack go under the radar, using very slow brute force attack.It can not be detected by detection criteria of Brute force protection feature of Advanced WAF/ASM reason being if you try to tweak the setting to catch slow brute force attack then its very hard for ASM/Advance WAF to distinguish between attack and legitimate user login atttempt. We may use other protection available in ASM/Advance WAF to protect from Slow brute force attack. In this chapter to protect from Slow brute force attack we will use TLS signature generated by behavioural DOS. But first: Benefits, Limitation and Requirement. Benefits Benefit of using TLS fingerprint: Good and bad Clients can be differentiated based on SSL handshake. Once the Advance WAF/ASM is 100% confident user does not have to do anything to find out unusual/attack traffic pattern. This can be also used to protect mobile application as it does not use Javascript. Limitations To get TLS fingerprinting signature using BADOS legitimate traffic should be learned by Advanced WAF/ASM On ASM, Behavioural DOS can be configured on max 2 virtual servers, where as on Advanced WAF, Although there is no license limitation of attaching DOS profile with BADOS enable to Virtual server but it is not recommended to configure more then 70 BADOS enabled Virtual server per box. Requirements ASM/Advanced WAF license. Appropriate rights to access/make changes from GUI and command line. Some of the reporting is available only if AFM is provisioned in addition to the above mentioned modules. (If AFM is not provisioned you can still find the information using CLI) Proactiveness As a general rule, instead of waiting for attack and then take necessary action, We should always be proactive in defending attack. Preparation for mitigating slow Brute force attack. Slow brute force is very hard to detect, So most important thing to protect application from slow brute force attack is Advanced WAF/ASM should know the normal traffic. For that we can use Behavioral & Stress-based (D)DoS Detection option under DoS Protection profile of Advance WAF/ASM. For Configuring DoS Protection profile, to protect against slow brute force attack using TLS fingerprinting follow the below mentioned steps. Important:For BIG-IP ASM/Advance WAF 14.1.0, you can access theTLS fingerprinting signaturesconfiguration sectiononly when you had previously selectedUse Legacy Application Dos viewin theHTTP Propertiesconfiguration pop-up. Go to Security››DOS Protection ›› Protection Profiles››click create. Enter the profile name as per your requirement, select the family as HTTP and press Commit Changes to System Click on newly shown HTTP and then click configure settings for HTTP Family settings. Next click on Use Legacy Application DOS View Go to Behavioral and stress-based detection under Application security. Change operation mode to blocking and Threshold mode to automatic. Under Behavioral Detection and Mitigation enable Request signature detection along with TLS fingerprinting signatures and Use approved signatures only (In case you don’t want to use unapproved signature). Leave all the settings unchanged and click save and finished. (Make Sure Bad actors behavior detection is unchecked as we want to use TLS signature) Select Mitigation to Standard or as per requirement from available options and then Click save Next apply the newly created dos profile to the appropriate https virtual server. Go toLocal Traffic>Virtual Servers. Select the name of the HTTPSvirtual server. Go toSecuritytab and selectPolicies. ForDoS Protection Profile, selectEnabled. ForProfile, select the DoS profile created in above steps. Select theUpdatebutton. Let the normal traffic pass through the VS. This will allow ASM to learn the traffic. How do we know ASM is ready and is 100% confident about the normal traffic? Login to cli of BIGIP Run command “admd -s vs./Common/<VSname>+/Common/<DOSprofilename.info.learning>” For exampleadmd -s vs./Common/BF-PHP+/Common/ASM-TLS-Fingerprinting.info.learning You will see output as similar to the one mentioned below. vs./Common/BF-Test+/Common/Brute-Force-test.info.learning:[0, 0, 0, 0] Once the traffic starts passing through vs these values will increase. Each value has its own meaning as described below. A.baseline_learning_confidence: Description: in % how confident the system is in the baseline learning. Desired Value: > 90% B. learned_bins_count: Description: number of learned bins Desired Value: > 0 C. good_table_size: Description: number of learned requests Desired Value: > 2000 D. good_table_confidence: Description: how confident, as %, the system is in the good table Desired Value: Must be 100 for signatures You may run the command again if the Behavioral DoS is still learning Still learning admd -s vs./Common/BF-PHP+/Common/ASM-TLS-Fingerprinting.info.learning Behavioural DOS feature is based on learning analysing all traffic to the web application, building baselines, and then identifying anomalies when server stress is detected.So its important to know when server is stress and how to check the server street level. To find out the stress level Go to Security››DoS Protection››Protected Objects(This option is only available if you have AFM Provisioned) Find out the VS for which you would like to check the status and Click the arrow below Attack status. Once you click you will detailed informed is displayed on the screen, which includes Server Stress To check the Server stress using CLI you may run below mentioned command. admd -s vs./Common/<VSname>+/Common/<DOSprofilename.sig.health> Server Stress value Range: If there is no traffic server value is 0.5 If server functions properly value is between (0,1) Value higher then 1 is considered as load and mitigation may be applied for example admd -s vs./Common/BF-PHP+/Common/ASM-TLS-Fingerprinting.sig.health Once the output of below command shows appropriate values (as mentioned above) which tells ASM is confident, ASM is ready to differentiate between normal and attack traffic. Below output shows ASM is 100% confident admd -s vs./Common/BF-PHP+/Common/ASM-TLS-Fingerprinting.info.learning Slow brute Force attack has been reported To check the status of attack and Server stress level. Go to Security››DoS Protection››Protected Objects(This option is only available if you have AFM Provisioned) Find out the VS for which you would like to check the status and Click the arrow below Attack status. Once you click you will see detailed informed is displayed on the screen. For example as show below Server Stress is 100 now. If AFM is not provisioned you may run below mentioned command to check if the server is under stress. admd -s vs./Common/<VSname>+/Common/<DOSprofilename.sig.health> Server Stress value Range: If there is no traffic server value is 0.5 If server functions properly value is between (0,1) Value higher then 1 is considered as load and mitigation may be applied For example admd -s vs./Common/BF-PHP+/Common/ASM-TLS-Fingerprinting.sig.health You may continue to monitor the output using command line or GUI to find out if attack has started. To check if attack has started you may check using command line. If the value is 0,0 then there is no attack if the value is 1 VS is under attack admd -s vs./Common/<VSname>+/Common/<DOSprofilename.info> for example: admd -s vs./Common/BF-PHP+/Common/ASM-TLS-Fingerprinting.info Using the GUI Go to Security››DoS Protection:Protected Objects Note: (To get this view AFM should be provisioned ) If you continue to monitor you may notice that BADOs has started generating signature. But accuracy in start will not be 100% and it may take some time to become 100% accurate. Using CLI admd -s vs./Common/BF-PHP+/Common/ASM-TLS-Fingerprinting.info Using GUI Security››DoS Protection››Protected Objects(This option is only available if you have AFM Provisioned) If the Dynamic Signature status is unready the signature is not ready and does not have 100% accuracy. Note: (To get this view AFM should be provisioned, If AFM is not provisioned you may continue monitor using CLI ) Once signature is ready Dynamic signature status will change as shown below. Note: (To get this view AFM should be provisioned, If AFM is not provisioned you may continue monitor using CLI ) Once the signature’s accuracy is 100%, It will be available underSecurity››DoS Protection:Signatures >> Dynamic. As shown below. You may notice in above screenshot that Accuracy of signature is 100% where as approval status is Unapproved, If you want to use only approved signature (which we have used in this case) you need to click the check box infront of the signature, as soon as you will enable check box a window on right side will pop up and you may enable check box in-front of Approved and then press update to manually approve the signature. Note: User approved signatures only under Behavioral & Stress-based (D)DoS Detection in the DOS profile should be enable. Once you approve the signature, Signature approval state will change to manually approved as shown below You may also check DOS logs by checking Security››Event Logs››DoS›› Application Events Another Graphical view option for DOS can be checked by going to Security››Reporting:DoS:Dashboard If you want to check a specific attack ID then please on right side under Attack IDs find the attack ID and click on it. As soon as you will click on it page will show the data related to specific attack ID as shown below. As shown above during attack, TLS signature generate by Behavioural DOS ismitigating the attack and normal requests are still passing through using Behavioural attack signature. Note: By default, when the systemidentifies signature pattern anomalies, itsilently drops the connection. You can change the mitigation mode and force the system to send a reset(RST) when the traffic matches a signature pattern. To change the mitigation mode fromdrop to reset, perform the following steps: 1. Log in to tmsh by typing the following command: tmsh 2. To change themitigation mode to reset, typethe following command: modify sys db adm.mitigation.accelerated.signatures.drop.mode value reset Note: If you want to generate HTTP signature using BADOS instead of TLS signature in DOS protection profile you can select accelerated signature and rest of the steps will remain same.
