Cookie-based DDoS protection

Dear Reader,

DDoS protection starts with being able to identify that you are under attack. Next, you will need to be able to differentiate between good and bad traffic. This will be particularly challenging if the attack (bad) traffic looks like your good traffic. So, how can you ideally filter out only the attack traffic without affecting the good traffic? Well, and finally, you need to have the capacity and performance to deal with the high number of flood packets.

In this article, I want to explain to you another layer of F5’s network DDoS protection, which perfectly meets these requirements: cookie-based DDoS protection.

Figure 1: Multi-layer DDoS mitigation architecture

Cookie-based protection

No, I´m not talking about HTTP, we are still within the Network protection. Even on the network layer, BIG-IP can deal with cookies and can use it in a very effective way to mitigate without false/positives.

The overall idea is to store connection- related information, encrypted within a packet, which is being forwarded. The response packet will have this cookie included as well and can be used to identify that the packet belongs to an existing connection. Therefore the packet is valid and not just a randomized packet to drive up CPU utilization. When the packet is “wrong” it can be dropped at an early stage by a dedicated chip (FPGA) and doesn´t consume CPU resources.

But let´s start talking about attack vectors.

SYN flood

SYN flood is a very old, but still very effective way for DDoS attacks. The idea is to overwhelm stateful devices with too many half-open connections, which mostly results in (too) high CPU usage and/or fully utilized memory consumption.

Attackers just send SYN packets towards the target (server, firewall, etc.). The first packet during a 3-way handshake of a connection setup will let

the target immediately allocate memory for the TCP socket data structure holding the tuple, sequence number, and “state” of the session. The target then replies to the client (attacker) with an “SYN-ACK” packet. Under normal circumstances, packet loss or slow communication can cause the SYN-ACK packet to be delayed or lost altogether. The server must decide how long to hold the socket in a half-open state before transmitting a reset and releasing the memory for other tasks. This means the higher this timer is, the faster memory can get filled up under SYN flood conditions. But, on the other hand, the lower the value is, the higher the risk that you drop a valid connection request (for example, because of latency).

SYN flood attackers usually spoof the source IP, so the SYN-ACK does not hit the initial source (attacker). This way the SYN flood can also result in an SYN-ACK flood done by the target (server, firewall, etc.), which will be especially painful because the spoof involves only one or a small number of IP addresses. This can be easily identified on the mitigation side and can get blocked or rate-limited. Keep in mind, however, that this could also be a strategy of the attacker…

Some interesting side effects are also, that depending on the TCP stack, servers send a RST packet when they release a timed-out connection. Some TCP stacks do also send a RST packet when they receive an SYN-ACK packet for which they don´t have a connection entry. Therefore, often SYN floods do also end in SYN-ACK and RST floods as well.

Figure 2: SYN flood

But, how can you protect against this type of attack?

• SYN rate-limiting via static DoS vectors is one option, but it can potentially lead to false/positives. This is true in particular when you cannot limit the mitigation to “attacked destination” and/or “bad actors” (source IPs causing the stress). For further information, please read my article “Increasing accuracy using Bad Actor and Attacked Destination detection“.
• Behavioral DoS (BDoS) is a very effective way but usually takes a few seconds to kick in.
• Another strong option is to use F5’s SYN-Cookie mitigation.

SYN-Cookie mitigation

SYN-Cookie mitigation is an effective way to resist SYN floods. It gets activated when the threshold of the configured number of half-open connections is reached. On AFM (Advanced Firewall Manager) or DHD (DDoS Hybrid Defender), the threshold can be configured via the “tcp-half-open” vector on the device level and also on the per VS/PO level via a DoS profile attached to it.

Figure 3: TCP Half Open vector configuration

Mitigation threshold means, activate the SYN-cookie process after (in this example) 10.000 half-open connections. Keep in mind, on the Device level you configure the threshold per TMM and within a DoS profile it is per PO/VS to which it is attached.

SYN-Cookie mitigation gets deployed by choosing initial TCP sequence numbers in the SYN-ACK response according to the properties of the originating TCP SYN. It encrypts connection information within the sequence number of the responding SYN-ACK packet. When the SYN was sent by a valid client, BIG-IP will receive a well-formed ACK response from the client including the cookie stored within the "ack" field and incremented by one. BIG-IP can now decrypt and reconstruct the connection details and establish the connection by performing the necessary SYN establishment handshakes with the server on behalf of the client.

The beauty of this technique is that it smartly protects the memory, but the process of “encrypt” and “decrypt” the cookies consumes CPU, which can be potentially dangerous. To enhance the capability of the BIG-IP systems, SYN cookie functionality is extended to the FPGA, which is radically more effective than software mitigation alone. In this way, BIG-IP can differentiate between “valid” and “bad” SYN packets without negatively impacting memory or CPU performance!

Here is a useful hint!

As mentioned above, you can configure the activation of the SYN cookie process by defining a threshold of the number of half-open connections. This can be done via the “tcp-half-open” vector on the Device level and also via the DoS profile for a VS/PO. This is my recommended approach. Keep the DoS config in one place and therefore disable the second option (via traditional TMOS configuration). This can be done under System/Configuration: Local Traffic: General, by setting the “Default Per Virtual Server SYN Check Threshold” and “Global SYN Check Threshold” to “0”. Also, disable “Hardware VLAN SYN Cookie Protection”.

Figure 4: LTM SYN-Cookie configuration

Now it’s all controlled by the “tcp-half-open” vector, which makes it less confusing.

Please also keep in mind, SYN Cookie does only work in FPGA when “Auto Last Hop” is enabled!

What do the stats of "tcp-half open" mean in the context of SYN-Cookie protection?

When we take a look into the stats via the script I provided in my article “Demonstration of Device DoS and Per-Service DoS protection” during an SYN flood attack, we see a rate for int_drops. This only appears when the mitigation threshold (number of half-open connections) is reached.

Figure 5: TCP Half Open vector shown in dos_stats output

The “stats_rate” shows the number of half-open connections in the last second and the “drop_rate” or “int_drop_rate” = (the number of cookies sent) – (number of valid ACKs received).

SYN-ACK flood

Let’s talk about another attack vector: SYN-ACK flood.

Again, this attack can be done in different ways. The attackers simply send a high rate of randomized SYN-ACK packets towards the victim. Or it is a result of an SYN flood with responding SYN-ACK packets towards a spoofed source IP of the SYN packet.

Anyhow, every high packet rate can cause massive damage to the performance of Firewalls, Servers, or other devices and this is what DoS attackers usually try to archive with this type of attack. And again, the question is: how to differentiate between “good” and “bad” SYN/ACK packets? Asking the connection table would increase the CPU and is therefore not the best choice. So, ideally, it is done upfront (CPU) within the hardware chip (FPGA) to be high performant.

SYN-ACK flood mitigation

By enabling “Only Count Suspicious Events” on the “TCP SYN ACK Flood” vector on the Device level you do enable a cookie mechanism that allows the BIG-IP to differentiate if an SYN-ACK packet belongs to an SYN which has passed the BIG-IP before.

Figure 6: TCP SYN ACK Flood vector configuration

This mechanism is processed within the FPGA and does therefore not consume CPU. This enables the BIG-IP to be extremely resistant against this type of flood.

Please keep in mind, this cookie function was added to the product with version 15.1. Before 15.1, you could also enable “Only Count Suspicious Events”, but it puts the vector into software mode. That means it can differentiate between “good” and “bad” SYN-ACK packets, but the mitigation is done within the CPU.

How does SYN-ACK cookie mitigation work

When the mechanism is enabled and an SYN packet received on a BIG-IP and is being forwarded, then the initial SNI gets replaced by an own SNI which includes properties of the originating TCP SYN in an encrypted format. When BIG-IP now receives SYN-ACK packets, it expects the initial SNI (cookie) with the ACK field to be incremented by 1.

BIG-IP can now decrement and decrypt that value to check if the “cookie properties” fit the properties of the initial SYN packet. If that is the case, then the "ack" field gets replaced with the incremented SNI of the initial SYN packet and it will then be returned to the original client.

If the cookie check fails, then it will be counted as a bad SYN-ACK and when the mitigation threshold on the vector is reached the rate-limiting for bad SYN-ACK packets gets activates in the hardware (FPGA).

Figure 7: TCP SYN ACK cookie mitigation

ACK flood and mitigation

Let’s get to a third attack vector: ACK flood.

This one is also widely used by attackers and can be very damaging. We already know it’s about the differentiation between valid and bad ACK packets plus being able to use the hardware (FPGA) for mitigation.

Here again, the overall idea is to encrypt connection properties within a cookie and store them in the TCP packet. Valid response packets do have this cookie included and can be validated.

When the number of invalid ACK packets reaches the mitigation threshold, then the rate-limiting will start.

This cookie protection mechanism can be enabled per VLAN on the “bad_ack” vector on the Device configuration.

Figure 8: TCP BADACK flood vector (cookie) configuration

Simply move the VLANs you want to have this feature being enabled to the “Selected” area.

Conclusion

TCP floods can be easily used to bring down even strong firewalls, servers, or other security devices. Generally, it is not the volume, in terms of bandwidth, that is at issue here. It is the high packet rate, which in particular puts stateful devices under pressure if they don´t have dedicated hardware (FPGA) supported defense mechanism.

With the cookie-based detection and mitigation defense technic, F5 delivers another strong layer within the layered network DDoS architecture. Even the whole cookie process is done within FPGAs, these chips can differentiate between valid and bad TCP packets without checking the connection state table.

SYN-ACK and ACK cookies, which got introduced into the product with version 15.1, are being processed even before the Behavioral DDoS layer. SYN cookies are done after the static vectors.

One of my next articles will provide an overview of best practices and an explanation on how to integrate DHD (DDoS Hybrid Defender) or AFM (Advanced Firewall Manager) into the Network for DDoS protection. But, next, I will explain the latest part of the Network protection layers, which is the first layer from the data path order: IP-Intelligence/ IP shunning and also RTBH (Remote triggered Black Hole Routing) and Flowspec.

I hope this article was useful to you and keep in touch!

Thank you, Sven Mueller