2. SYN Cookie: Operation

Introduction

 

As concluded in the last article, in order to avoid allocating space for TCB, the attacked device needs to reject TCP SYN packets sent by clients. In this article I will explain how a system can do this without causing service disruption, and more specifically I will explain how this work in BIG-IP.

 

Implementation

 

Since attacked device need to alter default TCP 3WHS behaviour the best option is implementing SYN Cookie countermeasure in an intermediate device, so you offload your servers from this task. In this way if connection is not legitimate then it is just never forwarded to the backend server and therefore it will not waste any kind of extra resourcess. Since BIG-IP is already in charge of handling application traffic directed towards servers it is the best place to implement SYN Cookie.

 

What BIG-IP does is adding an extra layer, which we can call SYN Cookie Agent, that basically implement a stateless buffer between client and BIG-IP TCP stack. This agent is in charge of handling client’s TCP SYN packets by modifying slightly the standard behaviour of TCP 3WHS, this modification comes in two flavours depending on what type of routing role BIG-IP is playing between client and server.

 

Symmetric routing

 

This is the typical situation. In this environment BIG-IP is sited in the middle of the TCP conversation and all the traffic from/to server pass through it. The SYN Cookie operation in this case is briefly described below:

 

  1. Client sends a TCP SYN packet to BIG-IP.
  2. BIG-IP uses a stateless buffer for answering each client SYN with a SYN/ACK.
  3. BIG-IP generates the 32 bits sequence number which will be included in SYN/ACK packet sent to the client.
    1. BIG-IP encodes essential and mandatory information of the connection in 24 bits.
    2. BIG-IP hashes the previous encoded information.
    3. BIG-IP also encodes other values like MSS in the remaining bits.
  4. BIG-IP generates the SYN/ACK response packet and includes the hash and other encoded information as sequence number for the packet. This is the so called SYN Cookie.
  5. BIG-IP sends SYN/ACK to client.
  6. BIG-IP discards the SYN from the stateless buffer, in other words, BIG-IP removes all information related to this TCP connection. At this point no memory has been allocated for TCP connection (TCB).
  7. Client sends correct ACK to BIG-IP acknowledging BIG-IP’s sequence number.
  8. BIG-IP validates ACK.
  9. BIG-IP subtracts one to this ACK.
  10. BIG-IP runs the hash function using connection information as input (see point 3-b above). Then it compares it with the hash provided in the ACK, if they match then it means client sent a correct SYN Cookie response and that client is legitimate.
  11. BIG-IP uses connection information in the ACK TCP/IP headers to create an entry in connflow. At this time is when BIG-IP builds TCB entry, so it's the first time BIG-IP uses memory to save connection information.
  12. BIG-IP increase related SYN Cookie stats.
  13. BIG-IP starts and complete a TCP 3WHS with backend server on behalf of the client since we are sure that client is legitimate.
  14. Now traffic for the connection is proxied by BIG-IP as usual attending to configured L4 profile in the virtual server.

*If ACK packet received by BIG-IP is a spurious ACK then BIG-IP will discard it and no entry will be created in connection table.

Since attackers will never send a correct response to a SYN/ACK then you can be sure that TCB entries are never created for them. Only legitimate clients will use BIG-IP resources as shown in diagram:

 

Fig4. TCP SYN Flood attack with SYN Cookie countermeasure

 

Asymmetric routing

 

In an asymmetric environment, also called nPath or DSR, you face a different problem because Big-IP cannot establish a direct TCP 3WHS with the server (point 11 in last section). As you can see in below diagram SYN/ACK packet sent from server to client would not traverse Big-IP, so method used for symmetric routing cannot be used in this case. You need another way to trust in clients and discard them as attackers, so clients then can complete TCP 3WHS directly with the server.

Fig5. TCP state diagram section for asymmetric routing + FastL4

 

In order to circumvent this problem BIG-IP takes the advantage of the fact that applications will try to start a second TCP connection if a first TCP 3WHS is RST by the server. What BIG-IP does in asymmetric routing environments is completing the TCP three way handshake with client, issuing SYN Cookie as I described for symmetric routing, and once BIG-IP confirms that client is trustworthy it will add its IP to SYN Cookie Whitelist, it will send a RST to the client and it will close the TCP connection. At this point client will try to start a new connection and this time BIG-IP will let the client talk directly to the server as it would do under normal circumstances (see diagram above).

The process is briefly described below:

  1. Client sends a TCP SYN packet to BIG-IP.
  2. BIG-IP generates the sequence number as explained for symmetric routing.
  3. BIG-IP generates the SYN/ACK response packet and includes the calculated sequence number for the packet.
  4. BIG-IP sends SYN/ACK to client and then remove all information related to this TCP connection. At this point no memory has been allocated for TCP connection (TCB).
  5. Client sends correct ACK to BIG-IP acknowledging sequence number.
  6. BIG-IP validates ACK.
  7. BIG-IP substract one to this ACK so it can decode needed information and check hash.
  8. BIG-IP increase related SYN Cookie stats.
  9. BIG-IP sends a RST to client and discards TCP connection.
  10. Big-IP adds client’s IP to Whitelist.
  11. Clients starts a new TCP connection.
  12. BIG-IP lets client to start this new TCP connection directly to the server since it knows that server is legitimate.

*If ACK packet received by BIG-IP is a spurious ACK then BIG-IP will discard it and no entry will be created in connection table.

Fig6. TCP 3WHS flow in asymmetric routing

 

DSR whitelist has some important characteristics you need to take into account:

  • By default IP is added 30 seconds to the whitelist (DB Key tm.flowstate.timeout).
  • Client’s IP is added to whitelist for 30 seconds but only if there is no traffic from this client, if BIG-IP have already seen ACK from this client then its IP is removed from whitelist since connection has been already established between client and server.
  • Whitelist is common to all virtual servers, so if a client is whitelisted it will be for all applications.
  • Whitelist it is not mirrored.
  • Whitelist it is not shared among blades.

 

SYN Cookie Challenge

 

When a TCP connection is initiated the TCP SYN packet sent by the client specify certain values that define the connection, some of these values are mandatory, like source and destination IP and port, otherwise you would not be able to identify the correct connection when packets arrive. Some other values are optional and they are used to improve performance, like TCP options or WAN optimizations. Under normal circumstances, upon TCP SYN reception the system creates a TCB entry where all this information is saved. BIG-IP will use this information to set up the TCP connection with the backend server.

The problem we face is that when SYN Cookie is in play device does not create the TCB entry, only a limited piece of connection information is collected, so the information that BIG-IP has about the TCP connection is limited.

Remember that SYN Cookie is not handled by TCP stack but by the stateless SYN Cookie Agent, so we cannot save the connection details. What BIG-IP does instead is encoding key information in 24 bits, this left only some bits for the rest of data. While there is no room for values like Window Scale information, however we have space for other values, for example 3 bits reserved for MSS value. This limits MSS possible values to 8, the most commonly used values, and the one chosen will the nearest to the original MSS value.

However, note that MSS limitation has a workaround through configuration in FastL4 profile that it can help in some cases. The option SYN Cookie MSS in this profile specifies a value that overrides the SYN cookie maximum segment size (MSS) value in the SYN-ACK packet that is returned to the client. Valid values are 0, and values from 256 through 9162. The default is 0, which means no override. You might use this option if backend servers use a different MSS value for SYN cookies than the BIG-IP system does.

If this is not enough for some customers, BIG-IP overcomes this problem by using TCP timestamp space to save extra information about TCP connection, in other words we create a second extra SYN Cookie. This space is used to record the client and server side Window Scale values, or the SACK info which are then made available to the TCP stack via this cookie if the connection is accepted. Note that TCP connection will only be accepted if both SYN Cookies (standard and Timestamp) are correct.

So all this means that if the client starts a connection with TCP TS set then Big-IP will have more space to encode information and hence the performance will be improved when under TCP SYN flood attack.

 

NOTE: SYN Cookie Timestamp extension (software or hardware) only work for standard virtual servers currently. FastL4 virtual servers only supports MMS option in SYN Cookie mode.

 

Conclusion

 

Now you know how SYN Cookie works under the hoods. In next article I will describe when SYN Cookie is activated and I will give specific details of BIG-IP SYN Cookie operation.

Note that limitations related to SYN Cookie commented in this article are not caused specifically by BIG-IP SYN Cookie implementation but by SYN Cookie standard itself. In fact, F5 Networks implements some improvements on SYN Cookie to get a better performance and provide some extra features.

Updated Apr 26, 2023
Version 3.0
AFM
dos
LTM
security
series-syn-cookie-the-definitive-guide
SYN Cookie

Was this article helpful?

3 Comments

Sort By
  • Dan_Pacheco's avatar
    Dan_Pacheco
    Icon for Cirrus rankCirrus
    Apr 03, 2021

    Thanks for clarifying the above. This article series goes a long way to explaining how the big-ip handles SYN cookies. I remember looking for a detailed explanation 18 months ago and didn’t seem anything this detailed. Great post!!!

  • Dan_Pacheco's avatar
    Dan_Pacheco
    Icon for Cirrus rankCirrus
    Apr 02, 2021

    Thanks for the excellent explanation. Does the VS type have an affect? I am under the impression for a standard VS type, the 3WHS needs to complete with the client before a SYN is sent to the pool member, so the above explanation aligns perfectly.

     

    But for a Performance L4 VS, it passed the SYN to the server before completing the 3WHS. https://support.f5.com/csp/article/K8082

     

    Does this mean that we negate the benefits of SYN cookie when using a perf L4 a VS type?

     

  • Javier_Velasco's avatar
    Javier_Velasco
    Icon for Employee rankEmployee
    Apr 03, 2021

    When SYN cookie is activated, regardless the type of the virtual server, BIG-IP needs to work in a full proxy mode for the initial TCP 3WHS with client in order to confirm that it is not an attacker. This means that BIG-IP will first handle SYN Cookie TCP handshake with client, and once BIG-IP confirms client is legitimate it will start a second TCP 3WHS with the server.

     

    SYN Cookie works in the same way for Standard or FastL4 virtual server, so it can work with FastL4 virtual servers as well. If this is the case BIG-IP modifies traditional FastL4 behaviour by disallowing direct initial TCP 3WHS negotiation between client and server, instead it will work as a typical Standard virtual server, but only for the initial TCP 3WHS.

    Note that FasL4 profile goal is accelerating the connection traffic between client and server (not for accelerating TCP 3WHS) by offloading eligible flows to the ePVA chip for acceleration. So once client is confirmed as legitiate then rest of the traffic for the connection will behaves as expected for a FastL4 virtual server.