9. SYN Cookie Troubleshooting: Logs
Introduction In this last article I will add the last piece of information you can check when troubleshooting TCP SYN Cookie attacks, logs. With this information together with all that you have learned until now you should be able to understand how SYN Cookie is behaving and decide if there is any change you should do in your configuration to improve it. Use cases LTM SYN Cookie at Global context Logs when Global SYN Check Threshold or Default Per Virtual Server SYN Check Threshold has been exceeded are similar, so in order to know in which context was SYN Cookie activated you need to focus on specific text in logs. For example, by having below config: turboflex profile feature => adc tmsh list sys db pvasyncookies.enabled => true tmsh list ltm global-settings connection default-vs-syn-challenge-threshold => 1500 <= tmsh list ltm global-settings connection global-syn-challenge-threshold => 2050 <= tmsh list ltm profile fastl4 syn-cookie-enable => enabled You will get logs similar to the ones below if Global SYN cache has been reached: Dec 7 03:03:02 B12050-R67-S8 warning tmm9[5507]: 01010055:4: Syncookie embryonic connection counter 2051 exceeded sys threshold 2050 Dec 7 03:03:02 B12050-R67-S8 warning tmm5[5507]: 01010055:4: Syncookie embryonic connection counter 2051 exceeded sys threshold 2050 Dec 7 03:03:02 B12050-R67-S8 notice tmm5[5507]: 01010240:5: Syncookie HW mode activated, server name = /Common/syncookie_test server IP = 10.10.20.212:80, HSB modId = 1 Dec 7 03:03:02 B12050-R67-S8 notice tmm9[5507]: 01010240:5: Syncookie HW mode activated, server name = /Common/syncookie_test server IP = 10.10.20.212:80, HSB modId = 2 As you can notice there are two different messages, the first one informs about Software SYN Cookie being activated at Global context, and the second one tells us that Hardware is offloading SYN Cookie from Software. Since there is a minimum delay before Hardware to start to offload SYN Cookie is expected to see a non zero value for the counter Current SYN Cache stats. See article in this SYN Cookie series for more information about stats. Global SYN cache value is configured per TMM, so you see in the log that 2050 threshold has been exceeded in the TMM, and therefore SYN Cookie is activated globally in the device. In this specific example the device has two HSBs and BIG-IP decided that tmm9 and tmm5 would activate each one of them this is why we see the logs repeated. LTM SYN Cookie at Virtual context For the same configuration example I showed above you will see log similar to one below if Virtual SYN cache has been reached: Oct 18 02:26:32 I7800-R68-S7 warning tmm[15666]: 01010038:4: Syncookie counter 251 exceeded vip threshold 250 for virtual = 10.10.20.212:80 Oct 18 02:26:32 I7800-R68-S7 notice tmm[15666]: 01010240:5: Syncookie HW mode activated, server name = /Common/wildcardCookie server IP = 10.10.20.212:80, HSB modId = 1 Oct 18 02:26:32 I7800-R68-S7 notice tmm[15666]: 01010240:5: Syncookie HW mode activated, server name = /Common/wildcardCookie server IP = 10.10.20.212:80, HSB modId = 2 Virtual SYN cache value is configured globally meaning that the configured value must be divided among TMMs to know when SYN cookie will be enabled on each TMM. Run below command to see physical number of cores: tmsh sho sys hard | grep core In this example device has 6 TMMs, so 1500/6 is 250. Note that you will see a warning message entry per TMM (I removed 3 log entries in above example order to summarize) and per HSB ID. Log does not always show the VIP’s IP, it depends on type of VIP. For example in below case: Oct 17 04:04:54 I7800-R68-S7 warning tmm2[22805]: 01010038:4: Syncookie counter 251 exceeded vip threshold 250 for virtual = 10.10.20.212:80 Oct 17 04:04:54 I7800-R68-S7 warning tmm3[22805]: 01010038:4: Syncookie counter 251 exceeded vip threshold 250 for virtual = 10.10.20.212:80 Oct 17 04:04:55 I7800-R68-S7 notice tmm2[22805]: 01010240:5: Syncookie HW mode activated, server name = /Common/wildcardCookie server IP = 10.10.20.212:80, HSB modId = 1 Oct 17 04:05:51 I7800-R68-S7 notice tmm2[22805]: 01010241:5: Syncookie HW mode exited,server name = /Common/wildcardCookie server IP = 10.10.20.212:80, HSB modId = 1 from HSB There is not any virtual configured with destination IP 10.10.20.212. In fact traffic is handled by a wildcard VIP listening on 0.0.0.0/0, this logged IP is the destination IP:Port in the request that triggered SYN Cookie. You can consider this IP as the most probable attacked IP since it was the one that exceeded the threshold, so you can assume there are more attacks to this IP, however attack could have a random destination IPs target. Important: Per-Virtual SYN Cookie threshold MUST be lower than Global threshold, if you configure Virtual Server threshold higher than Global, or 0, then internally BIG-IP will set SYN Cookie Global threshold equals to Per-Virtual SYN Cookie threshold. LTM SYN Cookie at VLAN context Configuration example for triggering LTM SYN Cookie at VLAN context: turboflex profile feature => adc tmsh list sys db pvasyncookies.enabled => true tmsh list ltm global-settings connection vlan-syn-cookie => enabled tmsh list net vlan hardware-syncookie => [vlan external: 2888] tmsh list ltm global-settings connection default-vs-syn-challenge-threshold => 0 tmsh list ltm global-settings connection global-syn-challenge-threshold => 2500 When SYN cookie is triggered you get log: Oct 17 10:27:23 I7800-R68-S7 notice tmm[15666]: 01010292:5: Hardware syncookie protection activated on VLAN 1160 (syncache:2916 syn flood pkt rate:0) In this case you will see that information related to virtual servers on this VLAN will show SYN cookie as ‘not activated’ because protection is at VLAN context: #tmsh show ltm virtual | grep ' status ' -i Statusnot-activated Statusnot-activated If you configure SYN Cookie per VLAN but Turboflex adc/security is not provisioned then you will get: Oct 17 04:39:52 I7800-R68-S7.sin.pslab.local warning mcpd[7643]: 01071859:4: Warning generated : This platform supports Neuron-based Syncookie protection on per VS basis (including wildcard virtual). Please use that feature instead AFM SYN Cookie at Global context Main different in AFM default log is that you will not get a message telling you the threshold it has been exceeded, instead log will inform you directly about the context that detected the attack. Configuration example for triggering AFM SYN Cookie at global context: turboflex profile feature=> security tmsh list ltm global-settings connection vlan-syn-cookie=> enabled tmsh list net vlan hardware-syncookie[not compatible with DoS device] tmsh list sys db pvasyncookies.enabled=> true tmsh list ltm global-settings connection default-vs-syn-challenge-threshold.=> 0 tmsh list ltm global-settings connection global-syn-challenge-threshold=> 2500 tmsh list security dos device-config default-internal-rate-limit (tcp-half-open)=> >2500 tmsh list security dos device-config detection-threshold-pps (tcp-half-open)=> 2500 tmsh list ltm profile fastl4 syn-cookie-enable=> enabled AFM Device DoS has preference over LTM Global SYN Cookie, so in above configuration AFM tcp half open vector will be triggered: Oct 19 02:23:41 I7800-R68-S7 err tmm[23288]: 01010252:3: A Enforced Device DOS attack start was detected for vector TCP half open, Attack ID 1213152658. Oct 19 02:29:23 I7800-R68-S7 notice tmm[23288]: 01010253:5: A Enforced Device DOS attack has stopped for vector TCP half open, Attack ID 1213152658. In the example above you can see that there are logs warning you about an attack that started and stopped, but there is not any log showing if attack is mitigated. This is because you have not configured AFM to log to local-syslog (/var/log/ltm). In this situation DoS logs are basic. If you want to see packets dropped or allowed you need to configure specific security log profile. Be aware that when SYN Cookie is active because Device TCP half open DoS vector’s threshold has been reached then you will not see any Virtual Server showing that SYN Cookie has been activated, as it happens when SYN Cookie VLAN is activated: SYN Cookies Statusnot-activated This is slightly different to LTM Global SYN Cookie, when LTM Global SYN Cookie is enabled BIG-IP will show which specific VIP has SYN Cookie activated (Status Full Hardware/Software). In case you have configured logging for DoS then you would get logs like these: Oct 23 03:56:15 I7800-R68-S7 err tmm[21638]: 01010252:3: A Enforced Device DOS attack start was detected for vector TCP half open, Attack ID 69679369. Oct 23 03:56:15 I7800-R68-S7 info tmm[21638]: 23003138 "Oct 23 2020 03:56:15","10.200.68.7","I7800-R68-S7.sin.pslab.local","Device","","","","","","","TCP half open","69679369","Attack Started","None","0","0","0000000000000000", "Enforced", "Volumetric, Aggregated across all SrcIP's, Device-Wide attack, metric:PPS" Oct 23 03:56:16 I7800-R68-S7 info tmm[21638]: 23003138 "Oct 23 2020 03:56:16","10.200.68.7","I7800-R68-S7.sin.pslab.local","Device","","","","","","","TCP half open","69679369","Attack Sampled","Drop","3023","43331","0000000000000000", "Enforced", "Volumetric, Aggregated across all SrcIP's, Device-Wide attack, metric:PPS" Oct 23 03:56:16 I7800-R68-S7 info tmm[21638]: 23003138 "Oct 23 2020 03:56:16","10.200.68.7","I7800-R68-S7.sin.pslab.local","Device","","","","","","","TCP half open","69679369","Attack Sampled","Drop","3017","69710","0000000000000000", "Enforced", "Volumetric, Aggregated across all SrcIP's, Device-Wide attack, metric:PPS” The meaning of below fields shown in above logs: "Drop","3023","43331","0000000000000000" "Drop","3017","69710","0000000000000000" Are as below: {action} {dos_packets_received} {dos_packets_dropped} {flow_id} Where: {dos_packets_received} - It counts the number of TCP SYNs received for which you have not received the ACK. Also called embryonic SYNs. {dos_packets_dropped} - It counts the number of TCP syncookies that you have sent for which you have not received valid ACK. If you see that {dos_packets_received}are high, but {dos_packets_dropped} are 0 or low, then it just means that AFM is responding to SYN packets with SYN cookies and it is receiving correct ACKs from client. Therefore, AFM is not dropping anything. So this could mean that this is not an attack but a traffic peak. It can happen that you configure a mitigation threshold lower than detection threshold, although you will get a message warning you, you could not realise about it. If this is the case you will not see any log informing you about that there is an attack. This will happen for example with below configuration: tmsh list ltm global-settings connection global-syn-challenge-threshold=> 3400 tmsh list security dos device-config default-internal-rate-limit (tcp-half-open)=> 3000 tmsh list security dos device-config detection-threshold-pps (tcp-half-open)=> 3900 tmsh list ltm profile fastl4 syn-cookie-enable=> disabled Due to this you will see in /var/log/ltm something like: Oct 23 03:38:12 I7800-R68-S7.sin.pslab.local warning mcpd[10516]: 01071859:4: Warning generated : DOS attack data (tcp-half-open): Since drop limit is less than detection limit, packets dropped below the detection limit rate will not be logged. AFM SYN Cookie at Virtual context All information provided in previous use case applies in here, so for below configuration example: tmsh list ltm global-settings connection global-syn-challenge-threshold=> 3400 tmsh list security dos device-config default-internal-rate-limit (tcp-half-open)=> 3000 tmsh list security dos device-config detection (tcp-half-open)=> 3900 list security dos profile SYNCookie dos-network default-internal-rate-limit (tcp-half-open)=> 2000 list security dos profile SYNCookie dos-network detection-threshold-pps (tcp-half-open)=> 1900 tmsh list ltm profile fastl4 <name>=> enabled AFM device SYN Cookie is activated for specific virtual server with security profile applied: Oct 23 04:10:26 I7800-R68-S7 notice tmm[21638]: 01010240:5: Syncookie HW mode activated, server = 0.0.0.0:0, HSB modId = 1 Oct 23 04:10:26 I7800-R68-S7 notice tmm5[21638]: 01010240:5: Syncookie HW mode activated, server = 0.0.0.0:0, HSB modId = 2 Oct 23 04:10:26 I7800-R68-S7 err tmm3[21638]: 01010252:3: A NETWORK /Common/SYNCookie_Test DOS attack start was detected for vector TCP half open, Attack ID 2147786126. Oct 23 04:10:28 I7800-R68-S7 info tmm[16005]: 23003156 "10.200.68.7","I7800-R68-S7.sin.pslab.local","Virtual Server","/Common/SYNCookie_Test","Cryptographic SYN Cookie","16973","0","0","0", Oct 23 04:10:57 I7800-R68-S7 notice tmm5[21638]: 01010253:5: A NETWORK /Common/SYNCookie_Test DOS attack has stopped for vector TCP half open, Attack ID 2147786126. Oct 23 04:12:46 I7800-R68-S7 notice tmm[21638]: 01010241:5: Syncookie HW mode exited,server = 0.0.0.0:0, HSB modId = 1 from HSB Oct 23 04:12:47 I7800-R68-S7 notice tmm5[21638]: 01010241:5: Syncookie HW mode exited,server = 0.0.0.0:0, HSB modId = 2 from HSB Troubleshooting steps When you need to troubleshoot how device is working against a SYN flood attack there are some steps you can follow. Check configuration to make a global idea of what should happen when SYN flood occurs: tmsh show sys turboflex profile feature tmsh list ltm global-settings connection vlan-syn-cookie tmsh list net vlan hardware-syncookie tmsh list sys db pvasyncookies.enabled tmsh list ltm global-settings connection default-vs-syn-challenge-threshold tmsh list ltm global-settings connection global-syn-challenge-threshold tmsh list ltm profile fastl4 syn-cookie-enable tmsh list ltm profile tcp all-properties | grep -E 'profile|syn-cookie' tmsh list ltm profile fastl4 all-properties| grep -E 'profile|syn-cookie' list security dos device-config syn-cookie-whitelist syn-cookie-dsr-flow-reset-by tscookie-vlans tmsh list security dos device-config dos-device-config | grep -A23 half tmsh list security dos profile dos-network {<profile> { network-attack-vector { tcp-half-open } } } *I can miss some commands since I cannot know specific configuration you are using, but above list can give you a good idea about what you have actually configured in your system. Are you using Hardware or Software SYN cookie? Are you using CMP or vCMP? Is device a Neuron platform? Is SYN cookie configured/working in AFM, in LTM or in both? Is SYN cookie enabled at Device, VLAN or Virtual Server context? If issue is at virtual server context, which virtual servers are affected? is the problem happening in a Standard or FastL4 VIP, …? Check logs (date/times) and stats to confirm what it has really happened and since when. Take captures to confirm your findings. Is this an attack? Were there other attacks at the same time (TCP BAD ACK, TCP RST maybe)? Are thresholds correctly configured attending to expected amount of traffic? If clients are hidden by a proxy maybe you could save resources by configuring Challenge and remember. If this is a Neuron platform, is there any error in /var/log/neurond? Check published IDs related to SYN Cookie for specific TMOS versions or/and platforms. Conclusion Now you have enough information to start troubleshooting your own BIG-IP devices if any issue happens, and also and maybe more important you have tools to create the most appropriate configuration attending to your specific platform and traffic patterns. So you can start to take the advantage of your knowledge to improve performance of your device when under TCP SYN flood attack.2. SYN Cookie: Operation
Introduction As concluded in the last article, in order to avoid allocating space for TCB, the attacked device needs to reject TCP SYN packets sent by clients. In this article I will explain how a system can do this without causing service disruption, and more specifically I will explain how this work in BIG-IP. Implementation Since attacked device need to alter default TCP 3WHS behaviour the best option is implementing SYN Cookie countermeasure in an intermediate device, so you offload your servers from this task. In this way if connection is not legitimate then it is just never forwarded to the backend server and therefore it will not waste any kind of extra resourcess. Since BIG-IP is already in charge of handling application traffic directed towards servers it is the best place to implement SYN Cookie. What BIG-IP does is adding an extra layer, which we can call SYN Cookie Agent, that basically implement a stateless buffer between client and BIG-IP TCP stack. This agent is in charge of handling client’s TCP SYN packets by modifying slightly the standard behaviour of TCP 3WHS, this modification comes in two flavours depending on what type of routing role BIG-IP is playing between client and server. Symmetric routing This is the typical situation. In this environment BIG-IP is sited in the middle of the TCP conversation and all the traffic from/to server pass through it. The SYN Cookie operation in this case is briefly described below: Client sends a TCP SYN packet to BIG-IP. BIG-IP uses a stateless buffer for answering each client SYN with a SYN/ACK. BIG-IP generates the 32 bits sequence number which will be included in SYN/ACK packet sent to the client. BIG-IPencodes essential and mandatory information of the connection in 24 bits. BIG-IP hashes the previous encoded information. BIG-IP also encodes other values like MSS in the remaining bits. BIG-IP generates the SYN/ACK response packet and includes the hash and other encoded information as sequence number for the packet. This is the so calledSYN Cookie. BIG-IP sends SYN/ACK to client. BIG-IP discards the SYN from the stateless buffer, in other words, BIG-IP removes all information related to this TCP connection. At this point no memory has been allocated for TCP connection (TCB). Client sends correct ACK to BIG-IP acknowledging BIG-IP’s sequence number. BIG-IP validates ACK. BIG-IP subtracts one to this ACK. BIG-IP runs the hash function using connection information as input (see point 3-b above). Then it compares it with the hash provided in the ACK, if they match then it means client sent a correct SYN Cookie response and that client is legitimate. BIG-IP uses connection information in the ACK TCP/IP headers to create an entry in connflow. At this time is when BIG-IP builds TCB entry, so it's the first time BIG-IP uses memory to save connection information. BIG-IP increase related SYN Cookie stats. BIG-IP starts and complete a TCP 3WHS with backend server on behalf of the client since we are sure that client is legitimate. Now traffic for the connection is proxied by BIG-IP as usual attending to configured L4 profile in the virtual server. *If ACK packet received by BIG-IP is a spurious ACK then BIG-IP will discard it and no entry will be created in connection table. Since attackers will never send a correct response to a SYN/ACK then you can be sure that TCB entries are never created for them. Only legitimate clients will use BIG-IP resources as shown in diagram: Fig4. TCP SYN Flood attack with SYN Cookie countermeasure Asymmetric routing In an asymmetric environment, also called nPath or DSR, you face a different problem because Big-IP cannot establish a direct TCP 3WHS with the server (point 11 in last section). As you can see in below diagram SYN/ACK packet sent from server to client would not traverse Big-IP, so method used for symmetric routing cannot be used in this case. You need another way to trust in clients and discard them as attackers, so clients then can complete TCP 3WHS directly with the server. Fig5. TCP state diagram section for asymmetric routing + FastL4 In order to circumvent this problem BIG-IP takes the advantage of the fact that applications will try to start a second TCP connection if a first TCP 3WHS is RST by the server. What BIG-IP does in asymmetric routing environments is completing the TCP three way handshake with client, issuing SYN Cookie as I described for symmetric routing, and once BIG-IP confirms that client is trustworthy it will add its IP to SYN Cookie Whitelist, it will send a RST to the client and it will close the TCP connection. At this point client will try to start a new connection and this time BIG-IP will let the client talk directly to the server as it would do under normal circumstances (see diagram above). The process is briefly described below: Client sends a TCP SYN packet to BIG-IP. BIG-IP generates the sequence number as explained for symmetric routing. BIG-IP generates the SYN/ACK response packet and includes the calculated sequence number for the packet. BIG-IP sends SYN/ACK to client and then remove all information related to this TCP connection. At this point no memory has been allocated for TCP connection (TCB). Client sends correct ACK to BIG-IP acknowledging sequence number. BIG-IP validates ACK. BIG-IP substract one to this ACK so it can decode needed information and check hash. BIG-IP increase related SYN Cookie stats. BIG-IP sends a RST to client and discards TCP connection. Big-IP adds client’s IP to Whitelist. Clients starts a new TCP connection. BIG-IP lets client to start this new TCP connection directly to the server since it knows that server is legitimate. *If ACK packet received by BIG-IP is a spurious ACK then BIG-IP will discard it and no entry will be created in connection table. Fig6. TCP 3WHS flow in asymmetric routing DSR whitelist has some important characteristics you need to take into account: By default IP is added 30 seconds to the whitelist (DB Key tm.flowstate.timeout). Client’s IP is added to whitelist for 30 seconds but only if there is no traffic from this client, if BIG-IP have already seen ACK from this client then its IP is removed from whitelist since connection has been already established between client and server. Whitelist is common to all virtual servers, so if a client is whitelisted it will be for all applications. Whitelist it is not mirrored. Whitelist it is not shared among blades. SYN Cookie Challenge When a TCP connection is initiated the TCP SYN packet sent by the client specify certain values that define the connection, some of these values are mandatory, like source and destination IP and port, otherwise you would not be able to identify the correct connection when packets arrive. Some other values are optional and they are used to improve performance, like TCP options or WAN optimizations. Under normal circumstances, upon TCP SYN reception the system creates a TCB entry where all this information is saved. BIG-IP will use this information to set up the TCP connection with the backend server. The problem we face is that when SYN Cookie is in play device does not create the TCB entry, only a limited piece of connection information is collected, so the information that BIG-IP has about the TCP connection is limited. Remember that SYN Cookie is not handled by TCP stack but by the stateless SYN Cookie Agent, so we cannot save the connection details. What BIG-IP does instead is encoding key information in 24 bits, this left only some bits for the rest of data. While there is no room for values like Window Scale information, however we have space for other values, for example 3 bits reserved for MSS value. This limits MSS possible values to 8, the most commonly used values, and the one chosen will the nearest to the original MSS value. However, note that MSS limitation has a workaround through configuration in FastL4 profile that it can help in some cases. The option SYN Cookie MSS in this profile specifies a value that overrides the SYN cookie maximum segment size (MSS) value in the SYN-ACK packet that is returned to the client. Valid values are 0, and values from 256 through 9162. The default is 0, which means no override. You might use this option if backend servers use a different MSS value for SYN cookies than the BIG-IP system does. If this is not enough for some customers, BIG-IP overcomes this problem by using TCP timestamp space to save extra information about TCP connection, in other words we create a second extra SYN Cookie. This space is used to record the client and server side Window Scale values, or the SACK info which are then made available to the TCP stack via this cookie if the connection is accepted. Note that TCP connection will only be accepted if both SYN Cookies (standard and Timestamp) are correct. So all this means that if the client starts a connection with TCP TS set then Big-IP will have more space to encode information and hence the performance will be improved when under TCP SYN flood attack. NOTE: SYN Cookie Timestamp extension (software or hardware) only work for standard virtual servers currently. FastL4 virtual servers only supports MMS option in SYN Cookie mode. Conclusion Now you know how SYN Cookie works under the hoods. In next article I will describe when SYN Cookie is activated and I will give specific details of BIG-IP SYN Cookie operation. Note that limitations related to SYN Cookie commented in this article are not caused specifically by BIG-IP SYN Cookie implementation but by SYN Cookie standard itself. In fact, F5 Networks implements some improvements on SYN Cookie to get a better performance and provide some extra features.7. SYN Cookie: Troubleshooting Stats
Introduction In this article I will explain what SYN Cookie stats you can consult and their meaning. There are more complex stats than the explained in this article but they are intended for helping F5 engineers when cases become complex. SYN Cookie stats First at all, in order to troubleshoot SYN Cookie you need to know how you can check SYN Cookie stats easily and understand what you are reading. The easiest way to see these stats in a device running LTM module based SYN Cookie is by running below command and focusing in SYN Cookies section of the output: # tmsh show ltm virtual <virtual> SYN Cookies Statusfull-hardware Hardware SYN Cookie Instances6 Software SYN Cookie Instances0 Current SYN Cache2.0K SYN Cache Overflow24 Total Software4.3M Total Software Accepted0 Total Software Rejected0 Total Hardware21.7M Total Hardware Accepted3 Let’s go through each field to know what they means: Status: [full-software/full-hardware|not-activated]. This value describes what type of SYN Cookie mode has been activated, software or hardware. Once an attack has finished it is normal that LTM takes some time to deactivate SYN Cookie mode after attack traffic stops. Calculation is complex and related to specific platform and several factors, also we do not want SYN Cookie entering in an activating/deactivating loop in case TCP SYN packets per second reaching device are near to the configured SYN cache threshold. So delay of 30-60 seconds before SYN Cookie being deactivated is in normal range. Hardware SYN Cookie Instances: How many TMMs are under Hardware SYN Cookie mode. Software SYN Cookie Instances: How many TMMs are under Software SYN Cookie mode. It indicates if software is currently generating SYN cookies. Current SYN Cache: Indicates how many embryonic connections are handled by BIG-IP (refreshed every 2 seconds). SYN cache is always counting embryonic connections, regardless if SYN Cookie is activated or not. So even if SYN Cookies is completely turned off, we still have embryonic flows that have not been promoted to full flows yet (SYNs which has not completed 3WHS), this means that cache counter will still be used regularly, so is normal having a value different than 0. Disabling SYN Cookie will not avoid this counter increase but thresholds will not be taken into account. Since SYN cache is no longer a cache, as explained in prior articles, and the stat is merely a counter of embryonic flows, it does not consume memory resources. As the embryonic flows are promoted or time out, this value will decrease. SYN Cache Overflow (per TMM): It is incremented whenever the SYN cache threshold is exceeded and SYN cookies need to be generated. It increments in one per each TMM instance and this value is a counter that only increases, so we can consult the value to know how many times SYN Cookie has been activated since last stats reset, remember, per TMM. Total Software:Number of challenges generated by Software. This is increased regardless if client sends a response, does not send any response or the response is not correct. Total Software Accepted:Number of TCP handshakes that were correctly handled with clients. Total Software Rejected:Number of wrong responses to challenges. Remember that ALL rejected SYN cookies are in software, there is no hardware rejected. Total Hardware: Number of challenges generated by Hardware. Total Hardware Accepted:Number of TCP handshakes that were correctly handled with clients. In case you have configured AFM based SYN Cookie then you can use two easy sources of information, the already explained LTM command above, but also you can check stats for TCP half open DoS vector, at device or virtual server context, as you would do with any other DoS vector. Let’s check the most important fields at device context. # tmsh show security dos device-config | grep -A 40 half Statistics TypeCount StatusReady Attack Detected1 *Attacked Dst Detected0 *Bad Actor Attack Detected0 Aggregate Attack Detected1 Attack Count2 *Stats 1h Samples0 Stats408 *Stats Rate408 Stats 1m104 Stats 1h0 Drops1063 *Drops Rate1063 Drops 1m187 Drops 1h4 *Int Drops0 *Int Drops Rate0 *Int Drops 1m0 *Int Drops 1h0 Status: This field confirm if this specific DoS vector is ready to detect TCP SYN flood attacks. You have to take into account that you could decide to configure this DoS vector with Auto-threshold, in which case AFM would be in charge of deciding the best threshold. Note that by enabling auto-threshold AFM would need some time to learn the traffic pattern of your environment, until it has enough information to create a correct threshold you will not see this field as Ready but as Learning. If vector is configured manually you always see it as Ready. Attack detected: It informs you if currently there is an ongoing attack and detected by AFM. Aggregate attack detected: Since DoS stats shown above are for device context the detected attack is aggregate. Attack Count: Gives information about how many attacks have been detected since stats were reset last time. It does not decrease. Stats: Number of embryonic connections at this current second. Remember that since this is a snapshot, the counter could go increase or decrease. This is different to other DoS vectors where it only increases. Stats 1m: Average of the Stats in the last minute. This is the average number ofembryonic connections that AFM has seen when taking sampling every 1s. Stats 1h: Average of the Stats 1m in the last hour. Drops: It counts the number of wrong ACKs received. In other words this is the current snapshot of: Number of SYN Cookies – Number of validated ACKs received Drops 1m: Average of the Drops in the last minute Drops 1h: Average of the Drops 1m in the last hour I have added an asterisk to some fields to point to values that has not real meaning for SYN Cookie DoS vector, or information is not really useful from my perspective, but they are included to have coherence with other DoS attack stats information. Note: Detection logic for this vector is not based on the Stats 1m, as other DoS vectors, instead it is based on the current number of embryonic connections, that is, value seen in Stats counter. Interpreting stats Once you know what each important stat mean I will give some advises when you interpret SYN Cookie stats. You should know some of them already if you have read all articles in this series: Hardware can offload TMM for validation, so you can see a number of software generated SYN Cookies much bigger than software validated SYN Cookie since software generates cookies that are validated by hardware as well. Since it is possible that a software generated SYN Cookie be accepted by hardware and vice versa, hardware generated SYN Cookies be accepted by software, you could think that value for Total Software is the same than the result of adding the Total Software Accepted plus Total Software rejected. But that is NOT true since it can be possible that a SYN Cookie sent by BIG-IP does not have a response (ACK), quite typical during a SYN flood attack indeed. In this case nothing adds up because generated SYN Cookie was not accepted nor rejected. Remember that SYN cookie’s responses discarded by hardware will be rejected by software, so all rejects are in software. This is why there is not counter for Total Hardware rejected. This means that Total Software Rejected can be increase by Total Hardware. When there is an attack vectors definition that match this attack will be increased. So, for example, during a TCP SYN flood attack you will see that Stats increases for TCP half open and TCP SYN flood (maybe others like low TTL,... as well). Also Stats will increase in all contexts (device and virtual server). The first vector in an specific context whose limit is exceeded it will start to mitigate. This is important because in cases where TCP SYN flood DoS vector has a lower threshold than TCP half open DoS vector, you will notice that traffic is dropped but you did not expect this behavior. Check article 5 for more information. Example In this part you will learn what stats changes you should expect to see when a TCP SYN flood attack is detected and SYN Cookie is activated, so it starts to mitigate the attack.Let’s interpret below stats: During attack Statusfull-hardware Hardware SYN Cookie Instances6 Software SYN Cookie Instances0 Current SYN Cache1 SYN Cache Overflow24 Total Software10.1K Total Software Accepted0 Total Software Rejected0 Total Hardware165.1M Total Hardware Accepted3 After attack Statusnot-activated Hardware SYN Cookie Instances0 Software SYN Cookie Instances0 Current SYN Cache15 SYN Cache Overflow24 Total Software10.1K Total Software Accepted0 Total Software Rejected0 Total Hardware171.0M Total Hardware Accepted3 Before the attack, before SYN Cookie is activated, you will see Current SYN Cache stats starts to increase quickly since TCP SYN packets are causing an increment of embryonic connections. Once SYN Cache threshold has been reached in a TMM then SYN Cache Overflow will increase attending to the number of TMMs that detected the attack. In above example you see 24 because this is a 6 CPUs device and 4 TCP SYN Flood attacks were detected by SYN Cookie. Remember this counter will never decrease unless we reset stats. During attack SYN cookie is activated so Current SYN Cache will start to decrease until reaching 0 because SYN Cookie Agent starts to handle TCP 3WHS, in other words, TCP stack stops to receive TCP SYN packets. We can check how many TMMs have SYN cookie activated currently looking at Hardware/Software SYN Cookie Instances counter. Note this match with what I explained for SYN Cache Overflow value. Legitimate connections are counted under Total Hardware/Software Accepted. In this case, we can see that although several millions of TCP SYN packets reached the device only 3 TCP 3WHS were correctly carried out. As you can see this device has Hardware SYN Cookie configured and working, but you can read that software also generates SYN Cookie challenges (Total Software). As commented in article number six of these series, this is expected and can be due to collisions or due to validations of first challenges when SYN Cookie is activated and TMMs handles TCP SYN packets until it enables hardware to do it. This does not affect device performance. In order to change from ‘Status full-hardware’ to ‘Status not-activated’ both HSBs must exit from SYN Cookie mode. Conclusion Now you know how to interpret stats, so you know can deduce information about the past and the present status of your device related to SYN Cookie. In next two article I will end up this series and this troubleshooting part talking about traffic captures and meaning of logs.8. SYN Cookie: Troubleshooting tcpdump
Introduction In this troubleshooting article, I will explain what traffic is expected to collect in a tcpdump capture during an ongoing attack. In this way you can compare with your own environment and understand why you see what you see. Use cases In this section I will show you the most common SYN Cookie configurations and how they affect to the traffic that reach BIG-IP. For each use case I add a diagram explaining the role of each party, so you can use it as a rerefence when reviewing captures. Note that, as a rule of thumb, in tcpdump captures you will see only traffic reaching red TMM arrow in diagrams. FastL4 + Hardware SYN Cookie Fig15. FastL4 + HW SYN Cookie When enabling SYN Cookie hardware offloading in FastL4 you must have into account that, regardless if FastL4 profile is configured as EST or SYN for ‘PVA TCP Offload State’ the capture will be similar to the one I paste below. Also note that colours in captures match with the diagram above, that is, blue (on the left) for client, red (next to blue and green) for BIG-IP and green (on the right) for server. BLUE: 10:55:30.650614 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [S], seq 1486117989, win 5840, options [mss 1460,sackOK,TS val 257012221 ecr 0,nop,wscale 6], length 0 10:55:30.651084 IP 10.10.20.212.80 > 10.10.10.10.33669: Flags [S.], seq 2770916048, ack 1486117990, win 4380, options [mss 1460], length 0 10:55:30.651115 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [.], ack 1, win 5840, length 0 10:55:30.651271 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 5840, length 167 RED: 10:48:27.666478 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [.], ack 2770916049, win 5840, length 0 in slot1/tmm3 lis= 10:48:27.666793 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [P.], seq 0:167, ack 1, win 5840, length 167: HTTP: GET /zarapito HTTP/1.1 in slot1/tmm3 lis=/Common/fwd 10:48:27.667073 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [S], seq 1486117989, win 5840, options [mss 1460], length 0 out slot1/tmm3 lis=/Common/fwd 10:48:27.667245 IP 10.10.20.212.80 > 10.10.10.10.33669: Flags [S.], seq 3169243726, ack 1486117990, win 64240, options [mss 1460], length 0 in slot1/tmm3 lis=/Common/fwd 10:48:27.667586 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [.], ack 1, win 5840, length 0 out slot1/tmm3 lis=/Common/fwd 10:48:27.667588 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 5840, length 167: HTTP: GET /zarapito HTTP/1.1 out slot1/tmm3 lis=/Common/fwd GREEN: 09:54:39.236239 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [S], seq 1486117989, win 5840, options [mss 1460], length 0 09:54:39.236264 IP 10.10.20.212.80 > 10.10.10.10.33669: Flags [S.], seq 3169243726, ack 1486117990, win 64240, options [mss 1460], length 0 09:54:39.236692 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [.], ack 1, win 5840, length 0 09:54:39.236707 IP 10.10.10.10.33669 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 5840, length 167: HTTP: GET /zarapito HTTP/1.1 09:54:39.236728 IP 10.10.20.212.80 > 10.10.10.10.33669: Flags [.], ack 168, win 64073, length 0 There are three important things you can observe in above captures: TCP options are lost in this example. This is because SYN Cookie is offloaded in hardware and for this mode currently only MSS value is trasfered from client side to server side, with some limitations. Check article dedicated to 'Hardware Vs Software' in this article series. In BIG-IP you cannot see the first two packets of TCP 3WHS (SYN and SYN/ACK). This is because hardware is in charge of validating SYN Cookie, since tcpdump is capturing traffic in TMM you only see last ACK packet. Also note that SYN sequence number sent by client to BIG-IP is the same than the one BIG-IP sends to Server. FastL4 + Software SYN Cookie Fig16. FastL4 + SW SYN Cookie In this case, although you are using an accelerated profile (FastL4), since software is handling the TCP 3WHS this allows you to see all packets when you run a traffic capture. Blue is on the left, red in the middle, green on the right. BLUE: 17:32:15.588706 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [S], seq 2986408498, win 5840, options [mss 1460,sackOK,TS val 262963455 ecr 0,nop,wscale 6], length 0 17:32:15.588973 IP 10.10.20.212.80 > 10.10.10.10.34527: Flags [S.], seq 2739000888, ack 2986408499, win 4380, options [mss 1460], length 0 17:32:15.588996 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [.], ack 1, win 5840, length 0 17:32:15.589119 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 5840, length 167 RED: 17:25:11.566071 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [S], seq 2986408498, win 5840, options [mss 1460,sackOK,TS val 262963455 ecr 0,nop,wscale 6], length 0 in slot1/tmm3 lis= 17:25:11.566095 IP 10.10.20.212.80 > 10.10.10.10.34527: Flags [S.], seq 2739000888, ack 2986408499, win 4380, options [mss 1460], length 0 out slot1/tmm3 lis=/Common/fwd 17:25:11.566321 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [.], ack 1, win 5840, length 0 in slot1/tmm3 lis= 17:25:11.566451 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 5840, length 167: HTTP: GET / HTTP/1.1 in slot1/tmm3 lis=/Common/fwd 17:25:11.566834 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [S], seq 2986408498, win 5840, options [mss 1460], length 0 out slot1/tmm3 lis=/Common/fwd 17:25:11.567058 IP 10.10.20.212.80 > 10.10.10.10.34527: Flags [S.], seq 4131626625, ack 2986408499, win 64240, options [mss 1460], length 0 in slot1/tmm3 lis=/Common/fwd 17:25:11.567344 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [.], ack 1, win 5840, length 0 out slot1/tmm3 lis=/Common/fwd 17:25:11.567346 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 5840, length 167: HTTP: GET / HTTP/1.1 out slot1/tmm3 lis=/Common/fwd GREEN: 16:31:24.339490 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [S], seq 2986408498, win 5840, options [mss 1460], length 0 16:31:24.339551 IP 10.10.20.212.80 > 10.10.10.10.34527: Flags [S.], seq 4131626625, ack 2986408499, win 64240, options [mss 1460], length 0 16:31:24.339987 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [.], ack 1, win 5840, length 0 16:31:24.340003 IP 10.10.10.10.34527 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 5840, length 167: HTTP: GET / HTTP/1.1 As it happened in the previous example, BIG-IP does not ACK the HTTP request (PSH) sent by client because a FastL4 profile is being used, what it means that BIG-IP will act as a TCP transparent proxy between server and client. In other words, once correct TCP handshake has done BIG-IP will forward rest of this connection traffic to server directly. Standard + Hardware SYN Cookie Fig17. Standard + Hardware SYN Cookie The main difference in this case is that for Standard virtual server BIG-IP must act as a TCP proxy between client and server during all connection time. This means that BIG-IP sends ACK as response to PSH packet sent by client before this PSH is in fact received by the server. This image has the same color scheme as the first image. BLUE: 14:02:03.606740 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [S], seq 2828374030, win 5840, options [mss 1460,sackOK,TS val 259810460 ecr 0,nop,wscale 6], length 0 14:02:03.606861 IP 10.10.20.212.80 > 10.10.10.10.34522: Flags [S.], seq 2971392569, ack 2828374031, win 4380, options [mss 1460,sackOK,TS val 870079512 ecr 259810460], length 0 14:02:03.606885 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [.], ack 1, win 5840, options [nop,nop,TS val 259810460 ecr 870079512], length 0 14:02:03.607176 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 5840, options [nop,nop,TS val 259810460 ecr 870079512], length 167 RED: 13:55:00.128183 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [.], ack 2971392570, win 5840, options [nop,nop,TS val 259810460 ecr 870079512], length 0 in slot1/tmm4 lis= 13:55:00.128272 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [S], seq 2348737573, win 14600, options [mss 1460,nop,wscale 0,sackOK,TS val 1607698719 ecr 0], length 0 out slot1/tmm4 lis=/Common/std 13:55:00.128496 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [P.], seq 0:167, ack 1, win 5840, options [nop,nop,TS val 259810460 ecr 870079512], length 167: HTTP: GET / HTTP/1.1 in slot1/tmm4 lis=/Common/std 13:55:00.128501 IP 10.10.20.212.80 > 10.10.10.10.34522: Flags [.], ack 167, win 14767, options [nop,nop,TS val 870079512 ecr 259810460], length 0 out slot1/tmm4 lis=/Common/std 13:55:00.128559 IP 10.10.20.212.80 > 10.10.10.10.34522: Flags [S.], seq 1953609172, ack 2348737574, win 65160, options [mss 1460,sackOK,TS val 1991641654 ecr 1607698719,nop,wscale 7], length 0 in slot1/tmm4 lis=/Common/std 13:55:00.128564 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [.], ack 1, win 14600, options [nop,nop,TS val 1607698719 ecr 1991641654], length 0 out slot1/tmm4 lis=/Common/std 13:55:00.128567 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 14600, options [nop,nop,TS val 1607698719 ecr 1991641654], length 167: HTTP: GET / HTTP/1.1 out slot1/tmm4 lis=/Common/std GREEN: 13:01:12.268012 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [S], seq 2348737573, win 14600, options [mss 1460,nop,wscale 0,sackOK,TS val 1607698719 ecr 0], length 0 13:01:12.268103 IP 10.10.20.212.80 > 10.10.10.10.34522: Flags [S.], seq 1953609172, ack 2348737574, win 65160, options [mss 1460,sackOK,TS val 1991641654 ecr 1607698719,nop,wscale 7], length 0 13:01:12.268347 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [.], ack 1, win 14600, options [nop,nop,TS val 1607698719 ecr 1991641654], length 0 13:01:12.268355 IP 10.10.10.10.34522 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 14600, options [nop,nop,TS val 1607698719 ecr 1991641654], length 167: HTTP: GET / HTTP/1.1 Note that SYN sequence number sent by client to BIG-IP is different than the one BIG-IP sends to Server (2828374030 Vs 2348737573). This is because we are using Standard virtual server, so there are two independent TCP connections, one between client and BIG-IP and the second between BigIP and server. Again, as commented since in this example Hardware SYN Cookie is used we cannot see first two TCP handshake packets in the capture. Standard + Software SYN Cookie Fig18. Standard + Software SYN Cookie Apparently there are not any difference when capturing traffic of a client request when SYN Cookie is activated or disabled in this case. Only difference is that BIG-IP creates a SYN Cookie challenge and it adds it as a sequence number in SYN/ACK, but looking to capture this change is not clearly visible. We need to compare with log and stats to confirm that in fact SYN Cookie is enabled, also depending in config we can check as well TCP options passed to server side. Color scheme here is the same as the second image. BLUE: 14:32:58.966168 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [S], seq 3344434108, win 5840, options [mss 1460,sackOK,TS val 260274299 ecr 0,nop,wscale 6], length 0 14:32:58.966358 IP 10.10.20.212.80 > 10.10.10.10.34525: Flags [S.], seq 2686010241, ack 3344434109, win 14600, options [mss 1460,nop,wscale 0,sackOK,TS val 3273182234 ecr 260274299], length 0 14:32:58.966398 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [.], ack 1, win 92, options [nop,nop,TS val 260274299 ecr 3273182234], length 0 14:32:58.966649 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 92, options [nop,nop,TS val 260274300 ecr 3273182234], length 167 RED: 14:25:55.405917 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [S], seq 3344434108, win 5840, options [mss 1460,sackOK,TS val 260274299 ecr 0,nop,wscale 6], length 0 in slot1/tmm1 lis= 14:25:55.405931 IP 10.10.20.212.80 > 10.10.10.10.34525: Flags [S.], seq 2686010241, ack 3344434109, win 14600, options [mss 1460,nop,wscale 0,sackOK,TS val 3273182234 ecr 260274299], length 0 out slot1/tmm1 lis=/Common/std 14:25:55.406048 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [.], ack 1, win 92, options [nop,nop,TS val 260274299 ecr 3273182234], length 0 in slot1/tmm1 lis= 14:25:55.406355 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 92, options [nop,nop,TS val 260274300 ecr 3273182234], length 167: HTTP: GET / HTTP/1.1 in slot1/tmm1 lis=/Common/std 14:25:55.406359 IP 10.10.20.212.80 > 10.10.10.10.34525: Flags [.], ack 168, win 14767, options [nop,nop,TS val 3273182234 ecr 260274300], length 0 out slot1/tmm1 lis=/Common/std 14:25:55.406131 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [S], seq 2235159701, win 14600, options [mss 1460,nop,wscale 0,sackOK,TS val 1609553998 ecr 0], length 0 out slot1/tmm1 lis=/Common/std 14:25:55.406524 IP 10.10.20.212.80 > 10.10.10.10.34525: Flags [S.], seq 3205875957, ack 2235159702, win 65160, options [mss 1460,sackOK,TS val 1993497007 ecr 1609553998,nop,wscale 7], length 0 in slot1/tmm1 lis=/Common/std 14:25:55.406527 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [.], ack 1, win 14600, options [nop,nop,TS val 1609553998 ecr 1993497007], length 0 out slot1/tmm1 lis=/Common/std 14:25:55.406531 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 14600, options [nop,nop,TS val 1609553998 ecr 1993497007], length 167: HTTP: GET / HTTP/1.1 out slot1/tmm1 lis=/Common/std GREEN: 13:32:07.645733 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [S], seq 2235159701, win 14600, options [mss 1460,nop,wscale 0,sackOK,TS val 1609553998 ecr 0], length 0 13:32:07.645820 IP 10.10.20.212.80 > 10.10.10.10.34525: Flags [S.], seq 3205875957, ack 2235159702, win 65160, options [mss 1460,sackOK,TS val 1993497007 ecr 1609553998,nop,wscale 7], length 0 13:32:07.646011 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [.], ack 1, win 14600, options [nop,nop,TS val 1609553998 ecr 1993497007], length 0 13:32:07.646022 IP 10.10.10.10.34525 > 10.10.20.212.80: Flags [P.], seq 1:168, ack 1, win 14600, options [nop,nop,TS val 1609553998 ecr 1993497007], length 167: HTTP: GET / HTTP/1.1 Note that, as in the previous example, SYN sequence number sent by client to BIG-IP is different than the one BIG-IP sends to Server (3344434108 Vs 2235159701). Conclusion At this point only one part left to have a complete picture of BIG-IP's SYN Cookie countermeasure functioning, so in next article I finish this series giving details about expected logs when SYN Cookie is working in our device.6. SYN Cookie: Hardware vs Software
Introduction Currently, you know the differences between LTM and AFM when talking about SYN Cookie capabilities and configuration. In this article I describe how SYN Cookie can perform its tasks using two different ways, in software or offloaded in hardware. The most important difference between these functioning modes is clearly the resource consuming. In hardware based platforms SYN Cookie is a countermeasure that will not consume extra memory nor CPU, while in software this task must be handled by TMM, so some load could be added comparing to normal operation. Hardware SYN Cookie If you have a hardware offloading capable platform and you configure SYN Cookie to work in hardware, the creation and validation of SYN Cookie challenges will be managed by dedicated FPGA (Programmable Field Array) or NSP (Neuron Search Processor) In a few words this means that SYN Cookie running in hardware basically allows you to forget about TCP SYN flood attacks. You do not have to care about it, if it is correctly configured, since BIG-IP will not only mitigate attack without impacting legitimate users, who will still access to application during the attack, but also no extra resources will be needed. The only handicap you have to take into account currently when hardware SYN Cookie is activated is that you cannot keep TCP options information. In order to keep this information you need to enable SYN Cookie in software and client connection need to activate TCP TS option (this is enabled by default in BIG-IP). See second article in this series for more details (SYN Cookie Operation). Neuron Platforms shipped with Neuron chip and running a TMOS version greater than 14, improve performance because this chip, which is directly connected to HSBe2, provide with extra functionalities to SYN Cookie. Neuron improves performance and solves limitations that you could find in HSBe2 only platforms when SYN Cookie is activated for wildcard virtual servers. A brief description of this limitation can be found in K50955355. Note that Neuron does not only offloads SYN Cookie tasks from software, if you have an AFM provisioned device you can take advantage of other features like extended whitelists and blacklists in hardware. However these article series are only focused on SYN Cookie. With this in mind the functioning of Neuron is simple, HSBe2 makes requests to Neuron when a TCP SYN reaches BIG-IP, and depending of the type of request Neuron then will create a SYN Cookie for the connection or it will validate SYN Cookie response presented by client. For example, as I showed in previous articles, when an ACK comes from a client then HSBe2 will ask Neuron if SYN Cookie is correct, then Neuron will response with the requested information, so TMM will know if it must drop or allow the connection through the datapath. SYN Cookie Neuron does not offload AFM Global SYN Cookie, it works for: LTM Global SYN Cookie Per VLAN SYN Cookie LTM Per Virtual SYN Cookie AFM Per Virtual SYN Cookie In order to enable SYN Cookie Neuron in your platform you need to activate the Turboflex profile for Security/Securityv1 (AFM) or ADC (LTM). There is many literature in AskF5 about Turboflex, but in a few words it allows you to group several related features to be accelerated in hardware. I will not give technical details about Neuron since it is not the intention of these article series. For troubleshooting issues with Neuron please raise a case to F5 Networks, so expert engineer can investigate it. Making internal changes to Neuron could lead to a worst result. Disabling SYN Cookie SYN Cookie is enabled by default in software and hardware, if we disable hardware SYN Cookie then software SYN Cookie comes into play. There is a DB key for disabling Hardware SYN Cookie in all contexts: tmsh list sys db pvasyncookies.enabled sys db pvasyncookies.enabled { value "false" } Of course, if the specific platform cannot offload SYN Cookie into hardware then above DB key has no effect. In this case software SYN Cookie is enabled by default. There is not such DB key for enabling/disabling software SYN Cookie as we have for hardware, instead, if you want to disable software SYN Cookie then you have different options depending on context that you can consult in table below. Remember that AFM SYN Cookie has precedence over LTM SYN Cookie, this means that if AFM SYN Cookie is configured and you want to disable completely SYN Cookie you need to disable both, AFM and LTM SYN Cookie. The simplest path to disable completely SYN Cookie assuming default config, that is, hardware SYN Cookie is enabled, it would be: Fig14. Disabling SYN Cookie In table below I summarize how to disable hardware and software SYN Cookie in each context for the different scopes. For AFM disabling SYN Cookie is quite easier, you just need to change DoS Device and DoS profiles TCP Half Open vector to 0/Infinite. Warnings It can be possible that SYN Cookie works in software in your device whilst you expected SYN Cookie working in hardware. Sometimes you end up in this situation due to a wrong configuration. Below I make a list of possible reasons that you can check: You have disabled autolasthop. This option must be enabled if you want to offload SYN Cookie into hardware (K16887). You have changed DB key connection.syncookies.algorithm manually. This key defines which algorithm is used for generating hash that is part of the SYN Cookie challenge (as I described in the first article of this series). When set a value of Hardware to this DB key the error detecting code used for validating the hash will be understood by both, software and hardware. If set to Software then it will only be understood by software. If your version also has the option ‘both’, this means that BIG-IP dynamically discovers which algorithm should be used rather than forcing to Software or Hardware. In summary, if virtual is configured for hardware SYN Cookie but algorithm is configured in software then hardware will inspect SYN Cookies, it will confirm it cannot validate them and then it will send it to TMM for validation. Until v14.1 SYN cookie hardware it is not recommended for protecting networks (wildcard virtual servers), this is because SYN Cookie will only protect a single flood network destination when flooding towards multiple network destination at a time. The other networks will be protected by software SYN cookie. This could cause unexpected extra CPU utilization. This is greatly improved in Neuron capable platforms. If there are collisions when SYN Cookie is configured in hardware you will still see non zero stats for software SYN Cookie. Also, these stats can increase due to validation of first challenges, when SYN Cookie is activated, and TMMs handles TCP SYN until it enables hardware to do it. This is expected and it does not mean that hardware SYN Cookie is not working. I will show examples in article dedicated to stats. In order to end up this section with clear ideas, remember that hardware SYN Cookie and hardware flow acceleration are two different concepts. Conclusion In this article you learnt how behaves SYN Cookie when working in software and hardware and the advantages or disadvantages. In next two articles I will show you how to troubleshoot SYN Cookie issues.5. SYN Cookie: LTM Vs AFM
Introduction At this point I have covered SYN Cookie from LTM perspective, in this article I will explain the important differences between LTM and AFM SYN Cookie. LTM Vs AFM AFM is a security platform, so it adds some extra options when configuring SYN Cookie that you cannot find in LTM. These additional abilities are added to AFM because in AFM SYN Cookie is just another DoS vector, so this allows you to configure characteristics reserved for DoS vectors. In fact AFM SYN Cookie is configured as a DoS vector called TCP Half Open. It is very important to note that when LTM and AFM SYN Cookie are configured at the same time in your BIG-IP then AFM SYN Cookie will work since it has precedence: Fig12. AFM Vs LTM However, before starting describing SYN Cookie in AFM be aware that comparing to other DoS vectors TCP half open has a couple of limitations: -Blacklisting is not allowed. -TCP SYN flood attackers handled by SYN Cookie are not included in IPI You have learnedmain details about LTM SYN Cookie, now the best way to see the advantages of AFM is by listing them: One of the most important advantages of AFM SYN Cookie is that it allows more granularity than LTM SYN Cookie since we can define a different threshold for each virtual server. This means that you can configure different SYN Cookie thresholds for each virtual server, for each VLAN and globally for the device. Remember that LTM SYN cookie only allows us to define a sole threshold for all virtual servers. Fig13. AFM granularity AFM handles SYN Cookie as another DoS vector, so administration is easier since you do not need to create parallel configurations for different types of countermeasures. AFM whitelist is unified, there is only one whitelist for DoS and SYN cookie. This ease the security administration. TCP Half Open DoS vector also allows you to use Auto-Threshold, so you can rely on AFM in order to define the best threshold attending to your environment, after a learning period of course. AFM provides extra information about SYN Cookie in form of events and stats. Logging As I pointed out one of the important advantages of AFM is that it provides extra information about SYN Cookie. In order to get this information first you need to create a Security Log Profile and enable specific options in it: Network firewall will provide specific SYN Cookie events, while DoS protection will provide DoS events for TCP Half open, as it does for any other DoS vector. But you will also get TCP half open stats automatically because AFM enables basic AVR under the hoods. So you will be able to see information in Dashboard. Below tables shows the information you can check in GUI after enabling above options in security log profile: Configuration Let’s see how to configure SYN Cookie in different contexts for AFM as we did for LTM in previous article: Device/Global context Details about the context provided for LTM applies here as well. Also note that SYN cookie MUST be enabled in the protocol profile applied to virtual server in order to AFM Global SYN cookie take the virtual server into account when counting embryonic connections. Configuration options for typical LTM protocol profiles: Remember that DSR whitelist is not the same than SYN Cookie whitelist. DSR whitelist is a special whitelist that automatically add legitimate clients to a temporal whitelist in order to allow SYN Cookie in DSR environments (see article 2 of this series). On the other hand, SYN Cookie whitelist is in fact a DoS whitelist where client’s IPs are added until you decide to remove them. During the time an IP is in this address list the client will not be matched against DoS vectors. Above table refers specifically to DSR whitelist. Configuration options for TCP half open vector through TMSH are common to other vectors, but if you try to configure a non supported option then you will get an error. For example: 01071b13:3: DOS attack data (tcp-half-open): bad actor and auto blacklisting features do not apply to tcp-half-open vector. Since we are working with DoS there are other options you can configure for this SYN Cookie vector, like auto-threshold, floors, multiplier, etc. but in this article I just want to describe the related SYN Cookie usage, so you can compare better with LTM version. VLAN context Details about this context described for LTM applies here. (*) Be aware of ID993269. At this point it is important to give more details about the difference between mitigating a TCP SYN flood attack by using TCP SYN flood DoS vector or by using SYN Cookie. TCP SYN flood vector will just count the number of TCP SYN packets that reach the specific context where it is configured, if this counter value exceeds the threshold that you configured for the SYN Flood vector, then SYN packets will be dropped until counter value is under the threshold again. As you can guess this could disrupt legitimate traffic during the attack. Sometimes this is better than lose totally an application but you need to configure this with care and always as a last resort. On the other hand, SYN Cookie will not drop any connection unless connection comes from an attacker. Virtual server context Details about the context described for LTM applies here. Example for a virtual server using a TCP profile: Conclusion Now you know the difference between SYN Cookie LTM and AFM. Clearly AFM has more versatility, so it could be interesting thinking in moving SYN Cookie configuration from LTM to AFM if we have this module licensed. Configuration will be centralized and might be clearer if you are get used to work with security DoS vectors. In next article I will write about the differences between hardware and software when working with SYN Cookie.4. SYN Cookie: LTM Configuration
Introduction Since you already know how SYN Cookie works now it is time to start configuring BIG-IP devices. In this article I explain how to configure BIG-IP LTM devices for protecting against TCP SYN flood attack at different contexts. Configuration BIG-IP is an application focused device, so from security perspective is in charge of protecting these applications, but it is also important to protect BIG-IP itself since compromising it means that all applications could be affected. This is why SYN Cookie can be configured not only for protecting applications but also for protecting whole BIG-IP, and even for protecting specific VLANs connected to BIG-IP. In this section I will explain how to configure SYN Cookie for each context. Device/Global At this context we protect BIG-IP itself, so the idea is defining how many embryonic connections are allowed to be handled by BIG-IP TCP stack, regardless virtual server destination. This means that SYN Cache counter will increase for each embryonic connection that device is handling globally. Since TMM is the core of the BIG-IP, the element that manipulate traffic, it makes sense to protect each TMM specifically. This is why SYN Cache at global context is defined per TMM. So for example, configuring a value of 1000 in a 4 TMM device means that the Big-IP will potentially (if connections are evenly balanced among TMMs) handle 4000 embryonic connections before activating SYN Cookie. Fig9. Device context SYN cookie will take into account all TCP SYN packets sent to any listener exposed, this includes also selfIPs and virtual servers, but note that itMUST be enabled in the protocol profile applied to virtual server (which is in fact the default value) in order to Global SYN cookie take the specific virtual server into account when counting embryonic connections. Configuration example for a device with one virtual server using a tcp profile: VLAN context The threshold is defined per TMM as in device context . This is only available for SYN Cookie hardware offloading platforms. See related AskF5 article for more details. With this feature you can protect a group of virtual servers listening in the same VLAN. Reasons why you would want to do this can vary, from architecture perspective you could have a dedicated VLAN serving applications on old servers and therefore expecting lower number of connections for example. Or you could want to configure SYN Cookie in a more granular way instead configuring a global value for whole device. In older versions also there could be a risk of collisions when multiple virtual servers were under attack at the same time, so using this method instead SYN Cookie per virtual could help in these cases. If SYN Cookie is enabled at Global context the SYN Cookie Per-VLAN is disabled because Device protection is ON at all-VLAN basis and it would interfere with Per VLAN SYN cookie. Fig10. VLAN context At VLAN context you can configure not only SYN Cookie but also TCP SYN flood DDoS vector, even with only LTM license. It is important to note that setting a threshold for this vector could have consequences, unlike SYN Cookie, this vector will start to drop TCP SYN packets once the configured number of TCP SYN packets per second is reached. This is because TCP SYN flood vector just counts TCP SYN packets reaching the VLAN, instead embryonic connections, and if a TCP SYN packet exceeds the threshold then it is just dropped. TCP SYN flood and TCP SYN Cookie are totally different and independent countermeasures. Virtual server context In this case the threshold is defined globally. This means that the value is defined for all TMMs, unlike SYN Cookie at Device/VLAN context. If you want to know how many embryonic connections will be handled by each TMM prior the TMM activates SYN Cookie then you must divide configured threshold among the number of existing TMMs. When configuring SYN Cookie in a LTM device you only can define a general threshold that it will be common for all virtual servers (in next article I will show differences with AFM SYN Cookie). So all virtual servers will have the same threshold. Fig11. Virtual Server context Configuring SYN Cookie at this context requires setting a common threshold for all virtual servers but also you MUST enable SYN Cookie in specific protocol profile that is applied to the virtual server in order to be able to enable the countermeasure for that virtual server. This will allow you to enable SYN Cookie in some virtual servers and disable it in others. Although you cannot have specific SYN Cache threshold for a virtual server, you can have different SYN Cookie behaviours depending on virtual server, like for example using whitelist or not (see second table below). Example for fastL4 configuration: Note that in protocol profile you could still have see twp options in TMSH that are deprecated from version 13 and have no effect anymore. They will be removed in a future release: tmsh modify ltm profile fastl4 fastL4 hardware-syn-cookie <value> tmsh modify ltm profile fastl4 fastL4 software-syn-cookie <value> As you can see in table above, as part of DSR configuration you can decide the device that will send the RST to the TCP connection: tmsh modify ltm profile fastl4 <name> syn-cookie-dsr-flow-reset-by <bigip | client | none> Sometimes it could be better choosing ‘client’ since some applications do not like Big-IP sending the reset. In order to get this, Big-IP sends the SYN Cookie in the ACK instead than in the sequence number, in this way client will RST the connection for this specific ACK, so when Big-IP gets the RST knows about the referred connection and it adds it into DSR whitelist. SYN Cookie can work in two different modes at this context. Example for tcp protocol profile: In case you have wildcard virtual servers and you want to protect them with SYN Cookie at virtual server context, then you will have to configure Software SYN Cookie unless you have a Neuron capable platform. I will give more details about this in next articles. Default values In TMOS versions higher than v12 we have the below default values: Three comments related to above values. DB Key pvasyncookies.virtual.maxsyncache is deprecated in version >v12. This means that you should ignore the value of this DB Key whatever it is, since it is not used anymore by the system. Reason why we disable SYN Cookie at virtual server context when device is installed from scratch, is that we do not know the capacity of the pool members attached to virtual servers, so we let that value be disabled for customers to decide this threshold. However after upgrade from v12 we do not want to modify whatever behaviour security administrator considered correct in v12, so we just keep the value. At virtual server context SYN Cookie threshold in TMOS ≤v12 was set per TMM. In TMOS >v12 we define this value globally, as explained above in this same article. This is why threshold after upgrade is calculated as the existing value configured in v12 multiplied by the number of TMMs. In this way we keep the same value, so customer will not notice any change in the behaviour of their device. But note that there are also a couple of things to take into account when upgrading from v12.x currently: If virtual server threshold in v12 (DB key pvasyncookies.virtual.maxsyncache) was < 4095 then value will be the same in >v12 after upgrade. If virtual server threshold in v12 (DB key pvasyncookies.virtual.maxsyncache) was ≥ 4095 then value after upgrade to ≥v12 will be 8192. This behaviour it is being improved for new versions, so threshold will be kept regardless its original value, as I have commented. Conclusion You know how to configure SYN Cookie in LTM module based system, in next article I will explain differences between LTM and AFM SYN Cookie, so you will be aware of the limitations of each one and have a global perspective of the feature.3. SYN Cookie: SYN Cache
Introduction In previous articles I have explained why it is so important to implement TCP SYN Cookie in order to protect exposed applications. In this article I will explain when SYN Cookie is activated and different aspects you should take into account when you configure it in LTM. SYN cache You know how SYN Cookie works in broad strokes, so now the pending question about implementation is to know when BIG-IP should activate SYN Cookie countermeasure. This is where security administrators come into play. Commonly SYN Cookie implementation works together with another countermeasure called SYN cache. SYN cache is based in the use of a cache for incomplete TCBs, this allows devices to save some resources comparing with standard TCP connection because full state allocation for TCB is delayed until the TCP 3WHS has been fully finished.I will not describe detailed functioning of SYN cache here but just note that this method could increase time to establish legitimate TCP connections in 15%. BIG-IP does not use a cache as is, instead it uses a TCP embryonic connection counter, if BIG-IP detects that the number of embryonic connections have been exceeded attending to a configured threshold then SYN Cookie is activated. This is more efficient than typical SYN cache implementation since it does not add any delay by creating incomplete TCBs prior to complete TCP 3WHS. Also, it does not need to allocate memory for these incomplete TCBs. The process of SYN Cookie activation can be described as below: Fig7. TCP SYN Flood attack + SYN Cache Our only task as Security Administrators is defining the best threshold for this counter. It is very important that threshold fits perfectly in your environment and there are two main reasons for it: Depending on our BIG-IP device platform SYN Cookie can be implemented in hardware or in software, if our device has not hardware offloading capability for SYN Cookie then the SYN Cookie process described in second article of this series must be run in TMM, that is, in software. Although the process is optimised there will be a slight penalty in CPU usage during an attack since TMM will have to create SYN Cookie challenges and validate client responses. This should not affect normal functioning of the device but it is relevant to take this into account since not always customer has a perfect BIG-IP sizing. So it is important to distinguish between a real attack or a wrong configured threshold,and avoid extra load to your devices by setting a correct SYN Cache value. As you will see with more detail in later articles in this series, activating SYN Cookie can have some consequences due to how SYN Cookie is implemented, not specifically in BIG-IP devices, but SYN Cookie standard in general. One important drawback is the limited space used for the SYN Cookie challenge which, as explained in previous article, it will limit the possible MSS sizes and also will remove some TCP options information (unless TCP Timestamp is used). This could cause some impact by slowing down the traffic between client and backend server when SYN Cookie is active. However, note something very important, SYN Cookie is only activated when under attack if it is correctly configured, so the possible choices during an attack are disruption if no countermeasure is configured or, in the worst cases, having a minimum impact in performance until attack is finished. Also, if SYN Cookie is correctly configured it should be only activated specifically on virtual servers under attack, so usually you will not notice any issue. Although F5 Networks creates a default configuration for SYN cache, that you will read about in the next article, the value that better fits with your environment is the value you define attending to expected traffic patterns, and you have the best knowledge of your network. Operation As a short summary you can check below flow diagram where the most important steps are shown, so you can have at a glance a global idea about what it happens when SYN cache is exceeded in a TMM and hence SYN Cookie is activated in hardware or software for this TMM. There are some important points to note: To prevent oscillation (activating/deactivating SYN Cookie loop), the entry/exit strategy from SYN cookie mode have some hysteresis. This is determined by several factors like the round-trip time, rate of SYN arrival, the 'exit' threshold, the value of the SYN cache,… This will avoid entering and exiting SYN Cookie continuously during an attack if the number of TCP SYN packets per second for this attackis near to the configured SYN Cache threshold. From TMOS v13 you can stop to challenge a source IP during some seconds if the client response with a successfulchallenge response, that is, if client is trustworthy This is detailed in next article where configuration is treated. By checking diagram above you will notice that regardless if Hardware or Software SYN Cookie is enabled we see TMM always rejecting connections. This is because hardware only generates and validates SYN Cookies but it never drops connections. This is important when we check SYN Cookie stats as you will see in future article. Conclusion At this point you have a clear knowledge of SYN Cookie and SYN Cache countermeasures and how they are implemented in BIG-IP. In next articles we will start to talk about configurations and troubleshooting.
