Huge number of TCP 3WHS rejected (bad ACK), chksum incorrect
Hi guys, Hope you can help me with this, for me, complete mystery. I'm getting lots of following: Wireshark text export from F5 tcpdump: 4815 17:27:58.597830 CLIENT_IP F5_VS_IP TCP 162 36562 → 443 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=681665250 TSecr=0 WS=128 4816 17:27:58.597846 F5_VS_IP CLIENT_IP TCP 193 443 → 36562 [SYN,ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 TSval=751619333 TSecr=681665250 SACK_PERM=1 4817 17:27:58.660439 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=1 Ack=1 Win=29200 Len=0 TSval=681665313 TSecr=751619333 4818 17:27:58.730179 CLIENT_IP F5_VS_IP TLSv1.2 380 Client Hello 4819 17:27:58.730201 F5_VS_IP CLIENT_IP TLSv1.2 4529 Server Hello 4820 17:27:58.792837 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=196 Ack=4345 Win=37648 Len=0 TSval=681665445 TSecr=751619465 4821 17:27:58.792854 F5_VS_IP CLIENT_IP TLSv1.2 706 Certificate, Server Hello Done 4822 17:27:58.855416 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=196 Ack=4866 Win=40544 Len=0 TSval=681665508 TSecr=751619528 4823 17:27:58.857719 CLIENT_IP F5_VS_IP TLSv1.2 543 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 4824 17:27:58.857731 F5_VS_IP CLIENT_IP TCP 185 443 → 36562 [ACK] Seq=4866 Ack=554 Win=4933 Len=0 TSval=751619593 TSecr=681665510 4825 17:27:58.859778 F5_VS_IP CLIENT_IP TLSv1.2 276 Change Cipher Spec, Encrypted Handshake Message 4826 17:27:58.923739 CLIENT_IP F5_VS_IP TLSv1.2 670 Application Data 4827 17:27:58.923793 F5_VS_IP CLIENT_IP TCP 185 443 → 36562 [ACK] Seq=4957 Ack=1039 Win=5418 Len=0 TSval=751619659 TSecr=681665576 4828 17:27:58.923981 F5_FLOAT_IP SERVER_IP TCP 193 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751619659 TSecr=0 SACK_PERM=1 4829 17:27:59.923626 F5_FLOAT_IP SERVER_IP TCP 193 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751620659 TSecr=0 SACK_PERM=1 4830 17:27:59.923874 SERVER_IP F5_FLOAT_IP TCP 173 [TCP ACKed unseen segment] 8080 → 1360 [ACK] Seq=1 Ack=993763571 Win=29845 Len=0 4831 17:27:59.923882 F5_FLOAT_IP SERVER_IP TCP 209 1360 → 8080 [RST] Seq=993763571 Win=0 Len=0 4832 17:28:00.923733 F5_FLOAT_IP SERVER_IP TCP 193 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751621659 TSecr=0 SACK_PERM=1 4833 17:28:01.923650 F5_FLOAT_IP SERVER_IP TCP 181 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 SACK_PERM=1 4834 17:28:01.923822 SERVER_IP F5_FLOAT_IP TCP 173 [TCP ACKed unseen segment] 8080 → 1360 [ACK] Seq=2408178215 Ack=1538671403 Win=30282 Len=0 4835 17:28:01.923845 F5_FLOAT_IP SERVER_IP TCP 209 1360 → 8080 [RST] Seq=1538671403 Win=0 Len=0 4836 17:28:02.923550 F5_VS_IP CLIENT_IP TCP 204 443 → 36562 [RST,ACK] Seq=4957 Ack=1039 Win=0 Len=0 4837 17:28:02.923561 F5_FLOAT_IP SERVER_IP TCP 204 [TCP ACKed unseen segment] 1360 → 8080 [RST, ACK] Seq=1 Ack=591246314 Win=0 Len=0 F5 tcpdump sees following (this is for different case): F5_FLOAT_IP.27216 > SERVER_IP.8080: Flags [R], cksum 0x95b2 (incorrect -> 0x0a17), seq 3911139265, win 0, length 0 out slot1/tmm10 lis=/Common/https_production flowtype=128 flowid=5618A9EEBE00 peerid=56189FD35F00 conflags=4000024 inslot=2 inport=9 haunit=1 priority=3 rst_cause="[0x2b07e6a:2314] TCP 3WHS rejected (bad ACK)" peerremote=00000000:00000000:X:X peerlocal=00000000:00000000:X:X remoteport=59656 localport=443 proto=6 vlan=4093 It is hitting constantly, and quite a lot. As per "K13223" this represent "The BIG-IP system failed to establish a TCP connection with the host (client or server) due to a failure during the TCP 3-way handshake process." In my case it is communication between F5 and server pool (all nodes affected). There is no firewall between F5 and server pool(s). It is happening with both AutoMap and SNAT. Are there any guides/cases how to debug this issue further? Mine test shows that it's not connected with client type (browser, curl, ...) or URL (same URL works in 99 percent of cases, that 1% is what's bothering me). Thank you!