Huge number of TCP 3WHS rejected (bad ACK), chksum incorrect

Hi guys,

Hope you can help me with this, for me, complete mystery. I'm getting lots of following:

Wireshark text export from F5 tcpdump:

4815 17:27:58.597830    CLIENT_IP       F5_VS_IP        TCP     162 36562 → 443 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=681665250 TSecr=0 WS=128
4816 17:27:58.597846    F5_VS_IP        CLIENT_IP       TCP     193 443 → 36562 [SYN,ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 TSval=751619333 TSecr=681665250 SACK_PERM=1
4817 17:27:58.660439    CLIENT_IP       F5_VS_IP        TCP     185 36562 → 443 [ACK] Seq=1 Ack=1 Win=29200 Len=0 TSval=681665313 TSecr=751619333
4818 17:27:58.730179    CLIENT_IP       F5_VS_IP        TLSv1.2 380     Client Hello
4819 17:27:58.730201    F5_VS_IP        CLIENT_IP       TLSv1.2 4529    Server Hello
4820 17:27:58.792837    CLIENT_IP       F5_VS_IP        TCP     185 36562 → 443 [ACK] Seq=196 Ack=4345 Win=37648 Len=0 TSval=681665445 TSecr=751619465
4821 17:27:58.792854    F5_VS_IP        CLIENT_IP       TLSv1.2 706     Certificate, Server Hello Done
4822 17:27:58.855416    CLIENT_IP       F5_VS_IP        TCP     185 36562 → 443 [ACK] Seq=196 Ack=4866 Win=40544 Len=0 TSval=681665508 TSecr=751619528
4823 17:27:58.857719    CLIENT_IP       F5_VS_IP        TLSv1.2 543     Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
4824 17:27:58.857731    F5_VS_IP        CLIENT_IP       TCP     185 443 → 36562 [ACK] Seq=4866 Ack=554 Win=4933 Len=0 TSval=751619593 TSecr=681665510
4825 17:27:58.859778    F5_VS_IP        CLIENT_IP       TLSv1.2 276     Change Cipher Spec, Encrypted Handshake Message
4826 17:27:58.923739    CLIENT_IP       F5_VS_IP        TLSv1.2 670     Application Data
4827 17:27:58.923793    F5_VS_IP        CLIENT_IP       TCP     185 443 → 36562 [ACK] Seq=4957 Ack=1039 Win=5418 Len=0 TSval=751619659 TSecr=681665576
4828 17:27:58.923981    F5_FLOAT_IP     SERVER_IP       TCP     193 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751619659 TSecr=0 SACK_PERM=1
4829 17:27:59.923626    F5_FLOAT_IP     SERVER_IP       TCP     193 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751620659 TSecr=0 SACK_PERM=1
4830 17:27:59.923874    SERVER_IP       F5_FLOAT_IP     TCP     173 [TCP ACKed unseen segment] 8080 → 1360 [ACK] Seq=1 Ack=993763571 Win=29845 Len=0
4831 17:27:59.923882    F5_FLOAT_IP     SERVER_IP       TCP     209 1360 → 8080 [RST] Seq=993763571 Win=0 Len=0
4832 17:28:00.923733    F5_FLOAT_IP     SERVER_IP       TCP     193 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751621659 TSecr=0 SACK_PERM=1
4833 17:28:01.923650    F5_FLOAT_IP     SERVER_IP       TCP     181 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 SACK_PERM=1
4834 17:28:01.923822    SERVER_IP       F5_FLOAT_IP     TCP     173 [TCP ACKed unseen segment] 8080 → 1360 [ACK] Seq=2408178215 Ack=1538671403 Win=30282 Len=0
4835 17:28:01.923845    F5_FLOAT_IP     SERVER_IP       TCP     209 1360 → 8080 [RST] Seq=1538671403 Win=0 Len=0
4836 17:28:02.923550    F5_VS_IP        CLIENT_IP       TCP     204 443 → 36562 [RST,ACK] Seq=4957 Ack=1039 Win=0 Len=0
4837 17:28:02.923561    F5_FLOAT_IP     SERVER_IP       TCP     204 [TCP ACKed unseen segment] 1360 → 8080 [RST, ACK] Seq=1 Ack=591246314 Win=0 Len=0

F5 tcpdump sees following (this is for different case):

F5_FLOAT_IP.27216 > SERVER_IP.8080:      Flags [R], cksum 0x95b2 (incorrect -> 0x0a17), seq 3911139265, win 0, length 0 
                                                out slot1/tmm10 lis=/Common/https_production flowtype=128 flowid=5618A9EEBE00 peerid=56189FD35F00 conflags=4000024 
                                                inslot=2 inport=9 haunit=1 priority=3 
                                                rst_cause="[0x2b07e6a:2314] 
                                                TCP 3WHS rejected (bad ACK)" 
                                                peerremote=00000000:00000000:X:X peerlocal=00000000:00000000:X:X 
                                                remoteport=59656 localport=443 proto=6 vlan=4093

It is hitting constantly, and quite a lot.

As per "K13223" this represent "The BIG-IP system failed to establish a TCP connection with the host (client or server) due to a failure during the TCP 3-way handshake process." In my case it is communication between F5 and server pool (all nodes affected).

There is no firewall between F5 and server pool(s).

It is happening with both AutoMap and SNAT.

Are there any guides/cases how to debug this issue further? Mine test shows that it's not connected with client type (browser, curl, ...) or URL (same URL works in 99 percent of cases, that 1% is what's bothering me).

Thank you!