Forum Discussion
John_McCulloch
Nimbostratus
NimbostratusBlocked 'Reverse Connections'
Hi all,
Recently we have ran into an issue with connections that involved our BIGIP devices and client server connections when there is a firewall as man in middle. The issue that is seen presents itself as a reverse connection i.e. ports flipped or source = 443 that gets blocked. Furthermore, this issue does not seem to stem from one set of client or servers devices and also seems to happen with health checks.
Now rules have been put into place on the MIM firewall that allow for the expected flow of traffic and this issue does not seem to be service affecting but it does generate a mass of logging for the blocks. I have attempted troubleshot what I can only describe as half closed behaviour but unfortunately have started to come up dry with further theory's as to why the F5 does not seem to tear down the connection along with the rest of the devices and sends one last packet that gets blocked when the other side closes the connection forcing the firewall to remove it from its state table.
My hope is that others may have experienced this or can add to the conversation so I can look into other avenues for investigation.
- Nikoolayy1
MVP
Maybe it is not the same but I had issue that the F5 and the servers may have different time_wait for connection reuse as for f5 is 2 seconds by default and you may change this for the newer tcp profiles to match the servers as their time_wait could be bigger or smaller (for windows servers it is much bigger):
https://support.f5.com/csp/article/K29377715
For fastl4 there is still no such option and adding a SNAT pool with many ip addresses and maybe preserve the source port for SNAT:
https://support.f5.com/csp/article/K14400019
Also randomizing the ports helps but this are system variables, so be carefull:
https://support.f5.com/csp/article/K8246
John_McCullochThanks for the response, unfortunately we are not using fastl4.
In regards to the TIME_WAIT, I did look down this path and they do seem to have a longer setting compared to the F5. Interestingly it does seem to be the F5 client side of the VS's that are having the issues, servers side seems to close 4 way without as many issues.
I have noticed that the F5s are responding to the FIN's sent with double ACKS. The client generally seems to be responding to this with a RST closing one half of the connection. I will look further into the KA's above and see if there is anything else I have missed though.
Just a quick addition for clarification, these reverse connection are not new connections they seem to be the tail end of already existing/closing connections.
- Nikoolayy1
Maybe it is better that you are not using fatl4 as I mentioned with normal standard vip you selected you customize the Time_Wait of the normal tcp profiles. You can also use "show sys connection" fulter by VS and do it outside working hours to see if the connections are in Time_Wait. My issue was the servers entering time_wait and I saw this with netstat -an on the server pool members.
https://support.f5.com/csp/article/K40033505
https://support.f5.com/csp/article/K53851362
Also you may test using diffent optimized tcp profiles for the client-side and server-side (wan-optimized for the clienside and lan-optimized for the serverside for example):
- John_McCulloch
Ok great, I will have a further look into that then report back if it helps. Thanks again for the support.
