Forum Discussion

Nimbostratus

Aug 07, 2014

Health check failing - tcp/54321 - is this more than a coincidence?

We are rolling out a new web application in a couple of weeks and the developers have noticed issues with the application that are very intermittent and difficult to quantify. Testers have reported ...

sdavids5670_166

Nimbostratus

Aug 14, 2014

Here's what F5 support told our admin to do for the tcp/54321 issue. Since I do not know for which version this was applied, I would suggest that you contact F5 support if you're having this issue. The workaround could be different for your version.

"Here are the steps to work around this issue:

Run the following commands to update your current iptables rules:

/sbin/iptables -D INPUT -p tcp --dport 54321 -j REJECT --reject-with icmp-port-unreachable /sbin/iptables -D INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset /sbin/iptables -A INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset

--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__

Run the following commands to add the iptables rules to /config/startup. This will ensure that these iptables rules persist upon reboot and upgrades:

echo "/sbin/iptables -D INPUT -p tcp --dport 54321 -j REJECT --reject-with icmp-port-unreachable" >> /config/startup echo "/sbin/iptables -D INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset" >> /config/startup echo "/sbin/iptables -A INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset" >> /config/startup"

sdavids5670_166
Nimbostratus
Aug 14, 2014
Sorry about the formatting (again). What's up with this site???
DSS_Gateway_158
Nimbostratus
Aug 14, 2014
Hi David, Good to hear you had success, F5 have given us the same commands to try. I am not sure our issue is exactly the same as for 7 monitor fails, only 1 of those used 54321 for all 5 attempts. We are seeing random src ports which the F5 gives port unreachable, then a 54321 port unreachable just appears which wasn't even part of the TCP conversation.
sdavids5670_166
Nimbostratus
Aug 15, 2014
Dave, is your environment virtualized? Does your environment use TSO or has it been disabled? We had tons of health check fails (and even situations in which the heartbeat between the two F5s failed) and all of that disappeared (with the exception of the tcp/54321) when we turned off TSO on the F5s.
DSS_Gateway_158
Nimbostratus
Aug 18, 2014
Hi Steven, We are virtualized and use cisco 1000v. I found your cisco thread here: https://supportforums.cisco.com/discussion/11883926/tcp-segmentation-offload-tso-and-vmxnet31000v-bug On our newer VCMP F5s running 11.5.1 we might only be getting monitor flapping due to the 54321 bug, we will be implementing the F5 change soon. As for our other F5 (11.1) we get a lot of monitor flapping. The 11.1 machine was showing flapping due to TCP reuse hitting the server TIME_WAIT. We setup our older F5s running 11.4 to probe the same webservers which also showed a lot of monitor flapping but didn't see the TCP reuse/TIME_WAIT issue, more just SYNs with no reply from the webserver and hence we closed the case with F5. We still get a lot of monitor flapping on the 11.1 F5 and hoping most of that will go away once we upgrade to 11.5. I don't know much about TSO and the 1000v, I will investigate. I know we had major issues with our first 1000v rebooting and also causing "blue screen of death" for servers. That was all resolved with the upgrade, current 1000v version 4.2(1)SV2(2.2). Cheers, Dave