Forum Discussion
NimbostratusHealth check failing - tcp/54321 - is this more than a coincidence?
We are rolling out a new web application in a couple of weeks and the developers have noticed issues with the application that are very intermittent and difficult to quantify. Testers have reported that from time to time, they either get a "Maintenance" page or a "Page cannot be displayed". This environment has virtualized F5s and virtualized IIS servers running on 1000v in an ESXi 5.1 environment. One thing that we've noticed, while analyzing logs on the F5, is that the F5 health check will fail on one or more IIS servers in the pool (haven't correlated this to anything yet) and then the server will be put back into the pool within 5 to 15 seconds (our health check runs at 5 second intervals). I've been collecting packet captures of this health check and I've noticed something strange. Twice I have seen the following three-way handshake:
F5 > IIS, 54321 > 80 [SYN] IIS < F5, 80 > 54321 [SYN,ACK] F5 > IIS, ICMP "Destination unreachable (port unreachable)" Then there are a series of attempt to reset this and when the IIS server tries to acknowledge this the F5 sends more ICMP messages back to the IIS server.
I want to know why would the F5 decide to use 54321 and then IMMEDIATELY tell the IIS server to go pound sand on the SYN,ACK? The F5 picked 54321. It should be available, right? Is it possible that 54321 is a special port number to the F5 and that this is some kind of bug?
35 Replies
sdavids5670_166Sorry about the formatting. I was hoping that the three-way handshake would appear on separate lines. The gist of this is that the F5 starts with a SYN and then immediately rejects the SYN,ACK from the IIS server because the port it picked (54321) is unreachable.
JGCumulonimbus
That is because the Windows server sends a "win=0" back. We had this problem as well in the past, and Microsoft Support regarded this as "normal"!
I don't remember how we got out of this, but it was in the early days of BIG-IP v11.2.0, and there were many engineering hostfixes we had to apply during that time. We are now on v11.3.0 with one of the latest hf applied and I don't see this problem anymore. But I can't say BIG-IP was the problem.
- sdavids5670_166
FYI, F5 came back and said that the tcp/54321 issue is a known bug with their product (although good luck finding any information on it). You basically need to configure the F5 to not use 54321 to health check (or any other reason). We were getting tons of health check errors and after we disabled 54321 AND (more importantly) disabled TSO ALL of our problems vanished and the end users are much happier. If you're not familiar with TSO (TCP segmentation offloading) get up to speed on it and turn it off.
- sdavids5670_166
Here's what F5 support told our admin to do for the tcp/54321 issue. Since I do not know for which version this was applied, I would suggest that you contact F5 support if you're having this issue. The workaround could be different for your version.
"Here are the steps to work around this issue:
- Run the following commands to update your current iptables rules:
--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__--__
- Run the following commands to add the iptables rules to /config/startup. This will ensure that these iptables rules persist upon reboot and upgrades:
- sdavids5670_166Sorry about the formatting (again). What's up with this site???
- DSS_Gateway_158Hi David, Good to hear you had success, F5 have given us the same commands to try. I am not sure our issue is exactly the same as for 7 monitor fails, only 1 of those used 54321 for all 5 attempts. We are seeing random src ports which the F5 gives port unreachable, then a 54321 port unreachable just appears which wasn't even part of the TCP conversation.
- sdavids5670_166Dave, is your environment virtualized? Does your environment use TSO or has it been disabled? We had tons of health check fails (and even situations in which the heartbeat between the two F5s failed) and all of that disappeared (with the exception of the tcp/54321) when we turned off TSO on the F5s.
For the record, this affects all versions of:
*v11.5.0* *v11.5.1* and most likely *v11.6.0* when it is released.-
Run the following commands to update your current iptables rules:
/sbin/iptables -D INPUT -p tcp --dport 54321 -j REJECT --reject-with icmp-port-unreachable/sbin/iptables -D INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset/sbin/iptables -A INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset
-
Run the following commands to add the iptables rules to /config/startup. This will ensure that these iptables rules persist upon reboot and upgrades:
echo "/sbin/iptables -D INPUT -p tcp --dport 54321 -j REJECT --reject-with icmp-port-unreachable" >> /config/startupecho "/sbin/iptables -D INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset" >> /config/startupecho "/sbin/iptables -A INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset" >> /config/startup
The synopsis of this bug is that there is an iptables rule that was implemented to secure a Websense vulnerability.
In the process, the iptables rule was created in a rule that would block ALL Traffic to ALL Interfaces of a BIG-IP device.
This means that all traffic sourced from the BIG-IP with an ephemeral port of :54321 will be RST when the peer device sends it's SYN,ACK (as you see above in DSS Gateway's post)
I highly recommend that anybody running v11.5+ perform the above steps to ensure your system is not incorrectly blocking traffic that has been sourced by the BIG-IP with an ephemeral port of 54321.
- JGThanks for confirming and sharing this fix. Is there a bug id for this so that we can track it?
- Bug ID 455284
-
For the record, this affects all versions of:
*v11.5.0* *v11.5.1* and most likely *v11.6.0* when it is released.-
Run the following commands to update your current iptables rules:
/sbin/iptables -D INPUT -p tcp --dport 54321 -j REJECT --reject-with icmp-port-unreachable/sbin/iptables -D INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset/sbin/iptables -A INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset
-
Run the following commands to add the iptables rules to /config/startup. This will ensure that these iptables rules persist upon reboot and upgrades:
echo "/sbin/iptables -D INPUT -p tcp --dport 54321 -j REJECT --reject-with icmp-port-unreachable" >> /config/startupecho "/sbin/iptables -D INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset" >> /config/startupecho "/sbin/iptables -A INPUT -p tcp -m tcp --dport 54321 --tcp-flags ACK,SYN SYN -j REJECT --reject-with tcp-reset" >> /config/startup
The synopsis of this bug is that there is an iptables rule that was implemented to secure a Websense vulnerability.
In the process, the iptables rule was created in a rule that would block ALL Traffic to ALL Interfaces of a BIG-IP device.
This means that all traffic sourced from the BIG-IP with an ephemeral port of :54321 will be RST when the peer device sends it's SYN,ACK (as you see above in DSS Gateway's post)
I highly recommend that anybody running v11.5+ perform the above steps to ensure your system is not incorrectly blocking traffic that has been sourced by the BIG-IP with an ephemeral port of 54321.
- JGThanks for confirming and sharing this fix. Is there a bug id for this so that we can track it?
- Bug ID 455284
-
shaggyDocumented in 11.6 APM release notes and apparently still an issue - ID 455284 -
http://support.f5.com/kb/en-us/products/big-ip_apm/releasenotes/product/relnote-apm-11-6-0.html
- The scope of this bug has been increased to include the following: 11.5.0 Release Notes Known Issue https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-5-0.htmlrn_ki 11.5.1 Release Notes Known Issue 11.6.0 Release Notes Known Issue https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-6-0.htmlrn_ki Known-Issue Solution Article: SOL15907: The BIG-IP system may incorrectly send an ICMP destination unreachable message to a server responding to health monitor traffic on TCP source port 54321 - https://support.f5.com/kb/en-us/solutions/public/15000/900/sol15907.html
- JGSOL15907: "This issue occurs because the default iptables rules on the BIG-IP system contain a rule that is configured to protect a BIG-IP APM service listening on TCP port 54321, even though APM or SWG is not provisioned." Does this mean if the system has APM provisioned it is not affected?
- It does not. If you are running APM, you are still affected by this bug. This means that BIG-IP software is affected, no matter what modules are provisioned.
- JGThanks. But of the 3 commands provided in the sol, the 2nd one gives an error when run: "iptables: No chain/target/match by that name"
- The scope of this bug has been increased to include the following: 11.5.0 Release Notes Known Issue https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-5-0.htmlrn_ki 11.5.1 Release Notes Known Issue 11.6.0 Release Notes Known Issue https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-6-0.htmlrn_ki Known-Issue Solution Article: SOL15907: The BIG-IP system may incorrectly send an ICMP destination unreachable message to a server responding to health monitor traffic on TCP source port 54321 - https://support.f5.com/kb/en-us/solutions/public/15000/900/sol15907.html
- JGSOL15907: "This issue occurs because the default iptables rules on the BIG-IP system contain a rule that is configured to protect a BIG-IP APM service listening on TCP port 54321, even though APM or SWG is not provisioned." Does this mean if the system has APM provisioned it is not affected?
- It does not. If you are running APM, you are still affected by this bug. This means that BIG-IP software is affected, no matter what modules are provisioned.
- JGThanks. But of the 3 commands provided in the sol, the 2nd one gives an error when run: "iptables: No chain/target/match by that name"
