Dear All, We are facing issue with monitoring of nodes in one of our virtual server pool, however when we check on the server side we didn’t found any discrepancy on server port side. Due to this F5 keep flipping those servers out of pool and hence we are facing issue . Appreciate please do let us know and what are the fix for this in case any bug or hotfix. Please do let us know in case if you required any further information please check the log : size 128KB download here on google drive thank you

It will be interesting to check the tcp dump. can you run a tcp dump on F5 by running tcpdum -ni "vlan/interface" dst host 10.64.32.16 and src host "F5 self IP" -w . It seems like tcp health check's timeout is expiring before server responds. But then again maybe I am stating the obvious. Are the timers on tcp health check default? or were they fine tuned at any stage?

Hi Buddy, Your destination port in tcp dump is always changing. What is the port number on which pool members of this pool are listening? Can you run following commands from tmsh and provide the input 1) list /ltm monitor tcp 2) list /ltm pool "pool_name" I am assuming that you are using tcp monitor for your health check, but if you are using some other health check please change its name instead of tcp.

Hi bro, yes im using TCP for monitor, is it wrong? ravi@(F5_1-PRD-Tibco)(cfg-sync In Sync)(Active)(/Common)(tmos) list /ltm monitor tcp ltm monitor tcp tcp { destination *:* interval 5 time-until-up 0 timeout 16 } ltm monitor tcp tcp_7456 { defaults-from tcp destination *:* interval 5 time-until-up 0 timeout 16 } ltm monitor tcp tcp_8081 { defaults-from tcp destination *.tproxy interval 5 time-until-up 0 timeout 16 } ltm monitor tcp tcp_8085 { defaults-from tcp destination *.8085 interval 30 time-until-up 0 timeout 91 } ltm monitor tcp tcp_8181 { defaults-from tcp destination *.8181 interval 5 time-until-up 0 timeout 16 } ravi@(F5_1-PRD-Tibco)(cfg-sync In Sync)(Active)(/Common)(tmos) list ltm pool P_VAS_Query_44070 ltm pool P_VAS_Query_44070 { members { 10.64.32.15:44070 { address 10.64.32.15 session monitor-enabled state up } 10.64.32.16:44070 { address 10.64.32.16 session monitor-enabled state up } 10.64.32.29:44070 { address 10.64.32.29 session monitor-enabled state up } 10.64.32.29:44071 { address 10.64.32.29 session monitor-enabled state down } 10.64.32.30:44070 { address 10.64.32.30 session monitor-enabled state down } 10.64.32.30:44071 { address 10.64.32.30 session monitor-enabled state down } Mumwin03:44070 { address 10.64.32.25 session monitor-enabled state up } Mumwin04:44070 { address 10.64.32.26 session monitor-enabled state up } Mumwin04:44071 { address 10.64.32.26 session monitor-enabled state up } Mumwin04:44072 { address 10.64.32.26 session monitor-enabled state down } Mumwin09:44070 { address 10.64.32.12 session monitor-enabled state up } Mumwin09:44071 { address 10.64.32.12 session monitor-enabled state down } Mumwin09:44072 { address 10.64.32.12 session monitor-enabled state down } Mumwin09:44073 { address 10.64.32.12 session monitor-enabled state down } } monitor tcp } (END)

Change your tcpdump to tcpdump -i 1.15 -ni vlan_internal host 10.64.32.16 and host 10.64.32.35 so you can see the response or lack there of coming back from your pool members. If they don't respond within 16 seconds they will be marked down based on the timeout of 16 seconds in your default tcp monitor. Also, have you enabled monitor logging for one of your flapping pool members to see why its being marked down? Once you do that each pool member that has that enabled will have a log at /var/log/monitors/.

F5 Pool Members flapping issue

26 Replies

adithyodw_18563
Nimbostratus
Sep 01, 2015
Hi Bro, Still i cant find the log,
- Brad_Parker
  Cirrus
  Sep 01, 2015
  Well apparently that wasn't introduced until 11.5. Here is an article how to get the same info in previous verisions via the bigd log, https://devcentral.f5.com/wiki/advdesignconfig.TroubleshootingLtmMonitors.ashx.
- adithyodw_18563
  Nimbostratus
  Sep 02, 2015
  Hi Brad, will check for this, thanks
GaganD
Nimbostratus
Sep 02, 2015
Hi Buddy, Looking at the tcpdump, your F5 is doing a TCP health check on different server ports. Normal behaviour would be that F5 will only try to connect to the port on which member server is listening on. So for example 10.64.32.16 is listening on port 44070, so health checks should only happen on port 44070. Can you confirm what monitor are you using to monitor the node 44070(may be problem is at node level)? If you are using TCP, then I can understand your problem. Please change it to icmp. You can find it out by typing list /ltm node 10.64.32.16 from tmsh
Shaun_Simmons1
Altostratus
Sep 02, 2015
Could it be a subnet mask issue of the Self IP?
adithyodw_18563
Nimbostratus
Sep 02, 2015
Hi gagan, based on gui, im using tcp for health monitors for 44070, so i must change it to gateway_icmp? and why? thank you
- Shaun_Simmons1
  Altostratus
  Sep 02, 2015
  ICMP can be used to quick-check the path to the server(s) are good. I use it when I know I have the LTM configured correctly but don't know if the network or server may be the issue.
- adithyodw_18563
  Nimbostratus
  Sep 02, 2015
  hi gagan, based on gui 10.64.32.16 using monitoring icmp. should i change?
- GaganD_191239
  Nimbostratus
  Sep 02, 2015
  If node is using icmp as its health check at node level then you don't need to change. But if its using tcp to monitor node at node level then yes please change it to icmp. ICMP is good enough health monitor at node level, all fancy monitors should be used at pool level to monitor health of application. Since by default tcp(or all health monitors in fact) does not specify destination port in the monitor config(its usually set to wildcard), it does not know which port to monitor, But when it is assigned to pool which has member listening to specific port(like 10.64.32.16:44070) in you case, it take the port number (44070) from pool member and use it in its monitoring. So it know what port to monitor. But when you assign the same monitor tcp to a node, which has no port number specified(and you actually cannot specify it), monitor does not know which port to monitor, so it can cause to problems.
adithyodw_18563
Nimbostratus
Sep 02, 2015
Hi All,

thank you, i already found the root cause, my method is tried to put down pool member one by one, and see the traffic. and the result is flip flopping in one pools, application issue.

thank you all

But please help me analyze also for another refrence.

tcpdump | google drive

Log grep down | google drive
GaganD
Nimbostratus
Sep 04, 2015
Hi Buddy,

After checking you tcpdump, I can see that your problematic server us sending TCP zerowindow to F5. upon which after waiting for some time, F5 tears down the session by sending resets, causing health check to fail. There is something which is making your server really busy, to an extent that it is sending message that it cannot receive any more data. What is cpu, memory, interface stats on the server.