F5 Pool Members flapping issue

Forums Technical Forums
, ,
1

Dear All,

We are facing issue with monitoring of nodes in one of our virtual server pool, however when we check on the server side we didn’t found any discrepancy on server port side. Due to this F5 keep flipping those servers out of pool and hence we are facing issue . Appreciate please do let us know and what are the fix for this in case any bug or hotfix. Please do let us know in case if you required any further information

please check the log : size 128KB download here on google drive

thank you

2

It will be interesting to check the tcp dump. can you run a tcp dump on F5 by running tcpdum -ni “vlan/interface” dst host 10.64.32.16 and src host “F5 self IP” -w . It seems like tcp health check’s timeout is expiring before server responds. But then again maybe I am stating the obvious. Are the timers on tcp health check default? or were they fine tuned at any stage?

3

deleted

4

Hi Buddy, Your destination port in tcp dump is always changing. What is the port number on which pool members of this pool are listening? Can you run following commands from tmsh and provide the input 1) list /ltm monitor tcp 2) list /ltm pool “pool_name” I am assuming that you are using tcp monitor for your health check, but if you are using some other health check please change its name instead of tcp.

5

Hi bro, yes im using TCP for monitor, is it wrong?

ravi@(F5_1-PRD-Tibco)(cfg-sync In Sync)(Active)(/Common)(tmos) list /ltm monitor tcp
ltm monitor tcp tcp {
    destination *:*
    interval 5
    time-until-up 0
    timeout 16
}
ltm monitor tcp tcp_7456 {
    defaults-from tcp
    destination *:*
    interval 5
    time-until-up 0
    timeout 16
}
ltm monitor tcp tcp_8081 {
    defaults-from tcp
    destination *.tproxy
    interval 5
    time-until-up 0
    timeout 16
}
ltm monitor tcp tcp_8085 {
    defaults-from tcp
    destination *.8085
    interval 30
    time-until-up 0
    timeout 91
}
ltm monitor tcp tcp_8181 {
    defaults-from tcp
    destination *.8181
    interval 5
    time-until-up 0
    timeout 16
}

ravi@(F5_1-PRD-Tibco)(cfg-sync In Sync)(Active)(/Common)(tmos) list ltm pool P_VAS_Query_44070
ltm pool P_VAS_Query_44070 {
    members {
        10.64.32.15:44070 {
            address 10.64.32.15
            session monitor-enabled
            state up
        }
        10.64.32.16:44070 {
            address 10.64.32.16
            session monitor-enabled
            state up
        }
        10.64.32.29:44070 {
            address 10.64.32.29
            session monitor-enabled
            state up
        }
        10.64.32.29:44071 {
            address 10.64.32.29
            session monitor-enabled
            state down
        }
        10.64.32.30:44070 {
            address 10.64.32.30
            session monitor-enabled
            state down
        }
        10.64.32.30:44071 {
            address 10.64.32.30
            session monitor-enabled
            state down
        }
        Mumwin03:44070 {
            address 10.64.32.25
            session monitor-enabled
            state up
        }
        Mumwin04:44070 {
            address 10.64.32.26
            session monitor-enabled
            state up
        }
        Mumwin04:44071 {
            address 10.64.32.26
            session monitor-enabled
            state up
        }
        Mumwin04:44072 {
            address 10.64.32.26
            session monitor-enabled
            state down
        }
        Mumwin09:44070 {
            address 10.64.32.12
            session monitor-enabled
            state up
        }
        Mumwin09:44071 {
            address 10.64.32.12
            session monitor-enabled
            state down
        }
        Mumwin09:44072 {
            address 10.64.32.12
            session monitor-enabled
            state down
        }
        Mumwin09:44073 {
            address 10.64.32.12
            session monitor-enabled
            state down
        }
    }
    monitor tcp 
}
(END)
6

Change your tcpdump to

tcpdump -i 1.15 -ni vlan_internal host 10.64.32.16 and host 10.64.32.35

so you can see the response or lack there of coming back from your pool members. If they don’t respond within 16 seconds they will be marked down based on the timeout of 16 seconds in your default tcp monitor. Also, have you enabled monitor logging for one of your flapping pool members to see why its being marked down? Once you do that each pool member that has that enabled will have a log at /var/log/monitors/.

7

Change your tcpdump to

tcpdump -i 1.15 -ni vlan_internal host 10.64.32.16 and host 10.64.32.35

so you can see the response or lack there of coming back from your pool members. If they don’t respond within 16 seconds they will be marked down based on the timeout of 16 seconds in your default tcp monitor. Also, have you enabled monitor logging for one of your flapping pool members to see why its being marked down? Once you do that each pool member that has that enabled will have a log at /var/log/monitors/.

8

Also, what version of Big-IP are you on? Could you be suffering from this bug, 404 Page not found | BIG-IP Documentation?

9

Also, what version of Big-IP are you on? Could you be suffering from this bug, 404 Page not found | BIG-IP Documentation?

10

deleted tcpdump

11

deleted tcpdump

12

deleted tcpdump

13

ls /var/log/monitors/. Please, don’t post a screen capture of packets, its impossible to read. If you want to share captures send them to a pcap file using -w /shared/tmp/name.pcap so you can analyse them off box. The monitor log will be very beneficial to try to isolate down why the monitor would be going down. To turn on monitor logging for a pool member, go into a pool, and click on a member. There should be a check box to enable monitor logging.

14

ls /var/log/monitors/. Please, don’t post a screen capture of packets, its impossible to read. If you want to share captures send them to a pcap file using -w /shared/tmp/name.pcap so you can analyse them off box. The monitor log will be very beneficial to try to isolate down why the monitor would be going down. To turn on monitor logging for a pool member, go into a pool, and click on a member. There should be a check box to enable monitor logging.

Hi Brad, thank you for still helping me,im newbie in F5.

1) i cant find ls /var/log/monitors

image_113353.png
image_113353.png1545×307 18.6 KB

2) i cant find how to check logging for pool member, so how can i enable it, and see the log when the pool is down anyway

image_113353.png
image_113353.png814×768 28.7 KB

3) here the tcpdump -i 1.15 -ni vlan_internal host 10.64.32.16 and host 10.64.32.35 -w /shared/tmp/testaja.pcap tcpdump googledrive

15

click on the members tab at the top of this screen capture, then click on an actual pool member. You should then see the check box.

16

Hi Bro, Still i cant find the log,

image_113355.png
image_113355.png1380×633 46.3 KB

image_113355.png
image_113355.png1365×637 30.3 KB

17

i think this one bro? Local Traffic > Profiles > Other > Request Logging

18

Well apparently that wasn’t introduced until 11.5. Here is an article how to get the same info in previous verisions via the bigd log, DevCentral - An F5 Technical Community.

19

Hi Buddy, Looking at the tcpdump, your F5 is doing a TCP health check on different server ports. Normal behaviour would be that F5 will only try to connect to the port on which member server is listening on. So for example 10.64.32.16 is listening on port 44070, so health checks should only happen on port 44070. Can you confirm what monitor are you using to monitor the node 44070(may be problem is at node level)? If you are using TCP, then I can understand your problem. Please change it to icmp. You can find it out by typing list /ltm node 10.64.32.16 from tmsh

20

Could it be a subnet mask issue of the Self IP?