Forum Discussion

Coso_17543's avatar
Coso_17543
Icon for Nimbostratus rankNimbostratus
Nov 10, 2010

Frequent Health monitors bouncing - Help

Hi all,

 

 

I need assistance on a strange issue on my bigip and I need help to understand what could be.

 

 

We have 2 bigip machines in sync (one active and one standby) that are obviously identically configured and the issue is about a frequent down/up of the nodes of a particular pool due to their monitor bouncing. Sometimes a node goes down, sometimes the other one and sometimes both. After a minute they back online.

 

 

The strange thing is that the nodes have no issues that can cause the down of their monitor and mainly that while the active bigip has these constant monitor down/up on those nodes monitor, the standby one had never reported a single monitor failure on them. This bouncing affects only active bigip while the standby has no issue and has never seen them down.

 

 

I can add also that nodes have no connectivity issue with the active bigip because everytime the monitor goes down their status (icmp) is always green.

 

 

The monitor is basically just a get of a web page, but I tried adding also a generic TCP get as a second monitor configuring pool to have just 1 of them up to be ok but without success. In this case sometimes a monitor goes down, sometimes the other one and frequently both ... and the node goes red.

 

 

What could cause this issue? Maybe a high number of concurrent connections explaying why the standby bigip had never seen those monitor down, even if I have no connection limit in VIP and nodes config? A sort of autodefence of bigip that put a monitor down when "something" happens? The servers that for some issue warms bigip to stop to use them and then it shutdown the monitor? Bug?

 

 

In other words, in wich states a health monitor of a node goes down even if the monitored service is correctly up and running?

 

 

 

Any help is appreciated

 

 

Thank you

 

  • How do your network interface stats look on the affected bigip? Seeing any errors, drops, or collisions?
  • You could also capture a tcpdump on the server VLAN to see whether bigd is sending a monitor request and if so, whether it gets a valid response.

     

     

    Another option would be to enable logging on bigd. But is very verbose logging and could eat up a fair amount of system CPU. For details, check this wiki page:

     

    http://devcentral.f5.com/wiki/default.aspx/AdvDesignConfig/TroubleshootingLtmMonitors.html

     

     

    Aaron
  • I've to check

     

     

    Anyway it's strange that we have a lot of pools and servers but only that pool reports this issue.

     

     

    For info, app server is using tomcat
  • I have a number of tomcat nodes. The only time I've had an issue with monitors going down when the nodes werre in fact fine was when some Cisco IDS rules got enabled that started blocking traffic. IDS interprets gets that don't specify an HTTP version as potentially malicious and I had been using the default HTTP health monitor. Have you tried failing over to the partner bigip to see if the problem follows?
  • I have a similar scenario:

     

     

    Monitors http for virtual servers switching between up and down o nly on the active unit. My application server is Tomcat (version 5.5.25) too.

     

     

     

  • Hi RAC,

     

     

    Do you have any questions about monitoring? Have you read the suggestions in this thread?

     

     

    Aaron
  • AlexJ's avatar
    AlexJ
    Icon for Nimbostratus rankNimbostratus
    Hello all,

     

     

    Did anyone get to the bottom of this as I am experiencing the same problem when targeting and IIS server...

     

     

    No Cisco IDS in operation..

     

     

    Cheers
  • as Aaron suggested, bigd debug and tcpdump are really useful.

     

     

    Troubleshooting Ltm Monitors

     

    Contributed by: hoolio - hooleylists at gmail dot com

     

    http://devcentral.f5.com/wiki/default.aspx/AdvDesignConfig/TroubleshootingLtmMonitors.html