External Monitor not working

Forums Technical Forums
,
1

Hi all,

I am having some odd issues with a monitor. I have a web application I need to monitor and for whatever reason a HTPS monitor does not seem to work. The status of the pool always stays as “Unknown (Enabled)”

In desperation I am now trying to use a external monitor. Using the sample_monitor.sh script in /usr/bin/monitors as a basis for my monitor. The script uses three arguments, hostname/ip, URI and expected string. If the output of curl contains the extected string then it returns UP, else it returns DOWN.

Now the script runs and works as expected when I execute it from the command line, but when I create a external monitor to run this script with the required parameters the status of the pool remains as “Unknown”. I decided to put in a debug line in the script that creates a logfile and writes to the logfile at certain points within the script. This logfile is never created which seems to inidicate the monitor is not being run.

Any ideas on why this would be the case? All I see is this in the LTM logfile once I apply the monitor to my test pool

Mar 16 15:59:12 local/bigip-u1 notice mcpd[1657]: 01070638:5: Pool member nnn.nnn.nnn.nnn:443 monitor status unchecked.

Here is the monitor shell script

!/bin/sh   
      
   LOGFN="/tmp/exam-services.log"   
   touch $LOGFN   
   echo "Starting exam-services monitor" >>$LOGFN   
      
    remove IPv6/IPv4 compatibility prefix (LTM passes addresses in IPv6 format)   
   IP=`echo ${1} | sed 's/::ffff://'`   
      
   PIDFILE="/var/run/`basename ${0}`.${IP}.pid"   
    kill of the last instance of this monitor if hung and log current pid   
   if [ -f $PIDFILE ]   
   then   
      kill -9 `cat $PIDFILE` > /dev/null 2>&1   
   fi   
   echo "$$" > $PIDFILE   
   echo "PIDFILE $PIDFILE created" >>$LOGFN   
      
    send request & check for expected response   
    using -k as Monash use a self signed certificate   
      
      
   echo "Executing CURL command" >>$LOGFN   
   curl -fNsvk https://${IP}/exam-services/student 2>&1 | grep -i "/exam-services" 2>&1 > /dev/null   
    mark node UP if expected response was received   
   if [ $? -eq 0 ]   
   then   
       echo "UP"   
       echo "Exam Services Website is up." >>$LOGFN   
   else   
       echo "DOWN"   
       echo "Exam Servicews Website is down." >>$LOGFN   
   fi   
      
   rm -f $PIDFILE   
   echo "Exiting exam-services monitor" >>$LOGFN   
   exit

and in the health monitor GUI setup screen I have

External Program: /usr/bin/monitors/examservices_monitor.sh

Arguments:

I have actually tried just specifying the scipt name unqualified in the External Program field as well and there was no difference.

Everything else is left at their default values.

As I mentioned, it all works when I run from the command line.

Regards

Craig

p.s. Sorry, omitted to mention we are running 10.0.1 Build 378.0 with HF3.

2

You might have better luck posting this in the monitoring section..

http://devcentral.f5.com/Default.aspx?tabid=53&view=topics&forumid=32

3

Check the permissions on the monitor, should be be read/write/execute by root. Moving this topic to monitor forum.

4

Hi all,

The permissions appear to be fine

-rwxr--r-x    1 root     root          978 Mar 16 16:02 examservices_monitor.sh

Any other ideas anyone?

Craig

5

- Can you send us the monitor’s setting? (eg. b monitor list and b pool list)

- Try running curl from shell, do you see the expected result?

eg. curl -kv https://x.x.x.x/exam-services/student 2>&1 | grep -i “/exam-services”

- FYI, printing “DOWN” and any words will mark the pool member up.

- Try enabling bigd debug (eg. b db bigd.debug enable ) and review /var/log/bigdlog.

- Check log file (eg. grep /var/log/ltm | tail -30)

6

And also to test your monitor setting, create a simple external script eg.

!/bin/sh 
 echo $1 $2 >> /tmp/test.log

Check whether the log file is increasing (eg. tail -f /tmp/test.log)

7

Hmm … Try restarting bigd, or if possible, restart the box.

Is there any issue with other monitor?

8

Hi Hum,

yes, the other monitors are also playing up. It seems that they work for a very short time on a reboot (well most of them do) and go into a UP state, but stay in the UP state regradless of whether the nodes are accessible or not.

What effect will restarting bigd have on the unit? Will it drop connections or cause service interuptions?

Regards,

Craig

9

Hi Craig,

bigd is responsible with health check. When it’s not running, monitors are not available.

If it’s a production unit, I recommend to do it in a maintenance window.

Or, you may want to work with F5 support? Anyone else has idea?

10

Hi Hum,

ok, I’ll try restarting the bigd service. Very odd, I was going to test this out on the standby unit and noticed the external health monitor is working on the standby unit but not the live unit. I think I will schedule a failover and see how things go.

Craig

11

Hmm that’s odd.. what’s the difference between the two boxes? How is the environment set up? Are these inline or do you typically SNAT?

If they’re inline and your doing a bunch of self IPs I would verify self IPs between boxes…

12

Hi,

the two boxes are identical models running the same software release (10.0.1 B378.0 with HF3). As I understand it we have a fairly bog standard high availability setup (active/standby).

I was just about to force a failover to the standby unit and thought I would just check to see if it all looked ok and now the pool that was using the external monitor is flagged as down. If I run the script manually it reports the remote system as up. This is very confusing.

I tried restarting the bigd service on the standby unit and it made no difference. The LTM log is reporting that it thinks the members in the pool are down

Mar 18 10:38:52 local/bigip-u2 notice mcpd[1672]: 01070638:5: Pool member nnn.nnn.nnn.nnn:443 monitor status down. 
 Mar 18 10:38:52 local/tmm err tmm[2485]: 01010028:3: No members available for pool Test_HM 
 Mar 18 10:43:20 local/bigip-u2 notice root: -bash  ==> /usr/bin/bigstart restart bigd 
 Mar 18 10:43:20 local/bigip-u2 notice mcpd[1672]: 01070410:5: Removed subscription with subscriber id BIGD_Subscriber

I’ll reboot the standby unit and see if there is any change from a fresh reboot.

Craig

13

Ok, after a clean reboot the pool the monitor is applied to is flagged as down now, depsite the fact when I run the script from the command line it works and reports the remote server as up.

This is all very confusing and somewhat frustrating. Might have to log a call over this one as I am running out of ideas.

Regards,

Craig

14

If you still want to experiment, try creating a simple monitor, eg.

!/bin/sh   
 echo $1 $2 >> /tmp/test.log   
 echo UP

Do you see /tmp/test.log file increasing and the pool member UP?

15

Hi Hum,

I’l give it a go, but to add to the confusion, the orginal internal health monitor I had which was NOT working has now started to work on the standby unit. There aren’t any issues having monitors querying the same pool members is there? It almost seems to be when one monitor works the other fails.

The internal monitor was just doing a

GET /exam-services/student HTTP/1.1\r\nHost: \r\nConnection:Close\r\n

and expecting HTTP/1.1 200 OK

Although this is a HTTPS monitor it was suggested I didn’t need to specify a client certificate or client key because the monitors don’t use them.

Regards,

Craig

16

Hi Hum,

just tried creating that simple monitor and assigning it to a poo. They pool is flagged as down and no log file is created.

I’m getting quite concerned that something is not right with our units.

Regards,

Craig

17

Hi Hum,

Will do. It’s getting beyond me. Thanks for all you help all the same.

Craig

18

Don’t mention it .. It would be my pleasure if I could help you. Feel free to let us know if you need help …

19

Hello, I’ve got a similar issue: http://devcentral-sea.f5.com/Forums/tabid/53/afv/topic/aff/32/aft/1172321/Default.aspx Have you had any luck? I’m about to raise a support call with F5.

Running BIG-IP 9.4.7 Build 320.1 Final

20

Hi There,

I have the very same problem. My script works just fine from the command line but F5 seems to not be able to run the script. Nothing is logged in the script debug file and the server is marked as down. Is there any update on this bug?