HTTPMonitor_cURL_ProbeTimeout

I am not sure how to implement the sample monitor posted on DevCentral

I cannot use the http monitor as the string I need to recieve is longer than 5120 bytes

The following script should work but I don’t know how to implement it

PIDFILE=“/var/run/`basename ${0}`.${NODE}_${PORT}.pid”

kill of the last instance of this monitor if hung and log current pid

if [ -f $PIDFILE ]

then

kill -9 `cat $PIDFILE` > /dev/null 2>&1

fi

echo “$$” > $PIDFILE

CAUCHOPORT=4000

POOL=cwmp.stage.443

URI=/caucho-status

P_TIMEOUT=5

send request & check for expected response

for i in `seq 1 2`; do

curl -fNs -m ${P_TIMEOUT} http://w50-$i.c50.domain.com:${CAUCHOPORT}${URI} | grep -io “w50-$i”‘.c50.domain.com:6802 .ok.’ 2>&1 > /dev/null

exstatus=$?

echo “exstatus = $exstatus”

let bigd mark node UP if expected response was received

otherwise force the node DOWN immediately

if [ $exstatus -eq 0 ]

then

echo bigpipe pool ${POOL} member 10.50.1.$i up 2>&1 > /dev/null

else

echo bigpipe pool ${POOL} member 10.50.1.$i down 2>&1 > /dev/null

fi

rm -f $PIDFILE

done

exit

The Adv Design & Config “ExternalMonitor” wiki page (Click here) has the implementation details you’re looking for.

/deb

The directions say to put it in /config/eav but the F5 insists it be in /usr/bin/monitors

Once I put the script there

I added the following name/value variables in the GUI

CAUCHOPORT=4000

POOL=cwmp.stage.443

URI=/caucho-status

P_TIMEOUT=5

and the following argument

curl -fNs -m ${P_TIMEOUT} http://w50-$i.c50.2wire.com

I set the alias service port to 4000

I changed the script to match the F5 example here

then

echo “UP”

else

bigpipe pool ${POOL} member ${IP}:${PORT} down 2>&1 > /dev/null

fi

but it makes no difference

The members are shown as red and down

Not sure what I am dong wrong

OK, wiki page is updated to indicate the correct directory, sorry for the confusion there.

Now as far as your variables are concerned, you need either set them in the monitor definition as name/value pairs OR set them within the script. If you do both, your monitor settings will be ignored and the script values will be used for every instance of the script. In this case, since they are the same, that isn’t the problem. (Also, if your pool members are defined on port 4000 already, you don’t need to specify that as a variable value in either place.)

I’m not sure I understand your intention regarding the argument you’ve specified, but since your script doesn’t look for any arguments (passed as $3 or greater), eliminate all arguments from the monitor definition.

LTM by default isn’t configured for name resolution, and even if it were, resolving a name within a monitor is not recommended since the induced latency will most likely result in intermittent false down’s.

You also have not included the required LTMv9 IP address parsing line. (This is the most likely reason your monitor is failing -- no valid destination address. I bet if you looked at a trace, you’d see no monitor traffic.)

I didn’t notice reading your script before, but it seems to be using a for loop to check both pool members, and the loop is terminated in the wrong place for proper pidfile management. You are also using a bigpipe command to manipulate the node status. Neither approach represents how LTM monitors are intended to work, and both of them will interfere with the normal operation of the monitor.

Hold on & I’ll post a script that will work and the monitor definition to deploy it…

/deb

OK…

Assuming your pool members are listening for monitor traffic on the same port as load balanced traffic, use this script:

remove IPv6/IPv4 compatibility prefix (LTM passes addresses in IPv6 format)
IP=`echo ${1} | sed 's/::ffff://'`
PORT=${2}
PIDFILE="/var/run/`basename ${0}`.${IP}_${PORT}.pid"
 kill of the last instance of this monitor if hung and log current pid
if [ -f $PIDFILE ]
then
   kill -9 `cat $PIDFILE` > /dev/null 2>&1
fi
echo "$$" > $PIDFILE
 send request & check for expected response
curl -fNs -m ${P_TIMEOUT} http://${IP}:${PORT}${URI} | grep -i "${RECV}" 2>&1 > /dev/null
 let bigd mark node UP if expected response was received
 otherwise force the node DOWN immediately
if [ $? -eq 0 ]
then
    echo "UP"
else
    /bin/bigpipe pool ${POOL} member ${IP}:${PORT} down 2>&1 > /dev/null
fi
rm -f $PIDFILE
exit

And configure your monitor with these variables:

RECV=w50-[12].c50.domain.com:6802 .ok.
POOL=cwmp.stage.443
URI=/caucho-status
P_TIMEOUT=5

HTH

/deb

The script above did not work

I added CAUCHOPORT=4000 to the GUI template

and added

curl -fNs -m ${P_TIMEOUT} http://${IP}:${CAUCHOPORT}${URI} | grep -i “${RECV}” 2>&1 > /dev/null

but it still did not work

the reason being is that I need the F5 to check the resin status page at port 4000.

Resin handles the db connections on the back end.

Each app server hosts a page called resin status page (this page is only for monitoring)

This is located at IPADDRESS:4000/caucho-status

I need to grep each app servers resin status page for w50-1.c50.domain.com:6802 (ok) for app server 1

I need to grep the applicable page for w50-2c50.domain.com:6802 (ok) for app server 2

the pool cwmp.stage.443 actually sends traffic to port 5004 not 4000

but it still did not work

the pool cwmp.stage.443 actually sends to 5004

so when it downs the member traffice should not go to this port

Then you would want to use the supplied script and define the monitor template with the alias port of 4000, and it will pass it to the monitor as $2 / $PORT, rather than adding yet another variable with the same value.

/deb

Whether I put the 4000 in the GUI template or the script it still marks all servers as down.

I am not sure why

the script in my original post should work fine if the monitor can detect the UP per member

Your original script might work fine as a standalone, but if you are going to run it as a monitor, there are some special considerations you must take into account when coding. Review my first response for details.

If you could please post the bigip.conf sections for the virtual server, the pool, and the monitor definitions, I may be able to help further.

/deb

Attached is the monitor pool and virtual server configs

It does not matter if the monitor is applied to the pool cwmp.stage.443 or cwmp.stage.80

10.50.1.3 w50-3 and 10.50.1.4 w50-4 are offline for other reasons

}

monitor curlcaucho {

defaults from external

dest *:4000

POOL “cwmp.stage.443”

P_TIMEOUT “5”

RECV “w50-[12].c50.domain.com:6802 .ok.”

run “curlcaucho”

URI “/caucho-status”

}

virtual cwmp.stage.443 {

destination 216.52.22.101:https

ip protocol tcp

profile cwmp.stage tcp

pool cwmp.stage.80

}

pool cwmp.stage.443 {

lb method member least conn

monitor all http and curlcaucho

member 10.50.1.1:5004

member 10.50.1.2:5004

member 10.50.1.3:5004

member 10.50.1.4:5004

}

sample html resin-status page to be parsed

resin-status

Server: w50-1.c50.domain.com

Config: /var/cms/conf/resin.conf

Server Start: Wed, 12 Dec 2007 14:28:23 -0800 (PST)

Server Reload: Wed, 12 Dec 2007 14:28:23 -0800 (PST)

Total Memory: 769.26Meg

Free Memory: 32.08Meg

TCP ports

Current Threads Config

Protocolort Active Keepalive Total thread-max keepalive-max request-timeout

srun:6802 1 0 6 200 100 65

http:8080 1 0 3 200 100 65

Srun Servers

Host Active Count live-time dead-time request-timeout

1. w50-1.c50.domain.com:6802 (ok) 0 15 15 30

2. w50-2.c50.domain.com:6802 (ok) 0 15 15 30

The script and monitor definition look fine, unless your servers are expecting an HTTPS request.

I see you have 2 monitors applied - the external monitor we’ve been discusssing + the default http monitor. I’d start by removing the default http monitor until I got the other working, then decide if it’s even necessary.

If the external monitor by itself is still is marking the pool members down, you’ll need to capture a packet trace from the server vlan, filtering on the LTM non-floating self-IP and port 4000, and look for monitor traffic from LTM to the servers & back.

HTH

/deb

removing http monitor makes no difference and I do need both.

I believe I did the tcpdump correct but I see see only one packet

Internal is 10.50.0.246 and is not the floating IP.

lb50-1:Active] / tcpdump -i internal:4000

tcpdump: WARNING: internal:4000: no IPv4 address assigned

tcpdump: listening on internal:4000

15:56:57.203234 802.1Q vlan4095 P0 tmm.9141 > localhost.localdomain.syslog: udp 64 (DF)

A full tcpdump from the GUI does not show any traffic either

I have also noticed doing a b load adds a second = in the GUI monitor variable secition

removing them makes no difference

Your tcpdump command line should be something like:

tcpdump -nni internal host  and port 4000

If you don’t see any traffic, your monitor is not sending the expected request, and you need to fall back to the command line to verify the script is working as expected.

To test the script at the command line, first export the variables you have defined:

export RECV="w50-[12].c50.domain.com:6802 .ok."
export POOL="cwmp.stage.443"
export URI="/caucho-status"
export P_TIMEOUT="5"

Verify the expected values have been properly exported by running the “set” command and examining the output. Then run the monitor from the command line, supplying the pool member address and port to which the monitor traffic should be sent:

/usr/bin/monitors/curlcaucho 10.50.1.1 5004

If you see the word “UP”, the monitor script is working as expected. (That should be the only output you ever see.)

If the pool member is up, but no output was seen from the script, run the script again with the shell debug flag set:

sh -x /usr/bin/monitors/curlcaucho 10.50.1.1 5004

and examine the output to determine what isn’t working as expected.

HTH

/deb

The script echoes UP as it should but it does not mark the pools as UP in the GUI template.

This statement w50-[12].c50.domain.com:6802 .ok." says if it sees any page that has

w50-1 or w50-2.domain in it and echoes as up

My original script is correct in that I need it to run a for loop

for i in `seq 1 2`; do

curl -fNs -m ${P_TIMEOUT} http://w50-$i.c50.domain.com:${CAUCHOPORT}${URI} | grep -io “w50-$i”‘.c50.domain.com:6802 .ok.’ 2>&1 > /dev/null

It should only down the $i for each app server

Each status page will look like the example I posted

1. w50-1.c50.domain.com:6802 (ok)
0
15
15
30
2. w50-2.c50.domain.com:6802 (ok)
0

above is part of one app servers resiin page Each app server will list all of the app servers in the cluster.

as shown below

echo bigpipe pool ${POOL} member 10.50.1.$i up 2>&1 > /dev/null

else

echo bigpipe pool ${POOL} member 10.50.1.$i down 2>&1 > /dev/null

I was not able to get this to work

tcpdump -nni host and port 4000

I tried tcpdump -nni hostname 10.50.0.246:4000

tcpdump: Interface name too long. Vlan names longer than 15 characters must use vlan as the interface name

My original script is correct in that I need it to run a for loop

for i in `seq 1 2`; do

curl -fNs -m ${P_TIMEOUT} http://w50-$i.c50.domain.com:${CAUCHOPORT}${URI} | grep -io “w50-$i”‘.c50.domain.com:6802 .ok.’ 2>&1 > /dev/null

As I mentioned before, this approach may work fine for a standalone script intended to monitor multiple hosts, but that is not what you are trying to build here. What you need is an LTM monitor, and an LTM monitor script is really a template for monitoring a single pool member. Whether you apply an LTM monitor to an individual pool member or to the entire pool in the GUI, a separate copy of the monitor runs for each pool member, passing only that specific IP & port to be tested and maintaining only that single tested pool member’s availability.

Since the provided monitor script returns the expected value when run from the command line, the next step is to apply it as an external monitor using the settings recommended previously, and look at a trace of the monitor traffic to see what’s going wrong.

Your tcpdump command should include the interface name before the host IP & port:

tcpdump -nni internal host 10.50.0.246 and port 4000

There is a logical problem you’re bound to encounter once you get the monitor working, though: Since all pool members’ status will be reflected in the output of this script, and the wildcard expression in the current RECV string will match all instances of the reported status, each pool member will be marked UP if any of them are UP (IOW, you will only ever get a DOWN status using this script against this page if all servers are DOWN.)

To work around that, you’ll need to make your RECV string something more specific per instance. Since the last octet of the IP address is the same as the host number value you need, you should be able to use this string as the new RECV string to substitute that value on the fly for each monitor instance:

w50-`echo $IP | cut -d. -f4`.c50.domain.com:6802 .ok.

Be sure to test the script itself at the command line as you did before to verify the intended functionality before defining it as an external monitor.

/deb

lb50-1-cs:Active] monitors tcpdump -nni internal host 10.50.0.246 and port 4000

tcpdump: listening on internal

0 packets received by filter

0 packets dropped by kernel

running tcp dump while curl caucho is enabled gives no packets

If I enable the following export then the script works

The IP export must be added first because the script does run manually does not determine the IP based on

this command

IP=`echo ${1} | sed ‘s/::ffff://’`

export IP=“10.50.1.1”

export RECV=“w50-`echo $IP | cut -d. -f4`.c50.domain.com:6802 .ok.”

export POOL=“cwmp.stage.80”

export URI=“/caucho-status”

export P_TIMEOUT=“5”

export CAUCHOPORT=4000

[root@lb50-1-cs:Active] monitors sh -xv /usr/bin/monitors/curlcaucho 10.50.1.1 4004

remove IPv6/IPv4 compatibility prefix (LTM passes addresses in IPv6 format)

IP=`echo ${1} | sed ‘s/::ffff://’`

echo ${1} | sed ‘s/::ffff://’

++ echo 10.50.1.1

++ sed s/::ffff://

+ IP=10.50.1.1

PORT=${2}

+ PORT=4004

RECV=“w50-$i”‘.c50.domain.com:6802 .ok.’

PIDFILE=“/var/run/`basename ${0}`.${IP}_${PORT}.pid”

basename ${0}

++ basename /usr/bin/monitors/curlcaucho

+ PIDFILE=/var/run/curlcaucho.10.50.1.1_4004.pid

kill of the last instance of this monitor if hung and log current pid

if [ -f $PIDFILE ]

then

kill -9 `cat $PIDFILE` > /dev/null 2>&1

fi

+ ‘[’ -f /var/run/curlcaucho.10.50.1.1_4004.pid ‘]’

echo “$$” > $PIDFILE

+ echo 4737

send request & check for expected response

for i in `seq 1 2`; do

curl -fNs -m ${P_TIMEOUT} http://${IP}:${CAUCHOPORT}${URI} | grep -io “${RECV}” 2>&1 > /dev/null

+ curl -fNs -m 5 http://10.50.1.1:4000/caucho-status

+ grep -io ‘w50-1.c50.domain.com:6802 .ok.’

let bigd mark node UP if expected response was received

otherwise force the node DOWN immediately

if [ $? -eq 0 ]

then

echo “UP”

else

echo /bin/bigpipe pool ${POOL} member ${IP}:${PORT} down 2>&1 > /dev/null

fi

+ ‘[’ 0 -eq 0 ‘]’

+ echo UP

UP

rm -f $PIDFILE

+ rm -f /var/run/curlcaucho.10.50.1.1_4004.pid

exit

+ exit

Oh, IP and PORT should have been on the export list for command line testing. Probably a cut & paste error on my part, sorry about that.

Geez, not sure what I was thinking here -- IP & port are specified as command line arguments as indicated in the command above… Neither needs to be set as an env var if they are properly specified on the command line when running the script manually.

So it looks like it’s working as expected. Did you have another question…?

/deb

It only works manually It does not work with the variables in the template

I see no traffic when I run

lb50-1-cs:Active] monitors tcpdump -nni internal host 10.50.0.246 and port 4000

tcpdump: listening on internal

0 packets received by filter

0 packets dropped by kernel

running tcp dump while curl caucho is enabled gives no packet

I don’t think it is parsing $1 and $2 automatically

Do you see the traffic with the same tcpdump filter when testing from the command line?

Please post your current monitor definition (output from "b list monitor " )

/d

If you’re not seeing the traffic in a trace in either case, perhaps you have the wrong self IP in the tcpdump filter. You can change it to the IP address of the pool member you are testing against (10.50.1.1) and see if you can see the target traffic then.

/deb