how have others conquered external monitor resource alloc problems?

How complex is your monitor script?

 

 

IMO, external monitor script is quite expensive due to the process forking. Try to reduce number of commands in the script.

 

 

As an alternative, offload the monitors to another system (eg. web-based monitor check). So F5 is using HTTP to check the condition of all back-end servers . By doing so, F5 doesn't need to fork new process for each SNMP Get.

the script isn't too complicated. But i've got about 600 of them running.

Offloading the checks onto a secondary system and polling a results page would work pretty well, but it's still an external check via curl/wget/whatever though I suppose the of commands spawned would be reduced to just one.

Here's one of the monitors. This one only runs on about 200 nodes currently. there's another that is very similar (more snmpgets) that runs on about 400.

!/bin/bash
DESTIP=`echo $1 | sed 's/:FFFF://'`
COMMUNITY=$3
MAXSESSIONS=$4
pidfile="/var/run/monitor_nodes.$1..$2.pid"
if [ -f $pidfile ] ; then
 kill -9 `cat $pidfile` > /dev/null 2>&1
fu

echo "$$" > $pidfile
MAX=`nice snmpget -v2c -c $COMMUNITY $DESTIP my.really.long.oid | awk '{print $4}'`
SESSION_THROTTLE=`nice snmpget -v2c -c $COMMUNITY $DESTIP my.really.long.oid | awk '{print $4}'`

DOWN=0
if [ $SESSION_THROTTLE -eq 2 ] ; then
  DOWN=1
fi

if [ "x$MAX" == "x" ] ; then
  DOWN=1
fi

if [ $MAX -lt $MAXSESSIONS ] ; then
  DOWN=0
fi

if [ $DOWN -eq 0 ] ; then
  echo "up"
fi

rm -f $pidfile
 

the TCP Port check is probably the least expensive from the F5's point of view, correct? Wonder if I could get creative with a Tcl or Perl script that runs as a monitor on each node. Have the script open or close a TCP port on the host based on availability. As long as the port is open, the service is alive. If the port shuts, the service is in an overload condition or not running.. seems like a lot of connections, maybe UDP instead.

sounds like a lot of work.

Jeff

 

 

I have used external monitors that execute on the back-end node via inetd/tcpserver ( assuming backends are un*x -like ) with success, using telnet on the bigip. However , should the backend have a http server I think it would be easier to use a httpmonitor and have a script on the webserver do the same checks the shell script does. You can then check for "up" in the reply.

 

 

Right now every 6900 is forking 1 shell with at least 4 sub-shells ( back-quotes ) per back-end

 

With 600 be's , that makes 3000 instances of bash every x seconds.

 

Seeing the part with the pidfile and the kill, I assume you have noticed the scripts overlap or hang.

 

 

kr,

 

Ib

 

 

 

 

 

 

 

 

 

Hey I thought I'd chime in here because I did some work with external monitors over the past couple of days. It seems that the LTM will not execute any commands after receiving anything in STDOUT (i.e. after the "echo" command). So when the check is successful, the "rm -f $pidfile" command in your script does not get executed. I was troubleshooting the fact that my PID file was not getting cleaned up when I expected it to (my script was similar to yours - echo, then rm). But then I noticed the last comment in this Tech Tip:

 

https://devcentral.f5.com/Tutorials/TechTips/tabid/63/articleType/ArticleView/articleId/151/LTM-External-Monitors-The-Basics.aspx

 

 

 

FYI: When the external monitor script outputs "UP" than the deletion of the PIDFILE never occurs, neither does any further commands after the output of "UP"

 

 

It does however delete the PID if the monitor fails. I can only assume the F5 kills the script immediately once any output is detected...

 

 

 

After I read that I did some more testing, and sure enough, that's what I found too despite not being able to find any documentation on it. Not sure that's directly related to your scalability issue, but thought you might want to be aware of it as you develop your automation.

I concur. I found that the script is killed after any STDOUT is sent:

 

 

https://devcentral.f5.com/wiki/AdvDesignConfig.TemplateForExternalLtmMonitors.ashx

 

Note that any standard output will result in the script execution being stopped

 

So do any cleanup before echoing to STDOUT

 

 

Aaron

Thanks for the replies guys.

 

 

I had seen the scripts overlap. The SNMP polls are expensive as the BE nodes are running a custom app that provides the SNMP interface and it's a little slow. On top of that, the line card the 6900's are on are a wee bit overloaded at the moment and pretty much everything is slowing down because of it.

 

 

And thanks Ib - I was going to look into the pid file thing once I caught my breath. You saved me a ton of legwork there.

 

 

I dunno why I didn't think of the HTTP check method. I've done that in the past too. Must be because I'm not getting enough sleep. I must have been a real horrible person in a past life. :)

 

 

Appreciate all the feedback guys!