Health Monitors for Exchange 2010

I was recently asked to develop health monitors on my 10.2.4HF3 LTMs for our Exchange 2010 environment. Two of the three monitors were a bit challenging, so I wanted to share what I developed.  The monitors have been sanitized to protect our environment - modify them to fit yours.

*** Active Sync ***

This monitor was challenging me, until I stumbled across this statement for versions 10.2.x and 11.x in SOL2167:
“When Basic Authentication is enabled by configuring a User Name and Password in the monitor definition, the system inserts the Authorization header and a terminating double CR/LF sequence (0x0d 0x0a 0x0d 0x0a) after the last character in the Send String.”

Once I read that, I removed my standard trailing “\r\n\r\n” sequence at the end of the Send String, and the monitor immediately started working. It saved me from having to use an External monitor. You will need to insert a username, password, and host header value in the send string to fit your environment:

<span id="lnum1" style="color: rgb(96, 96, 96);">   1:</span> ltm monitor https active-sync {
<span id="lnum2" style="color: rgb(96, 96, 96);">   2:</span> cipherlist <span style="color: rgb(0, 96, 128);">"DEFAULT:+SHA:+3DES:+kEDH"</span>
<span id="lnum3" style="color: rgb(96, 96, 96);">   3:</span> compatibility <span style="color: rgb(0, 96, 128);">"enabled"</span>
<span id="lnum4" style="color: rgb(96, 96, 96);">   4:</span> defaults-from https
<span id="lnum5" style="color: rgb(96, 96, 96);">   5:</span> interval 10
<span id="lnum6" style="color: rgb(96, 96, 96);">   6:</span> password <span style="color: rgb(0, 96, 128);">"<password>"</span>
<span id="lnum7" style="color: rgb(96, 96, 96);">   7:</span> recv <span style="color: rgb(0, 96, 128);">"MS-ASProtocolCommands: Sync,SendMail,SmartForward,SmartReply,GetAttachment,GetHierarchy,CreateCollection,DeleteCollection,MoveCollection,FolderSync"</span>
<span id="lnum8" style="color: rgb(96, 96, 96);">   8:</span> send <span style="color: rgb(0, 96, 128);">"OPTIONS /Microsoft-Server-ActiveSync/ HTTP/1.1\r\nHost: <host header value>"</span>
<span id="lnum9" style="color: rgb(96, 96, 96);">   9:</span> time-<span style="color: rgb(0, 0, 255);">until</span>-up 0
<span id="lnum10" style="color: rgb(96, 96, 96);">  10:</span> timeout 31
<span id="lnum11" style="color: rgb(96, 96, 96);">  11:</span> username <span style="color: rgb(0, 96, 128);">"<domain>\<username>"</span>
<span id="lnum12" style="color: rgb(96, 96, 96);">  12:</span> }

*** Outlook Web Access ***

This monitor was straightforward. You will need to replace the FQDN in both the URI and the Host: header in the Send String:

<span id="lnum1" style="color: rgb(96, 96, 96);">   1:</span> ltm monitor https outlook-web-access {
<span id="lnum2" style="color: rgb(96, 96, 96);">   2:</span> cipherlist <span style="color: rgb(0, 96, 128);">"DEFAULT:+SHA:+3DES:+kEDH"</span>
<span id="lnum3" style="color: rgb(96, 96, 96);">   3:</span> compatibility <span style="color: rgb(0, 96, 128);">"enabled"</span>
<span id="lnum4" style="color: rgb(96, 96, 96);">   4:</span> defaults-from https
<span id="lnum5" style="color: rgb(96, 96, 96);">   5:</span> interval 10
<span id="lnum6" style="color: rgb(96, 96, 96);">   6:</span> recv <span style="color: rgb(0, 96, 128);">"OutlookSession="</span>
<span id="lnum7" style="color: rgb(96, 96, 96);">   7:</span> send <span style="color: rgb(0, 96, 128);">"GET /owa/auth/logon.aspx?url=https://<hostname>/owa/&reason=0 HTTP/1.1\r\nUser-Agent: Mozilla/4.0\r\nHost: <hostname>\r\n\r\n"</span>
<span id="lnum8" style="color: rgb(96, 96, 96);">   8:</span> time-<span style="color: rgb(0, 0, 255);">until</span>-up 0
<span id="lnum9" style="color: rgb(96, 96, 96);">   9:</span> timeout 31
<span id="lnum10" style="color: rgb(96, 96, 96);">  10:</span> }

*** Outlook Anywhere ***

This monitor was complicated enough that it could not be done by any of the built-in monitor types, so I had to run this with CURL as an External monitor. Here is the LTM monitor definition:

<span id="lnum1" style="color: rgb(96, 96, 96);">   1:</span> ltm monitor external outlook-anywhere {
<span id="lnum2" style="color: rgb(96, 96, 96);">   2:</span> defaults-from external
<span id="lnum3" style="color: rgb(96, 96, 96);">   3:</span> interval 10
<span id="lnum4" style="color: rgb(96, 96, 96);">   4:</span> run <span style="color: rgb(0, 96, 128);">"outlook-anywhere.sh"</span>
<span id="lnum5" style="color: rgb(96, 96, 96);">   5:</span> time-<span style="color: rgb(0, 0, 255);">until</span>-up 0
<span id="lnum6" style="color: rgb(96, 96, 96);">   6:</span> timeout 31
<span id="lnum7" style="color: rgb(96, 96, 96);">   7:</span> user-defined debug <span style="color: rgb(0, 96, 128);">"0"</span>
<span id="lnum8" style="color: rgb(96, 96, 96);">   8:</span> user-defined debugfile <span style="color: rgb(0, 96, 128);">"/shared/tmp/outloo-anywhere_debug.log"</span>
<span id="lnum9" style="color: rgb(96, 96, 96);">   9:</span> user-defined receivestring <span style="color: rgb(0, 96, 128);">"200 Success"</span>
<span id="lnum10" style="color: rgb(96, 96, 96);">  10:</span> user-defined username <span style="color: rgb(0, 96, 128);">"<domain>\<username>"</span>
<span id="lnum11" style="color: rgb(96, 96, 96);">  11:</span> }

In order to get this monitor to run in your environment you must update the username variable. In our environment, I had to precede the username with our domain name in order for it to return successful. The outlook-anywhere.sh script looks like this. It needs to be placed in /config/monitors with execute permission:

<span id="lnum1" style="color: rgb(96, 96, 96);">   1:</span> #!/bin/bash
<span id="lnum2" style="color: rgb(96, 96, 96);">   2:</span> #
<span id="lnum3" style="color: rgb(96, 96, 96);">   3:</span> # Exchange 2010 Outlook Anywhere external health monitor
<span id="lnum4" style="color: rgb(96, 96, 96);">   4:</span> #
<span id="lnum5" style="color: rgb(96, 96, 96);">   5:</span> # Syntax: /config/monitors/outlook-anywhere.sh
<span id="lnum6" style="color: rgb(96, 96, 96);">   6:</span> #
<span id="lnum7" style="color: rgb(96, 96, 96);">   7:</span> # Author:
<span id="lnum8" style="color: rgb(96, 96, 96);">   8:</span> # Date:
<span id="lnum9" style="color: rgb(96, 96, 96);">   9:</span> #
<span id="lnum10" style="color: rgb(96, 96, 96);">  10:</span> # Important Notes:
<span id="lnum11" style="color: rgb(96, 96, 96);">  11:</span> # * Username must be preceded by the <span style="color: rgb(0, 96, 128);">"<domain>\"</span> string
<span id="lnum12" style="color: rgb(96, 96, 96);">  12:</span> # * There should be four Variables configured <span style="color: rgb(0, 0, 255);">in</span> the monitor properties:
<span id="lnum13" style="color: rgb(96, 96, 96);">  13:</span> # - USERNAME: the user credentials used <span style="color: rgb(0, 0, 255);">to</span> perform the check
<span id="lnum14" style="color: rgb(96, 96, 96);">  14:</span> # - RECEIVESTRING: The string <span style="color: rgb(0, 0, 255);">to</span> look <span style="color: rgb(0, 0, 255);">for</span> <span style="color: rgb(0, 0, 255);">in</span> a successful response
<span id="lnum15" style="color: rgb(96, 96, 96);">  15:</span> # - DEBUG: Enables output debugging
<span id="lnum16" style="color: rgb(96, 96, 96);">  16:</span> # - DEBUGFILE: File which tores debug output (<span style="color: rgb(0, 0, 255);">if</span> DEBUG is enabled)
<span id="lnum17" style="color: rgb(96, 96, 96);">  17:</span> # * The password <span style="color: rgb(0, 0, 255);">for</span> USERNAME should only be stored <span style="color: rgb(0, 0, 255);">in</span> this file. This way,
<span id="lnum18" style="color: rgb(96, 96, 96);">  18:</span> # it is not accessible via TMSH commands or by viewing the LTM config file
<span id="lnum19" style="color: rgb(96, 96, 96);">  19:</span> # * <span style="color: rgb(0, 0, 255);">To</span> execute this script manually, uncomment the variables and execute
<span id="lnum20" style="color: rgb(96, 96, 96);">  20:</span> # the shell script. The output will be stored <span style="color: rgb(0, 0, 255);">in</span> DEBUGFILE.
<span id="lnum21" style="color: rgb(96, 96, 96);">  21:</span> # * CAUTION: <span style="color: rgb(0, 0, 255);">If</span> you execute this script manually, Make sure you comment
<span id="lnum22" style="color: rgb(96, 96, 96);">  22:</span> # the USERNAME, RECEIVESTRING, DEBUG, and DEBUGFILE variables (<span style="color: rgb(0, 0, 255);">do</span> not
<span id="lnum23" style="color: rgb(96, 96, 96);">  23:</span> # commend the PASSWORD variable). <span style="color: rgb(0, 0, 255);">If</span> you <span style="color: rgb(0, 0, 255);">do</span> not comment these variables,
<span id="lnum24" style="color: rgb(96, 96, 96);">  24:</span> # they will override the variables configured <span style="color: rgb(0, 0, 255);">in</span> the Monitor properties
<span id="lnum25" style="color: rgb(96, 96, 96);">  25:</span> #
<span id="lnum26" style="color: rgb(96, 96, 96);">  26:</span> # Revisions:
<span id="lnum27" style="color: rgb(96, 96, 96);">  27:</span> #
<span id="lnum28" style="color: rgb(96, 96, 96);">  28:</span> 
<span id="lnum29" style="color: rgb(96, 96, 96);">  29:</span> # <span style="color: rgb(0, 0, 255);">Do</span> not comment out the PASSWORD variable
<span id="lnum30" style="color: rgb(96, 96, 96);">  30:</span> PASSWORD='<password>'
<span id="lnum31" style="color: rgb(96, 96, 96);">  31:</span> 
<span id="lnum32" style="color: rgb(96, 96, 96);">  32:</span> # Uncomment these variables temporarily <span style="color: rgb(0, 0, 255);">if</span> you want <span style="color: rgb(0, 0, 255);">to</span> execute the
<span id="lnum33" style="color: rgb(96, 96, 96);">  33:</span> # script manually
<span id="lnum34" style="color: rgb(96, 96, 96);">  34:</span> #USERNAME='\'
<span id="lnum35" style="color: rgb(96, 96, 96);">  35:</span> #RECEIVESTRING='200 Success'
<span id="lnum36" style="color: rgb(96, 96, 96);">  36:</span> #DEBUG=1
<span id="lnum37" style="color: rgb(96, 96, 96);">  37:</span> #DEBUGFILE=/shared/tmp/outlook-anywhere_debug.log
<span id="lnum38" style="color: rgb(96, 96, 96);">  38:</span> 
<span id="lnum39" style="color: rgb(96, 96, 96);">  39:</span> # Remove IPv6/IPv4 compatibility prefix (LTM passes addresses <span style="color: rgb(0, 0, 255);">in</span> IPv6 format)
<span id="lnum40" style="color: rgb(96, 96, 96);">  40:</span> IP=`echo ${1} | sed 's/::ffff://'`
<span id="lnum41" style="color: rgb(96, 96, 96);">  41:</span> PORT=${2}
<span id="lnum42" style="color: rgb(96, 96, 96);">  42:</span> 
<span id="lnum43" style="color: rgb(96, 96, 96);">  43:</span> # Create a PID file
<span id="lnum44" style="color: rgb(96, 96, 96);">  44:</span> PIDFILE=<span style="color: rgb(0, 96, 128);">"/var/run/`basename ${0}`.${IP}_${PORT}.pid"</span>
<span id="lnum45" style="color: rgb(96, 96, 96);">  45:</span> 
<span id="lnum46" style="color: rgb(96, 96, 96);">  46:</span> # Kill of the last instance of this monitor <span style="color: rgb(0, 0, 255);">if</span> hung and log current pid
<span id="lnum47" style="color: rgb(96, 96, 96);">  47:</span> <span style="color: rgb(0, 0, 255);">if</span> [ -f $PIDFILE ]
<span id="lnum48" style="color: rgb(96, 96, 96);">  48:</span> <span style="color: rgb(0, 0, 255);">then</span>
<span id="lnum49" style="color: rgb(96, 96, 96);">  49:</span> kill -9 `cat $PIDFILE` ><span style="color: rgb(0, 128, 0);">; /dev/null 2>&1</span>
<span id="lnum50" style="color: rgb(96, 96, 96);">  50:</span> fi
<span id="lnum51" style="color: rgb(96, 96, 96);">  51:</span> echo <span style="color: rgb(0, 96, 128);">"$$"</span> ><span style="color: rgb(0, 128, 0);">; $PIDFILE</span>
<span id="lnum52" style="color: rgb(96, 96, 96);">  52:</span> 
<span id="lnum53" style="color: rgb(96, 96, 96);">  53:</span> 
<span id="lnum54" style="color: rgb(96, 96, 96);">  54:</span> <span style="color: rgb(0, 0, 255);">if</span> [ $DEBUG -eq 0 ]
<span id="lnum55" style="color: rgb(96, 96, 96);">  55:</span> <span style="color: rgb(0, 0, 255);">then</span>
<span id="lnum56" style="color: rgb(96, 96, 96);">  56:</span> 
<span id="lnum57" style="color: rgb(96, 96, 96);">  57:</span> curl -v -k --request RPC_IN_DATA -A MSRPC -u <span style="color: rgb(0, 96, 128);">"${USERNAME}:${PASSWORD}"</span> --ntlm -H <span style="color: rgb(0, 96, 128);">"Host: <hostname>"</span> <span style="color: rgb(0, 96, 128);">"https://${IP}/rpc/rpcproxy.dll?:6001"</span> 2><span style="color: rgb(0, 128, 0);">;&1 | grep -ic "${RECEIVESTRING}" > /dev/null 2>&1</span>
<span id="lnum58" style="color: rgb(96, 96, 96);">  58:</span> 
<span id="lnum59" style="color: rgb(96, 96, 96);">  59:</span> <span style="color: rgb(0, 0, 255);">else</span>
<span id="lnum60" style="color: rgb(96, 96, 96);">  60:</span> 
<span id="lnum61" style="color: rgb(96, 96, 96);">  61:</span> echo <span style="color: rgb(0, 96, 128);">"*******************"</span> 2><span style="color: rgb(0, 128, 0);">;&1 > "${DEBUGFILE}"</span>
<span id="lnum62" style="color: rgb(96, 96, 96);">  62:</span> echo -e <span style="color: rgb(0, 96, 128);">"curl -v -k --request RPC_IN_DATA -A MSRPC -u '${USERNAME}:${PASSWORD}' --ntlm -H \"</span>Host: <hostname>\<span style="color: rgb(0, 96, 128);">" \"</span>https://${IP}/rpc/rpcproxy.dll?:6001\<span style="color: rgb(0, 96, 128);">" 2>&1 | grep '200 Success'\n"</span> 2><span style="color: rgb(0, 128, 0);">;&1 >> "${DEBUGFILE}"</span>
<span id="lnum63" style="color: rgb(96, 96, 96);">  63:</span> echo <span style="color: rgb(0, 96, 128);">"USERNAME: ${USERNAME}"</span> 2><span style="color: rgb(0, 128, 0);">;&1 >> "${DEBUGFILE}"</span>
<span id="lnum64" style="color: rgb(96, 96, 96);">  64:</span> echo <span style="color: rgb(0, 96, 128);">"PASSWORD: ${PASSWORD}"</span> 2><span style="color: rgb(0, 128, 0);">;&1 >> "${DEBUGFILE}"</span>
<span id="lnum65" style="color: rgb(96, 96, 96);">  65:</span> echo <span style="color: rgb(0, 96, 128);">"RECEIVESTRING: ${RECEIVESTRING}"</span> 2><span style="color: rgb(0, 128, 0);">;&1 >> "${DEBUGFILE}"</span>
<span id="lnum66" style="color: rgb(96, 96, 96);">  66:</span> echo <span style="color: rgb(0, 96, 128);">"DEBUG: ${DEBUG}"</span> 2><span style="color: rgb(0, 128, 0);">;&1 >> "${DEBUGFILE}"</span>
<span id="lnum67" style="color: rgb(96, 96, 96);">  67:</span> echo <span style="color: rgb(0, 96, 128);">"DEBUGFILE: ${DEBUGFILE}"</span> 2><span style="color: rgb(0, 128, 0);">;&1 >> "${DEBUGFILE}"</span>
<span id="lnum68" style="color: rgb(96, 96, 96);">  68:</span> echo -e <span style="color: rgb(0, 96, 128);">"*******************\n"</span> 2><span style="color: rgb(0, 128, 0);">;&1 >> "${DEBUGFILE}"</span>
<span id="lnum69" style="color: rgb(96, 96, 96);">  69:</span> 
<span id="lnum70" style="color: rgb(96, 96, 96);">  70:</span> curl -v -k --request RPC_IN_DATA -A MSRPC -u <span style="color: rgb(0, 96, 128);">"${USERNAME}:${PASSWORD}"</span> --ntlm -H <span style="color: rgb(0, 96, 128);">"Host: <hostname>"</span> <span style="color: rgb(0, 96, 128);">"https://${IP}/rpc/rpcproxy.dll?:6001"</span> 2><span style="color: rgb(0, 128, 0);">;&1 | tee -a "${DEBUGFILE}" | grep -ic "${RECEIVESTRING}" > /dev/null 2>&1</span>
<span id="lnum71" style="color: rgb(96, 96, 96);">  71:</span> fi
<span id="lnum72" style="color: rgb(96, 96, 96);">  72:</span> 
<span id="lnum73" style="color: rgb(96, 96, 96);">  73:</span> # Retain the <span style="color: rgb(0, 0, 255);">return</span> code of the grep command.
<span id="lnum74" style="color: rgb(96, 96, 96);">  74:</span> EXITCODE=$?
<span id="lnum75" style="color: rgb(96, 96, 96);">  75:</span> 
<span id="lnum76" style="color: rgb(96, 96, 96);">  76:</span> # Need <span style="color: rgb(0, 0, 255);">to</span> remove PIDFILE here because the LTM terminates the script after
<span id="lnum77" style="color: rgb(96, 96, 96);">  77:</span> # receiving anythin <span style="color: rgb(0, 0, 255);">in</span> STDOUT
<span id="lnum78" style="color: rgb(96, 96, 96);">  78:</span> rm -f $PIDFILE
<span id="lnum79" style="color: rgb(96, 96, 96);">  79:</span> 
<span id="lnum80" style="color: rgb(96, 96, 96);">  80:</span> <span style="color: rgb(0, 0, 255);">if</span> [ $EXITCODE -eq 0 ]
<span id="lnum81" style="color: rgb(96, 96, 96);">  81:</span> <span style="color: rgb(0, 0, 255);">then</span>
<span id="lnum82" style="color: rgb(96, 96, 96);">  82:</span> echo <span style="color: rgb(0, 96, 128);">"UP"</span>
<span id="lnum83" style="color: rgb(96, 96, 96);">  83:</span> fi

  • In order to get this script to run in your environment you must update the PASSWORD variable, and insert your <hostname> into Host: header and the URI string in all three CURL command references.
  • To execute the script manually, you must define the variables that are normally passed by the LTM by uncommenting them near the top of the script. If you do uncomment these variables, make sure you comment them out when you are finished or they will override the variables in the Monitor definition:

USERNAME

RECEIVESTRING

DEBUG (if desired)

DEBUGFILE (if DEBUG desired)

Nice work SMP! That’s really helpful info.

With the addition of NTLM support for HTTP/S monitors in 11.1 you should be able to use an inbuilt monitor for the Outlook Anywhere monitor.

Aaron

hi SMP Thanks for your Post. I’m new to this topic and would like to monitor our ActiveSync server. Is there any way to get the script running on a windows server? Thanks.

’