Forum Discussion
Burrell
Nimbostratus
NimbostratusExchange 2016 iApp RC2 issue
We have implemented our 2 node exchange 2016 environment with the RC2 iApp template. The cluster uses certificates to secure/encrypt traffic, and all appears to be working except EWS related functions with advanced monitors, if we step down to simple monitors everything works. We would like to have the additional application layer validation if possible.
Our issue is the EWS advanced health monitor is failing. How can we troubleshoot this pre-scripted monitor?
Below is the script the health monitor runs using credentials we provided. The credentials authenticate and are also used for OWA, POP3, IMAP and these health monitors are working/succeeding. It is only EWS that fails.
!/bin/sh
These arguments supplied automatically for all external monitors:
$1 = IP (nnn.nnn.nnn.nnn notation)
$2 = port (decimal, host byte order)
This script expects the following Name/Value pairs:
USER = the username associated with a mailbox
PASSWORD = the password for the user account
DOMAIN = the Windows domain in which the account lives
EMAIL = the email address associated with the user mailbox
Remove IPv6/IPv4 compatibility prefix (LTM passes addresses in IPv6 format)
NODE=`echo ${1} | sed 's/::ffff://'`
if [[ $NODE =~ ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}$ ]]; then
node is v4
NODE=${NODE}
else
node is v6
NODE=[${NODE}]
fi
PORT=${2}
PIDFILE="/var/run/`basename ${0}`.my_new_iapp_test_2010_${USER}_${NODE}_ews.pid"
DIR="http://schemas.microsoft.com/exchange/services/2006"
XML1=$( cat<IdOnly
EOF
)
XML2=$( cat<
EOF
)
XMLFULL=${XML1}${EMAIL}${XML2}
EWSURI='/EWS/Exchange.asmx'
RECV=''
kill of the last instance of this monitor if hung and log current pid
if [ -f $PIDFILE ]
then
echo "EAV exceeded runtime needed to kill ${NODE}:${PORT}" | logger -p local0.error
kill -9 `cat $PIDFILE` > /dev/null 2>&1
fi
echo "$$" > $PIDFILE
curl-apd -g -s --ntlm -k -X POST -H 'Content-Type: text/xml; charset=utf-8' -d "${XMLFULL}" -u ${DOMAIN}\\${USER}:${PASSWORD} https://${NODE}${EWSURI} | grep -i "${RECV}" > /dev/null 2>&1
STATUS=$?
rm -f $PIDFILE
if [ $STATUS -eq 0 ]
then
echo "UP"
fi
exit
Burrell, what response do you get if you run the curl command used by the monitor from the command line:
curl-apd -g -s --ntlm -k -X POST -H 'Content-Type: text/xml; charset=utf-8' -d "IdOnlyyouremail@domain.com" -u domain\user:password http:///EWS/Exchange.asmx | grep -i ""
BurrellThere doesn't appear to be a response when run from an ssh session and substituting the various data for our servername, email, and user account.
Its possible I am executing it wrong I suppose. I have removed any sensitive information from the version I ran. We have logged into the mailbox and sent a message so there would be content in the folder to ensure its working properly.
curl-apd -g -s --ntlm -k -X POST -H 'Content-Type: text/xml; charset=utf-8' -d "<?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages"><soap:Header></soap:Header><soap:Body><GetFolderxmlns="http://schemas.microsoft.com/exchange/services/2006/messages"><FolderShape><t:BaseShape>IdOnly</t:BaseShape></FolderShape><FolderIds><t:DistinguishedFolderId Id="inbox"><t:Mailbox><t:EmailAddress>F5HealthCheck1@****.com</t:EmailAddress></t:Mailbox></t:DistinguishedFolderId></FolderIds></GetFolder></soap:Body></soap:Envelope>" -u OURADDOMAIN.com\saf5healthcheck1:******* http://123.123.123.123:443/EWS/Exchange.asmx | grep -i "<m:GetFolderResponseMessage ResponseClass="Success">"Try adding the "-vv" option to the curl command to use verbose mode. Also, the monitor uses the IP address instead of the FQDN of the service, so try something like "http://1.2.3.4/EWS/Exchange.asmx".
- Burrell
-vv gave a response, I presume the issue is authentication related.
curl-apd -vv -g -s --ntlm -k -X POST -H 'Content-Type: text/xml; charset=utf-8' -d "<?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages"><soap:Header></soap:Header><soap:Body><GetFolderxmlns="http://schemas.microsoft.com/exchange/services/2006/messages"><t:BaseShape>IdOnly</t:BaseShape><t:DistinguishedFolderId Id="inbox"><t:Mailbox><t:EmailAddress>F5HealthCheck1@webdomain.com</t:EmailAddress></t:Mailbox></t:DistinguishedFolderId></soap:Body></soap:Envelope>" -u 'AD_DOMAIN.com\saf5healthcheck1':******** http://10.130.172.31:443/EWS/Exchange.asmx | grep -i "<m:GetFolderResponseMessage ResponseClass="Success">"Thanks so much for your assistance! I feel like this is enough to keep pursuing with the Exchange admin.
- Burrell
I just noticed I neglected to add the port 443 to the Node IP and now it seems to respond!
* timeout on name lookup is not supported * About to connect() to 10.130.172.31 port 443 (0) * Trying 10.130.172.31... * connected * Connected to 10.130.172.31 (10.130.172.31) port 443 (0) * Server auth using NTLM with user 'OURDOMAIN.com\saf5healthcheck1' > POST /EWS/Exchange.asmx HTTP/1.1 > Authorization: NTLM TlRMTVNTUAABAAAABoIIAAAAAAAAAAAAAAAAAAAAAAA= > User-Agent: curl/7.25.0 (x86_64-redhat-linux-gnu) libcurl/7.25.0 OpenSSL/1.0.1j zlib/1.2.3 libidn/0.6.5 > Host: 10.130.172.31:443 > Accept: */* > Content-Type: text/xml; charset=utf-8 > Content-Length: 0 > * Recv failure: Connection reset by peer * Closing connection 0 Is your pool member on 443 or 444? The iApp assumes 443, so this is going to fail if EWS is actually listening on 444. AFAIK, 444 is used for intra-server Exchange communications, so using it for a client service might not be the best way to go.
- Burrell
I concur, the 444 is used by internal exchange system and the iApp wouldn't and shouldn't use that, however testing it via the Node's default 443 produced same result.
So now I think we are past the binding issue, but not sure why the exchange server is resetting the connection.
You don't happen to have NTLMv2 enforced in your domain, do you? The advanced monitors only support Basic and NTLMv1 authentication.
- Burrell
Yes current default domain policy is to require NTLMv2 session security, we will test disabling that setting and see how it affects the monitor.
Is there any plan to implement support for NTLMv2 or a way to work around this roadblock. We have to maintain NTLMv2 for PCI compliance.
There is no plan for NTLMv2 support, AFAIK that's up to the developers of curl to implement and I haven't been able to find any evidence of plans to do that.
Unfortunately, with the architectural changes in Exchange 2013 and 2016, our advanced monitors aren't as useful. Marking the Exchange server down because one mailbox is unavailable doesn't give us a correct picture of the server health; in other words, if any mailbox is available though client access services, the server should be marked up, but adding a separate monitor for each mailbox database doesn't scale.
The plan for now with our Exchange solution is to recommend using only the "simple" monitors (which are what Microsoft officially recommends), and F5 will explore addtional advanced monitoring options in the future.