Clean Up Monitor Error: "SSL23_GET_SERVER_HELLO:unknown protocol"

Problem this snippet solves:

This script produces a report of which monitors are causing the "SSL23GETSERVER_HELLO:unknown protocol" error messages in /var/log/ltm.

I would be very interested in any feedback/tips.

See the source for known issues (caveats).

Background

v11 introduced the logging of SSL monitor errors to /var/log/ltm which look similar to: “SSL23GETSERVER_HELLO:unknown protocol”

SOL13768 -- Change in Behavior: SSL based monitors log handshake failures to the /var/log/ltm file

https://support.f5.com/kb/en-us/solutions/public/13000/700/sol13768.html

Code :

#!/bin/sh
# sslHandShakeError_MonitorSearch.sh
# 2014.01.17-08:39
# Daniel Tavernier
# tavdan@gmail.com
# F5 DevCentral: tabernarious

# This script analyzes and tests an F5 v11.x.x configuration for
#   failing ssl monitors that fill /var/log/ltm with messages like:
#   "routines:SSL23_GET_SERVER_HELLO:unknown protocol"
#
# Known Issues:
# --Strips out administrative partitions (e.g. /Common/monitorName)
#     You may have issues if you have identically named monitors in multiple partitions
# --The script will test ssl against port 443 for any built-in monitors; this is not
#     a problem for "https" and "https_443", but could be if others exist
# --If custom ssl ciphers defined on the monitor are the cause of the ssl errors
#     then the monitor will not show on the final list; you will have to cross-reference
#     the list of marked-down https-based pool member monitors

# List of files produced:
httpsMonitorListFile="/tmp/sslHandshakeError_httpsMonitorList.tmp"
downMonitorListRaw1="/tmp/sslHandshakeError_downMonitorListRaw1.tmp"
downMonitorListRaw2="/tmp/sslHandshakeError_downMonitorListRaw2.tmp"
downMonitorListFile="/tmp/sslHandshakeError_downMonitorList.tmp"
opensslErrorFile="/tmp/sslHandshakeError_openssl.tmp"
failingMonitorListFile="/tmp/sslHandshakeError_failingMonitorList.tmp"

# Produce a list of all https-based monitors from all partitions
# For simplicity, we will strip all partition designations
# (Part of the issue is that "tmsh show ltm pool members" inconsistently
#   includes the partition in the name (e.g. /Common/objectName)
grep "ltm monitor https" /config/bigip.conf |sed 's/.*\/.*\/\(.*\) {/\1/' > $httpsMonitorListFile
# DEBUG
echo "############################"
echo "## HTTPS-Based Monitor List"
echo "############################"
cat $httpsMonitorListFile
echo

# Add built-in https-based monitors
echo "https" >> $httpsMonitorListFile
echo "https_443" >> $httpsMonitorListFile

# Create bash array of https-based monitors
#IFS=$'\r\n' httpsMonitors=($(cat $httpsMonitorListFile))
httpsMonitors=`cat $httpsMonitorListFile`

# Produce a list of all marked-down pool member monitors with the format:
# poolName poolMemberIP monitorName
#tmsh show ltm pool members all-properties |grep -A7 "Ltm::Pool Member" |grep -B7 "Monitor Status.*down" |grep -C5 "marked down by a monitor" |grep "Ltm::Pool\|Monitor  " |sed 'N;s/.*Pool Member: \([0-9.:a-Z]*\).*Monitor.*: \(.*\) .*/\1 \2/' > /tmp/poolMembersMonitors.tmp
tmsh show ltm pool members all-properties \
|grep -A9 "Ltm::Pool Member" \
|grep "Ltm::Pool\|Reason\|Monitor\|Pool Name" \
|grep -C3 "Monitor Status.*down" \
|grep -C3 "marked down by a monitor" \
|grep "Ltm::Pool\|Monitor  \|Pool Name" \
|awk '{ if (NR % 3) printf("%s ", $0); else printf("%s\n", $0) }' > $downMonitorListRaw1
cat $downMonitorListRaw1 |sed 's/.*Pool Member: \([0-9.:_a-Z]*\).*Monitor.*: \([0-9._ /a-Z]*\).*Pool Name.*: \([0-9._a-Z]*\).*/\3 \1 \2/' > $downMonitorListRaw2

# Save default IFS (Internal Field Separator)
oldIFS=$IFS

# Set new IFS
IFS=$'\r\n'

# Clear $downMonitorListFile; this process must append not overwrite
echo "" > $downMonitorListFile

# Iterate through pool members to check for multiple monitors and verify
#   monitors are https-based
for member in `cat $downMonitorListRaw2`; do 
    # Iterate through each monitor for pool members that are assigned
    #   multiple monitors; tmsh displays "monitor1 and monitor2 and ..."
    monitorCount=`echo $member |grep -c " and "`
    monitorCount=$(( $monitorCount + 1 ))
    while [ $monitorCount -gt 0 ]; do
        # Determine which
        monitorPosition=$(( 1 + $monitorCount * 2 ))
        member=`echo $member | sed 's/\/.*\///'`
        #echo $member |awk -v monitorPosition=$monitorPosition '{print $1 " " $2 " " $( monitorPosition )}' >> $downMonitorListFile
        poolName=`echo $member |awk '{print $1}'`
        memberIP=`echo $member |awk '{print $2}'`
        monitorName=`echo $member |awk -v monitorPosition=$monitorPosition '{print $( monitorPosition )}'`
        for httpsMonitor in ${httpsMonitors[@]}; do
            if [ $monitorName == $httpsMonitor ]; then
                echo "$poolName $memberIP $monitorName" >> $downMonitorListFile
            fi
        done
        monitorCount=$(( $monitorCount - 1 ))
    done
done
echo "################################################"
echo "## Marked-Down HTTPS-Based Pool Member Monitors"
echo "################################################"
echo "POOL_NAME  POOL_MEMBER_IP/PORT  MONITOR_NAME"
cat $downMonitorListFile
echo

# Reduce list to all non-443 pool members (most likely having handshake issues)
echo "#########################################################################"
echo "## Marked-Down HTTPS-Based Pool Member Monitors for non-443 Pool Members"
echo "#########################################################################"
echo "POOL_NAME  POOL_MEMBER_IP/PORT  MONITOR_NAME"
grep -v ":443" $downMonitorListFile
echo

# Clear $failingMonitorListFile; this process must append not overwrite
echo "" > $failingMonitorListFile

# Test suspect pool members and report monitors triggering this SSL error:
#   "routines:SSL23_GET_SERVER_HELLO:unknown protocol"
for httpsMonitor in `cat $downMonitorListFile`; do
    poolName=`echo $httpsMonitor |awk '{print $1}'`
    memberIP_port=`echo $httpsMonitor |awk '{print $2}'`
    memberIP=`echo $memberIP_port |sed 's/\(.*\):\(.*\)/\1/'`
    memberPort=`echo $memberIP_port |sed 's/\(.*\):\(.*\)/\2/'`
    monitorName=`echo $httpsMonitor |awk '{print $3}'`
    # Get the monitor's destination IP (alias or pool member default)
    monitorIP=`grep -A5 "ltm monitor https /.*/$monitorName " /config/bigip.conf |grep "destination" |sed 's/.*destination \(.*\):\(.*\)/\1/'`
    if [ "$monitorIP" == '*' ] || [ "$monitorIP" == "" ]; then
        monitorIP=$memberIP
    fi
    # Get the monitor's destination port (alias or pool member default)
    monitorPort=`grep -A5 "ltm monitor https /.*/$monitorName " /config/bigip.conf |grep "destination" |sed 's/.*destination \(.*\):\(.*\)/\2/'`
    if [ "$monitorPort" == '*' ] || [ "$monitorPort" == "" ]; then
        monitorPort=$memberPort
    fi
    monitorIP_port="${monitorIP}:${monitorPort}"
    # Attempt ssl connection to pool member based on monitor's port (alias or pool member default)
    echo close |openssl s_client -quiet -connect "$monitorIP_port" 1> /dev/null 2> $opensslErrorFile
    sslError=`grep -m1 "routines:SSL23_GET_SERVER_HELLO:unknown protocol" $opensslErrorFile`
    if [[ "$sslError" =~ "routines:SSL23_GET_SERVER_HELLO:unknown protocol" ]]; then
        echo "$poolName $memberIP_port $monitorName $monitorIP_port" >> $failingMonitorListFile
    fi
done
echo "################################################"
echo "## Monitors Failing with SSL23_GET_SERVER_HELLO"
echo "################################################"
echo "POOL_NAME  POOL_MEMBER_IP/PORT  MONITOR_NAME  MONITOR_DEST_IP/PORT"
cat $failingMonitorListFile

# Revert IFS
IFS=$oldIFS