Forum Discussion

F5_Digger_13600's avatar
Apr 23, 2018
Solved

Multiple external health monitors do not work except one. Any alternative?

I need some help from DevCentral F5 expert regarding external monitors.

 

In our organization, I created a bash script file, which will be associated with 4 different external monitors. Basically the script makes SSH connection to a backend server, grasp the status string, close SSH, and reporting the status to bigd.

 

The issue is only one external monitor out of 4 is called correctly. All the others are not called at all.

 

BIG-IP version: 12.1.3.3

 

The setup is like below: -One external bash file (Pasted below) -There are 4 external health monitors associated with the same external bash file

 

Issues identified -It seems bigd(A process manages health monitors) doesn’t call all external health monitors (In a bash script, it logs its status to an external file, not STDOUT) oOne external monitor is called correctly all the time. oAll the other three are never be called (No logging to external file)

 

I am using an external monitor because this is only method to properly handle complex backend server health status.

 

Now I wonder using external monitor is bad idea. If using external monitor is not a good choice of checking complex backend server health status what would be its alternative?

 

Thank you.

 

Here is my bash code.

 

!/bin/bash
 Modified F5 DNS External Health Monitor
 Script Name: interactive_ssh_extmon.sh (Interactive SSH External Health Monitor)

 About external monitor CLI parameters
 interactive_ssh_extmon.sh doesn't need any CLI parameters (param1 and param2 are automatically fed by F5 LTM when used with F5 LTM monitor)
 - param1: Pool member IP
 - param2: Service Port Number(With this external monitor, service port will be SSH (TCP 22))
 - param3: SSH Username
 Test Example with shell: 
 ./interactive_ssh_extmon.sh 1.2.3.4 22 ssh_user_name
 Test Example with /config/filestore/files_d/Common_d/external_monitor_d/custom_external_monitors
 /bin/sh ":Common:interactive_ssh_extmon_35393_ver" 1.2.3.4 22 f5montor
 F5 External script directory: /config/filestore/files_d/Common_d/external_monitor_d

 Pool member UP/Down log is sent to extmon.log
 Log File location: /var/log/extmon.log

 How to use this external monitor with F5 LTM (BIG-IP LTM 11.x or later)
 Step0 - Prerequisite (Public-Key based auto login)
         : Public key needs to be exported to remote system by using the following command
          - ssh-copy-id -i /home/user/.ssh/id_rsa.pub username@remote_host_ip
 Step1 - Import this external monitor script onto LTM (System->File Management->External Monitor Program File List)
         : Choose the script file and assign a name
 Step2 - Create a monitor (Local Traffic -> Monitor -> Create)
         : General Properties
           - Assign a Name
           - Choose "External" with Parent Monitor
         : Configuration (Advanced)
           - Choose the imported external monitor name
 Step3 - Associate this external monitor with a pool or a pool member accordingly


 If the Node IP address includes IPv6 format, remove IPv6 format.
node_ip=`echo $1 | sed 's/::ffff://'`

echo "`date` ${node_ip}: Script started" >> /var/log/extmon.log

 If the number of parameters is not 3, print the usage of the script
if [ "$" -ne 3 ]
then
    echo "Usage: interactive_ssh_extmon.sh Pool_member_IP Member_port ssh_user_name"
    exit 1
fi

un="$3"

 create a pidfile based on calling health monitor
if [ "$DEBUG1" == "1" ];then 
    pidfile="/var/run/`basename $0`.$node_ip.$2.pid"
elif [ "$DEBUG2" == "1" ];then 
    pidfile="/var/run/`basename $0`.$node_ip.$2.pid"
elif [ "$DEBUG3" == "1" ];then 
    pidfile="/var/run/`basename $0`.$node_ip.$2.pid"
elif [ "$DEBUG4" == "1" ];then 
    pidfile="/var/run/`basename $0`.$node_ip.$2.pid"
else 
    pidfile="/var/run/`basename $0`.$node_ip.$2.pid"
fi

echo "`date` PIDFILE: ${pidfile} PID: $$" >> /var/log/extmon.log

 If pidfile exists (-f: regular file), kill the process associated with the pidfile
if [ -f $pidfile ]
then
    kill -9 `cat $pidfile` > /dev/null 2>&1
fi

 Save the process ID and use it later to kill the process once everything is done.
echo "$$" > $pidfile

 Run SSH command and save the output
MTSstate=`ssh ${un}@${node_ip} -t "bash -l -c 'exit'" 2>/dev/null`

 Save the return status of executed command
 0 - success, non-zero - failure
status=$?

 Command execution should be success. Otherwise exit 1
if [ $status -ne 0 ]; then exit 1; fi

 "=~" operator: Consider RHS as Regex expression.
if [[ $MTSstate =~ .*UP.* ]]
then
    rm -f $pidfile
    echo "`date` ${node_ip}: UP" >> /var/log/extmon.log
    echo "UP"
else
    rm -f $pidfile
     Do Not send any data to STDOUT for the monitor failure.
     Failure: No data to stdout
    echo "DOWN"
    echo "`date` ${node_ip}: DOWN" >> /var/log/extmon.log
fi

exit 0
  • The issue has been resolved and I am sharing the solution for the people who may need multiple external monitors using SSH connection within their script.

    Configuration and Environment

    - BIG-IP version: 12.1.3.3

    - Use multiple External monitors

    - Script(External script file) include SSH connection to backend pool member on tcp port 22

    Issues identified and its solution

    - Issue 1: Redirection logging data to a file for the troubleshooting purpose doesn't work well.

    Solution: Use logger command to send logging data to /var/log/ltm (Credit to Jie. Thanks Jie)

    echo "`date` ${node_ip}: Script started" | logger -p local0.debug
    

    -Issue 2: From the second SSH connection used in any external health monitors, it doesn't use the Client SSH cipher configuration under /config/ssh/ssh_config.

    Ciphers aes128-cbc,aes256-cbc,aes256-ctr,aes128-ctr,aes192-ctr
    

    Instead it starts using only ciphers aes128-cbc,aes256-cbc where the pool members (backend servers) require aes128-ctr,aes192-ctr,aes256-ctr. Because of this, SSH connection fails and finally ssh command returns 255, which says SSH connection failure - Refer to wireshark capture above

    -Solution: Specify specific SSH ciphers within ssh command

    RET_STATUS=`ssh -c aes128-ctr,aes192-ctr,aes256-ctr ${un}@${node_ip} -t "bash -l -c 'exit'" 2>/dev/null`
    

    Thank you.

  • JG's avatar
    JG
    Icon for Cumulonimbus rankCumulonimbus

    It's highly unlikely, as it is the default config, but did you by any chance change the settings of "Availability Requirement" in the pool configuration to be not "All Health Monitor(s)"?

     

  • The issue has been resolved and I am sharing the solution for the people who may need multiple external monitors using SSH connection within their script.

    Configuration and Environment

    - BIG-IP version: 12.1.3.3

    - Use multiple External monitors

    - Script(External script file) include SSH connection to backend pool member on tcp port 22

    Issues identified and its solution

    - Issue 1: Redirection logging data to a file for the troubleshooting purpose doesn't work well.

    Solution: Use logger command to send logging data to /var/log/ltm (Credit to Jie. Thanks Jie)

    echo "`date` ${node_ip}: Script started" | logger -p local0.debug
    

    -Issue 2: From the second SSH connection used in any external health monitors, it doesn't use the Client SSH cipher configuration under /config/ssh/ssh_config.

    Ciphers aes128-cbc,aes256-cbc,aes256-ctr,aes128-ctr,aes192-ctr
    

    Instead it starts using only ciphers aes128-cbc,aes256-cbc where the pool members (backend servers) require aes128-ctr,aes192-ctr,aes256-ctr. Because of this, SSH connection fails and finally ssh command returns 255, which says SSH connection failure - Refer to wireshark capture above

    -Solution: Specify specific SSH ciphers within ssh command

    RET_STATUS=`ssh -c aes128-ctr,aes192-ctr,aes256-ctr ${un}@${node_ip} -t "bash -l -c 'exit'" 2>/dev/null`
    

    Thank you.