Thank you for your answers. We suspected a monitor problem, but we have more than 1200 pools, this is very difficult to know which ones are generating this kind of log. Is there a way to activate debug on these log? Thank you for your help.
You could set SSL logging to debug and see if that gives you more information, but don’t leave it in debug mode for very long. Turn it on only for as long as you need to gather information on the problem and then set it back to its default.
To turn on debug logging: tmsh modify sys db log.ssl.level value “Debug”
To set it back to the default: tmsh modify sys db log.ssl.level value “Error”
In our case the poolmembers were just marked down by the monitor as they did not enter the SSL handshake. So parsing the list of all poolmembers for their status was answering the question for us.
In case this is a intermittent problem, watching the /var/log/ltm should reveal suspect poolmembers.
Finally a specific tcpdump proved the assumption.
Btw, 1.200 pools is really a lot. If each of these is having i.e. 2 poolmembers to be monitored with https I would also be a bit concerned about scalability of the BIG-IP´s monitoring capacity. You are using a very short monitoring interval?
! /bin/bash
i=0
for line in `tmsh list ltm monitor one-line | grep https | awk '{print $4}'`
do
arr[$i]=$line
i=`expr $i + 1`
done
for i in "${arr[@]}"
do
tmsh list ltm pool one-line | grep -E ":(80|http) .*session monitor-enabled state down.*${i}" | awk '{print $3}'
done
! /bin/bash
for line in `tmsh list ltm monitor one-line | grep https | awk '{print $4}'`
do
tmsh list ltm pool one-line | grep -E ":(80|http) .*session monitor-enabled state down.*${line}" | awk '{print $3}'
done
I have written a fairly comprehensive script that should root out these troubled pool members. I have included it in the DevCentral CodeShare in hopes that it can help others out. Let me know if you have any issues with the script.
I’ve seen multiple solutions that require to run multiple tmsh command on the running BigIP.
I’ve made a (dirty) script that parse the bigip.conf and return a CSV list of node on port 80 with a https monitor. The only downside is all the monitor doing https check must have the word “https” in it.
! /usr/bin/sh
nbl=0 ;
runningPool=""
runningIRule=""
runningVS=""
runningNode=""
while IFS= read -r bigipconf
do
if [[ $bigipconf == ltm\ pool\ \/Common\/* ]]
then
runningPool=$(echo $bigipconf | awk '{print $3}' | awk -F '/' '{print $3}')
echo $runningPool
poolNode=""
fi
if [[ $runningPool != "" ]]
then
if [[ $bigipconf == *\/Common\/*\:80* ]]
then
runningNode=$(echo $bigipconf | awk -F '/' '{print $3 }' | awk '{ print $1 }')
poolNode=$(echo "$poolNode;$runningNode")
fi
if [[ $bigipconf == *monitor* ]]
then
monitor=$(echo $bigipconf | awk -F '/' '{ print $3 }')
if [[ $monitor == *https* ]]
then
echo -e "$runningPool;$monitor;$poolNode" | grep ':'
fi
fi
fi
if [[ $bigipconf == ltm\ virtual\ \/Common\/* ]]
then
runningPool=""
fi
done < input/bigip.conf
We had the same problem at my office and it turns out that it was a Cipher issue with the HTTPS monitor. As part of web server hardening, everything but TLS1.2 was disabled on the web servers. The HTTPS monitor did have “DEFAULT:+SHA:+3DES:+kEDH” in the ciphers list and the ‘DEFAULT’ group does also include TLSv1.2 cipher suites but somehow this was causing problems.
I changed the cipher list to TLSv1_2+AES and this immediately fixed the problem - though it did create other problems as I did have a pool where some servers were hardened while others weren’t. Adding this cipher suites caused the pool members pointing to the unhardened servers to go down.
In the end I couldn’t find a compromise so I just put it to “tcp”. Not the best solution but I need to get the server and application guys to fix it on the servers.