F5 sending all synthetic transactions to one host

We found the root cause. The extensive logging was enabled on the script and we found the script actually made 2 connections. One to test the connection and the 2nd one actually sent the SOAP request. Since these devices are not live yet, the only traffic on them are the synthetic transactions. So it appears that the F5 was only sending traffic to one device.

We proved this by looking at the logs and finding which device was getting the soap request from the monitoring tool. We then sent a SoapUI request so it would hit the same device.

On the next synthetic transaction soap request the direction would change and started hitting the other device.

Another indication that an initial test connection was being made by the monitoring script was that we were seeing SSL handshake errors and peer did not send certificate errors on the device that appeared not to receive any traffic at almost the exact same time the synthetic transaction SOAP request was hitting the other device.

Hence monitoring script would do a test connection without sending a certificate to device A and then a second connection to send the actual SOAP request with the certificate hitting device B.