The load balancing on GTM is not working properly. Preferred is Global Availability and rest are none.

The "iqmgmt_ssl_connect: SSL error" is not normal. I still suspect there is an iquery connection failure somewhere. If I understand correctly you have 2 GTMs-they will have iquery connection with each other. Both of these GTMs will also have an iquery connection with every LTM server object present in the GTM config. This is how GTM delegates monitoring tasks to bigips in your data centers. With the default config GTM rotates monitoring tasks among all the bigips in a data center. I suspect there is an LTM that the GTM is failing to reach when it tries to delegate monitor. The error no reply from big3d: timed out also leads me in that direction.

Another possible issue could be that the bigip being asked to run the monitor is incapable of reaching the device that is to be monitored. This could account for the fact that the primary is up sometimes but down at other times; it would depend on which bigip is performing the monitoring. Are all bigips able to communicate with one another over tcp port 4353? Are all bigips able to reach every vip that is being monitored?

If all the GTM vips are hosted on LTMs then the best practice is to not add additional GTM monitors to the servers or pools. Let the LTMs report vip status to the GTMs through the bigip monitor and iquery connection.

I would also check big3d version on every bigip, GTM and LTM. They must all be compatible.

If you haven't seen these articles I highly recommend reading them:

sol13703