FAILED failover: Virtual servers were disabled on standby unit.

Thank you. I reply:

Were all the pools offline on lb1 or just the ones you mention while others were still green/online?

I did find one or two VS that remained green. One is a sort of place holder VS, which was set up to eventually take over for a virtual server on a totally separate standalone F5. The other is the testing pool. I do believe that all the other enabled VS were in the red diamond state. We did intentionally force down several VS during the other configuration I mentioned, and those were reported correctly during the incident. Only the majority of VS that were "enabled" on F52 were found with red diamonds on F5_1 during the incident.

Re: " built-in snmp trap alerts "

can you share the log to search for this alert? We downloaded the tar of existing logs at the time of reloading the LB_2 as the active unit.

re: does not continuously log offline status

Thank you. It clears up that part for me. No way to expect it to log continual offline, even when failing over.

Re: Monitors should checking every 10 secs ...

It is what we expected for sure. So that they aught to have been noticed and come online. I think this is the most distressing thing to solve, but the fact that the original offline condition was triggered so long ago, and never recovered we are at a loss to understand or explain. It is most frustrating. Especially to explain for the boss.

Re: it seems there was some issue (obviously) with lb1 internally or its ability to reach the servers.

While testing with LB_1 in the broken condition. I could curl the web servers test pages and got the expected result from the LB_1 ssh command line. So, we are all the more confused to discover what cause, apparently, the F5 to not be sending the monitor test.

"Regarding the "In sync" status,"

Thank you again for clearing up the misunderstanding. I assumed that the status would be shared as part of "syncing" Thus, i think we are facing an unexplainable fail on LB_1 to continue or successfully accomplish the web server monitoring.

Re: ...F5 Support ...

No. The system is up, so we will likely just move on. We have had scheduled upgrade time anyway for few weeks away.

I think we must devise a script to capture the VS statuses on F5_1 and 2. The script will simply compare them and alert us if the statuses are not the same at any time. Even a hourly interval check should be fine.