Nimbostratus
Nimbostratus
I did more digging into the logs and found out that it WASN'T the BigIP monitors that caused the issue. My first post was the morning after the maintenance window, on very little sleep, before I had a chance to look at the logs and into the configuration.
The problem is with the Data Center Links that are defined and being monitored. There are 3 links, pinging the ISP routers, using the default gateway_icmp monitor. We can ping them from the CLI just fine. However, when on 10.2.0HF2, only one of them is actually up. The other two are down and from reviewing the gtm logs directly after boot, they never actually come up - I can see their state go from blue -> red. Looking at the monitor statistics in the GUI, they are flatlined. All zeroes for input/output packets for the two that are down. Tcpdumps also confirm that the monitors aren't even attempting to send out any traffic.
Now when we upgraded to 10.2.3HF1, the same thing happened to the 3rd link. Blue -> red directly after boot. I didn't have time to troubleshoot as we were already past the maint. window and I was very tired. But if I had to guess, it wasn't even trying to send traffic out. Once all 3 links were down, it brought down everything defined for that data center. Which ultimately forced us to roll back.
Any ideas what the problem is here? I'm thinking of creating custom monitors and see if the default is just acting up but really, that's just a stab in the dark.
Apr 7 04:53:32 local/dc1-lb-03 alert gtmd[3754]: 011ae0f2:1: Monitor instance gateway_icmp x.x.x.3:0 UNKNOWN_MONITOR_STATE --> DOWN from gtmd (no reply from big3d: timed out)
Apr 7 04:53:32 local/dc1-lb-03 alert gtmd[3754]: 011a2003:1: SNMP_TRAP: Link dc-link-02 (ip=x.x.x.3) state change blue --> red (Link dc-link-02: Monitor gateway_icmp from gtmd : no reply from big3d: timed out)