29 hour cycle in status monitors
I found a weird thing in the logs.
Each of the various pools on the f5 has a status monitor, reporting whether it is up or not.
Each of those pool monitors drops out for a short period every 29 hours, give-or-take a few seconds. (Mostly. Occasionally they’ll miss one, and drop out at the next multiple of 29 hours…) They stay down for a handful of seconds – up to a minute or so - then come back up for another 29 hours.
When the monitor sees a pool drop out, it sees it drop out from both f5s simultaneously (makes sense, but just checking…)
The different pools drop out at different times (so its presumably not the whole network going on the blink…)
Even though its less than a minute a day, I feel like our other status monitoring would be shouting at us if we were actually losing the service, so I wondered if we could be losing the monitor, not the pool. One of our sysadmins pointed out to me that 29 hours is the default Microsoft IIS recycling interval. Is the ststus monitor perhaps using Microsoft IIS somehow?