For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

RobM's avatar
RobM
Icon for Cirrus rankCirrus
Mar 21, 2022

29 hour cycle in status monitors

I found a weird thing in the logs. Each of the various pools on the f5 has a status monitor, reporting whether it is up or not. Each of those pool monitors drops out for a short period every 29 hou...
application delivery
RobM's avatar
RobM
Icon for Cirrus rankCirrus
Mar 22, 2022

Its http monitors that I'm seeing it with, but then thats most of what we have on the system.  Here's some sample log data (redacted) so you can see what I'm talking about:

[root@bigip_2:Active:Standalone] log # zgrep -hi ' Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80' ltm* | sort

Mar 11 15:29:32 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status up. [ /NZP/nzp-aspxrptgen-http: up ]  [ was down for 0hr:0min:10sec ]
Mar 14 01:29:45 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status down. [ /NZP/nzp-aspxrptgen-http: down; last error: /NZP/nzp-aspxrptgen-http:  Response Code: 200 (OK) @2022/03/14 01:29:45.  ]  [ was up for 58hrs:0min:13sec ]
Mar 14 01:29:47 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status up. [ /NZP/nzp-aspxrptgen-http: up ]  [ was down for 0hr:0min:2sec ]
Mar 15 06:29:50 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status down. [ /NZP/nzp-aspxrptgen-http: down; last error: /NZP/nzp-aspxrptgen-http:  Response Code: 200 (OK) @2022/03/15 06:29:50.  ]  [ was up for 29hrs:0min:3sec ]
Mar 15 06:30:00 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status up. [ /NZP/nzp-aspxrptgen-http: up ]  [ was down for 0hr:0min:10sec ]
Mar 16 11:30:00 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status down. [ /NZP/nzp-aspxrptgen-http: down; last error: /NZP/nzp-aspxrptgen-http:  Response Code: 200 (OK) @2022/03/16 11:30:00.  ]  [ was up for 29hrs:0min:0sec ]
Mar 16 11:30:17 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status up. [ /NZP/nzp-aspxrptgen-http: up ]  [ was down for 0hr:0min:17sec ]
Mar 17 16:30:12 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status down. [ /NZP/nzp-aspxrptgen-http: down; last error: /NZP/nzp-aspxrptgen-http:  Response Code: 200 (OK) @2022/03/17 16:30:12.  ]  [ was up for 28hrs:59mins:55sec ]
Mar 17 16:30:21 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status up. [ /NZP/nzp-aspxrptgen-http: up ]  [ was down for 0hr:0min:9sec ]
Mar 18 21:30:17 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status down. [ /NZP/nzp-aspxrptgen-http: down; last error: /NZP/nzp-aspxrptgen-http:  Response Code: 200 (OK) @2022/03/18 21:30:17.  ]  [ was up for 28hrs:59mins:56sec ]
Mar 18 21:30:22 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status up. [ /NZP/nzp-aspxrptgen-http: up ]  [ was down for 0hr:0min:5sec ]
Mar 20 02:30:28 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status down. [ /NZP/nzp-aspxrptgen-http: down; last error: /NZP/nzp-aspxrptgen-http:  Response Code: 200 (OK) @2022/03/20 02:30:28.  ]  [ was up for 29hrs:0min:6sec ]
Mar 20 02:30:32 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status up. [ /NZP/nzp-aspxrptgen-http: up ]  [ was down for 0hr:0min:4sec ]
Mar 21 07:30:39 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status down. [ /NZP/nzp-aspxrptgen-http: down; last error: /NZP/nzp-aspxrptgen-http:  Response Code: 200 (OK) @2022/03/21 07:30:39.  ]  [ was up for 29hrs:0min:7sec ]

[root@bigip_1:Active:Standalone] log # zgrep -hi ' Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80' ltm* | sort
Mar 21 07:30:37 bigip_1 notice mcpd[6631]: 01070638:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status down. [ /NZP/nzp-aspxrptgen-http: down; last error: /NZP/nzp-aspxrptgen-http:  Response Code: 200 (OK) @2022/03/21 07:30:37.  ]  [ was up for 29hrs:0min:1sec ]
Mar 21 07:30:47 bigip_1 notice mcpd[6631]: 01070727:5: Pool /NZP/nzp-aspxrptgen-pool member /NZP/server_1:80 monitor status up. [ /NZP/nzp-aspxrptgen-http: up ]  [ was down for 0hr:0min:10sec ]

[root@bigip_2:Active:Standalone] log # zgrep -hi ' Pool /NZP/nzp-sdmp-apollo-pool' ltm* | sort
Mar 12 21:38:12 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/12 21:38:12.  ]  [ was up for 86hrs:58mins:54sec ]
Mar 12 21:39:22 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:10sec ]
Mar 12 22:33:07 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/12 22:33:07.  ]  [ was up for 57hrs:58mins:47sec ]
Mar 12 22:34:17 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:10sec ]
Mar 15 07:38:14 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/15 07:38:14.  ]  [ was up for 57hrs:58mins:52sec ]
Mar 15 07:39:24 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:10sec ]
Mar 15 08:33:04 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/15 08:33:04.  ]  [ was up for 57hrs:58mins:47sec ]
Mar 15 08:34:18 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:14sec ]
Mar 16 12:38:14 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/16 12:38:14.  ]  [ was up for 28hrs:58mins:50sec ]
Mar 16 12:39:27 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:13sec ]
Mar 16 13:33:05 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/16 13:33:05.  ]  [ was up for 28hrs:58mins:47sec ]
Mar 16 13:34:20 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:15sec ]
Mar 17 17:38:15 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/17 17:38:15.  ]  [ was up for 28hrs:58mins:48sec ]
Mar 17 17:39:28 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:13sec ]
Mar 17 18:33:06 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/17 18:33:06.  ]  [ was up for 28hrs:58mins:46sec ]
Mar 17 18:34:18 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:12sec ]
Mar 18 22:38:16 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/18 22:38:16.  ]  [ was up for 28hrs:58mins:48sec ]
Mar 18 22:39:27 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:11sec ]
Mar 18 23:33:06 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/18 23:33:06.  ]  [ was up for 28hrs:58mins:48sec ]
Mar 18 23:34:17 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:11sec ]
Mar 20 03:38:17 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/20 03:38:17.  ]  [ was up for 28hrs:58mins:50sec ]
Mar 20 03:39:29 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:12sec ]
Mar 20 04:33:07 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/20 04:33:07.  ]  [ was up for 28hrs:58mins:50sec ]
Mar 20 04:34:18 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:11sec ]
Mar 21 08:38:18 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/21 08:38:18.  ]  [ was up for 28hrs:58mins:49sec ]
Mar 21 08:39:31 bigip_2 notice mcpd[7758]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:13sec ]
Mar 21 09:33:08 bigip_2 notice mcpd[7758]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/21 09:33:08.  ]  [ was up for 28hrs:58mins:50sec ]

[root@bigip_1:Active:Standalone] log # zgrep -hi ' Pool /NZP/nzp-sdmp-apollo-pool' ltm* | sort
Mar 21 08:38:18 bigip_1 notice mcpd[6631]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/21 08:38:18.  ]  [ was up for 28hrs:58mins:49sec ]
Mar 21 08:39:30 bigip_1 notice mcpd[6631]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_2:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:12sec ]
Mar 21 09:33:09 bigip_1 notice mcpd[6631]: 01070638:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status down. [ /Common/http: down; last error: /Common/http: Unable to connect @2022/03/21 09:33:09.  ]  [ was up for 28hrs:58mins:51sec ]
Mar 21 09:34:18 bigip_1 notice mcpd[6631]: 01070727:5: Pool /NZP/nzp-sdmp-apollo-pool member /NZP/server_3:80 monitor status up. [ /Common/http: up ]  [ was down for 0hr:1min:9sec ]