19-Sep-2022 08:40 - edited 19-Sep-2022 09:04
We have a LTM i2600 with some nonPROD VS having not that much load, but the 3 hours CPU performance diagram shows a regular peak of around 85% load every 5 minutes:
In the detailed view, we see it's related to "Control Plane" and "Analysis Plane" usage, but there is currently no analytics profile in use. I also noticed, this is the same on the active AND standby LTM. Any idea where this is coming from and what does this mean? How can I further analyze this? These LTMs are managed via BIG-IQ and are polled via SNMPv2.
Regards Stefan 🙂
anything standing out in the logs during those spikes? Could write an iCall script that triggers on high CPUs to kick off other troubleshooting steps. @Rodrigo_Albuque has a great article on steps to take and insights.
additional to that indeed great article from @Rodrigo_Albuque there is also a solution article about understand CPU usage. That may also help. K71764661: Understanding BIG-IP CPU usage
20-Sep-2022 01:02 - edited 20-Sep-2022 02:09
This is expected/normal behaviour. Spikes can be due to AVR/ASM daemons described in K16469 / K15606 and in this case it's AVR-related daemons such as avrd, monpd and mysqld, even if AVR is not provisioned and no analytics profile loaded. There's no impact to performance. I briefly mentioned this in my article above.
the AVR was also our first idea and checking your provided article this makes most sence. We first removed the AVR-profiles from the VS, but the spikes were still there. And now I also tried to delete/reset the AVR statistics as described in article K14956 but still spikes every 5 minutes. I also understand, that we don't need to worry about performance, because AVR uses separate logical cores and are throttled down when TMM needs the resources.
But I'm wondering, where this behavior comes from, because we have several clusters running, all with AVR provisioned, but all others are not showing these spikes.
Regards Stefan 🙂
The spikes happen in boxes with HyperThreading (HT+) technology. In HT+ boxes, TMM resides in even-numbered cores (0, 2, 4) and the control plane in odd-numbered cores (1, 3, 5 etc).It depends on the version you're running and whether your box has HyperThreading (HT+). Notice that the spikes are in the control plane CPUs which indicates these are due to the daemons I told you about. You can delete the hell out of AVR and deprovision it but it won't stop the spikes. The hardware boxes listed with HT+ in K14358, all support HyperThreading technology. Have a quick look at it.