Forum Discussion
After upgrade to 11.6 HF4 mysqld causes high cpu usage
Hi devcentral community,
Just wondering if somebody else noticed the same behaviour after upgrade to 11.6 HF4 (LTM/APM/ASM).
After upgrade to 11.6 HF4 on a BIG-IP 2000s cluster (from 11.4.1 HF4) average cpu consumption is much more higher (active and standby unit!). After some investigation I found out that the "mysqld" process consumes up to 100% of one cpu-core in 5min intervals, only for 3-5 secs. On a heavy loaded system this could have serious performance impacts of course. See graphs below:
CPU 30 days graph:
CPU 7 days graph
CPU 3 hours graph
Actually all services are running stable with 11.6 HF4 far, but of course CPU usage statistics are approx. 20% higher with 11.6 HF4 than before upgrade!
Both cluster members are affected and I was also able to reproduce the issue on my lab environment (vmware-esx).
I tried to deactivate AVR, but the issue remains with and without AVR!
Does anybody else noticed this higher cpu usage after upgrade to 11.6 HF4 with LTM/APM/ASM?
Thanks a lot Regards Lukas
15 Replies
- Gianrico
Employee
Both APM and ASM use mysqld although ASM more extensively.
In your lab, can you "bigstart stop asm " ans see if there is a change?
-- gianrico
Hi Gianrico,
Just tried with "bigstart stop asm" but still the same high CPU issue. I also de-provisioned the ASM module, rebooted the VM and still the same issue with LTM/APM.
In the meantime I disabled AVR again and did a reboot of both cluster members. Right now it seems that the high CPU peaks are still there but only around 40-60% and much shorter then before.
Anyway I will monitor the CPU usage in the next few hours to see if AVR caused the high CPU usage. Keep you posted.
Thx Lukas
12 hours later CPU usage is still around 40% average on both cluster members! Enabled modules LTM/APM. I opened a ticket at f5 support but no response so far...
Anybody else out there in the community having the same issue with 11.6 HF4 in combination with LTM/APM? Thx Lukas
- JG
Cumulonimbus
I think that's probably what everybody is getting. My observation is that only the last CPU core that gets used a lot, for the non-dada-plane tasks, such as intra-blade clustering.
- Belanger__Yves
Altostratus
Hi,
I have the same issue with SharePoint on APM at 11.6 HF4.
Yves
Hi all,
I just downgraded my lab vm to 11.6 HF3 and as you can see in the graph below, cpu usage peaks went down to the half:
So for me it's obvious that f5 changed something in the AVR code from HF3 to HF4.
In the meantime I got an answer from f5 support that this kind of mysql high cpu usage should be the normal behaviour beginning with 11.6, doesn't matter if AVR is enabled or not...;-)
I will try to reset the customers AVR statistics database by tomorrow morning according to SOL14956, maybe this will help...
Keep you posted Regards Lukas
- Hussein_Ghazy
Nimbostratus
Hi All,
I am facing the same issue on my Viprion Blades
Following the SOL 14956, did it solve the mysql high cpu issue?
Regards
- Emad_26973
Cirrus
If you are running ASM then check learning suggestions in ASM and remove if not required. There is a mysql query for learning suggestions which run every 3-4 minutes and in case of large no of Learning Suggestion in ASM, can increase CPU utilization.
- Hussein_Ghazy
Nimbostratus
I got a strange feedback from F5 support, this is a normal behavior that the ASM uses CPU7 !
Regarding the high CPU 7 usage,
The high CPU on the last core ( in this case core 7) is a frequently reported concern from many customers. The CPU usage here is misleading- it is high because many ASM processes are pinned to that core for GUI performance reasons. This was done after much testing by developers ; the associated with ID432080. By pinning ASM processes to a single CPU, the processes benefit from local processor caching and communication overhead.
That said, F5 PD is well aware that the high CPU usage reporting is worrying here needlessly. They have made some changes in 11.5.1 hotfixes and in 11.5.2 to improve the CPU reporting, specifically with regard to AVR spiking the CPU on the last core every 5 minutes (again, expected behavior; ASM leverages AVR for its reporting) but these changes have not been ported to 11.6.0 yet.
I have escalated the issue, as we are not able to access some features in the ASM GUI due to the High CPU.
- Rudi_95802
Nimbostratus
We also got the feedback that ASM will use the highest core. Thing is though. We dont have ASM enabled. My gripe is that the AVR spikes are much more aggressive after HF3 and we're not getting an answer on that.
- BinaryCanary_19Historic F5 AccountBasically, the Combination of HTSplit and Process pinning (some processes are pinned to highest CPU core) mean that you should not really worry about spikes related to AVR and ASM processes. This is because: The critical processes will always get priority access to the CPU. Because critical processes always get priority access to the CPU, it is better to let the non-critical processes run as fast as possible and with a low nice value so that they can finish their tasks as soon as possible when the CPU is otherwise busy.
- BinaryCanary_19Historic F5 Accounthttps://support.f5.com/kb/en-us/solutions/public/15000/600/sol15606.html?sr=45970987
- BinaryCanary_19Historic F5 Accountsmall correction: "finish their tasks as soon as possible when the CPU is otherwise idle"
Help guide the future of your DevCentral Community!
What tools do you use to collaborate? (1min - anonymous)Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com