cancel
Showing results for 
Search instead for 
Did you mean: 
Login & Join the DevCentral Connects Group to watch the Recorded LiveStream (May 12) on Basic iControl Security - show notes included.

High Server CPU after 16.2.1 Upgrade

dhubick
Altocumulus
Altocumulus

Inkedcpu_namprodgw1_LI.jpg

 Our SysAdmin team has reported a sharp increase in CPU usage on the nodes in our ltm pools after upgrading to BIG-IP 16.2.1. from 14.2.1. The servers are maintained by a different team, but I can confirm they are running Apache 2.2.27

Has anyone experienced similar outcomes during upgrades?

10 REPLIES 10

Hamish
Cirrocumulus
Cirrocumulus

My first question back to them, would be

'What's in the access and error logs for your apache server'? Both before and after the BigIP upgrade

If it's the BigIP then the traffic profiles hitting the apache will be different. 

 

 

Still working with the server team and vendor to determine this 😥

Better check from the servers whuch process causes the issue not on the F5. The issue could be that F5 now uses more complex SSL ciphers or something needs to be changed in the server side TCP profile on the F5 device to optimize the traffic.

https://support.f5.com/csp/article/K50411377

https://support.f5.com/csp/article/K72605755

 

 

Still check the F5 LTM logs for some new errors with the VIP/nodes/pools/poolmembers and also see the f5 interfaces for some errors just in case.

Best I can tell, it was using "ECDHE-RSA-AES128-GCM-SHA256" before and after upgrade.

Thanks for the leads though. Still checking into TCP profiles.

Sorry I couldn't follow the context, is the load high on the servers or on the F5 ?

If it's on the F5, run the top cmd and see which deamon is causing it. See if it's the odd or even cores, so that we'll know if it's the tmm or non tmm.

Sorry if I wasn't clear. CPU is high on the servers in the LTM pools... F5 CPU is just fine.

JRahm
Community Manager
Community Manager

Some good feedback here. I'd add:

  • First off, do you mean release 16.1.2? There isn't a 16.2.x.
  • Secondly, using default profiles in the config can create unexpected behaviors during major upgrades, as options within them can and do change over time. I'd review the tcp and server ssl profiles first, and perhaps your monitors as well to see if any of those are different after the upgrade. 
  • Lastly, if you haven't upgraded the standby box yet, take a packet capture on your upgraded active device, then fail over to the not yet upgraded standby device and take another packet capture so you can compare and analyze between the two environments.

THE Jason Rahm? I'm honoured! I enjoy your Lightboard Lessons.

  • Yes, I do mean 16.1.2.1.  Although, I see 16.1.2.2 was released a couple weeks ago and I am preparing to upgrade to that.
  • We primarily use iApps from our BIG-IP 12 and 14 days. Most are using using custom TCP profiles derived from tcp-lan-optimized and they appear unchanged before/after the upgrade.

JRahm
Community Manager
Community Manager

good deal. Keep us posted, would love to hear the resolution on this!

And thank you, I appreciate that!

namgw1_cpu.png

We updated from 16.1.2.1 to 16.1.2.2 this week. Can you guess what day it was?

So, the resolution is updating to 16.1.2.2, but I am still investigating to determine a root cause. I have not had a chance to compare before/after packet captures yet.

One detail I left out of the initial post- these apache webservers are Identity proxies that sit in our DMZ and forward traffic between our external f5 hosts and our internal f5 hosts.

I had assumed that the high CPU was caused by the external f5 hosts forwarding to the Identity proxies. But, perhaps it was between the Identity proxies and our internal f5 hosts.