High Server CPU after 16.2.1 Upgrade

[dhubick]

 Our SysAdmin team has reported a sharp increase in CPU usage on the nodes in our ltm pools after upgrading to BIG-IP 16.2.1. from 14.2.1. The servers are maintained by a different team, but I can confirm they are running Apache 2.2.27

Has anyone experienced similar outcomes during upgrades?

[Hamish]

My first question back to them, would be

'What's in the access and error logs for your apache server'? Both before and after the BigIP upgrade

If it's the BigIP then the traffic profiles hitting the apache will be different. 

[dhubick]

Still working with the server team and vendor to determine this 😥

[Nikoolayy1]

Better check from the servers whuch process causes the issue not on the F5. The issue could be that F5 now uses more complex SSL ciphers or something needs to be changed in the server side TCP profile on the F5 device to optimize the traffic.

https://support.f5.com/csp/article/K50411377

https://support.f5.com/csp/article/K72605755

Still check the F5 LTM logs for some new errors with the VIP/nodes/pools/poolmembers and also see the f5 interfaces for some errors just in case.

[dhubick]

Best I can tell, it was using "ECDHE-RSA-AES128-GCM-SHA256" before and after upgrade.

Thanks for the leads though. Still checking into TCP profiles.

[jaikumar_f5]

Sorry I couldn't follow the context, is the load high on the servers or on the F5 ?

If it's on the F5, run the top cmd and see which deamon is causing it. See if it's the odd or even cores, so that we'll know if it's the tmm or non tmm.

[dhubick]

Sorry if I wasn't clear. CPU is high on the servers in the LTM pools... F5 CPU is just fine.

[JRahm]

Some good feedback here. I'd add:

• First off, do you mean release 16.1.2? There isn't a 16.2.x.
• Secondly, using default profiles in the config can create unexpected behaviors during major upgrades, as options within them can and do change over time. I'd review the tcp and server ssl profiles first, and perhaps your monitors as well to see if any of those are different after the upgrade. 
• Lastly, if you haven't upgraded the standby box yet, take a packet capture on your upgraded active device, then fail over to the not yet upgraded standby device and take another packet capture so you can compare and analyze between the two environments.

[dhubick]

THE Jason Rahm? I'm honoured! I enjoy your Lightboard Lessons.

• Yes, I do mean 16.1.2.1.  Although, I see 16.1.2.2 was released a couple weeks ago and I am preparing to upgrade to that.
• We primarily use iApps from our BIG-IP 12 and 14 days. Most are using using custom TCP profiles derived from tcp-lan-optimized and they appear unchanged before/after the upgrade.

[JRahm]

good deal. Keep us posted, would love to hear the resolution on this!

And thank you, I appreciate that!

[dhubick]

We updated from 16.1.2.1 to 16.1.2.2 this week. Can you guess what day it was?

So, the resolution is updating to 16.1.2.2, but I am still investigating to determine a root cause. I have not had a chance to compare before/after packet captures yet.

One detail I left out of the initial post- these apache webservers are Identity proxies that sit in our DMZ and forward traffic between our external f5 hosts and our internal f5 hosts.

I had assumed that the high CPU was caused by the external f5 hosts forwarding to the Identity proxies. But, perhaps it was between the Identity proxies and our internal f5 hosts.