Forum Discussion

dhubick's avatar
dhubick
Icon for Altocumulus rankAltocumulus
May 04, 2022

High Server CPU after 16.2.1 Upgrade

 Our SysAdmin team has reported a sharp increase in CPU usage on the nodes in our ltm pools after upgrading to BIG-IP 16.2.1. from 14.2.1. The servers are maintained by a different team, but I can confirm they are running Apache 2.2.27

Has anyone experienced similar outcomes during upgrades?

application delivery
  • Hamish's avatar
    Hamish
    Icon for Cirrocumulus rankCirrocumulus
    May 04, 2022

    My first question back to them, would be

    'What's in the access and error logs for your apache server'? Both before and after the BigIP upgrade

    If it's the BigIP then the traffic profiles hitting the apache will be different. 

     

     

    • dhubick's avatar
      dhubick
      Icon for Altocumulus rankAltocumulus
      May 06, 2022

      Still working with the server team and vendor to determine this 😥

  • Nikoolayy1's avatar
    Nikoolayy1
    Icon for MVP rankMVP
    May 04, 2022

    Better check from the servers whuch process causes the issue not on the F5. The issue could be that F5 now uses more complex SSL ciphers or something needs to be changed in the server side TCP profile on the F5 device to optimize the traffic.

    https://support.f5.com/csp/article/K50411377

    https://support.f5.com/csp/article/K72605755

     

     

    Still check the F5 LTM logs for some new errors with the VIP/nodes/pools/poolmembers and also see the f5 interfaces for some errors just in case.

    • dhubick's avatar
      dhubick
      Icon for Altocumulus rankAltocumulus
      May 06, 2022

      Best I can tell, it was using "ECDHE-RSA-AES128-GCM-SHA256" before and after upgrade.

      Thanks for the leads though. Still checking into TCP profiles.

  • jaikumar_f5's avatar
    jaikumar_f5
    Icon for MVP rankMVP
    May 05, 2022

    Sorry I couldn't follow the context, is the load high on the servers or on the F5 ?

    If it's on the F5, run the top cmd and see which deamon is causing it. See if it's the odd or even cores, so that we'll know if it's the tmm or non tmm.

    • dhubick's avatar
      dhubick
      Icon for Altocumulus rankAltocumulus
      May 06, 2022

      Sorry if I wasn't clear. CPU is high on the servers in the LTM pools... F5 CPU is just fine.

  • JRahm's avatar
    JRahm
    Icon for Admin rankAdmin
    May 06, 2022

    Some good feedback here. I'd add:

    • First off, do you mean release 16.1.2? There isn't a 16.2.x.
    • Secondly, using default profiles in the config can create unexpected behaviors during major upgrades, as options within them can and do change over time. I'd review the tcp and server ssl profiles first, and perhaps your monitors as well to see if any of those are different after the upgrade. 
    • Lastly, if you haven't upgraded the standby box yet, take a packet capture on your upgraded active device, then fail over to the not yet upgraded standby device and take another packet capture so you can compare and analyze between the two environments.
    • dhubick's avatar
      dhubick
      Icon for Altocumulus rankAltocumulus
      May 06, 2022

      THE Jason Rahm? I'm honoured! I enjoy your Lightboard Lessons.

      • Yes, I do mean 16.1.2.1.  Although, I see 16.1.2.2 was released a couple weeks ago and I am preparing to upgrade to that.
      • We primarily use iApps from our BIG-IP 12 and 14 days. Most are using using custom TCP profiles derived from tcp-lan-optimized and they appear unchanged before/after the upgrade.
      • JRahm's avatar
        JRahm
        Icon for Admin rankAdmin
        May 06, 2022

        good deal. Keep us posted, would love to hear the resolution on this!

        And thank you, I appreciate that!