Forum Discussion

SanYang's avatar
SanYang
Icon for Cirrus rankCirrus
Sep 15, 2023

High CPU utilization for only cpu 0

Hello,

I have 26 CPUs in multi thread format but only CPU 0 utilization goes 100 and other CPUs utilization remains under 30%.

# tmsh show /sys tmm-info

--------------------------
Sys::TMM: 0.0      
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id         0
  TMM Count              1
  CPU Id                 0
                   
Memory (bytes)     
  Total              29.6G
  Used                4.7G
                   
CPU Usage Ratio (%)
  Last 5 Seconds       100
  Last 1 Minute        100
  Last 5 Minutes       100

--------------------------
Sys::TMM: 0.2      
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id         2
  TMM Count              1
  CPU Id                 2
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        25
  Last 1 Minute         25
  Last 5 Minutes        25

--------------------------
Sys::TMM: 0.4      
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id         4
  TMM Count              1
  CPU Id                 4
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        22
  Last 1 Minute         22
  Last 5 Minutes        22

--------------------------
Sys::TMM: 0.6      
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id         6
  TMM Count              1
  CPU Id                 6
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        30
  Last 1 Minute         28
  Last 5 Minutes        28

--------------------------
Sys::TMM: 0.8      
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id         8
  TMM Count              1
  CPU Id                 8
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        21
  Last 1 Minute         21
  Last 5 Minutes        21

--------------------------
Sys::TMM: 0.10     
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id        10
  TMM Count              1
  CPU Id                10
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        62
  Last 1 Minute         62
  Last 5 Minutes        62

--------------------------
Sys::TMM: 0.12     
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id        12
  TMM Count              1
  CPU Id                12
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        23
  Last 1 Minute         21
  Last 5 Minutes        22

--------------------------
Sys::TMM: 0.14     
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id        14
  TMM Count              1
  CPU Id                14
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        22
  Last 1 Minute         21
  Last 5 Minutes        21

--------------------------
Sys::TMM: 0.16     
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id        16
  TMM Count              1
  CPU Id                16
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        24
  Last 1 Minute         24
  Last 5 Minutes        25

--------------------------
Sys::TMM: 0.18     
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id        18
  TMM Count              1
  CPU Id                18
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        22
  Last 1 Minute         21
  Last 5 Minutes        22

--------------------------
Sys::TMM: 0.20     
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id        20
  TMM Count              1
  CPU Id                20
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        24
  Last 1 Minute         22
  Last 5 Minutes        22

--------------------------
Sys::TMM: 0.22     
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id        22
  TMM Count              1
  CPU Id                22
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        21
  Last 1 Minute         20
  Last 5 Minutes        20

--------------------------
Sys::TMM: 0.24     
--------------------------
Global             
  TMM Process Id     15246
  Running TMM Id        24
  TMM Count              1
  CPU Id                24
                   
Memory (bytes)     
  Total                  0
  Used                   0
                   
CPU Usage Ratio (%)
  Last 5 Seconds        22
  Last 1 Minute         21
  Last 5 Minutes        21

 Why is this happening

Any help is appreciate.

  • Hi SanYang , 

    try to run this Command : 

    tmsh show sys tmm-traffic

    To see how is the traffic distributed accross TMMs instances
    This Article will help you : https://my.f5.com/manage/s/article/K36483224


    - if you found uneven balancing throught tTMM instances , maybe you will need something Like ( F5 DAG ) to adjust this traffic ditribution , For DAG please have a look in this : https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-service-provider-generic-message-administration-13-0-0/5.html

    you can configure it on vlans ( GUI or CLI ). 

    - If you define the correct DAG , you shouldn't see that one of TMM instances CPU is 100 % , Correct DAG value protects your from Context-swiching ( which means one of your TMM is 100% so it couldn't handle the data traffic for clients so this leads to redirect connrctions to another TMM or drop it , also if TMM is fully utilized it may request more resources to handle this high pressure on it whereas the other TMMs not handle this quantity of data traffic ) 

    This is from my perspective , but I want to know the updates till reach the optimal value of TMM CPU maybe I am not fully correct, so I will keep the attention for this thread here 🙂 

    • SanYang's avatar
      SanYang
      Icon for Cirrus rankCirrus

      Hi Mohamed_Ahmed_Kansoh ,

      It seems to be evenly distributed.

      # tmsh show sys tmm-traffic
      
      -------------------------------------------------
      Sys::TMM: 0.0            
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       49.6T       38.5T
        Bits Out                      17.9T       17.9T
        Packets In                    86.2G       68.8G
        Packets Out                    2.0G        1.7G
        Current Connections             311         279
        Maximum Connections           20.2K       20.2K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             66.9G       66.9G
        Total Requests                58.6M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                   490.0K
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 537
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.2            
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       50.4T       41.0T
        Bits Out                      21.4T       21.5T
        Packets In                    96.6G       78.1G
        Packets Out                    2.4G        2.0G
        Current Connections            1.9K        1.7K
        Maximum Connections           63.8K       63.3K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             76.5G       76.5G
        Total Requests                70.4M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       34
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 504
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.4            
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       53.8T       44.0T
        Bits Out                      20.5T       20.6T
        Packets In                   100.6G       81.7G
        Packets Out                    2.3G        1.9G
        Current Connections           58.7K       58.2K
        Maximum Connections           62.8K       62.1K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             80.5G       80.5G
        Total Requests                67.3M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       33
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 585
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.6            
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       80.9T       46.5T
        Bits Out                      20.1T       20.2T
        Packets In                   149.2G       86.6G
        Packets Out                    2.2G        1.9G
        Current Connections            1.9K        1.7K
        Maximum Connections           20.4K       20.3K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             85.5G       85.5G
        Total Requests                65.7M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                        4
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 519
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.8            
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       46.4T       37.1T
        Bits Out                      20.1T       20.2T
        Packets In                    89.3G       72.0G
        Packets Out                    2.2G        1.9G
        Current Connections            1.8K        1.5K
        Maximum Connections           20.5K       20.4K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             70.6G       70.6G
        Total Requests                65.8M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       10
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 547
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.10           
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                      271.8T      257.5T
        Bits Out                      20.0T       20.1T
        Packets In                   132.0G      115.3G
        Packets Out                    2.2G        1.9G
        Current Connections            1.9K        1.6K
        Maximum Connections           20.4K       20.4K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             70.4G       70.4G
        Total Requests                65.5M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       19
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 582
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.12           
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       55.1T       42.5T
        Bits Out                      20.1T       20.3T
        Packets In                    98.7G       76.9G
        Packets Out                    2.2G        1.9G
        Current Connections            1.7K        1.4K
        Maximum Connections           20.4K       20.4K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             75.8G       75.8G
        Total Requests                65.7M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       42
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 538
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.14           
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       52.9T       40.6T
        Bits Out                      20.0T       20.1T
        Packets In                    92.3G       74.9G
        Packets Out                    2.2G        1.9G
        Current Connections            1.7K        1.5K
        Maximum Connections           20.3K       20.3K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             73.1G       73.1G
        Total Requests                65.5M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       22
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 557
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.16           
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       64.9T       47.9T
        Bits Out                      20.0T       20.1T
        Packets In                   110.0G       83.4G
        Packets Out                    2.2G        1.9G
        Current Connections            1.8K        1.5K
        Maximum Connections           20.3K       20.3K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             81.8G       81.8G
        Total Requests                65.6M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       19
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 532
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.18           
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       47.1T       37.7T
        Bits Out                      20.0T       20.1T
        Packets In                    81.1G       68.3G
        Packets Out                    2.2G        1.9G
        Current Connections            1.8K        1.6K
        Maximum Connections           20.3K       20.3K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             66.9G       66.9G
        Total Requests                65.5M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       27
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 536
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.20           
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       50.0T       38.0T
        Bits Out                      19.8T       19.9T
        Packets In                    89.6G       70.3G
        Packets Out                    2.2G        1.8G
        Current Connections            1.7K        1.5K
        Maximum Connections           20.0K       19.9K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             68.8G       68.8G
        Total Requests                64.7M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       10
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 490
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.22           
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       47.1T       37.5T
        Bits Out                      19.8T       19.9T
        Packets In                    89.2G       71.1G
        Packets Out                    2.2G        1.8G
        Current Connections            1.7K        1.5K
        Maximum Connections           20.3K       20.2K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             70.0G       70.0G
        Total Requests                64.9M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       30
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 527
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      
      -------------------------------------------------
      Sys::TMM: 0.24           
      -------------------------------------------------
      TMM Traffic                ClientSide  ServerSide
        Bits In                       49.8T       39.3T
        Bits Out                      19.9T       20.0T
        Packets In                    90.7G       73.4G
        Packets Out                    2.2G        1.9G
        Current Connections            1.8K        1.6K
        Maximum Connections           20.0K       20.0K
        Evicted Connections               0           0
        Slow Connections Killed           0           0
        Total Connections             72.3G       72.3G
        Total Requests                64.9M           -
                               
      Errors                   
        Maintenance Mode                  0
        Virtual Addr Limit                0
        Virtual Server Limit              0
        Wrong Address                     0
        No handler                       39
        No Staged Handler                 0
        No license                        0
        Connection Memory                 0
        Packets Dropped                 546
        Packets In Errors                 0
        Packets Out Errors                0
      Connections Redirected              0
      • Hi SanYang

        As I don't see errors in passing traffic and somehow balanced traffic for all TMM instances ( maybe only TMM0.10 handles high connections so it's CPU is +60% ) but over all TMMs CPU is within 20%. 

        So this is an indication of that TMM0 represents the Total CPU & memory usage in your blade and because you have till 24 tmm instances it gave you 100 % as this is a high number of TMMs. 
        This is Clarified in this Article : https://my.f5.com/manage/s/article/K02895424 

        Now , you should keen on in the overall CPU & memory usage as average between all TMMs cores. 

        If you go to ( Statistics >>> Performance reports >>> Chech the CPU & memory graphs ) 
        Also adjust time range ( Last 3 hours then Last week , then last month ) 
        I'm sure you will find the overall Cpu is consistent following the same trend for all other graphs , I mean you will find the overall/average CPU for the ( last month/week/day/hours ) not high or reached to 100% 

        Do u know if you found this overall CPU reached to 100 % your box will failover immediately. 

        So I need you to check that performance report and let me know if my analysis is correct or not. 

        Thanks

  • As Mohamed_Ahmed_Kansoh mentioned check the TMM outside of that in older versions the CMP demoded VS caused such issues but we do not know your version https://my.f5.com/manage/s/article/K13033 / https://my.f5.com/manage/s/article/K14358 and this something F5 ihealth can show you.

     

    As CPU is even number it means this core is for data plane as seen in the article here, so checking the TMM is good step: https://my.f5.com/manage/s/article/K23505424 / https://my.f5.com/manage/s/article/K15003

     

    Ma

    • SanYang's avatar
      SanYang
      Icon for Cirrus rankCirrus

      Hi Nikoolayy1 ,

       

      Thanks for your reply.
      My version is the latest (17)

      # tmsh show sys tmm-info | grep Sys::TMM
      Sys::TMM: 0.0      
      Sys::TMM: 0.2      
      Sys::TMM: 0.4      
      Sys::TMM: 0.6      
      Sys::TMM: 0.8      
      Sys::TMM: 0.10     
      Sys::TMM: 0.12     
      Sys::TMM: 0.14     
      Sys::TMM: 0.16     
      Sys::TMM: 0.18     
      Sys::TMM: 0.20     
      Sys::TMM: 0.22     
      Sys::TMM: 0.24