14-Sep-2023 20:29
Hello,
I have 26 CPUs in multi thread format but only CPU 0 utilization goes 100 and other CPUs utilization remains under 30%.
# tmsh show /sys tmm-info
--------------------------
Sys::TMM: 0.0
--------------------------
Global
TMM Process Id 15246
Running TMM Id 0
TMM Count 1
CPU Id 0
Memory (bytes)
Total 29.6G
Used 4.7G
CPU Usage Ratio (%)
Last 5 Seconds 100
Last 1 Minute 100
Last 5 Minutes 100
--------------------------
Sys::TMM: 0.2
--------------------------
Global
TMM Process Id 15246
Running TMM Id 2
TMM Count 1
CPU Id 2
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 25
Last 1 Minute 25
Last 5 Minutes 25
--------------------------
Sys::TMM: 0.4
--------------------------
Global
TMM Process Id 15246
Running TMM Id 4
TMM Count 1
CPU Id 4
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 22
Last 1 Minute 22
Last 5 Minutes 22
--------------------------
Sys::TMM: 0.6
--------------------------
Global
TMM Process Id 15246
Running TMM Id 6
TMM Count 1
CPU Id 6
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 30
Last 1 Minute 28
Last 5 Minutes 28
--------------------------
Sys::TMM: 0.8
--------------------------
Global
TMM Process Id 15246
Running TMM Id 8
TMM Count 1
CPU Id 8
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 21
Last 1 Minute 21
Last 5 Minutes 21
--------------------------
Sys::TMM: 0.10
--------------------------
Global
TMM Process Id 15246
Running TMM Id 10
TMM Count 1
CPU Id 10
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 62
Last 1 Minute 62
Last 5 Minutes 62
--------------------------
Sys::TMM: 0.12
--------------------------
Global
TMM Process Id 15246
Running TMM Id 12
TMM Count 1
CPU Id 12
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 23
Last 1 Minute 21
Last 5 Minutes 22
--------------------------
Sys::TMM: 0.14
--------------------------
Global
TMM Process Id 15246
Running TMM Id 14
TMM Count 1
CPU Id 14
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 22
Last 1 Minute 21
Last 5 Minutes 21
--------------------------
Sys::TMM: 0.16
--------------------------
Global
TMM Process Id 15246
Running TMM Id 16
TMM Count 1
CPU Id 16
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 24
Last 1 Minute 24
Last 5 Minutes 25
--------------------------
Sys::TMM: 0.18
--------------------------
Global
TMM Process Id 15246
Running TMM Id 18
TMM Count 1
CPU Id 18
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 22
Last 1 Minute 21
Last 5 Minutes 22
--------------------------
Sys::TMM: 0.20
--------------------------
Global
TMM Process Id 15246
Running TMM Id 20
TMM Count 1
CPU Id 20
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 24
Last 1 Minute 22
Last 5 Minutes 22
--------------------------
Sys::TMM: 0.22
--------------------------
Global
TMM Process Id 15246
Running TMM Id 22
TMM Count 1
CPU Id 22
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 21
Last 1 Minute 20
Last 5 Minutes 20
--------------------------
Sys::TMM: 0.24
--------------------------
Global
TMM Process Id 15246
Running TMM Id 24
TMM Count 1
CPU Id 24
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 22
Last 1 Minute 21
Last 5 Minutes 21
Why is this happening
Any help is appreciate.
15-Sep-2023 19:11
Hi @SanYang ,
try to run this Command :
tmsh show sys tmm-traffic
To see how is the traffic distributed accross TMMs instances
This Article will help you : https://my.f5.com/manage/s/article/K36483224
- if you found uneven balancing throught tTMM instances , maybe you will need something Like ( F5 DAG ) to adjust this traffic ditribution , For DAG please have a look in this : https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-service-provider-generic-...
you can configure it on vlans ( GUI or CLI ).
- If you define the correct DAG , you shouldn't see that one of TMM instances CPU is 100 % , Correct DAG value protects your from Context-swiching ( which means one of your TMM is 100% so it couldn't handle the data traffic for clients so this leads to redirect connrctions to another TMM or drop it , also if TMM is fully utilized it may request more resources to handle this high pressure on it whereas the other TMMs not handle this quantity of data traffic )
This is from my perspective , but I want to know the updates till reach the optimal value of TMM CPU maybe I am not fully correct, so I will keep the attention for this thread here 🙂
17-Sep-2023 18:57
It seems to be evenly distributed.
# tmsh show sys tmm-traffic
-------------------------------------------------
Sys::TMM: 0.0
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 49.6T 38.5T
Bits Out 17.9T 17.9T
Packets In 86.2G 68.8G
Packets Out 2.0G 1.7G
Current Connections 311 279
Maximum Connections 20.2K 20.2K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 66.9G 66.9G
Total Requests 58.6M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 490.0K
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 537
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.2
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 50.4T 41.0T
Bits Out 21.4T 21.5T
Packets In 96.6G 78.1G
Packets Out 2.4G 2.0G
Current Connections 1.9K 1.7K
Maximum Connections 63.8K 63.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 76.5G 76.5G
Total Requests 70.4M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 34
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 504
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.4
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 53.8T 44.0T
Bits Out 20.5T 20.6T
Packets In 100.6G 81.7G
Packets Out 2.3G 1.9G
Current Connections 58.7K 58.2K
Maximum Connections 62.8K 62.1K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 80.5G 80.5G
Total Requests 67.3M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 33
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 585
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.6
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 80.9T 46.5T
Bits Out 20.1T 20.2T
Packets In 149.2G 86.6G
Packets Out 2.2G 1.9G
Current Connections 1.9K 1.7K
Maximum Connections 20.4K 20.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 85.5G 85.5G
Total Requests 65.7M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 4
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 519
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.8
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 46.4T 37.1T
Bits Out 20.1T 20.2T
Packets In 89.3G 72.0G
Packets Out 2.2G 1.9G
Current Connections 1.8K 1.5K
Maximum Connections 20.5K 20.4K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 70.6G 70.6G
Total Requests 65.8M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 10
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 547
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.10
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 271.8T 257.5T
Bits Out 20.0T 20.1T
Packets In 132.0G 115.3G
Packets Out 2.2G 1.9G
Current Connections 1.9K 1.6K
Maximum Connections 20.4K 20.4K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 70.4G 70.4G
Total Requests 65.5M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 19
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 582
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.12
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 55.1T 42.5T
Bits Out 20.1T 20.3T
Packets In 98.7G 76.9G
Packets Out 2.2G 1.9G
Current Connections 1.7K 1.4K
Maximum Connections 20.4K 20.4K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 75.8G 75.8G
Total Requests 65.7M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 42
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 538
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.14
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 52.9T 40.6T
Bits Out 20.0T 20.1T
Packets In 92.3G 74.9G
Packets Out 2.2G 1.9G
Current Connections 1.7K 1.5K
Maximum Connections 20.3K 20.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 73.1G 73.1G
Total Requests 65.5M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 22
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 557
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.16
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 64.9T 47.9T
Bits Out 20.0T 20.1T
Packets In 110.0G 83.4G
Packets Out 2.2G 1.9G
Current Connections 1.8K 1.5K
Maximum Connections 20.3K 20.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 81.8G 81.8G
Total Requests 65.6M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 19
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 532
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.18
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 47.1T 37.7T
Bits Out 20.0T 20.1T
Packets In 81.1G 68.3G
Packets Out 2.2G 1.9G
Current Connections 1.8K 1.6K
Maximum Connections 20.3K 20.3K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 66.9G 66.9G
Total Requests 65.5M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 27
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 536
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.20
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 50.0T 38.0T
Bits Out 19.8T 19.9T
Packets In 89.6G 70.3G
Packets Out 2.2G 1.8G
Current Connections 1.7K 1.5K
Maximum Connections 20.0K 19.9K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 68.8G 68.8G
Total Requests 64.7M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 10
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 490
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.22
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 47.1T 37.5T
Bits Out 19.8T 19.9T
Packets In 89.2G 71.1G
Packets Out 2.2G 1.8G
Current Connections 1.7K 1.5K
Maximum Connections 20.3K 20.2K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 70.0G 70.0G
Total Requests 64.9M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 30
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 527
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
-------------------------------------------------
Sys::TMM: 0.24
-------------------------------------------------
TMM Traffic ClientSide ServerSide
Bits In 49.8T 39.3T
Bits Out 19.9T 20.0T
Packets In 90.7G 73.4G
Packets Out 2.2G 1.9G
Current Connections 1.8K 1.6K
Maximum Connections 20.0K 20.0K
Evicted Connections 0 0
Slow Connections Killed 0 0
Total Connections 72.3G 72.3G
Total Requests 64.9M -
Errors
Maintenance Mode 0
Virtual Addr Limit 0
Virtual Server Limit 0
Wrong Address 0
No handler 39
No Staged Handler 0
No license 0
Connection Memory 0
Packets Dropped 546
Packets In Errors 0
Packets Out Errors 0
Connections Redirected 0
17-Sep-2023 19:20
Hi @SanYang,
As I don't see errors in passing traffic and somehow balanced traffic for all TMM instances ( maybe only TMM0.10 handles high connections so it's CPU is +60% ) but over all TMMs CPU is within 20%.
So this is an indication of that TMM0 represents the Total CPU & memory usage in your blade and because you have till 24 tmm instances it gave you 100 % as this is a high number of TMMs.
This is Clarified in this Article : https://my.f5.com/manage/s/article/K02895424
Now , you should keen on in the overall CPU & memory usage as average between all TMMs cores.
If you go to ( Statistics >>> Performance reports >>> Chech the CPU & memory graphs )
Also adjust time range ( Last 3 hours then Last week , then last month )
I'm sure you will find the overall Cpu is consistent following the same trend for all other graphs , I mean you will find the overall/average CPU for the ( last month/week/day/hours ) not high or reached to 100%
Do u know if you found this overall CPU reached to 100 % your box will failover immediately.
So I need you to check that performance report and let me know if my analysis is correct or not.
Thanks
17-Sep-2023 20:15
Thanks for your reply.
I checked the charts.
CPU0 has been running at 100% for a while now...
18-Sep-2023 01:08
Hello @SanYang,
yes this does make sense with the TMM0 CPU value in CLI.
But as I see in the historical graphs last monday and tuesday it was normal.
I see now you need to open a case with F5 support , to validate the health of TMM0.
maybe this is the reason : https://my.f5.com/manage/s/article/K02895424
but you need to verify this , because if one of your TMMs crashes ( data plane ones I mean ) this will effect badly in your overall performance.
So I recommend opening a case.
then when you get the RCA for this high utilization you can share it with us.
16-Sep-2023 02:29 - edited 16-Sep-2023 02:32
As @Mohamed_Ahmed_Kansoh mentioned check the TMM outside of that in older versions the CMP demoded VS caused such issues but we do not know your version https://my.f5.com/manage/s/article/K13033 / https://my.f5.com/manage/s/article/K14358 and this something F5 ihealth can show you.
As CPU is even number it means this core is for data plane as seen in the article here, so checking the TMM is good step: https://my.f5.com/manage/s/article/K23505424 / https://my.f5.com/manage/s/article/K15003
Ma
17-Sep-2023 19:00
Hi @Nikoolayy1 ,
Thanks for your reply.
My version is the latest (17)
# tmsh show sys tmm-info | grep Sys::TMM
Sys::TMM: 0.0
Sys::TMM: 0.2
Sys::TMM: 0.4
Sys::TMM: 0.6
Sys::TMM: 0.8
Sys::TMM: 0.10
Sys::TMM: 0.12
Sys::TMM: 0.14
Sys::TMM: 0.16
Sys::TMM: 0.18
Sys::TMM: 0.20
Sys::TMM: 0.22
Sys::TMM: 0.24