Forum Discussion
High CPU utilization for only cpu 0
Hello,
I have 26 CPUs in multi thread format but only CPU 0 utilization goes 100 and other CPUs utilization remains under 30%.
# tmsh show /sys tmm-info
--------------------------
Sys::TMM: 0.0
--------------------------
Global
TMM Process Id 15246
Running TMM Id 0
TMM Count 1
CPU Id 0
Memory (bytes)
Total 29.6G
Used 4.7G
CPU Usage Ratio (%)
Last 5 Seconds 100
Last 1 Minute 100
Last 5 Minutes 100
--------------------------
Sys::TMM: 0.2
--------------------------
Global
TMM Process Id 15246
Running TMM Id 2
TMM Count 1
CPU Id 2
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 25
Last 1 Minute 25
Last 5 Minutes 25
--------------------------
Sys::TMM: 0.4
--------------------------
Global
TMM Process Id 15246
Running TMM Id 4
TMM Count 1
CPU Id 4
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 22
Last 1 Minute 22
Last 5 Minutes 22
--------------------------
Sys::TMM: 0.6
--------------------------
Global
TMM Process Id 15246
Running TMM Id 6
TMM Count 1
CPU Id 6
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 30
Last 1 Minute 28
Last 5 Minutes 28
--------------------------
Sys::TMM: 0.8
--------------------------
Global
TMM Process Id 15246
Running TMM Id 8
TMM Count 1
CPU Id 8
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 21
Last 1 Minute 21
Last 5 Minutes 21
--------------------------
Sys::TMM: 0.10
--------------------------
Global
TMM Process Id 15246
Running TMM Id 10
TMM Count 1
CPU Id 10
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 62
Last 1 Minute 62
Last 5 Minutes 62
--------------------------
Sys::TMM: 0.12
--------------------------
Global
TMM Process Id 15246
Running TMM Id 12
TMM Count 1
CPU Id 12
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 23
Last 1 Minute 21
Last 5 Minutes 22
--------------------------
Sys::TMM: 0.14
--------------------------
Global
TMM Process Id 15246
Running TMM Id 14
TMM Count 1
CPU Id 14
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 22
Last 1 Minute 21
Last 5 Minutes 21
--------------------------
Sys::TMM: 0.16
--------------------------
Global
TMM Process Id 15246
Running TMM Id 16
TMM Count 1
CPU Id 16
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 24
Last 1 Minute 24
Last 5 Minutes 25
--------------------------
Sys::TMM: 0.18
--------------------------
Global
TMM Process Id 15246
Running TMM Id 18
TMM Count 1
CPU Id 18
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 22
Last 1 Minute 21
Last 5 Minutes 22
--------------------------
Sys::TMM: 0.20
--------------------------
Global
TMM Process Id 15246
Running TMM Id 20
TMM Count 1
CPU Id 20
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 24
Last 1 Minute 22
Last 5 Minutes 22
--------------------------
Sys::TMM: 0.22
--------------------------
Global
TMM Process Id 15246
Running TMM Id 22
TMM Count 1
CPU Id 22
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 21
Last 1 Minute 20
Last 5 Minutes 20
--------------------------
Sys::TMM: 0.24
--------------------------
Global
TMM Process Id 15246
Running TMM Id 24
TMM Count 1
CPU Id 24
Memory (bytes)
Total 0
Used 0
CPU Usage Ratio (%)
Last 5 Seconds 22
Last 1 Minute 21
Last 5 Minutes 21
Why is this happening
Any help is appreciate.
Hi SanYang ,
try to run this Command :tmsh show sys tmm-traffic
To see how is the traffic distributed accross TMMs instances
This Article will help you : https://my.f5.com/manage/s/article/K36483224
- if you found uneven balancing throught tTMM instances , maybe you will need something Like ( F5 DAG ) to adjust this traffic ditribution , For DAG please have a look in this : https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-service-provider-generic-message-administration-13-0-0/5.htmlyou can configure it on vlans ( GUI or CLI ).
- If you define the correct DAG , you shouldn't see that one of TMM instances CPU is 100 % , Correct DAG value protects your from Context-swiching ( which means one of your TMM is 100% so it couldn't handle the data traffic for clients so this leads to redirect connrctions to another TMM or drop it , also if TMM is fully utilized it may request more resources to handle this high pressure on it whereas the other TMMs not handle this quantity of data traffic )
This is from my perspective , but I want to know the updates till reach the optimal value of TMM CPU maybe I am not fully correct, so I will keep the attention for this thread here 🙂- SanYang
Cirrus
Hi Mohamed_Ahmed_Kansoh ,
It seems to be evenly distributed.
# tmsh show sys tmm-traffic ------------------------------------------------- Sys::TMM: 0.0 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 49.6T 38.5T Bits Out 17.9T 17.9T Packets In 86.2G 68.8G Packets Out 2.0G 1.7G Current Connections 311 279 Maximum Connections 20.2K 20.2K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 66.9G 66.9G Total Requests 58.6M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 490.0K No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 537 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.2 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 50.4T 41.0T Bits Out 21.4T 21.5T Packets In 96.6G 78.1G Packets Out 2.4G 2.0G Current Connections 1.9K 1.7K Maximum Connections 63.8K 63.3K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 76.5G 76.5G Total Requests 70.4M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 34 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 504 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.4 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 53.8T 44.0T Bits Out 20.5T 20.6T Packets In 100.6G 81.7G Packets Out 2.3G 1.9G Current Connections 58.7K 58.2K Maximum Connections 62.8K 62.1K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 80.5G 80.5G Total Requests 67.3M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 33 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 585 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.6 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 80.9T 46.5T Bits Out 20.1T 20.2T Packets In 149.2G 86.6G Packets Out 2.2G 1.9G Current Connections 1.9K 1.7K Maximum Connections 20.4K 20.3K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 85.5G 85.5G Total Requests 65.7M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 4 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 519 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.8 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 46.4T 37.1T Bits Out 20.1T 20.2T Packets In 89.3G 72.0G Packets Out 2.2G 1.9G Current Connections 1.8K 1.5K Maximum Connections 20.5K 20.4K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 70.6G 70.6G Total Requests 65.8M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 10 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 547 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.10 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 271.8T 257.5T Bits Out 20.0T 20.1T Packets In 132.0G 115.3G Packets Out 2.2G 1.9G Current Connections 1.9K 1.6K Maximum Connections 20.4K 20.4K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 70.4G 70.4G Total Requests 65.5M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 19 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 582 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.12 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 55.1T 42.5T Bits Out 20.1T 20.3T Packets In 98.7G 76.9G Packets Out 2.2G 1.9G Current Connections 1.7K 1.4K Maximum Connections 20.4K 20.4K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 75.8G 75.8G Total Requests 65.7M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 42 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 538 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.14 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 52.9T 40.6T Bits Out 20.0T 20.1T Packets In 92.3G 74.9G Packets Out 2.2G 1.9G Current Connections 1.7K 1.5K Maximum Connections 20.3K 20.3K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 73.1G 73.1G Total Requests 65.5M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 22 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 557 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.16 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 64.9T 47.9T Bits Out 20.0T 20.1T Packets In 110.0G 83.4G Packets Out 2.2G 1.9G Current Connections 1.8K 1.5K Maximum Connections 20.3K 20.3K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 81.8G 81.8G Total Requests 65.6M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 19 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 532 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.18 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 47.1T 37.7T Bits Out 20.0T 20.1T Packets In 81.1G 68.3G Packets Out 2.2G 1.9G Current Connections 1.8K 1.6K Maximum Connections 20.3K 20.3K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 66.9G 66.9G Total Requests 65.5M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 27 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 536 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.20 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 50.0T 38.0T Bits Out 19.8T 19.9T Packets In 89.6G 70.3G Packets Out 2.2G 1.8G Current Connections 1.7K 1.5K Maximum Connections 20.0K 19.9K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 68.8G 68.8G Total Requests 64.7M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 10 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 490 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.22 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 47.1T 37.5T Bits Out 19.8T 19.9T Packets In 89.2G 71.1G Packets Out 2.2G 1.8G Current Connections 1.7K 1.5K Maximum Connections 20.3K 20.2K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 70.0G 70.0G Total Requests 64.9M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 30 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 527 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0 ------------------------------------------------- Sys::TMM: 0.24 ------------------------------------------------- TMM Traffic ClientSide ServerSide Bits In 49.8T 39.3T Bits Out 19.9T 20.0T Packets In 90.7G 73.4G Packets Out 2.2G 1.9G Current Connections 1.8K 1.6K Maximum Connections 20.0K 20.0K Evicted Connections 0 0 Slow Connections Killed 0 0 Total Connections 72.3G 72.3G Total Requests 64.9M - Errors Maintenance Mode 0 Virtual Addr Limit 0 Virtual Server Limit 0 Wrong Address 0 No handler 39 No Staged Handler 0 No license 0 Connection Memory 0 Packets Dropped 546 Packets In Errors 0 Packets Out Errors 0 Connections Redirected 0
Hi SanYang,
As I don't see errors in passing traffic and somehow balanced traffic for all TMM instances ( maybe only TMM0.10 handles high connections so it's CPU is +60% ) but over all TMMs CPU is within 20%.
So this is an indication of that TMM0 represents the Total CPU & memory usage in your blade and because you have till 24 tmm instances it gave you 100 % as this is a high number of TMMs.
This is Clarified in this Article : https://my.f5.com/manage/s/article/K02895424Now , you should keen on in the overall CPU & memory usage as average between all TMMs cores.
If you go to ( Statistics >>> Performance reports >>> Chech the CPU & memory graphs )
Also adjust time range ( Last 3 hours then Last week , then last month )
I'm sure you will find the overall Cpu is consistent following the same trend for all other graphs , I mean you will find the overall/average CPU for the ( last month/week/day/hours ) not high or reached to 100%
Do u know if you found this overall CPU reached to 100 % your box will failover immediately.
So I need you to check that performance report and let me know if my analysis is correct or not.
Thanks
As Mohamed_Ahmed_Kansoh mentioned check the TMM outside of that in older versions the CMP demoded VS caused such issues but we do not know your version https://my.f5.com/manage/s/article/K13033 / https://my.f5.com/manage/s/article/K14358 and this something F5 ihealth can show you.
As CPU is even number it means this core is for data plane as seen in the article here, so checking the TMM is good step: https://my.f5.com/manage/s/article/K23505424 / https://my.f5.com/manage/s/article/K15003
Ma
- SanYang
Cirrus
Hi Nikoolayy1 ,
Thanks for your reply.
My version is the latest (17)# tmsh show sys tmm-info | grep Sys::TMM Sys::TMM: 0.0 Sys::TMM: 0.2 Sys::TMM: 0.4 Sys::TMM: 0.6 Sys::TMM: 0.8 Sys::TMM: 0.10 Sys::TMM: 0.12 Sys::TMM: 0.14 Sys::TMM: 0.16 Sys::TMM: 0.18 Sys::TMM: 0.20 Sys::TMM: 0.22 Sys::TMM: 0.24
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com