Forum Discussion
rjordan
Nimbostratus
NimbostratusHigh memory usage from TMM
Hey Everyone,
I've been experiencing some weird issues on my LTM HA cluster. Whenever I make a config change through Web Admin or CLI (using b load), it see a hundred or so monitors flap up and down. The flapping only occurs on my active unit. If I make similar changes on the standby unit, I don't see any flapping. I've also failed over to the secondary unit and confirmed that it has the same issue. Increasing the timeouts to around 4 or 5 times the interval seems help, but I don't want to wait for 4-5 failures before a member is marked down.
I don't know when this issue started so I can't really associate it with a specific config change. We use our LTMs as a shared appliance to provide load balancing and redundancy to our enterprise customers. We don't do much with iRules other than pointing a hostname or URI to a specific member.
So far, I've done the following:
1. full_box_reboot on each unit
2. ran EUD - both came back clean
3. upgraded from 9.4.6 to 9.4.8 HF4
The host memory is pretty much maxed out at any given time on our active and standby units, so I don't think it is a bad iRule. I see the following process (ps -aux) on both units using ~40% CPU and ~80% memory:
/bin/tmm -m -s 1556 --command=rtc_frequency 4096
Has anyone had similar problems? Does anyone know what that process is?
hoolioCirrostratus
TMM is the Traffic Management Microkernel and is responsible for handling load balanced traffic. See SOL8035 for details on TMM and other system daemons.
SOL8035: Overview of BIG-IP daemons
http://support.f5.com/kb/en-us/solutions/public/8000/000/sol8035.html
I suggest opening a case with F5 Support on the resource utilization issues. They'll be able to review your full configuration and recent logs to help troubleshoot this.
Aaron
rjordanThanks for the suggestion. I opened the case a week ago, but progress has been very slow. I just wanted to see if anyone else has experienced similar issues. I'll be sure to provide an update as soon as I find a fix.
L4L7_53191To add a bit: because TMM is a microkernel it'll occupy CPU - it's got real time priority (you'll see it as RT in the priority column in top). As a result it'll sit on idle cycles or has first right of refusal at the very least.I've not spent cozy time with 9.x for a good long while, but I do remember that TMM typically ran hot. TMM sitting on 40% CPU in 9.x doesn't sound alarming to me, for what it's worth. On a 10.x system it's common to see it sitting at around 8-11% utilization per core.
Despite all that, Aaron's advice is good, as usual - I'd open a case here too just to be sure.
-Matt- I hate to necromancer a thread like this, but I'm curious if there ever was a resolution to this issue. We're on 9.4.8 HF4 and experiencing similar issues. A case is open with F5 but it's progressing slowly.
Thanks! - nitass
Employee
as Aaron suggested, please open a support case if you have not yet done.
additonally, i think providing data below could be helpful for support team.
1. generate qkview
2. turn on bigd debug
3. start tcpdump
4. reproduce issue i.e. change config
5. stop tcpdump
6. turn off bigd debug
7. generate qkview again
Troubleshooting Ltm Monitors by Aaron
http://devcentral.f5.com/wiki/AdvDesignConfig.TroubleshootingLtmMonitors.ashx
