Forum Discussion

rjordan's avatar
rjordan
Icon for Nimbostratus rankNimbostratus
Nov 22, 2010

High memory usage from TMM

Hey Everyone,

 

I've been experiencing some weird issues on my LTM HA cluster. Whenever I make a config change through Web Admin or CLI (using b load), it see a hundred or so monitors flap up and down. The flapping only occurs on my active unit. If I make similar changes on the standby unit, I don't see any flapping. I've also failed over to the secondary unit and confirmed that it has the same issue. Increasing the timeouts to around 4 or 5 times the interval seems help, but I don't want to wait for 4-5 failures before a member is marked down.

 

I don't know when this issue started so I can't really associate it with a specific config change. We use our LTMs as a shared appliance to provide load balancing and redundancy to our enterprise customers. We don't do much with iRules other than pointing a hostname or URI to a specific member.

 

So far, I've done the following:

 

1. full_box_reboot on each unit

 

2. ran EUD - both came back clean

 

3. upgraded from 9.4.6 to 9.4.8 HF4

 

 

The host memory is pretty much maxed out at any given time on our active and standby units, so I don't think it is a bad iRule. I see the following process (ps -aux) on both units using ~40% CPU and ~80% memory:

 

/bin/tmm -m -s 1556 --command=rtc_frequency 4096

 

 

Has anyone had similar problems? Does anyone know what that process is?

 

management
monitoring
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Nov 23, 2010
    TMM is the Traffic Management Microkernel and is responsible for handling load balanced traffic. See SOL8035 for details on TMM and other system daemons.

     

     

    SOL8035: Overview of BIG-IP daemons

     

    http://support.f5.com/kb/en-us/solutions/public/8000/000/sol8035.html

     

     

    I suggest opening a case with F5 Support on the resource utilization issues. They'll be able to review your full configuration and recent logs to help troubleshoot this.

     

     

    Aaron
  • rjordan's avatar
    rjordan
    Icon for Nimbostratus rankNimbostratus
    Nov 23, 2010
    Thanks for the suggestion. I opened the case a week ago, but progress has been very slow. I just wanted to see if anyone else has experienced similar issues. I'll be sure to provide an update as soon as I find a fix.
  • L4L7_53191's avatar
    L4L7_53191
    Icon for Nimbostratus rankNimbostratus
    Dec 06, 2010
    To add a bit: because TMM is a microkernel it'll occupy CPU - it's got real time priority (you'll see it as RT in the priority column in top). As a result it'll sit on idle cycles or has first right of refusal at the very least.I've not spent cozy time with 9.x for a good long while, but I do remember that TMM typically ran hot. TMM sitting on 40% CPU in 9.x doesn't sound alarming to me, for what it's worth. On a 10.x system it's common to see it sitting at around 8-11% utilization per core.

     

     

    Despite all that, Aaron's advice is good, as usual - I'd open a case here too just to be sure.

     

     

    -Matt
  • JoshBecigneul's avatar
    JoshBecigneul
    Icon for MVP rankMVP
    Feb 14, 2012
    I hate to necromancer a thread like this, but I'm curious if there ever was a resolution to this issue. We're on 9.4.8 HF4 and experiencing similar issues. A case is open with F5 but it's progressing slowly.

     

     

    Thanks!

     

  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    Feb 15, 2012
    as Aaron suggested, please open a support case if you have not yet done.

     

     

    additonally, i think providing data below could be helpful for support team.

     

    1. generate qkview

     

    2. turn on bigd debug

     

    3. start tcpdump

     

    4. reproduce issue i.e. change config

     

    5. stop tcpdump

     

    6. turn off bigd debug

     

    7. generate qkview again

     

     

    Troubleshooting Ltm Monitors by Aaron

     

    http://devcentral.f5.com/wiki/AdvDesignConfig.TroubleshootingLtmMonitors.ashx