Forum Discussion

Satriaji's avatar
Satriaji
Icon for Cirrus rankCirrus
Mar 23, 2023

TMM Restarting (OS Version 12.1.1)

Hi Everyone,

I have a problem with my F5 device. The problem is the tmm service always restarting continuesly, so its make the device INOPERATIVE (intermittent active-standby).  Here the logs :

"Mar 23 23:12:17 JTLSF-DNS-PR-SOA-PREPROD emerg logger: Re-starting tmm"

Mar 23 22:09:29 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 01140045:5: HA reports tmm NOT ready.
Mar 23 22:09:29 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0050:5: Sod requests links down.
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD warning sod[5195]: 01140029:4: HA daemon_heartbeat tmm fails action is go offline down links and restart.
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: HalFailover_::set: Cannot clear /dev/ttyS1 DTR/RTS. errno=5
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: halSetFailover: set error
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0066:5: halSetFailover (clear) fails with status 11.
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0054:5: Offline for traffic group /Common/traffic-group-1.
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c003e:5: Offline
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err mcpd[6576]: 01070069:3: Subscription not found in mcpd for subscriber Id BIGD_Subscriber.
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006d:5: Leaving Offline for Active for dbvar not redundant.
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0053:5: Active for traffic group /Common/traffic-group-1.
Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0019:5: Active
Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006c:5: proc stat: [0] pid:24126 comm:(tmm.0) state:S utime:1474241 stime:648064 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 [-1] pid:24126 comm:(tmm.0) state:S utime:1474241 stime:648047 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 [-2] pid:24126 comm:(tmm.0) state:R utime:1474241 stime:648029 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 .
Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD warning sod[5195]: 01140029:4: HA proc_running tmm fails action is go offline and down links.
Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: HalFailover_::set: Cannot clear /dev/ttyS1 DTR/RTS. errno=5
Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: halSetFailover: set error
Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0066:5: halSetFailover (clear) fails with status 11.
Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0054:5: Offline for traffic group /Common/traffic-group-1.
Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c003e:5: Offline
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD info bigstart: Start bigd in single-process mode
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD info bigd.start[31188]: Execing bigd: /usr/bin/bigd bigd -S
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice promptstatusd[3640]: 01460006:5: semaphore tmm.running(1) held
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus datastor
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dedup_admin
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus tmrouted
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus wamd
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart tmrouted
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus websso
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart localdbmgr
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus acctd
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus eam
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus rba
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus eca
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice tmrouted[23455]: 01910005:5: Tmrouted exiting after getting termination request.
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id tmrouted
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus nlad
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus vdi
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus urldb
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus datasyncd
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus fpuserd
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus bdosd
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus antserver
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus admd
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus asm
Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus avrd
Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart avrd
Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dosl7d
Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dwbld
Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus mgmt_acld
Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus wr_urldbd
Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070410:5: Removed subscription with subscriber id avrd
Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id TMM
Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus avrd
Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart restart avrd
Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus dosl7d
Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus asm
Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus datasyncd
Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus admd
Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus mgmt_acld
Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus dwbld
Mar 23 22:09:51 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006c:5: proc stat: [0] pid:31188 comm:(bigd) state:S utime:12 stime:2 cutime:21 cstime:4 starttime:321443168 vsize:44949504 rss:6251 wchan:0 blkio_ticks:0 [-1] pid:1846 comm:(bigd) state:S utime:12181 stime:6605 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 [-2] pid:1846 comm:(bigd) state:S utime:12181 stime:6604 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 .
Mar 23 22:09:52 JTLSF-DNS-PR-SOA-PREPROD notice tmm[32121]: 01010001:5: pgo_use x86_64 padc TMM Version 12.1.1.0.0.184 starting
Mar 23 22:09:53 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070404:5: Add a new Publication for publisherID TMM and filterType (nil)
Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 01140030:5: HA proc_running tmm is now responding.
Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006d:5: Leaving Offline for Active for dbvar not redundant.
Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0053:5: Active for traffic group /Common/traffic-group-1.
Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0019:5: Active

How to solve this problem ? I want to upgrade this device ... but the CLI [INOPERATIVE], so we can do anyting at the CLI for running the script right ? Anyone have any steps for solve this TMM Restart Continuesly ?

 

Thankyouuu ...

 

application delivery
inopperative
tmm
tmm restart
  • alayne786's avatar
    alayne786
    Icon for Altostratus rankAltostratus
    Nov 02, 2023

    Any resolution?? We have bigip LTM 14.1 and it's been doing this for 24 hours now, but it's this exact situation. How did you fix it??

  • CA_Valli's avatar
    CA_Valli
    Icon for MVP rankMVP
    Mar 23, 2023

    If the unit is covered by a support contract and this is service impacting, your best bet is to open a support case to investigate the issue live.

    Most common reason for a unit to behave like this is if you have a VLAN failover enabled, and unit has the monitored links down.
    I haven't seen this specific evidence in your logs, but you do have a deamon heartbeat failure error. That means a system process is failing to respond to HA daemon heartbeat messages, which triggers the failover. 

    Again, I'd suggest you have support look into this. 

    • alayne786's avatar
      alayne786
      Icon for Altostratus rankAltostratus
      Nov 02, 2023

      Any resolution?? We have bigip LTM 14.1 and it's been doing this for 24 hours now, but it's this exact situation. How did you fix it?? We do have a vLan failover ..Been waiting for 24 hours from f5 support with no resolution in sight .

    • Satriaji's avatar
      Satriaji
      Icon for Cirrus rankCirrus
      Mar 23, 2023

      The technical support for this OS device has been out of support (Technical EOS at 2021), so we cannot open TAC to support.

      For now the log still appear :

      [msi@JTLSF-DNS-PR-SOA-PREPROD:INOPERATIVE:Standalone] ~ # Mar 23 23:51:45 JTLSF-DNS-PR-SOA-PREPROD emerg logger: Re-starting tmm

       

      This device is vCMP Guest, the other guest is normal. Just thist guest that having problem tmm restarting.

       

       

       

      • DuncanJones's avatar
        DuncanJones
        Icon for Nimbostratus rankNimbostratus
        Mar 24, 2023
        Satriaji wrote:

        The technical support for this OS device has been out of support (Technical EOS at 2021), so we cannot open TAC to support.

        For now the log still appear : Tower Defense

        [msi@JTLSF-DNS-PR-SOA-PREPROD:INOPERATIVE:Standalone] ~ # Mar 23 23:51:45 JTLSF-DNS-PR-SOA-PREPROD emerg logger: Re-starting tmm

         

        This device is vCMP Guest, the other guest is normal. Just thist guest that having problem tmm restarting.

         

         

         

        The Traffic Management Microkernel (tmm) process may restart without a stack trace or core file after becoming disconnected from the Master Control Program Daemon (mcpd) process.

        This issue occurs when all of the following conditions are met:

        • The Linux kernel memory becomes fragmented.
        • Another memory-intensive operation occurs, which may include:
          • You perform a configuration synchronization (ConfigSync) with full reload.
          • You run a tcpdump session.

        If tmm fails to start or restart properly as a result of this issue, you can try to recover the system temporarily by restarting the BIG-IP system.