TMM Restarting (OS Version 12.1.1)

Question

Hi Everyone,I have a problem with my F5 device. The problem is the tmm service always restarting continuesly, so its make the device INOPERATIVE (intermittent active-standby).&nbsp; Here the logs :"Mar 23 23:12:17 JTLSF-DNS-PR-SOA-PREPROD emerg logger: Re-starting tmm"Mar 23 22:09:29 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 01140045:5: HA reports tmm NOT ready.Mar 23 22:09:29 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0050:5: Sod requests links down.Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD warning sod[5195]: 01140029:4: HA daemon_heartbeat tmm fails action is go offline down links and restart.Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: HalFailover_::set: Cannot clear /dev/ttyS1 DTR/RTS. errno=5Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: halSetFailover: set errorMar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0066:5: halSetFailover (clear) fails with status 11.Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0054:5: Offline for traffic group /Common/traffic-group-1.Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c003e:5: OfflineMar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err mcpd[6576]: 01070069:3: Subscription not found in mcpd for subscriber Id BIGD_Subscriber.Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006d:5: Leaving Offline for Active for dbvar not redundant.Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0053:5: Active for traffic group /Common/traffic-group-1.Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0019:5: ActiveMar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006c:5: proc stat: [0] pid:24126 comm:(tmm.0) state:S utime:1474241 stime:648064 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 [-1] pid:24126 comm:(tmm.0) state:S utime:1474241 stime:648047 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 [-2] pid:24126 comm:(tmm.0) state:R utime:1474241 stime:648029 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 .Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD warning sod[5195]: 01140029:4: HA proc_running tmm fails action is go offline and down links.Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: HalFailover_::set: Cannot clear /dev/ttyS1 DTR/RTS. errno=5Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: halSetFailover: set errorMar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0066:5: halSetFailover (clear) fails with status 11.Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0054:5: Offline for traffic group /Common/traffic-group-1.Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c003e:5: OfflineMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD info bigstart: Start bigd in single-process modeMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD info bigd.start[31188]: Execing bigd: /usr/bin/bigd bigd -SMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice promptstatusd[3640]: 01460006:5: semaphore tmm.running(1) heldMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus datastorMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus dedup_adminMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus tmroutedMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus wamdMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart restart tmroutedMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus webssoMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart restart localdbmgrMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus acctdMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus eamMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus rbaMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus ecaMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice tmrouted[23455]: 01910005:5: Tmrouted exiting after getting termination request.Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id tmroutedMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus nladMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus vdiMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus urldbMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus datasyncdMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus fpuserdMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus bdosdMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus antserverMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus admdMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus asmMar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus avrdMar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart restart avrdMar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus dosl7dMar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus dwbldMar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus mgmt_acldMar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==&gt; /usr/bin/bigstart singlestatus wr_urldbdMar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070410:5: Removed subscription with subscriber id avrdMar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id TMMMar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==&gt; /usr/bin/bigstart singlestatus avrdMar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==&gt; /usr/bin/bigstart restart avrdMar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==&gt; /usr/bin/bigstart singlestatus dosl7dMar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==&gt; /usr/bin/bigstart singlestatus asmMar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==&gt; /usr/bin/bigstart singlestatus datasyncdMar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==&gt; /usr/bin/bigstart singlestatus admdMar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==&gt; /usr/bin/bigstart singlestatus mgmt_acldMar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==&gt; /usr/bin/bigstart singlestatus dwbldMar 23 22:09:51 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006c:5: proc stat: [0] pid:31188 comm:(bigd) state:S utime:12 stime:2 cutime:21 cstime:4 starttime:321443168 vsize:44949504 rss:6251 wchan:0 blkio_ticks:0 [-1] pid:1846 comm:(bigd) state:S utime:12181 stime:6605 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 [-2] pid:1846 comm:(bigd) state:S utime:12181 stime:6604 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 .Mar 23 22:09:52 JTLSF-DNS-PR-SOA-PREPROD notice tmm[32121]: 01010001:5: pgo_use x86_64 padc TMM Version 12.1.1.0.0.184 startingMar 23 22:09:53 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070404:5: Add a new Publication for publisherID TMM and filterType (nil)Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 01140030:5: HA proc_running tmm is now responding.Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006d:5: Leaving Offline for Active for dbvar not redundant.Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0053:5: Active for traffic group /Common/traffic-group-1.Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0019:5: ActiveHow to solve this problem ? I want to upgrade this device ... but the CLI [INOPERATIVE], so we can do anyting at the CLI for running the script right ? Anyone have any steps for solve this TMM Restart Continuesly ?&nbsp;Thankyouuu ...&nbsp;

ca_valli · Answer

If the unit is covered by a support contract and this is service impacting, your best bet is to open a support case&nbsp;to investigate the issue live.
Most common reason for a unit to behave like this is if you have a VLAN failover enabled, and unit has the monitored links down.I haven't seen this specific evidence in your logs, but you do have a deamon heartbeat failure error. That means a&nbsp;system process is failing to respond&nbsp;to HA daemon heartbeat messages, which triggers the failover.&nbsp;
Again, I'd suggest you have support look into this.&nbsp;

satriaji · Answer

The technical support for this OS device has been out of support (Technical EOS at 2021), so we cannot open TAC to support.

For now the log still appear :

[msi@JTLSF-DNS-PR-SOA-PREPROD:INOPERATIVE:Standalone] ~ # Mar 23 23:51:45 JTLSF-DNS-PR-SOA-PREPROD emerg logger: Re-starting tmm

This device is vCMP Guest, the other guest is normal. Just thist guest that having problem tmm restarting.

duncanjones · Answer

Satriaji&nbsp;wrote:The technical support for this OS device has been out of support (Technical EOS at 2021), so we cannot open TAC to support.For now the log still appear :&nbsp;Tower Defense[msi@JTLSF-DNS-PR-SOA-PREPROD:INOPERATIVE:Standalone] ~ # Mar 23 23:51:45 JTLSF-DNS-PR-SOA-PREPROD emerg logger: Re-starting tmm&nbsp;This device is vCMP Guest, the other guest is normal. Just thist guest that having problem tmm restarting.&nbsp;&nbsp;&nbsp;The Traffic Management Microkernel (tmm) process may restart without a stack trace or core file after becoming disconnected from the Master Control Program Daemon (mcpd) process.This issue occurs when all of the following conditions are met:The Linux kernel memory becomes fragmented.Another memory-intensive operation occurs, which may include:You perform a configuration synchronization (ConfigSync) with full reload.You run a&nbsp;tcpdump&nbsp;session.If&nbsp;tmm&nbsp;fails to start or restart properly as a result of this issue, you can try to recover the system temporarily by restarting the BIG-IP system.

alayne786 · Answer

Any resolution?? We have bigip LTM 14.1 and it's been doing this for 24 hours now, but it's this exact situation. How did you fix it??

alayne786 · Answer

Any resolution?? We have bigip LTM 14.1 and it's been doing this for 24 hours now, but it's this exact situation. How did you fix it?? We do have a vLan failover ..Been waiting for 24 hours from f5 support with no resolution in sight .

Forum Discussion

TMM Restarting (OS Version 12.1.1)

Recent Discussions

Rewriting Location Header in iRule

010713d0:3: Symmetric Unit Key decrypt failure - decrypt failure Rseries Tenant ..

Help with Migrating Netscaler Rewrite Policy to F5 LTM

Need to change HTML tag in HTTP response

Outlook application issue

Related Content

iRule to restart APM session

Big IP version 12 API

BIGdiff Version 2

configuration utility restarting (upgrade)

Creating a tmsh script with iControl REST and using it to restart HTTPD