TMM RESTARTING! (14.1.5)
It's almost identical to this issue below..Opened a case with f5 support but it's now been 25 hours and no resolution! The standby unit keeps going into standby over and over..and gives me messages about tmm restarting and possible issues with mpcd. even setting the unit offline and rebooting isn't stopping this issue. so we're not completely down but still it's worrying.. we do have VLAN failsafe enabled but i've played with the timeouts and dont notice a difference either https://community.f5.com/t5/technical-forum/tmm-restarting-os-version-12-1-1/td-p/312273 mgmt_acld Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus wr_urldbd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070410:5: Removed subscription with subscriber id avrd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id TMM Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus avrd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart restart avrd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus dosl7d Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus asm Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus datasyncd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus admd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus mgmt_acld Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus dwbld Mar 23 22:09:51 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006c:5: proc stat: [0] pid:31188 comm:(bigd) state:S utime:12 stime:2 cutime:21 cstime:4 starttime:321443168 vsize:44949504 rss:6251 wchan:0 blkio_ticks:0 [-1] pid:1846 comm:(bigd) state:S utime:12181 stime:6605 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 [-2] pid:1846 comm:(bigd) state:S utime:12181 stime:6604 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 . Mar 23 22:09:52 JTLSF-DNS-PR-SOA-PREPROD notice tmm[32121]: 01010001:5: pgo_use x86_64 padc TMM Version 12.1.1.0.0.184 starting Mar 23 22:09:53 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070404:5: Add a new Publication for publisherID TMM and filterType (nil) Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 01140030:5: HA proc_running tmm is now responding. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006d:5: Leaving Offline for Active for dbvar not redundant. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0053:5: Active for traffic group /Common/traffic-group-1. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0019:5: Active Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c003e:5: Offline Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD info bigstart: Start bigd in single-process mode Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD info bigd.start[31188]: Execing bigd: /usr/bin/bigd bigd -S Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice promptstatusd[3640]: 01460006:5: semaphore tmm.running(1) held Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus datastor Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dedup_admin Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus tmrouted Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus wamd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart tmrouted Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus websso Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart localdbmgr Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus acctd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus eam Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus rba Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus eca Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice tmrouted[23455]: 01910005:5: Tmrouted exiting after getting termination request. Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id tmrouted Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus nlad Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus vdi Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus urldb Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus datasyncd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus fpuserd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus bdosd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus antserver Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus admd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus asm Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus avrd Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart avrd Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dosl7d Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dwbld Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus mgmt_acld Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus wr_urldbd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070410:5: Removed subscription with subscriber id avrd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id TMM Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus avrd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart restart avrd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus dosl7d Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus asm Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus datasyncd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus admd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus mgmt_acld Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus dwbld Mar 23 22:09:51 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006c:5: proc stat: [0] pid:31188 comm:(bigd) state:S utime:12 stime:2 cutime:21 cstime:4 starttime:321443168 vsize:44949504 rss:6251 wchan:0 blkio_ticks:0 [-1] pid:1846 comm:(bigd) state:S utime:12181 stime:6605 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 [-2] pid:1846 comm:(bigd) state:S utime:12181 stime:6604 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 . Mar 23 22:09:52 JTLSF-DNS-PR-SOA-PREPROD notice tmm[32121]: 01010001:5: pgo_use x86_64 padc TMM Version 12.1.1.0.0.184 starting Mar 23 22:09:53 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070404:5: Add a new Publication for publisherID TMM and filterType (nil) Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 01140030:5: HA proc_running tmm is now responding. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006d:5: Leaving Offline for Active for dbvar not redundant. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0053:5: Active for traffic group /Common/traffic-group-1. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0019:5: Active
TMM Restarting (OS Version 12.1.1)
Hi Everyone, I have a problem with my F5 device. The problem is the tmm service always restarting continuesly, so its make the device INOPERATIVE (intermittent active-standby). Here the logs : "Mar 23 23:12:17 JTLSF-DNS-PR-SOA-PREPROD emerg logger: Re-starting tmm" Mar 23 22:09:29 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 01140045:5: HA reports tmm NOT ready. Mar 23 22:09:29 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0050:5: Sod requests links down. Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD warning sod[5195]: 01140029:4: HA daemon_heartbeat tmm fails action is go offline down links and restart. Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: HalFailover_::set: Cannot clear /dev/ttyS1 DTR/RTS. errno=5 Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: halSetFailover: set error Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0066:5: halSetFailover (clear) fails with status 11. Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0054:5: Offline for traffic group /Common/traffic-group-1. Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c003e:5: Offline Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD err mcpd[6576]: 01070069:3: Subscription not found in mcpd for subscriber Id BIGD_Subscriber. Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006d:5: Leaving Offline for Active for dbvar not redundant. Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0053:5: Active for traffic group /Common/traffic-group-1. Mar 23 22:09:40 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0019:5: Active Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006c:5: proc stat: [0] pid:24126 comm:(tmm.0) state:S utime:1474241 stime:648064 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 [-1] pid:24126 comm:(tmm.0) state:S utime:1474241 stime:648047 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 [-2] pid:24126 comm:(tmm.0) state:R utime:1474241 stime:648029 cutime:0 cstime:0 starttime:276391877 vsize:2618109952 rss:18897 wchan:0 blkio_ticks:0 . Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD warning sod[5195]: 01140029:4: HA proc_running tmm fails action is go offline and down links. Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: HalFailover_::set: Cannot clear /dev/ttyS1 DTR/RTS. errno=5 Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD err sod[5195]: 012a0003:3: halSetFailover: set error Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0066:5: halSetFailover (clear) fails with status 11. Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0054:5: Offline for traffic group /Common/traffic-group-1. Mar 23 22:09:41 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c003e:5: Offline Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD info bigstart: Start bigd in single-process mode Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD info bigd.start[31188]: Execing bigd: /usr/bin/bigd bigd -S Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice promptstatusd[3640]: 01460006:5: semaphore tmm.running(1) held Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus datastor Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dedup_admin Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus tmrouted Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus wamd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart tmrouted Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus websso Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart localdbmgr Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus acctd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus eam Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus rba Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus eca Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice tmrouted[23455]: 01910005:5: Tmrouted exiting after getting termination request. Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id tmrouted Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus nlad Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus vdi Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus urldb Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus datasyncd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus fpuserd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus bdosd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus antserver Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus admd Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus asm Mar 23 22:09:42 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus avrd Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart restart avrd Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dosl7d Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus dwbld Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus mgmt_acld Mar 23 22:09:43 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh ./finish 2079 9 ==> /usr/bin/bigstart singlestatus wr_urldbd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070410:5: Removed subscription with subscriber id avrd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070406:5: Removed publication with publisher id TMM Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus avrd Mar 23 22:09:44 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart restart avrd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus dosl7d Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus asm Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus datasyncd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus admd Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus mgmt_acld Mar 23 22:09:50 JTLSF-DNS-PR-SOA-PREPROD notice logger: /bin/sh /etc/bigstart/scripts/avr.provision ==> /usr/bin/bigstart singlestatus dwbld Mar 23 22:09:51 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006c:5: proc stat: [0] pid:31188 comm:(bigd) state:S utime:12 stime:2 cutime:21 cstime:4 starttime:321443168 vsize:44949504 rss:6251 wchan:0 blkio_ticks:0 [-1] pid:1846 comm:(bigd) state:S utime:12181 stime:6605 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 [-2] pid:1846 comm:(bigd) state:S utime:12181 stime:6604 cutime:15 cstime:2 starttime:276188235 vsize:44949504 rss:6253 wchan:0 blkio_ticks:0 . Mar 23 22:09:52 JTLSF-DNS-PR-SOA-PREPROD notice tmm[32121]: 01010001:5: pgo_use x86_64 padc TMM Version 12.1.1.0.0.184 starting Mar 23 22:09:53 JTLSF-DNS-PR-SOA-PREPROD notice mcpd[6576]: 01070404:5: Add a new Publication for publisherID TMM and filterType (nil) Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 01140030:5: HA proc_running tmm is now responding. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c006d:5: Leaving Offline for Active for dbvar not redundant. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0053:5: Active for traffic group /Common/traffic-group-1. Mar 23 22:09:55 JTLSF-DNS-PR-SOA-PREPROD notice sod[5195]: 010c0019:5: Active How to solve this problem ? I want to upgrade this device ... but the CLI [INOPERATIVE], so we can do anyting at the CLI for running the script right ? Anyone have any steps for solve this TMM Restart Continuesly ? Thankyouuu ...
iRule logging [LB::server pool] crashes tmm on 11.4.1 HF3
F5 support suggested I post on DevCentral to ask if selecting a pool in the Client_Accepted event would mark that pool active (yes) and if making this selection so early in the process I am able to "override" the default pool configured for a virtual server (as far as I can tell, yes). Instead of posting the question to DevCentral I thought I should just write a rule and test with some log messages. I found out I was right, but I also found out the rule I was using to test crashed tmm on 11.4.1 HF3 three times in a row. This did not happen in my DR environment which is on 11.6.0 HF3. Basically, the rule and procs just log which pool is selected multiple times between events and function calls. I suspect there must be some difference in how [LB::server pool] is being used between versions. It seems like I have experienced a ton of odd behavior while working with my LTMs. In the messages below there appears to be a time difference between the log message from the rule and the emerg logger messages. This was not the case in reality. I was monitoring the ltm log for my test messages with tail -f ltm | grep TEST and issuing a request I knew would trigger the rule execution. I saw my first log statement as expected and immediately the emerg logger statements were injected into my console. Has someone else experienced this? Does anyone know what causes this? Do I have to upgrade my production units to 11.6.0 HF3 to fix this? Messages from daemon.log: Mar 19 16:33:52 f5cm2 emerg logger: Re-starting tmm Mar 19 16:33:52 f5cm2 emerg logger: Re-starting tmm1 Mar 19 16:37:24 f5cm2 emerg logger: Re-starting tmm Mar 19 16:37:24 f5cm2 emerg logger: Re-starting tmm1 Mar 19 16:38:54 f5cm2 emerg logger: Re-starting tmm Mar 19 16:38:54 f5cm2 emerg logger: Re-starting tmm1 ` Messages from ltm log: `Mar 19 16:33:20 f5cm2 debug tmm1[8361]: Rule /Common/test : TEST: start pool in rule CLIENT_ACCEPT /Common/expected_pool_name Mar 19 16:33:20 f5cm2 notice sod[5545]: 01140045:5: HA reports tmm NOT ready. Mar 19 16:33:20 f5cm2 notice sod[5545]: 010c0050:5: Sod requests links down. Mar 19 16:33:20 f5cm2 info lacpd[5552]: 01160016:6: Failover event detected. (Switchboard failsafe disabled while offline) Mar 19 16:33:20 f5cm2 err bcm56xxd[6270]: 012c0010:3: Failover event detected. Marking external interfaces down. bsx.c(3724) ...snip... Mar 19 16:36:54 f5cm2 debug tmm[15613]: Rule /Common/test : TEST: start pool in rule CLIENT_ACCEPT /Common/expected_pool_name Mar 19 16:36:54 f5cm2 notice sod[5545]: 01140045:5: HA reports tmm NOT ready. Mar 19 16:36:54 f5cm2 notice sod[5545]: 010c0050:5: Sod requests links down. Mar 19 16:36:54 f5cm2 info lacpd[5552]: 01160016:6: Failover event detected. (Switchboard failsafe disabled while offline) Mar 19 16:36:55 f5cm2 err bcm56xxd[6270]: 012c0010:3: Failover event detected. Marking external interfaces down. bsx.c(3724) ...snip... Mar 19 16:38:24 f5cm2 debug tmm1[16328]: Rule /Common/test : TEST: start pool in rule CLIENT_ACCEPT /Common/expected_pool_name Mar 19 16:38:24 f5cm2 notice sod[5545]: 01140045:5: HA reports tmm NOT ready. Mar 19 16:38:24 f5cm2 notice sod[5545]: 010c0050:5: Sod requests links down. Mar 19 16:38:24 f5cm2 err bcm56xxd[6270]: 012c0010:3: Failover event detected. Marking external interfaces down. bsx.c(3724) ` test2 rule: when CLIENT_ACCEPTED { log local0.debug "TEST: start pool in rule CLIENT_ACCEPT [LB::server pool]" call test::client_accepted log local0.debug "TEST: pool in CLIENT_ACCEPT after function [LB::server pool]" } when HTTP_REQUEST { log local0.debug "TEST: start pool in rule HTTP_REQUEST [LB::server pool]" call test::http_request log local0.debug "TEST: pool in HTTP_REQUEST after function [LB::server pool]" } test rule proc client_accepted { log local0.debug "TEST: pool coming into client_accept [LB::server pool]" set var 1 switch $var { "0" { log local0.debug "TEST: var was set to $var." } "1" { log local0.debug "TEST: var was set to $var." pool pool_name log local0.debug "TEST: pool selected in switch on var of client_accept [LB::server pool]" } } log local0.debug "TEST: pool selected in client_accept [LB::server pool]" } proc http_request { log local0.debug "TEST: pool coming into http_request [LB::server pool]" pool pool2_name log local0.debug "TEST: pool selected in http_request [LB::server pool]" }
Cannot add F5 net route
Hello! I am scratching my head because I can't see why this static route is not being added to the config. The "x" is replaced with a usable 1-254 IP. Existing config: net self INT-350_NET1_PRIMARY_IP { address x.159.222.101/24 allow-service { default } traffic-group traffic-group-local-only vlan INT-350 } net self INT-350_NET1_FLOATING_IP { address x.159.222.103/24 allow-service { default } floating enabled traffic-group traffic-group-1 unit 1 vlan INT-350 } net vlan INT-350 { if-index 448 interfaces { n7k-Po16 { tag-mode service tagged } } tag 350 } root@(sfltm1)(cfg-sync Standalone)(Active)(/Common)(tmos) load sys config merge from-terminal Enter configuration. Press CTRL-D to submit or CTRL-C to cancel. net route INT-350_NET1_ROUTE { gw x.159.222.254 network x.159.222.0/24 } Loading configuration... 01070666:3: Static route duplicates Self IP x.159.222.0 / 255.255.255.0 implied route Unexpected Error: Loading configuration process failed.
IRULE TO REMOVE LOGS FROM FORWARD PROXY IRULE
What irule can be used to remove logs from an irule. The situation is that, the irule applied to a virtual server is filling up the /var/log folder and tmm is rebooting. example, if { $static::enable_logging_L4_VIP_GPRS_TRANSPARENT } { set logging_handle [HSL::open -proto UDP -pool ${static::log_destination_L4_VIP_GPRS_TRANSPARENT} ] i WANT ALL logs removed or tmm not to activate themIRULE TO REMOVE LOGS FROM FORWARD PROXY IRULE
What irule can be used to remove logs from an irule. The situation is that, the irule applied to a virtual server is filling up the /var/log folder and tmm is rebooting. example, if { $static::enable_logging_L4_VIP_GPRS_TRANSPARENT } { set logging_handle [HSL::open -proto UDP -pool ${static::log_destination_L4_VIP_GPRS_TRANSPARENT} ] i WANT ALL logs removed or tmm not to activate them
Have you ever run out of file descriptors while updating an AS3 declaration?
Running AS3 version 3.26.0 on Virtual Edition 15.1.5. Upon PATCHing the declaration I'm persistently getting nonsensical 422s for objects that can't be found. Upon checking the restnode log (/var/log/restnoded/restnoded.log), I see errors which indicate an exhaustion of file descriptors which appear to be limited to 4096 for the user restnoded, which the service runs as. Is there any supported way to work around this? Here are the aforementioned logs: Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-cert/~Common~default.crt failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"} Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-key/~Common~default.key failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"} Wed, 08 Jun 2022 18:37:31 GMT - warning: [appsvcs] {"message":"unable to digest declaration. Error: Unable to find /Common/default.crt for /redirect_w3_test_no_users/Shared/defaultCert/certificate","level":"warning"} Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"DELETE http://admin:XXXXXX@localhost:8100/mgmt/tm/ltm/data-group/internal/~Common~appsvcs~____appsvcs_lock Attempting to release global lock failed (getaddrinfo EBUSY localhost:8100)","level":"error"} Wed, 08 Jun 2022 18:37:31 GMT - severe: [appsvcs] {"message":"GET http://localhost:8100/mgmt/tm/sys/folder/~Common~appsvcs failed (getaddrinfo EBUSY localhost:8100)","level":"error"} Wed, 08 Jun 2022 18:37:32 GMT - severe: [appsvcs] {"message":"An error occured while deleting stored declaration: Error: spawn /bin/sh EMFILE","level":"error"} Wed, 08 Jun 2022 18:37:34 GMT - finest: socket 201 opened Wed, 08 Jun 2022 18:37:39 GMT - finest: socket 201 closed Wed, 08 Jun 2022 18:40:21 GMT - finest: socket 202 opened Wed, 08 Jun 2022 18:40:30 GMT - finest: socket 202 closed Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-cert/~Common~default.crt failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"} Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://admin:XXXXXX@localhost:8100/mgmt/tm/sys/file/ssl-key/~Common~default.key failed (connect EMFILE 127.0.0.1:8100 - Local (undefined:undefined))","level":"error"} Wed, 08 Jun 2022 18:40:37 GMT - warning: [appsvcs] {"message":"unable to digest declaration. Error: Unable to find /Common/default.crt for /redirect_w3_test_no_users/Shared/defaultCert/certificate","level":"warning"} Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"DELETE http://admin:XXXXXX@localhost:8100/mgmt/tm/ltm/data-group/internal/~Common~appsvcs~____appsvcs_lock Attempting to release global lock failed (getaddrinfo EBUSY localhost:8100)","level":"error"} Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"GET http://localhost:8100/mgmt/tm/sys/folder/~Common~appsvcs failed (getaddrinfo EBUSY localhost:8100)","level":"error"} Wed, 08 Jun 2022 18:40:37 GMT - severe: [appsvcs] {"message":"An error occured while deleting stored declaration: Error: spawn /bin/sh EMFILE","level":"error"}Are NTP and DNS traffic management type or not?
Hello everyone, I'm system engineer in integrator company and currently I have one PoC of AWAF project with a customer. I have little experience of working with f5 devices, so I have one question and it'll help me a lot in future to analyze how BIG-IP devices. I've done some research in documentations but I couldn't find clear answer on topics, which type of traffic is considered as Data Traffic and which one is Management? For example NTP and DNS traffic should use management route or TMM route (I mean the case when there is no direct path to the destination DNS/NTP servers)? I thought that BIG-IP devices will use management route (management gateway) to do DNS queries and time synchronization, so I asked customer to grant access on firewall from management interface to the destination servers, but it didn't work. Then I've captured traffic via tcpdump and I realized that BIG-IP devices try to use TMM default route instead. But I've read in this article - https://support.f5.com/csp/article/K13284 that NTP is management traffic. Also this article - https://support.f5.com/csp/article/K7017 says that during the device boot, ntpd daemon is starting before TMM, so if it has no route via management interface, time synchronization will fail. So, I'm a little confused, what should I ask customer, open access from TMM interface for DNS, NTP, also for Signature Updates? I just do not understand logically, why NTP, DNS and system update do not use management routes? If all of them are considered as a data traffic, than what is management route used for? Only for accessing management GUI and SSH, is that correct? Sorry for a long question, but I really want to understand the platform's logic of traffic routing, to be able to operate it and correctly implement it with the customer. Thanks in advance. // Giorgi
TMM memory leaking due to log publisher configuration
Hi dears I use F5 CGNAT module and I define a log publisher on F5 for remote logging. I have memory leaking problem: tmm memory increase during the time. F5 version: 14.1.2.3 picture1: https://ibb.co/YDN88Bz After analyzing "show sys memory" output we notice that memory leaking is because of Log profile: picture2: https://ibb.co/4m3snGt For troubleshooting I look at logs in tmm using "# cat /var/log/tmm" and notice that there are a lot of logs telling here is a problem with publisher: # cat /var/log/tmm <13> Jul 7 03:51:24 slot1/f5-one notice alg_hs_log_alg_event/1190: errdefs publisher log failure <13> Jul 7 03:51:24 slot1/f5-one notice alg_hs_log_alg_event/1190: errdefs publisher log failure <13> Jul 7 03:51:24 slot1/f5-one notice alg_hs_log_alg_event/1190: errdefs publisher log failure . . Now the question is: How can I analyze these logs and find solution for tmm memory leaking? Any one knows "logger" tool ? or Someone has encountered this problem before? We have this problem for a long time. And it has not been resolved by updating the version. picture3: https://ibb.co/R9dYLdM picture4: https://ibb.co/nfYJvd0 Sys::Provision ModuleCPU (%)Memory (MB)Host-Memory (MB)Disk (MB) --------------------------------------------------------- afm0000 am0000 apm0000 asm0000 avr118887687800 dos0000 fps0000 gtm0000 host10200300527856 ilx10121020 lc0000 ltm1000 pem0000 sslo0000 swg0000 tmos87424787000 urldb0000 vcmp0000 thanks.