Forum Discussion

Luca_55898

Nimbostratus

Jul 11, 2011

Failover event detected.... not sure why

Can anyone shed some light on the following logs:

We have two LTMs in a HA config. Not sure if the devices are actually failing over or not....

Mon Jul 11 11:48:24 EST 2011 notice local/LB003 sod[5151] 01140045 HA reports tmm NOT ready.

Mon Jul 11 11:48:24 EST 2011 notice local/LB003 sod[5151] 010c0050 Sod requests links down.

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160016 Failover event detected. (Switchboard failsafe disabled while offline)

Mon Jul 11 11:48:25 EST 2011 err local/LB003 bcm56xxd[4993] 012c0010 Failover event detected. Marking external interfaces down. bsx.c(3276)

Mon Jul 11 11:48:25 EST 2011 info local/LB003 bcm56xxd[4993] 012c0015 Link: 1.15 is DOWN

Mon Jul 11 11:48:25 EST 2011 info local/LB003 bcm56xxd[4993] 012c0015 Link: 1.16 is DOWN

Mon Jul 11 11:48:25 EST 2011 info local/LB003 bcm56xxd[4993] 012c0015 Link: 3.1 is DOWN

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160016 Interface 1.15, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160010 Link 1.15 removed from aggregation

Mon Jul 11 11:48:25 EST 2011 info local/LB003 bcm56xxd[4993] 012c0015 Link: 3.2 is DOWN

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160016 Interface 1.16, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160010 Link 1.16 removed from aggregation

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160016 Interface 3.1, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160010 Link 3.1 removed from aggregation

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160016 Interface 3.2, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down

Mon Jul 11 11:48:25 EST 2011 info local/LB003 lacpd[5133] 01160010 Link 3.2 removed from aggregation

Mon Jul 11 11:48:35 EST 2011 notice local/LB003 sod[5151] 01140029 HA daemon_heartbeat tmm5 fails action is go offline down links and restart.

Mon Jul 11 11:48:35 EST 2011 notice local/LB003 sod[5151] 010c003e Offline

Mon Jul 11 11:48:35 EST 2011 notice local/LB003 sod[5151] 01140044 HA reports tmm ready.

Mon Jul 11 11:48:35 EST 2011 notice local/LB003 sod[5151] 010c0018 Standby

Mon Jul 11 11:48:36 EST 2011 info local/LB003 lacpd[5133] 01160016 Connected to failover service.

Mon Jul 11 11:48:36 EST 2011 info local/LB003 bcm56xxd[4993] 012c0012 Connected to failover service.

Mon Jul 11 11:48:36 EST 2011 notice local/LB003 sod[5151] 010c0048 Bcm56xxd and lacpd connected - links up.

Mon Jul 11 11:48:38 EST 2011 info local/LB003 bcm56xxd[4993] 012c0015 Link: 3.1 is UP

Mon Jul 11 11:48:38 EST 2011 info local/LB003 bcm56xxd[4993] 012c0015 Link: 3.2 is UP

Mon Jul 11 11:48:39 EST 2011 info local/LB003 bcm56xxd[4993] 012c0015 Link: 1.15 is UP

Mon Jul 11 11:48:39 EST 2011 info local/LB003 bcm56xxd[4993] 012c0015 Link: 1.16 is UP

Mon Jul 11 11:48:40 EST 2011 info local/LB003 lacpd[5133] 01160009 Link 1.15 added to aggregation

Mon Jul 11 11:48:40 EST 2011 info local/LB003 lacpd[5133] 01160009 Link 1.16 added to aggregation

Mon Jul 11 11:48:40 EST 2011 info local/LB003 lacpd[5133] 01160009 Link 3.1 added to aggregation

Mon Jul 11 11:48:40 EST 2011 info local/LB003 lacpd[5133] 01160009 Link 3.2 added to aggregation

config

design

Eduardo_de_Oliv
Nimbostratus
Aug 27, 2015
I oppened a case at F5 and they said that the problem was as bellow:

"The cause of the crash was because TMM encountered a segmentation fault after parsing a malformed URL."

This is fixed in HF5.

So I upgrade my system's e waiting for knows if it really solved!
AdrienR_219328
Nimbostratus
Aug 27, 2015
That's a good lead, still I daresay it's NOT fixed in HF5 since I run this very one. Do you have the slightest idea what your "malformed URL" might have been? Something on the possibly customized portal? I'm currently looking in the logs for "url" or "parse".

Thanks for your news, they're of great help! Best regards,
AdrienR_219328
Nimbostratus
Sep 01, 2015
Hello Eduardo,

Any update for you? Only option to somewhat "stabilize" the failovers was to "bigstart restart sod", sod being the process hertbeat responsible for failover. But this morning, I had 7 failovers already, and I can't find any root cause. I'm considering rolling back to 11.6HF2 since my issues came after upgrading to 11.6HF5.

Please let me know if you have anything at all. Regards,

Adrien Restaut
Eduardo_de_Oliv
Nimbostratus
Sep 01, 2015
For me is working, i had no problem anymore since we put HF5. I think better you open a support case with F5 and they can look into this. I read a lot of probably causes for it happens and concluded that it is a bug.! the unique thing I know is, this problem started (with me) after using ASM... but if you don't uses ASM i really have no idea.
- Ndlovumm
  Cirrus
  Jan 14, 2022
  Had the same issue today while busy with ASM
AdrienR_219328
Nimbostratus
Sep 01, 2015
Good news for you. I'm running 11.6HF5 with the issue :-\ What is ASM, by the way? Regards, Adrien
- Eduardo_de_Oliv
  Nimbostratus
  Sep 01, 2015
  Application Security Feature
AdrienR_219328
Nimbostratus
Sep 02, 2015
Thanks Eduardo. Yesterday was just nightmare: failovers literally every 2 minutes, impossible for remote users to work. This early morning we rolled back to 11.6HF2. So far, no failovers. BIG ISSUE HERE! Anyway, glad we don't have the problem anymore, so far. Regards.
Balancing_Harmo
Nimbostratus
Jan 23, 2016
Hello, I know it's a bit late but have you tried this SOL article? https://support.f5.com/kb/en-us/solutions/public/17000/100/sol17155.html
Eduardo_de_Oliv
Nimbostratus
Jan 25, 2016
I oppened a case at F5 and they said that the problem was as bellow:

"The cause of the crash was because TMM encountered a segmentation fault after parsing a malformed URL."

This is fixed in HF5.

So I upgrade my system's it really solved!
Austin_Geraci
MVP
Jan 25, 2016
I can concur - I've seen HF5 resolve this issue as well in the field.
Hajar
Nimbostratus
Aug 30, 2016
Hello, I have 11.6.0 Hotfix 6 and today I had same problem of failing over with the same error. Does anyone know the cause of this failover issue? It happened few months ago also. I opened a case then and I was told it is probably a networking issue which I am sure it is not. I opened a new case today, waiting for an answer. Thanks
- Ford_Prefect
  Cirrus
  Nov 13, 2020
  Hello,
  I'm confused, but we had a failover on 15.1.0.4 today when F5 switched twice and the symptoms are very simillar:
  HA reports tmm NOT ready. Sod requests links down. Failover event detected. (Switchboard failsafe disabled while offline) Bringing down interfaces on links-up change
  Can you please, try to remember the support team answer? ASM is running on BIG-IP