Forum Discussion
Dazzla_20011
Nimbostratus
NimbostratusHelp Major network outage involving F5 LTM
Hi,
I'm really hoping someone can help me. Last Friday we had a major problem which affected access to all our Core Systems. The initial problem was caused due to a bug within the Cisco Nexus IOS which caused loopguard to block the vlans on a port-channel and then unblock them.
The 3 vlans used by the F5 (real, virtual servers and heartbeat) between our two LTM's became blocked for a few microseconds.
2010 Nov 19 14:31:43 GR_Core2 %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel1 on VLAN0205. 2010 Nov 19 14:31:43 GR_Core2 %STP-2-LOOPGUARD_UNBLOCK: Loop guard unblocking port port-channel1 on VLAN00205
We have two LTM's, in active (data centre1) - standby (data centre2).
When we came to investigate why users couldn't access the systems it was because the servers couldn't reach their default gateway which is a floating ip on the F5 LTM. To solve the problem I pressed update on the F5 self ip used as the DG. Suddenly the servers could reach their DG and access to systems was restored. I'm interested to know what this would have done. I suspect it sent out a gratuitous arp?
Having checked the logs the Standby LTM became Active. The LTM also reported address conflicts for some of the IP's which are used for the Virtual Servers.
Any help to determine the cause will be very much appreciated as we are new to the F5 world so troubleshooting is difficult as we are used to Cisco products. our support company isn't being very helpful.
One thing I have noticed as that we are not using MAC masquerade.
Many Thanks
Darren
nitassEmployee
i suggest u opening a support case. user is able to open a case with f5 support directly - no need to go through partner.
btw, is there any relevant message before address conflicting?? i meant in /var/log/ltm. it may have something there.
HamishCirrocumulus
Yeah. That sounds like the gratuitous arp was missed. Not sure you'll be able to gather a lot of data post problem. If it happens again, you could try a few things like flushing the arp table of one of the servers that are on the direct attached vlans and see if that brings a few things alive.
I'd suggest two things. Use vlan failsafe and a secondary (or primary and use the infra as secondary) heartbeat link that doesn't depend on your network infrastructure- Chris_Miller
Altostratus
I used to run into the fairly often.
As Hamish says, a heartbeat link is a great idea. As you've already said, MAC Masquerade would be wise too. I hope to see MAC Masquerade somehow be implemented by default...and maybe even seeing a "heartbeat link" as part of the initial config setup. - HamishFWIW I'm not a great fan of MAC masquerading... Mainly because of issues we used to have with cisco switches and them getting a bit confused when MAC addresses used to move from one part of the network to another suddenly (It was always easier to fixup ARP entries than try to force updates of where a MAC had moved to).
- HamishOh... The cisco issues where when running with multiple switches and a quite large network... Single switches worked fine...
- HamishOh... The cisco issues where when running with multiple switches and a quite large network... Single switches worked fine...
Dazzla_20011Thanks for the replies. The High Availability - Redundancy - Link Down time on Failover is set to 0.1 seconds. Is that basically saying if the Standby LTM receives nothing from Active LTM within 0.1 seconds then it will be come Active. If so this seems very low?- That option is disabled by default... but yes that's what it's saying... In my experience network failover was more trouble than it was worth... but you are in an unique setup where it seems your backup unit is in another DC... I do not believe that is an ideal HA pair scenario... It seems F5s goal is to backup a unit apples to apples usually within the same DC... but I don't see why they couldn't be in different DCs and utilize network failover...
A couple questins come to mind... Are these units on the same subnet? I'd think they would have to be if they were part of an HA pair....
LTMs aside, are your Data centers Active Active?
Did a Sales Engineer from F5 help you design your setup?
Without knowning anything of your envirnoment... If you have active gear in both Datacenters I would look to purchase another HA pair for your other DC and possibly look at some GTMs to help balance between Data Centers.. - Dazzla_20011We have two pairs of LTM's, each pair pair has different functions. The pair of LTM's I'm refering to were bought to replace a pair of Cisco CSS load balancers which ran active - active. These sit on our internal LAN and are used to load a balance a specific set of core applications. The core applications reside on servers at both data centres. The VLAN the servers and F5 LTM sit in spans across each data centre. so yes the units are in the same subnet. The F5 consultant who designed and installed them recommended we didn't use an active - active set up.
One thing I noticed was the standby LTM at the other data centre is reporting it is going active and has an alarm. My concern is that the inter site links can become quite heavily utilized so this could affect the heartbeats between each LTM.
We also have another pair of LTM's in our external DMZ in active standby which are used to load balance requests across our web servers. These link to two GTM's (active - active) which serve dns requests to our websites. The issue with this we've been asked if we can use the LTM's to load balance and monitor requests from the web servers to the servers within the internal dmz. This is proving difficult with active - standby. It seems we need another two LTM's in this tier too.
I think I'll get our support company to assess our set-up. - HamishMy turn.
I always configure network failover. Even when you have a direct cable (Which is only good for a few metres). Belt & Braces... After all that's why you have 2 boxes. Because things can fail (And going active/active just because you lose one cable really annoys me).
I've used active/active in the past... It works. Not a problem. The reason most 'consultants' seem to recommend against it is because they're scared that you'll load each system > 50% and wind up with >100% trying to be processed on one box... Apart from that, I have never found anything to be worried about on that front.
Dazzla, not sure if you're suggesting that you want to load balance internal and DMZ traffic at the same time. I'd always recommend separate devices for this (I dislike effectively bridging firewalls with anything... Including where it would be 'accidental' and not by design).
H
