I’m really hoping someone can help me. Last Friday we had a major problem which affected access to all our Core Systems. The initial problem was caused due to a bug within the Cisco Nexus IOS which caused loopguard to block the vlans on a port-channel and then unblock them.
The 3 vlans used by the F5 (real, virtual servers and heartbeat) between our two LTM’s became blocked for a few microseconds.
2010 Nov 19 14:31:43 GR_Core2 %STP-2-LOOPGUARD_BLOCK: Loop guard blocking port port-channel1 on VLAN0205. 2010 Nov 19 14:31:43 GR_Core2 %STP-2-LOOPGUARD_UNBLOCK: Loop guard unblocking port port-channel1 on VLAN00205
We have two LTM’s, in active (data centre1) - standby (data centre2).
When we came to investigate why users couldn’t access the systems it was because the servers couldn’t reach their default gateway which is a floating ip on the F5 LTM. To solve the problem I pressed update on the F5 self ip used as the DG. Suddenly the servers could reach their DG and access to systems was restored. I’m interested to know what this would have done. I suspect it sent out a gratuitous arp?
Having checked the logs the Standby LTM became Active. The LTM also reported address conflicts for some of the IP’s which are used for the Virtual Servers.
Any help to determine the cause will be very much appreciated as we are new to the F5 world so troubleshooting is difficult as we are used to Cisco products. our support company isn’t being very helpful.
One thing I have noticed as that we are not using MAC masquerade.
Yeah. That sounds like the gratuitous arp was missed. Not sure you’ll be able to gather a lot of data post problem. If it happens again, you could try a few things like flushing the arp table of one of the servers that are on the direct attached vlans and see if that brings a few things alive.
I’d suggest two things. Use vlan failsafe and a secondary (or primary and use the infra as secondary) heartbeat link that doesn’t depend on your network infrastructure
As Hamish says, a heartbeat link is a great idea. As you’ve already said, MAC Masquerade would be wise too. I hope to see MAC Masquerade somehow be implemented by default…and maybe even seeing a “heartbeat link” as part of the initial config setup.
FWIW I’m not a great fan of MAC masquerading… Mainly because of issues we used to have with cisco switches and them getting a bit confused when MAC addresses used to move from one part of the network to another suddenly (It was always easier to fixup ARP entries than try to force updates of where a MAC had moved to).
Thanks for the replies. The High Availability - Redundancy - Link Down time on Failover is set to 0.1 seconds. Is that basically saying if the Standby LTM receives nothing from Active LTM within 0.1 seconds then it will be come Active. If so this seems very low?
That option is disabled by default… but yes that’s what it’s saying… In my experience network failover was more trouble than it was worth… but you are in an unique setup where it seems your backup unit is in another DC… I do not believe that is an ideal HA pair scenario… It seems F5s goal is to backup a unit apples to apples usually within the same DC… but I don’t see why they couldn’t be in different DCs and utilize network failover…
A couple questins come to mind… Are these units on the same subnet? I’d think they would have to be if they were part of an HA pair…
LTMs aside, are your Data centers Active Active?
Did a Sales Engineer from F5 help you design your setup?
Without knowning anything of your envirnoment… If you have active gear in both Datacenters I would look to purchase another HA pair for your other DC and possibly look at some GTMs to help balance between Data Centers..
We have two pairs of LTM’s, each pair pair has different functions. The pair of LTM’s I’m refering to were bought to replace a pair of Cisco CSS load balancers which ran active - active. These sit on our internal LAN and are used to load a balance a specific set of core applications. The core applications reside on servers at both data centres. The VLAN the servers and F5 LTM sit in spans across each data centre. so yes the units are in the same subnet. The F5 consultant who designed and installed them recommended we didn’t use an active - active set up.
One thing I noticed was the standby LTM at the other data centre is reporting it is going active and has an alarm. My concern is that the inter site links can become quite heavily utilized so this could affect the heartbeats between each LTM.
We also have another pair of LTM’s in our external DMZ in active standby which are used to load balance requests across our web servers. These link to two GTM’s (active - active) which serve dns requests to our websites. The issue with this we’ve been asked if we can use the LTM’s to load balance and monitor requests from the web servers to the servers within the internal dmz. This is proving difficult with active - standby. It seems we need another two LTM’s in this tier too.
I think I’ll get our support company to assess our set-up.
I always configure network failover. Even when you have a direct cable (Which is only good for a few metres). Belt & Braces… After all that’s why you have 2 boxes. Because things can fail (And going active/active just because you lose one cable really annoys me).
I’ve used active/active in the past… It works. Not a problem. The reason most ‘consultants’ seem to recommend against it is because they’re scared that you’ll load each system > 50% and wind up with >100% trying to be processed on one box… Apart from that, I have never found anything to be worried about on that front.
Dazzla, not sure if you’re suggesting that you want to load balance internal and DMZ traffic at the same time. I’d always recommend separate devices for this (I dislike effectively bridging firewalls with anything… Including where it would be ‘accidental’ and not by design).
We use a dedicated vlan for the network failover. During the night when the backup runs I’m seeing the inter-site links between the data centres hit 90% at times. I would have thought this would impact all vlans. I need to look in to policing this backup traffic.
I totally agree that spanning-vlans across both data centres is not good practice.
The problem is this this network requires layer 2 across each data centre for v-motion, css boxes, f5 boxes, MS cluster,
In terms of load balancing for the websites we host the traffic comes in via the public ip, is translated to the F5 virtual server and then load balanced across the web servers in our external dmz.
I was asked can we utilize the F5 devices to monitor and load balance requests sent from the web servers in our external dmz to the application servers in our internal dmz. Basically if a server or service fails within the the internal dmz we want the F5 to mark this as down. Currently we have to implement manual processes if a server or service fails.
The external and internal dmz’s are on seperate networks so have to be routed via 2 sets of firewalls for them to communicate with each. No bridging is taking place. I am using source NAT to get around the routing issue as the external dmz servers do not have a route back to the internal dmz application servers via the F5. The firewall rules permit the f5 to the internal dmz servers. This is working perfectly well with the active LTM. The problem is with the standby LTM which sits in our other data centre. This is because its default gateway points to DC1 firewall (same as active LTM) but the routes back to it from the internal dmz point to DC2 so we have assymetric routing. My plan is to NAT the traffic of the standby LTM to an address from DC1 to get around this. My first plan was to have a different default route on the standby ltm. I was told I can’t do this as the routes on the LTM’s are sync’ed between each other.
Not sure anyone will have a clue what’s going on without a diagranm. I will try to upload one when I return to work.
One of the worst outages I’ve ever seen had to do with a network heartbeat VLAN between two DCs that was severed. The BigIPs were split, as were 10 or so other devices that rely on GARP for failover. They all did what they were configured to do: they went active and all started sending GARPs out to every VLAN. What’s worse is that only the failover VLAN was taken down and all of the other vlans were still connected. The arp storm was quite a spectacle to behold!
My personal angle is that network FO is fine, but you need to know the ramifications very well and think out all of the failure scenarios and their behaviors. That way, you’ll at least know what to expect outage wise and it’ll shorten your recovery times substantially. When it bites, layer 2 is merciless…
All very true… When configuring network failover I make it mandatory to have a NON core network path between the two boxes (And avoid 9.4 because it’ll only use ONE path for network failover traffic. WIth 10.x you can configure multiple connections).
Non core network means at it’s simplest an optical SFP in each F5 and a piece of dark-fibre. Or you can uses a pair of dedicated heartbeat switches with dark fibre between them etc.
What’s worse is that only the failover VLAN was taken down and all of the other vlans were still connected. The arp storm was quite a spectacle to behold!
Then configuring network failover I make it mandatory to have a NON core network path between the two boxes (And avoid 9.4 because it’ll only use ONE path for network failover traffic. WIth 10.x you can configure multiple connections).
Absolutely, highly important to have a redundant path for network failover traffic! That’s why the multiple heartbeat paths feature was added (it actually started in v9.6.1)