Cisco ACI Endpoint Learning with a BIG-IP HA Failover

Customer engagement always presents a great learning opportunity.  In this blog, I am sharing some lessons learned to guide the design and troubleshooting of the BIG-IP HA and failover with Cisco Application Centric Infrastructure (ACI).

Enterprises are deploying Cisco ACI and F5 BIG-IP to evolve their modern data center, to deliver mission-critical applications. Oftentimes, a high level of redundancy is built into such data center architectures, to prevent unplanned outages. Unfortunately, failures can sometimes happen. When it did happen, we observed some different patterns, and one complaint was having some Layer2 learning issues between ACI and BIG-IP, resulting in F5 BIG-IP failovers not working. Are the ARPs properly handled? Is the ARP or MAC table updated correctly with BIG-IP failover? What’s the best practice to configure BIG-IP High-Availability (HA)? As always, the correct answer is “it depends”.

To start off, let's take a look at how endpoints in the ACI fabric are leanred, and this understanding can greatly simplify the troubleshooting process.

Cisco ACI endpoint learning behavior

The Cisco ACI uses endpoints to forward traffic. An endpoint consists of one MAC address and zero or more IP addresses. Each endpoint represents a single networking device connected to an ACI leaf switch interface or a port-channel. These endpoints are associated with how Cisco leaf switches forwards traffic to these network ports.

In a traditional network, three tables are used to maintain the network addresses of external devices: a MAC address table for Layer 2 forwarding, a Routing Information Base (RIB) for Layer 3 forwarding, and an ARP table for the combination of IP addresses and MAC addresses. Cisco ACI, however, maintains this information in a different way. Cisco ACI replaced the MAC address table and ARP table with a single table called the endpoint table. This change implies that Cisco ACI learns that information in a different way than in a traditional network. Cisco ACI learns MAC and IP addresses in hardware by looking at the packet source MAC address and source IP address in the data plane instead of relying on ARP to obtain a next-hop MAC address for IP addresses. This approach reduces the number of resources needed to process and generate ARP traffic. It also allows detection of IP address and MAC address movement without the need to wait for GARP (Gratuitous Address Resolution Protocol) if some traffic is sent from the new host.

Endpoint movement with a BIG-IP Failover

When BIG-IP failover takes place, the newly active BIG-IP sends GARP for floating self-IPs and VIPs. This is done so that endpoints and network devices in the same broadcast domain can update the ARP table and MAC address table.

The ACI fabric has “Move Frequency (per second)” configuration in “Endpoint Retention Policy” that is referred from bridge domains to limit the maximum number of endpoint moves allowed per second in the bridge domain. The number is counted as total movements of any endpoint in the given bridge domain, whether it is a single endpoint flap, a simultaneous move of multiple endpoints, or a combination of both. If the number of movements per second is exceeded, the “Move Frequency” (256 by default) and the “Hold interval” (300 seconds by default) will trigger, and the learning new endpoint in the bridge domain is disabled until the “Hold Interval” expires. This feature is called BD Move Frequency or Endpoint Move Dampening.

If there are many IP addresses in a bridge domain that are expected to move at the same time, for example BIG-IP owns many IPs in a given bridge domain, you might need to increase the “Move Frequency” to prevent endpoint learning from being disabled in the bridge domain. The APIC configuration location for “End Point Retention Policy” is at Tenant > Policies > Protocol > End Point Retention, which is referred from bridge domains.

The other option to prevent endpoint learning from being disabled in the bridge domain is to enable “Rogue EP Control”. If the Rogue EP Control is enabled, Endpoint Move Dampening via Endpoint Retention Policy explained above will not take effect. The APIC configuration location for “Rogue EP Control” is at System > System Settings > Endpoint Controls > Rogue EP Control. This configuration is a fabric-wide setting and is disabled by default.

MAC masquerade

MAC masquerading is a feature that allows you to manually allocate a MAC address to a traffic group across a BIG-IP pair configured for high availability. More specifically, this MAC address floats between the devices in an HA pair, along with the floating self-IPs and virtual addresses within the same traffic group.

We highly recommend using MAC masquerade under the following conditions:

• To improve reliability and failover speed in lossy networks by minimizing Address Resolution Protocol (ARP) table updates on servers and network devices that are in the same broadcast domain with BIG-IP.
• When using Policy-Based Redirect (PBR) on Cisco ACI

For more information and configura tion, refer to SOL13502: Configuring MAC masquerade (11.x)

 Failover Case Considerations

In this section, we will highlight some of the failover cases experienced by our customers. Although they are not always common scenarios, if you perform BIG-IP failover, please pay attention to these behaviors.

Scenario 1:   a race condition between the GARPs sent from the new active and the new standby still sending out packets belonging to the same IP address

One scenario possible after BIG-IP failover takes place is that the new standby BIG-IP still sends traffic using the floating self-IPs and VIPs as source IP addresses for existing connections. This will result in the ACI fabric learning the IPs from multiple locations via the data plane.

Recommended action:

This issue can be avoided by disabling IP Data-plane Learning, if you have a possibility that the ACI fabric receives traffics with the same-source IP address from different locations.

Alternatively, you can configure MAC masquerading on your BIG-IP HA pair, which causes BIG-IP to change MAC addresses upon fail-over. This ensures the active unit retains the same MAC address upon a failover, and moves any problems associated with IP/MAC address changes to the standby unit.

Scenario 2:  GARPs may be lost after a BIG-IP failover event

If a large number of IP addresses are configured for the BIG-IP, the GARPs may be sent out too quickly for in the same broadcast domain to process. This may result in stale ARP entries on the device.

Devices limiting the rate of ARP packets or network loss can cause some devices to miss the GARP packets, resulting in a situation in which a virtual server or floating self IP address on an active BIG-IP becomes unreachable due to network devices or hosts sending packets to the wrong device or to the standby BIG-IP.

Recommended action:

If this is a recurring issue due to ARP rate-limiting on a receiving upstream network device or host, consider modifying the You can modify the BIG-IP gratuitous ARP behavior by following K11985.

If you want to minimize ARP communication during failover events, consider configuring MAC masquerade on the traffic group which allows a MAC address to be shared between BIG-IP systems.

If enabling MAC masquerading on the BIG-IP is not an option, F5 recommends that you make the following modifications on the relevant network devices:
- Increase the number of allowed broadcast frames, per second, before dropping the rest
- Increase the input buffer size
- Allow ARP updates to occur on the device

Scenario 3: BIG-IP objects may not send GARP requests during failover

BIG-IP system allows you to configure (virtual servers, NATs, or SNATs) on a different subnet than the self IP address or associated with VLANs that have no self IP addresses.  For BIG-IP redundant systems during a failover, the system taking over as the active unit does not advertise GARP requests for BIG-IP local traffic objects that are configured in this manner. When the BIG-IP system does not advertise GARP requests for a virtual address, the neighboring network devices may continue to send traffic to the previously active unit. As a result, traffic may be disrupted.

Recommended action:

Consider adding a self IP address that belongs to the subnet of the affected BIG-IP local traffic object on the VLAN that is receiving traffic for this object

If adding a self IP address is not desirable, consider configuring MAC masquerade for the traffic group that contains the BIG-IP local traffic object.

More Resources

For more details on ACI endpoint learning behavior and the configurations above, refer to “Endpoint Retention Policy” and “Rogue EP Control” sections in the ACI Fabric Endpoint Learning White Paper: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html.

For more details on ACI IP Data-plane Learning and its use case, refer to “IP Data-plane Learning” section in the ACI Fabric Endpoint Learning White Paper: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html#IPDataplaneLearning