Forum Discussion

Ron_Peters_2122's avatar
Ron_Peters_2122
Icon for Altostratus rankAltostratus
Sep 15, 2016

ARP/MAC Tables Not Updating on Core Switches After F5 LTM Failover (GARP Issue?)

We have two F5 LTM 5250v appliances configured with 2 vCMP instances each in an HA pair (Active/Standby). Each F5 5250v has a 10G uplink to two core switches (Cisco Nexus 7010) configured as an LACP ...
application delivery
arp
fail over
gratuitous arp
LTM
Ron_Peters_2122's avatar
Ron_Peters_2122
Icon for Altostratus rankAltostratus

We have finally resolved this issue and as promised I said I would comment on what the issue was. We confirmed 100% with a tcpdump on the F5s that they were sending Gratuitous ARPs out its 10G interfaces for all virtual-addresses after a failover event.

 

We opened a TAC case with Cisco and found that there is a hardware rate-limiter in place on the particular F1 card (very old card) that these F5's were terminating into. The rate-limit for class rl-4, which ARP was assigned to was set to 100 packets-per-second. This is way too low to support the amount of ARP traffic the F5 generates and we had millions of ARP drops on this particular card.

 

We analyzed the pcap file and found the rate at which the F5 transmitted these GARPs and adjusted the rate-limit on the rl-4 class to 3000 packets per second. We performed failover tests and the MAC addresses on both 7Ks updated immediately for all virtual-addresses.

 

Thanks for all the input you guys provided.

 

Ron_Peters_2122's avatar
Ron_Peters_2122
Icon for Altostratus rankAltostratus
Aug 25, 2017

This particular issue we were having was for the hardware rate limiter on older F1 cards. I found the following two links, which may be helpful. When I worked with TAC, I was able to prove via the packet captures that the F5s were in fact sending the GARPs and since they were directly attached to the 7Ks and the ARP tables were not updating (but the GARPs were being SPAN'd to a separate destination port on the 7K - it was an obvious issue with the 7Ks).

 

Perhaps it is an issue with how ARP works over OTV:

 

https://supportforums.cisco.com/t5/network-infrastructure-documents/troubleshooting-arp-issues-across-otv/ta-p/3120293

 

While the links below are for the 7000 series, it should be applicable in regards to the L3 glean Class on 77xx:

 

https://www.cisco.com/c/en/us/support/docs/switches/nexus-7000-series-switches/200677-Nexus-7000-Understanding-hardware-ip-g.html

 

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/5_x/nx-os/unicast/configuration/guide/l3_cli_nxos/l3_ip.html26569

 

In any case, be persistent with Cisco and escalate if need be. Good Luck!