Troubleshooting OSPF on BIG-IP systems
Quick Intro This is more of a what to look for when troubleshooting guide than a step-by-step guide as I believe that troubleshooting is not an exact science.. Here's a rough road map of what we need to know and what I'm gonna show you in this article: Know what OSPF is and how to enable it on BIG-IP Know what RHI is Determine if OSPF problem is about Route Health Injection (RHI) or pure OSPF RHI Troubleshooting Check if K route is present Given that K route is present, check if it is injected into OSPF Debug NSM (BIG-IP's Network Services Module) Pure OSPF Troubleshooting OSPF neighbour/adjacency not established DBD MTU Mismatch Route is present in OSPF database but no in Routing Table Make sure TMM is forwarding data to BIG-IP's Control plane and vice-versa OSPF Debug Know what OSPF is and how to enable it on BIG-IP OSPF is a routing protocol that roughly tells each router in the OSPF domain where to send the packet to. If there was no OSPF then we'd have to figure it out by ourselves and configure static routes. There is much more to it and in order to know more about OSPF I'd recommend this friendly illustrated book:Bryant's advantage ROUTE book. Apart from that, in order to enable OSPF we need to do it via Route Domain configuration: And here is how we access the CLI configuration: Here's where ZebOS/OSPF configuration is stored (per routing domain) because ZebOS is not aware of Routing Domains: If we had enabled it on Route Domain 1 (for example), then the path would be /config/zebos/rd1 and so on. Know what RHI is K14267explains what it is and even though the article is about BGP, it also applies for OSPF. Roughly speaking, RHI allows us to inject a virtual-address into ZebOS' routing table and advertise it into your OSPF/BGP domain and it is very easy to do it. Here's my routing table when I have no RHI: Now I use one of my VIPs as example: Instead of clicking on the VIP itself I go to the virtual-address list: I picked 10.199.3.143 and enabled Route Advertisement: Disabledis the default Enabledmeans route will always be advertised regardless Selectivemeans route will only be advertised if virtual-address isavailable, i.e. Availability field above is Green. Availability of virtual-address is based onAvailability Calculationfield above Anymeans when ANY virtual server using this virtual address is Available Allmeans ONLY advertise route when ALL virtual servers using this virtual address are Available PS:Selective is the usual choice but in older versions of BIG-IP we might find only two options (Enabled and Disabled). When this is the case, Enabled behaves like Selective does in newer versions. Please checkthis AskF5 article about that. Now look at my routing table and notice that a K route that matches my virtual address magically appeared: Now we just need to redistribute this route into OSPF with redistribute kernel command similar to whatK14267describes. Determine if OSPF problem is about Route Health Injection (RHI) or pure OSPF If it is RHI, we're talking about the K routes above K route is not being advertised via OSPF (e.g. old bug -ID529977OSPF may not process updates to redistributed routes) K route doesn't even appear at all Pure OSPF is everything else, e.g: OSPF neighbour relationship/adjacency not established DBD MTU mismatch Route is present in OSPF database but not in routing table (i.e. route is seen in 'show ip ospf database' but not in 'show ip route'). RHI Troubleshooting Check if K route is present Enable NSM debug and learn how to interpret it If we suspect BIG-IP's control plane is not receiving route from tmm (or mcpd in in 10.x/11.x) enable tmrouted debug (tmrouted is BIG-IP's routing control plane daemon) Given that K route is present, check if it is injected into OSPF Check 'show ip ospf database'in local peer or 'show ip route' and 'show ip ospf database' in remote peer Check 'redistribute' command in BIG-IP configuration. K route is only present when 'redistribute kernel' is present. Check for the present of route-maps in redistribute kernel command. E.g. redistribute kernel route-map my-filter Sometimes we might be filtering by mistake routes are redistributed with a route-map Debuging NSM We can type this on BIG-IP: All RHI relevant information should be on /var/tmp/my-box.log. We can even issue tail -f command to follow along in real time. Pure OSPF Troubleshooting OSPF neighbour/adjacency not established check 'show ip ospf neighbour' 2-way means neighbour relationship is established but we have not (or will not) exchange route with this particular neighbour Full means adjacency is established and this means we did exchange routes with this particular neighbour DR/BDR are only present in Ethernet network type for efficiency purposes which means a DR (designated router) and BDR (backup designated router) are elected and all the other routers maintain adjacency with both DR/BDR but only neighbour relationship among them. Any changes in the network are supposed to be advertised to the DR/BDR and only then it's spread to all the other routers. We can imagine how inefficient it would be in a network with hundreds of routers sharing the same network if any topology change was supposed to be advertised to all routers hence the idea of DR/BDR. Our job here is to make sure OSPF is not stuck in any of the intermediate stagesdue to misconfiguration,typically one of the options below do not match: Brief explanation of above highlighted fields for reference: Area ID: This is always 32 bit and area 0 is 0.0.0.0. Keep in mind that virtual-links would also appear to be originating from area 0 too. Auth Type: 0 is Null, 1 is password protected. Auth Data: Nothing if set to Null, MD5 hash if set to MD5 or clear-text password if set to Plain Text. Hello Interval[sec]: how often Hello packet is sent. This must match on both sides. N: When enabled (1), this means area isnot-so-stubby¹, i.e. does not accept type 5 External LSAs (O E1andO E2fromshow ip routecommand) but converts type 7 LSAs into type 5 in order to advertise route to other areas MC: When enabled (1), this means BIG-IP also supports multicast routing (MOSPF) apart from unicast routing. E: Disabled (0) means area isstub², i.e. does not accept type 5 External LSAs (O E1andO E2route types fromshow ip routecommand) Router Dead Interval[sec]: number of seconds neighbour is declared down when OSPF stops receiving Hello packets. Hello packets reset RouterDeadInterval counter. ¹area <area number> nssa command underrouter ospfmode ²area <area number> stub command underrouter ospfmode Take a packet capture: #tcpdump -nvi <VLAN name>:nnn -s0 -w /var/tmp/ospf_neighbor-tmm-net.pcap ip proto ospf -v In the pcap we should see the OSPF router ID of peer in theActive Neighborlist and this indicates that they're either neighbours or adjacent: Then, if we look at a packet capture and we see that they're both active neighbours for a while and suddenly they're not, then this means the side that removed the neighbour fromActive Neighborlistlikely flapped, restarted or disconnected. DBD MTU Mismatch After BIG-IP receives firstHello Packet, it enters2-waystate and then moves toExStartwhere it exchangesDBD packets.At this stage, MTU size is checked and neighbour relationship won't go any further if they do not match. To confirm this we can check the MTU configuration in the VLAN on BIG-IP and compare it to the one on the peer. Take a packet capture and look for the DBD's Interface MTU field: Route is present in OSPF database but not in Routing table Yes, OSPF has its own routing table where it decides what to add to the regular routing table that we all know and love To troubleshoot this we need to check the details of the LSA with show ip ospf database command. There are times when a route with a better Administrative Distance might be in place (e.g. static route) Make sure TMM is forwarding data to BIG-IP's Control Plane and vice-versa TMM <-> Network: TMM <-> Control Plane: The first packet capture will record the communication externally between BIG-IP and the external device. The second one will record the communication (internally) between BIG-IP's forwarding plane (tmm) and BIG-IP's control plane daemon responsible to process the routes and install in routing table. OSPF Debug The above commands will print debug information about OSPF networking as well as control plane.
OSPF stuck in Exstart
Hello, I am facing an issue with enabling OSPF between a Cisco Router and F5 LTM. The OSPF adjacency process is getting stuck at exstart phase. This is how my toplogy looks like: Cisco Router<-------Cisco L2 Switch------->F5 LTM LTM-01[2]>show ip os neighbor OSPF process 26: Neighbor ID Pri State Dead Time Address Interface 10.10.26.9 1 ExStart/Backup 00:00:37 172.16.207.2 /Inbound 10.10.26.10 1 ExStart/DROther 00:00:34 172.16.207.3 /Inbound LTM-01[2]> From debug in Cisco Router: 863027: Aug 7 11:37:12.920 PDT: OSPF-26 ADJ Gi0/1: Send DBD to 172.16.207.254 seq 0x238E opt 0x52 flag 0x7 len 32 863028: Aug 7 11:37:12.920 PDT: OSPF-26 ADJ Gi0/1: Retransmitting DBD to 172.16.207.254 [8] Here I can see that the router is actively sending DBD packets to F5, but no reply back. All the device are running interface MTU of 1500 bytes. I am not sure how to debug OSPF in F5 device. Any help would be appreciated!
Creating an active-active DSC configuration with OSPF routing on 11.x
Hi Experts I have to do a deployment of active/active VIPRION 2400 chassis, running 2250 blades. As per below link, we need to create two traffic groups for each VLAN. How will the default routing work from the server pool if there are two VLAN floating gateways? Can the two different floating VIPs be part of the same subnet? Also, is it mandatory to have the OSPF self IP peer, part of any floating traffic group? Will it also receive traffic, even if it is part of local traffic group, instead? (http://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-device-service-clustering-11-2-1/3.html)OSPF warnings on F5 BIG-IP 11.4.1
Hello, I have an issue with F5 BIG-IP LTM 11.4.1. I am getting numerous warning messages from OSPF process in logs. OSPF seems to work fine, but I don't know what these messages mean. Should I ignore them? Their OSPF neighbors are Cisco Catalyst 4900 switches. Please give me a clue. Thank you! Tue Feb 11 04:15:43 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: process_packet, unhandled PDU, Operational state Tue Feb 11 04:15:43 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: process_packet (Operational state), Response-PDU, 78966:78965 Tue Feb 11 04:15:43 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: requested pdu : 1 Tue Feb 11 04:15:43 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: process_packet, unhandled PDU, Operational state Tue Feb 11 04:21:57 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: process_packet (Operational state), Response-PDU, 78995:78994 Tue Feb 11 04:21:57 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: requested pdu : 1 Tue Feb 11 04:21:57 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: process_packet, unhandled PDU, operational state Tue Feb 11 04:28:07 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: process_packet (Operational state), Response-PDU, 79025:79021 Tue Feb 11 04:28:07 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: requested pdu : 1 Tue Feb 11 04:28:07 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: process_packet, unhandled PDU, Operational state Tue Feb 11 04:28:07 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: process_packet (Operational state), Response-PDU, 79025:79022 Tue Feb 11 04:28:07 MSK 2014 warning m1-rt-f5-1 OSPF[12267] OSPF: AgentX: requested pdu : 1
Default Route into OSPF
I am unable to advertise a default route 0.0.0.0/0 from the F5 into ospf. I have an F5 VE running 12.1.1 on KVM-QEMU. IMI is running and I have neighbor relationships with the appropriate routers. All other routes that I test are added without issues, but I do not see the 0.0.0.0/0 route being advertised into ospf. MY ZebOS config: [root@F5-INTERNET-01:Active:In Sync] config cat zebos/rd0/ZebOS.conf ! no service password-encryption ! interface lo ! interface tmm ! interface Core ip ospf priority 0 ip ospf mtu-ignore ! interface Internet ! router ospf ospf router-id 10.246.3.250 redistribute kernel passive-interface Internet network 10.246.3.0 0.0.0.255 area 0.0.0.0 ! line con 0 login line vty 0 39 login ! end Here is the LTM Configuration: ltm virtual /Common/Test { destination /Common/0.0.0.0:0 ip-protocol tcp mask any profiles { /Common/fastL4 { } } source 0.0.0.0/0 translate-address enabled translate-port disabled } ltm virtual /Common/test2 { destination /Common/10.10.10.1:80 ip-protocol tcp mask 255.255.255.255 profiles { /Common/tcp { } } source 0.0.0.0/0 translate-address enabled translate-port enabled } ltm virtual /Common/test3 { destination /Common/20.20.20.0:0 ip-protocol tcp mask 255.255.255.0 profiles { /Common/tcp { } } source 0.0.0.0/0 translate-address enabled translate-port disabled } ltm virtual-address /Common/0.0.0.0 { address any arp disabled icmp-echo disabled mask any route-advertisement enabled server-scope none traffic-group /Common/traffic-group-1 } ltm virtual-address /Common/10.10.10.1 { address 10.10.10.1 arp enabled icmp-echo enabled mask 255.255.255.255 route-advertisement enabled server-scope none traffic-group /Common/traffic-group-1 } ltm virtual-address /Common/20.20.20.0 { address 20.20.20.0 arp disabled icmp-echo disabled mask 255.255.255.0 route-advertisement enabled server-scope none traffic-group /Common/traffic-group-1 } What is the issue?
Multiples Route Domain with OSPF
Hi everyone, We're trying to redistribute 3 differents DNS Listener via OSPF with Multiples Route Domain in a lab environment, in this case "DNS Listener 0" is working with RD0, "DNS Listener 1" with RD1 and "DNS Listener 2" with RD2 but every RD is working with OSPF. We got ARM licensed and each one of the IMI Shell is redistributing his respective Kernel Adress. The RD0 has adyancency with the Router that is directly connected with the F5 BIG-IP, but neither RD1 and RD2 has. We tried with differents OSPF process and areas with no luck whatsoever. It is necessary to do an additional configuration in the F5 BIG-IP? Or this is a problem related to Networking? Here's a "sh run" command of the IMI Shell of RD2: no service password-encryption ! interface lo ! interface /Resolver-IT/VLAN40 ip ospf network non-broadcast ip ospf cost 1 ! router ospf 2 redistribute kernel network 40.0.0.0 0.0.0.255 area 0 neighbor 40.0.0.1 !F5 appliances failing to establish OSPF with attached devices
Hi all, there is a little known 'feature' in the underlying Linux OS that has a hard limit of 20 network statements. If you go beyond 20, the additional networks will not be advertised in OSPF. I hit this limit after migrating services to my Big-IP. It took F5 support a while to find the cause as the feature isn’t widely known within F5 despite been over 10 years old. My workaround was to super-net a number of /24 subnets into /20, /21 statements which brought me back under the limit of 20 networks (conf t network statements in imish). If this isn’t possible you need to change the net.ipv4.igmp_max_memberships configuration & restart the OSPF process sysctl net.ipv4.igmp_max_memberships=25 zebos -r 0 cmd clear ip ospf process show ip ospf neighbor Above I am setting the hard limit to 25 networks & restarting the OSPF process. Note, adding with sysctl should allow the setting to survive a restart/upgrade – omit it & the increase will not survive a reboot. Showing the neighborships will now show expected results for the missing networks (statements 21-25)
BIG-IP OSPF with Palo Alto
I have 2 BIG-IP 2200s units in an active/standby pair. Both of them OSPF peer with a Palo Alto 3060 failover pair. Whenever there is a topology change, such as a failover of the Palo Alto - I cannot get a full adjacency to establish between the Palo Alto and either of the F5s. It is stuck in the "exchange" state. Packet capture shows hellos being exchanged normally. The Palo Alto is repeatedly sending its Database Descriptor to the F5s, but not the other way around. The issue is resolved when I clear the OSPF process on the F5s. The F5 finally sends its Database Descriptor to the Palo Alto and I get full adjacency at that point. However if there is a real failover event on the firewalls, this would effectively bring our network down until someone can manually intervene. BIG-IP version: 11.5.4 HF2 Palo Alto version: 6.1.14 Anyone ever see this issue? It looks like a bug to me.
is it possible to have virtual servers in a different subnet than that of the interface on the F5 ?
Hi All, is it possible to have virtual servers in a different subnet than that of the interface on the F5 ? i want to be able to peer to the upstream router with the F5s interface IP using OSPF. I want to be able to advertise the virtual server ip addresses using ospf to the upstream router. There is a requirement to use a different IP subnet than that of the F5 interface. is this possible ? so that for all virtual server IP addresses, the upstream router has a route to the F5 interface ip address. thanks Bhav
