Troubleshooting OSPF on BIG-IP systems
Quick Intro This is more of a what to look for when troubleshooting guide than a step-by-step guide as I believe that troubleshooting is not an exact science.. Here's a rough road map of what we need to know and what I'm gonna show you in this article: Know what OSPF is and how to enable it on BIG-IP Know what RHI is Determine if OSPF problem is about Route Health Injection (RHI) or pure OSPF RHI Troubleshooting Check if K route is present Given that K route is present, check if it is injected into OSPF Debug NSM (BIG-IP's Network Services Module) Pure OSPF Troubleshooting OSPF neighbour/adjacency not established DBD MTU Mismatch Route is present in OSPF database but no in Routing Table Make sure TMM is forwarding data to BIG-IP's Control plane and vice-versa OSPF Debug Know what OSPF is and how to enable it on BIG-IP OSPF is a routing protocol that roughly tells each router in the OSPF domain where to send the packet to. If there was no OSPF then we'd have to figure it out by ourselves and configure static routes. There is much more to it and in order to know more about OSPF I'd recommend this friendly illustrated book:Bryant's advantage ROUTE book. Apart from that, in order to enable OSPF we need to do it via Route Domain configuration: And here is how we access the CLI configuration: Here's where ZebOS/OSPF configuration is stored (per routing domain) because ZebOS is not aware of Routing Domains: If we had enabled it on Route Domain 1 (for example), then the path would be /config/zebos/rd1 and so on. Know what RHI is K14267explains what it is and even though the article is about BGP, it also applies for OSPF. Roughly speaking, RHI allows us to inject a virtual-address into ZebOS' routing table and advertise it into your OSPF/BGP domain and it is very easy to do it. Here's my routing table when I have no RHI: Now I use one of my VIPs as example: Instead of clicking on the VIP itself I go to the virtual-address list: I picked 10.199.3.143 and enabled Route Advertisement: Disabledis the default Enabledmeans route will always be advertised regardless Selectivemeans route will only be advertised if virtual-address isavailable, i.e. Availability field above is Green. Availability of virtual-address is based onAvailability Calculationfield above Anymeans when ANY virtual server using this virtual address is Available Allmeans ONLY advertise route when ALL virtual servers using this virtual address are Available PS:Selective is the usual choice but in older versions of BIG-IP we might find only two options (Enabled and Disabled). When this is the case, Enabled behaves like Selective does in newer versions. Please checkthis AskF5 article about that. Now look at my routing table and notice that a K route that matches my virtual address magically appeared: Now we just need to redistribute this route into OSPF with redistribute kernel command similar to whatK14267describes. Determine if OSPF problem is about Route Health Injection (RHI) or pure OSPF If it is RHI, we're talking about the K routes above K route is not being advertised via OSPF (e.g. old bug -ID529977OSPF may not process updates to redistributed routes) K route doesn't even appear at all Pure OSPF is everything else, e.g: OSPF neighbour relationship/adjacency not established DBD MTU mismatch Route is present in OSPF database but not in routing table (i.e. route is seen in 'show ip ospf database' but not in 'show ip route'). RHI Troubleshooting Check if K route is present Enable NSM debug and learn how to interpret it If we suspect BIG-IP's control plane is not receiving route from tmm (or mcpd in in 10.x/11.x) enable tmrouted debug (tmrouted is BIG-IP's routing control plane daemon) Given that K route is present, check if it is injected into OSPF Check 'show ip ospf database'in local peer or 'show ip route' and 'show ip ospf database' in remote peer Check 'redistribute' command in BIG-IP configuration. K route is only present when 'redistribute kernel' is present. Check for the present of route-maps in redistribute kernel command. E.g. redistribute kernel route-map my-filter Sometimes we might be filtering by mistake routes are redistributed with a route-map Debuging NSM We can type this on BIG-IP: All RHI relevant information should be on /var/tmp/my-box.log. We can even issue tail -f command to follow along in real time. Pure OSPF Troubleshooting OSPF neighbour/adjacency not established check 'show ip ospf neighbour' 2-way means neighbour relationship is established but we have not (or will not) exchange route with this particular neighbour Full means adjacency is established and this means we did exchange routes with this particular neighbour DR/BDR are only present in Ethernet network type for efficiency purposes which means a DR (designated router) and BDR (backup designated router) are elected and all the other routers maintain adjacency with both DR/BDR but only neighbour relationship among them. Any changes in the network are supposed to be advertised to the DR/BDR and only then it's spread to all the other routers. We can imagine how inefficient it would be in a network with hundreds of routers sharing the same network if any topology change was supposed to be advertised to all routers hence the idea of DR/BDR. Our job here is to make sure OSPF is not stuck in any of the intermediate stagesdue to misconfiguration,typically one of the options below do not match: Brief explanation of above highlighted fields for reference: Area ID: This is always 32 bit and area 0 is 0.0.0.0. Keep in mind that virtual-links would also appear to be originating from area 0 too. Auth Type: 0 is Null, 1 is password protected. Auth Data: Nothing if set to Null, MD5 hash if set to MD5 or clear-text password if set to Plain Text. Hello Interval[sec]: how often Hello packet is sent. This must match on both sides. N: When enabled (1), this means area isnot-so-stubby¹, i.e. does not accept type 5 External LSAs (O E1andO E2fromshow ip routecommand) but converts type 7 LSAs into type 5 in order to advertise route to other areas MC: When enabled (1), this means BIG-IP also supports multicast routing (MOSPF) apart from unicast routing. E: Disabled (0) means area isstub², i.e. does not accept type 5 External LSAs (O E1andO E2route types fromshow ip routecommand) Router Dead Interval[sec]: number of seconds neighbour is declared down when OSPF stops receiving Hello packets. Hello packets reset RouterDeadInterval counter. ¹area <area number> nssa command underrouter ospfmode ²area <area number> stub command underrouter ospfmode Take a packet capture: #tcpdump -nvi <VLAN name>:nnn -s0 -w /var/tmp/ospf_neighbor-tmm-net.pcap ip proto ospf -v In the pcap we should see the OSPF router ID of peer in theActive Neighborlist and this indicates that they're either neighbours or adjacent: Then, if we look at a packet capture and we see that they're both active neighbours for a while and suddenly they're not, then this means the side that removed the neighbour fromActive Neighborlistlikely flapped, restarted or disconnected. DBD MTU Mismatch After BIG-IP receives firstHello Packet, it enters2-waystate and then moves toExStartwhere it exchangesDBD packets.At this stage, MTU size is checked and neighbour relationship won't go any further if they do not match. To confirm this we can check the MTU configuration in the VLAN on BIG-IP and compare it to the one on the peer. Take a packet capture and look for the DBD's Interface MTU field: Route is present in OSPF database but not in Routing table Yes, OSPF has its own routing table where it decides what to add to the regular routing table that we all know and love To troubleshoot this we need to check the details of the LSA with show ip ospf database command. There are times when a route with a better Administrative Distance might be in place (e.g. static route) Make sure TMM is forwarding data to BIG-IP's Control Plane and vice-versa TMM <-> Network: TMM <-> Control Plane: The first packet capture will record the communication externally between BIG-IP and the external device. The second one will record the communication (internally) between BIG-IP's forwarding plane (tmm) and BIG-IP's control plane daemon responsible to process the routes and install in routing table. OSPF Debug The above commands will print debug information about OSPF networking as well as control plane.
