No traffic on network card in virtual Big-IP
Hi, We have two virtual deployments of F5 version 17.1.2.2 in a VMware environment, Each deployment has 3 network cards: MGMT external (for VSs) internal (for backend pool) Lately we started to experience a very strange behavior, where randomly, the external NIC stops receiving traffic. To resolve the issue we can restart the Big-IP, or disable/enable the NIC on the VMside. When taking pcaps on the BIG-IP there's just nothing coming in, this is why the F5 support told us they can't troubleshoot if there's no traffic, and pointed us to VMware support. We took pcaps and logs on the vmware side and it looks like traffic is not passing from the hypervisor to the VM but the support doesn't see any errors. Because the issue is random we can't simulate it. We tried to correlate the issue with vm migration, or external backups but couldn't pin point anything. We have a large virtual environment it this issue only occurs on two F5 machine (out of 400+ other VMs). I'm not entirely sure, but I think the issue started after upgrading from version 16.1.4.1 to 17.1.2.2. I was wondering if anyone stumbled upon something similar? Thanks
Intermittent Net::ERR_CONNECTION_RESET Error and Incomplete Loading over HTTPS
I have an F5 load balancing setup configured with two servers. My MVC web application, which incorporates Kendo UI, Jquery, and bootstrapping, is hosted on an IIS server with an SSL certificate. However, when accessing the application via HTTPS from outside the server, it often or sometimes results in a 'net::ERR_CONNECTION_RESET' error, with intermittent failures to load javascript and CSS files to the client browser. Strangely, upon reloading the page, the assets load properly, and the page functions as expected. This issue did not occur when the application was accessed via HTTP, where it worked properly without any issues. What could be the reason behind this problem?F5 appliances failing to establish OSPF with attached devices
Hi all, there is a little known 'feature' in the underlying Linux OS that has a hard limit of 20 network statements. If you go beyond 20, the additional networks will not be advertised in OSPF. I hit this limit after migrating services to my Big-IP. It took F5 support a while to find the cause as the feature isn’t widely known within F5 despite been over 10 years old. My workaround was to super-net a number of /24 subnets into /20, /21 statements which brought me back under the limit of 20 networks (conf t network statements in imish). If this isn’t possible you need to change the net.ipv4.igmp_max_memberships configuration & restart the OSPF process sysctl net.ipv4.igmp_max_memberships=25 zebos -r 0 cmd clear ip ospf process show ip ospf neighbor Above I am setting the hard limit to 25 networks & restarting the OSPF process. Note, adding with sysctl should allow the setting to survive a restart/upgrade – omit it & the increase will not survive a reboot. Showing the neighborships will now show expected results for the missing networks (statements 21-25)
ARP requests for 127.2.0.1
Our GTM (BIG-IP 11.4.1 Build 635.0 HF2) is constantly—several times a second—sending out ARP requests out the mgmt interface for the IP address 127.2.0.1 with a source address of 127.2.0.2. For example: 2014-11-26 11:59:44.543142 00:01:d7:d5:2c:01 -> ff:ff:ff:ff:ff:ff ARP Who has 127.2.0.1? Tell 127.2.0.2 2014-11-26 11:59:44.543149 00:01:d7:d5:2c:01 -> ff:ff:ff:ff:ff:ff ARP Who has 127.2.0.1? Tell 127.2.0.2 2014-11-26 11:59:45.543200 00:01:d7:d5:2c:01 -> ff:ff:ff:ff:ff:ff ARP Who has 127.2.0.1? Tell 127.2.0.2 2014-11-26 11:59:45.543205 00:01:d7:d5:2c:01 -> ff:ff:ff:ff:ff:ff ARP Who has 127.2.0.1? Tell 127.2.0.2 2014-11-26 11:59:46.543134 00:01:d7:d5:2c:01 -> ff:ff:ff:ff:ff:ff ARP Who has 127.2.0.1? Tell 127.2.0.2 2014-11-26 11:59:46.543140 00:01:d7:d5:2c:01 -> ff:ff:ff:ff:ff:ff ARP Who has 127.2.0.1? Tell 127.2.0.2 Does anyone know why this is happening or how I can make it stop?
New i2800 to Cisco 93180YC-FX3 Twinax
I've seen older post and I've used these before but cannot get them working now. Does anyone know if you can still use the Cisco Twinax cables (SFP-H10GB-CU3M)? My F5 is showing the following: Net::Interface Name Status Bits Bits Pkts Pkts Drops Errs Media In Out In Out --------------------------------------------------------------- 1.0 up 0 0 0 0 0 0 1000CX-FD 2.0 miss 0 0 0 0 0 0 none 3.0 miss 0 0 0 0 0 0 none 4.0 miss 0 0 0 0 0 0 none 5.0 down 0 0 0 0 0 0 none 6.0 down 0 0 0 0 0 0 none mgmt up 20.2G 6.2G 1.7M 670.7K 0 0 1000T-FD net interface 5.0 { if-index 352 mac-address 14:a9:d0:06:80:88 media-max 10000T-FD module-description "Unsupported Optic detected" mtu 9198 serial JPC23220CWT vendor CISCO-JPC vendor-oui 001897 vendor-partnum P3410UB03000-1 vendor-revision A0
Connection loss Client -> F5 BIGIP LTM
Hi all, I am currently experiencing an issue with an application that is being used on 3 application servers (windows server 2003), loadbalanced behind the F5 BIGIP. Users are sometimes losing connection to the server, which makes the application crash. I have launched a capture for one of these clients and I'm seeing the following when this issue occurs (capture.png): Client: 10.229.237.235, IP of virtual server on BIGIP: 172.20.5.41 From what I can see there is no SYN-ACK being returned from BIGIP. There are also a lot of messages in the log containing TCP Window Full & TCP out of order. When we let the user connect directly to an application server instead of passing through BIGIP, they have no issues. The capture is also very clean in that case, no retransmissions, no duplicate acks or TCP resets.. The TCP protocol being used is Protocol Profile (Client) - TCP LAN Optimized and for Protocol Profile (Server) - TCP WAN Optimized. Does anyone have an idea why BIGIP doesn't send a SYN-ACK in this case? I was thinking maybe an issue with receiving window & send buffers.. Or would I need a capture on the virtual server to further analyze this behaviour? Any help would be greatly appreciated! Thank you Kind regards Ron
How to use Ansible with Cisco routers
Quick Intro For those who don't know, there is an Ansible plugin called network_cli to retrieve network device configuration for backup, inspection and even execute commands. So, let's assume we have Ansible already installed and 2 routers: I used Debian Linux here and I had to install Python 3: I've also installed pip as we can see above because I wanted to install a specific version of Ansible (2.5.+) that would allow myself to use network_cli plugin: Note: we can list available Ansible versions by just typing pip install ansible== I've also created a user named ansible: Edited Linux sudoers file with visudo command: And added Ansible user permission to run root commands without prompting for password so my file looked liked this: Quick Set Up This is my directory structure: These are the files I used for this lab test: Note: we can replace cisco1.rodrigo.example for an IP address too. In Ansible, there is a default config file (ansible.cfg) where we store the global config, i.e. how we want Ansible to behave. We also keep the list of our hosts into an inventory file (inventory.yml here). There is a default folder (group_vars) where we can store variables that would apply to any router we ran Ansible against and in this case it makes sense as my router credentials are the same. Lastly, retrieve_backup.yml is my actual playbook, i.e. where I tell Ansible what to do. Note: I manually logged in to cisco1.rodrigo.example and cisco2.rodrigo.example to populate ssh known_hosts files, otherwise Ansible complains these hosts are untrusted. Populating our Playbook file retrieve_backup.yml Let's say we just want to retrieve the OSPF configuration from our Cisco routers. We can use ios_command to type in any command to Cisco router and use register to store the output to a variable: Note: be careful with the indentation. I used 2 spaces here. We can then copy the content of the variable to a file in a given directory. In this case, we copied whatever is in ospf_output variable to ospf_config directory. From the Playbook file above we can work out that variables are referenced between {{ }} and we might probably be wondering why do we need to append stdout[0] to ospf_output right? If you know Python, you might be interested in knowing a bit more about what's going on under the hood so I'll clarify things a bit more here. The variable ospf_output is actually a dictionary and stdout is one of its keys. In reality, ospf_output.stdout could be represented as ospf_output['stdout'] We add the [0] because the object retrieved by the key stdout is not a string. It's a list! And [0] just represents the first object in the list. Executing our Playbook I'll create ospf_config first: And we execute our playbook by issuing ansible-playbook command: Now let's check if our OSPF config was retrieved: We can pretty much type in any IOS command we'd type in a real router, either to configure it or to retrieve its configuration. We could also append the date to the file name but it's out of the scope of this article. That's it for now.Bidirectional Forwarding Detection (BFD) Protocol Cheat Sheet
Definition This is a protocol initially described in RFC5880 and IPv4/IPv6 specifics in RFC5881. I would say this is an aggressive 'hello-like' protocol with shorter timers but very lightweight on the wire and requiring very little processing as it is designed to be implemented in forwarding plane (although RFC does not forbid it to be implemented in control plane). It also contains a feature called Echo that further leaves cpu processing cycle to roughly ZERO which literally just 'loops' BFD control packets sent from peer back to them without even 'touching' (processing) it. BFD helps routing protocol detects peers failure at sub-second level and BIG-IP supports it on all its routing protocols. On BIG-IP it is control-plane independent as TMM that takes care of BFD sending/receiving unicast probes (yes, no multicast!) and BIG-IP's Advanced Routing Module® being responsible only for its configuration (of course!) and to receive state information from TMM that is displayed in show commands. BIG-IP's control plane daemon communicates with TMM is oamd. This daemon starts when BFD is enabled in the route domain like any other routing protocol. BFD Handshake Explained 218: BFD was configured on Cisco Router but not on BIG-IP so neighbour signals BIG-IP sessionstate is Down and no flags 219: I had just enabled BFD on BIG-IP, session state is now Init and only Control Plane Independent flag set¹ 220: Poll flag is set to validate initial bidirectional connectivity (expecting Final flag set in response) 221: BIG-IP sets Final flag and handshake is complete² ¹Control Plane Independent flag is set because BFD is not actively performed by BIG-IP's control plane. BIG-IP's BFD control plane daemon (oamd) just signals TMM what BFD sessions are required and TMM takes care of sending/receiving of all BFD control traffic and informs session state back to Advanced Routing Module's daemon. ²Packets 222-223 are just to show that after handshake is finished all flags are cleared unless there is another event that triggers them. Packet 218 is how Cisco Router sees BIG-IP when BFD is not enabled. Control Plane Independent flag on BIG-IP remains though for the reasons already explained above. Protocol fields Diagnostic codes 0 (No Diagnostic): Typically seen when BFD session is UP and there are no errors. 1 (Control Detection Time Expired): BFD Detect Timer expired and session was marked down. 2 (Echo Function Failed): BFD Echo packet loop verification failed, session is marked down and this is the diagnostic code number. 3 (Neighbor Signaled Session Down): If either neighbour advertised state or our local state is down you will see this diagnostic code 4 (Forwarding Planet Reset): When forwarding-plane (TMM) is reset and peer should not rely on BFD so session is marked down. 5 (Path Down): On demand mode external application can signal BFD that path is down so we can set session down using this code 6 Concatenated Path Down): 7 (Administratively Down): Session is marked down by an administrator 8 (Reverse Concatenated Path Down): 9-31: Reserved for future use BFD verification 'show' commands ³Type IP address to see specific session Modes Asynchronous (default): hello-like mode where BIG-IP periodically sends (and receives) BFD control packets and if control detection timer expires, session is marked as down. It uses UDP port 3784. Demand: BFD handshake is completed but no periodic BFD control packets are exchanged as this mode assumes system has its own way of verifying connectivity with peer and may trigger BFD verification on demand, i.e. when it needs to use it according to its implementation. BIG-IP currently does not support this mode. Asynchronous + Echo Function: When enabled, TMM literally loops BFD echo-specific control packets on UDP port 3785 sent from peers back to them without processing it as it wasn't enough that this protocol is already lightweight. In this mode, a less aggressive timer (> 1 second) should be used for regular BFD control packets over port 3784 and more aggressive timer is used by echo function. BIG-IP currently does not support this mode. Header Fields Protocol Version: BFD version used. Latest one is v1 (RFC5880) Diagnostic Code: BFD error code for diagnostics purpose. Session State: How transmitting system sees the session state which can be AdminDown, Down, Init or Up. Message Flags: Additional session configuration or functionality (e.g. flag that says authentication is enabled) Detect Time Multiplier: Informs remote peer BFD session is supposed to be marked down if Desired Min TX Interval multiplied by this value is reached Message Length (bytes): Length of BFD Control packet My Discriminator: For each BFD session each peer will use a unique discriminator to differentiate multiple session. Your Discriminator: When BIG-IP receives BFD control message back from its peer we add peer's My Discriminator to Your Discriminator in our header. Desired Min TX Interval (microseconds): Fastest we can send BFD control packets to remote peer (no less than configured value here) Required Min RX Interval (microseconds): Fastest we can receive BFD control packets from remote peer (no less than configured value here) Required Min Echo Interval (microseconds): Fastest we can loop BFD echo packets back to remote system (0 means Echo function is disabled) Session State AdminDown: Administratively forced down by command Down: Either control detection time expired in an already established BFD session or it never came up. If probing time (min_tx) is set to 100ms for example, and multiplier is 3 then no response after 300ms makes system go down. Init: Signals a desire to bring session up in the beginning of BFD handshake. Up: Indicates session is Up Message Flags Poll: Pool flag is just a 'ping' that requires peer box to respond with Final flag. In BFD handshake as well as in Demand mode pool message is a request to validate bidirectional connectivity. Final: Sent in response to packet with Poll bit set Control Plane Independent: Set if BFD can continue to function if control plane is disrupted¹ Authentication Present: Only set if authentication is being used Demand: If set, it is implied that periodic BFD control packets are no longer sent and another mechanism (on demand) is used instead. Multipoint: Reserved for future use of point-to-multipoint extension. Should be 0 on both sides. ¹ This is the case for BIG-IP as BFD is implemented in forwarding plane (TMM) BFD Configuration Configure desired transmit and receive intervals as well as multiplier on BIG-IP. And Cisco Router: You will typically configure the above regardless of routing protocol used. BFD BGP Configuration And Cisco Router: BFD OSPFv2/v3 Configuration BFD ISIS Configuration BFD RIPv1/v2 Configuration BFD Static Configuration All interfaces no matter what: Specific interface only: Tie BFD configuration to static route:
