Packet Analysis with Scapy and tcpdump: Checking Compatibility with F5 SSL Orchestrator
In this guide I want to demonstrate how you can use Scapy (https://scapy.net/) and tcpdump for reproducing and troubleshooting layer 2 issues with F5 BIG-IP devices. Just in case you get into a finger-pointing situation... Starting situation This is a quite recent story from the trenches: My customer uses a Bypass Tap to forward or mirror data traffic to inline tools such as IDS/IPS, WAF or threat intelligence systems. This ByPass Tap offers a feature called Network Failsafe (also known as Fail-to-Wire). This is a fault tolerance feature that protects the flow of data in the event of a power outage and/or system failure. It allows traffic to be rerouted while the inline tools (IDS/IPS, WAF or threat intelligence systems) are shutting down, restarting, or unexpectedly losing power (see red line namedFallbackin the picture below). Since the ByPass Tap itself does not have support for SSL decryption and re-encryption, an F5 BIG-IP SSL Orchestrator shall be introduced as an inline tool in a Layer 2 inbound topology. Tools directly connected to the Bypass Tap will be connected to the SSL Orchestrator for better visibility. To check the status of the inline tools, the Bypass Tap sends health checks through the inline tools. What is sent on one interface must be seen on the other interface and vice versa. So if all is OK (health check is green), traffic will be forwarded to the SSL Orchestrator, decrypted and sent to the IDS/IPS and the TAP, and then re-encrypted and sent back to the Bypass Tap. If the Bypass Tap detects that the SSL Orchestrator is in a failure state, it will just forward the traffic to the switch. This is the traffic flow of the health checks: Target topology This results in the following topology: Problem description During commissioning of the new topology, it turned out that the health check packets are not forwarded through the vWire configured on the BIG-IP. A packet analysis with Wireshark revealed that the manufacturer uses ARP-like packets with opcode 512 (HEX 02 00). This opcode is not defined in the RFC that describes ARP (https://datatracker.ietf.org/doc/html/rfc826), the RFC only describes the opcodes Request (1 or HEX 00 01) and Reply (2 or HEX 00 02). NOTE:Don't get confused that you see ARP packets on port 1.1 and 1.2. They are not passing through, the Bypass Tap is just send those packets from both sides of the vWire, as explained above. The source MAC on port 1.1 and 1.2 are different. Since the Bypass Tap is located right behind the customer's edge firewall, lengthy and time-consuming tests on the live system are not an option, since it would result in a massive service interruption. Therefore, a BIG-IP i5800 (the same model as the customer's) was set up as SSL Orchestrator and a vWire configuration was build in my employers lab. The vWire configuration can be found in this guide (https://clouddocs.f5.com/sslo-deployment-guide/chapter2/page2.7.html). INFO:For those not familiar with vWire: "Virtual wire … creates a layer 2 bridge across the defined interfaces. Any traffic that does not match a topology listener will pass across this bridge." Lab Topology The following topology was used for the lab: I build a vWire configuration on the SSL Orchestrator, as in the customer's environment. A Linux system with Scapy installed was connected to Interface 1.1. With Scapy TCP, UDP and ARP packets can be crafted or sent like a replay from a Wireshark capture. Interface 1.3 was connected to another Linux system that should receive the ARP packets. All tcpdumps were captured on the F5 and analyzed on the admin system (not plotted). Validating vWire Configuration To check the functionality of the F5 and the vWire configuration, two tests were performed. A replay of the Healthcheck packets from the Bypass Tap and a test with RFC-compliant ARP requests. Use Scapy to resend the faulty packets First, I used Wireshark to extract a single packet from packet analysis we took in the customer environment and saved it to a pcap file. I replayed this pcap file to the F5 with Scapy. The sendp() function will work at layer 2, it requires the parametersrdpcap(location of the pcap file for replay) andiface(which interface it shall use for sending). webserverdude@tux480:~$ sudo scapy -H WARNING: IPython not available. Using standard Python shell instead. AutoCompletion, History are disabled. Welcome to Scapy (2.5.0) >>> sendp(rdpcap("/home/webserverdude/cusomter-case/bad-example.pcap"),iface="enp0s31f6") . Sent 1 packets. This test confirmed the behavior that was observed in the customer's environment. The F5 BIG-IP does not forward this packet. Use PING and Scapy to send RFC-compliant ARP packets To create RFC-compliant ARP requests, I first sent an ARP request (opcode 1) through the vWire via PING command. As expected, this was sent through the vWire. To ensure that this also works with Scapy, I also resent this packet with Scapy. >>> sendp(rdpcap("/home/webserverdude/cusomter-case/good-example.pcap"),iface="enp0s31f6") . Sent 1 packets. In the Wireshark analysis it can be seen that this packet is incoming on port 1.1 and then forwarded to port 1.3 through the vWire. Solving the issue with the help of the vendor It became evident that the BIG-IP was dropping ARP packets that failed to meet RFC compliance, rendering the Bypass Tap from this particular vendor seemingly incompatible with the BIG-IP. Following my analysis, the vendor was able to develop and provide a new firmware release addressing this issue. To verify that the issue was resolved in this firmware release, my customer's setup, the exact same model of the Bypass Tap and a BIG-IP i5800, were deployed in my lab, where the new firmware underwent thorough testing. With this approach I could test the functionality and compatibility of the systems under controlled conditions. In this Wireshark analysis it can be seen that the Healthcheck packets are incoming on port 1.1 and then forwarded to port 1.3 through the vWire (marked in green) and also the other way round, coming in on port 1.3 and then forwarded to port 1.1 (marked in pink). Also now you can see that the packet is a proper gratuitous ARP reply (https://wiki.wireshark.org/Gratuitous_ARP). Because the Healthcheck packets were not longer dropped by the BIG-IP, but were forwarded through the vWire the Bypass Tap subsequently marked the BIG-IP as healthy and available. The new firmware resolved the issue. Consequently, my customer could confidently proceed with this project, free from the constraints imposed by the compatibility issue.
MAC address not showing up in LLDP packet
Hello, If you look at the picture above, you can see that Arista's chassis ID is written as mac-address. Is it possible to change the chassis ID of the F5 device to the MAC address? Currently, the chassis ID is transmitted as a serial Number. This is an inquiry related to lldp. please answer about my question. thank youOrchestrated Infrastructure Security - Getting Started
Note:TheBeaconcapabilities referenced in this article hosted on F5 Cloud Services are planning a migration to a new SaaS Platform - Check out the latesthere. Introduction A typical daisy-chained security stack is difficult to manage and make changes.All devices in the chain are physically wired to each other in a serial arrangement.Each device performs SSL decryption and re-encryption when needed.All devices in the chain need to have similar performance capabilities.All devices in the chain need to be properly configured to route traffic to their neighboring devices, and likely will need to be manually configured to trust SSL certificates used by neighboring devices for decryption and re-encryption. Failure of any device in the security chain brings the entire chain down.Capacity cannot be increased simply by adding another like-device (i.e. a NGFW) to the chain.Capacity can only be increased by replacing a single device with a higher capacity device. Removing or adding a device to the chain is problematic.For one, the entire security stack will need to be unavailable while removing or adding a device.Proper routing between devices must be maintained or the whole chain will not pass traffic.Certificate trust and other factors may need to be addressed as well. High availability is also problematic.The only way to ensure high availability is to create another daisy-chain, identical to the first.This chain needs to wait in standby mode until the primary chain fails or is taken down, and then the standby chain can take over for the primary. Managing a single daisy chain security stack is not easy.Managing two for high availability is significantly more complicated and overly expensive. Some security devices are deployed differently and cannot operate together in the security stack.Those devices would need their own separate deployment from the devices in the daisy chain, further complicating the configuration.As an example, it’s not an uncommon security practice to employ network TAP devices, explicit proxies, ICAP servers as well as Layer2/3 devices. All of these devices cannot be configured to properly route traffic in a daisy chain. SSL Orchestrator solves almost all of these challenges, and enables you to have a nimble security solution capable of adapting to almost any type of threat. High Level Network Topology The network topology used for this setup is below.BIG-IP-11 and 12 are deployed in Layer 2 mode. The Advanced WAF and AFM devices will also be deployed in Layer 2 and will be physically wired to the SSL Orchestrators.This is a high availability environment where there is one Active BIG-IP and one ready on Standby.The Port Objects (511 & 512) allow traffic to flow through either BIG-IP, in case of a failure.The applications being protected are represented by the Ubuntu servers connected to the South switch. BIG-IP Network Topology A zoomed in view of the BIG-IP devices is below.This shows the physical connectivity and the specific interfaces used by SSL Orchestrator, Advanced WAF and AFM devices. Summary This article is part of a series on implementing Orchestrated Infrastructure Security. It includes High Availability, Central Management with BIG-IQ, Application Visibility with Beacon and the protection of critical assets using F5 Advanced WAF and Protocol Inspection (IPS) with AFM.It is assumed that SSL Orchestrator is already deployed, and basic network connectivity is working. Next Steps Click Next to proceed to the next article in the series.Implementing SSL Orchestrator - L2 Mode Deployment
Introduction This article shows you how to deploy SSL Orchestrator in Layer 2 (L2) mode This article is divided into the following high level sections: Network Topology Requirements Best Practices Known Limitations Please forgive me for using SSL and TLS interchangeably in this article. Software versions used in this article: BIG-IP Version: 14.1.2 SSL Orchestrator Version: 5.5 BIG-IQ Version: 7.0.1 Network Topology The diagram below is a representation of the actual lab network where SSL Orchestrator was deployed and tested. Before SSL Orchestrator After SSL Orchestrator All routing configuration is static routing – no dynamic routing protocols is implemented in this design.Dynamic Routing was not considered for this article series. Example Port Object/Channel and MLAG configuration from North Switch1: interface Port-Channel511 switchport trunk allowed vlan 511 switchport mode trunk mlag 511 interface Port-Channel512 switchport trunk allowed vlan 511 switchport mode trunk mlag 512 mlag configuration domain-id mlag1 heartbeat-interval 2500 local-interface Vlan4094 peer-address 172.16.0.2 peer-link Port-Channel10 reload-delay 150 Requirements BIG-IP Version: 14.1.2 SSL Orchestrator Version: 5.5 BIG-IQ Version: 7.0.1 NGWF Version: Details are beyond the scope of this discussion – note that NGWF is configured with vWire and performs its inspection as a transparent L2 device. Adv. WAF: Details are beyond the scope of this discussion – for this implementation, the Advanced WAF module is running on a separate BIG-IP platform. Configuration of BIG-IP deployed as SSL Orchestrator can be downloaded fromherein GitLab. Demo videos are available for both Inbound and Outbound use cases: Outbound Traffic Inspection Inbound Traffic Inspection Best Practices for SSL Orchestrator Deployment BIG-IP Recommendations The following provides salient recommendations, these and others are discussed in detail in theSSL Orchestrator Document. AutoMap AutoMap is a secure network address translation (SNAT) described in Knowledge articleK7336. AutoMap should not be used where possible in BIG-IP SSL Orchestrator deployments. Please refer toK7820for SNAT uses and best practices.With BIG-IP configurations (SSLO or other modules), whenever a large number of connections are going to require SNAT, you want to make sure that SNAT pools are used to avoid port collisions (running out of ephemeral ports to initiate the connection). Debug Logging Traffic through BIG-IP SSL Orchestrator will transit through different services as defined in the service chain.Enabling debug will generate logs as traffic traverses the different services – this can be very verbose and generate large amounts of traffic. It is recommended to leverage debug logging for troubleshooting only. Security Recommendations The following recommendations can also be found in theSSL Orchestrator Document. SSL Orchestrator and Service Proximity SSL Orchestrator is almost never first in an enterprise architecture, so other security devices (proxies, firewalls, IPSs, sandboxes, etc.) are already deployed. If you ask any network and/or security admin, 99% of the time they have no interest in moving those security devices or re-architecting anything else when SSL Orchestrator is introduced. However, remember that the traffic being sent to and from the security devices is unencrypted, so sending traffic across an existing enterprise network to some security device is also sending passwords, credit card numbers, and other protected data (HIPAA, PCI, etc.), across an uncontrolled span of network where any connected device can see it in clear text. It is therefore brutally important that customers understand this, and that they do move those security devices to networks that are behind and protected by SSL Orchestrator. In a perfect situation, no traffic should be able to reach these security devices except through SSL Orchestrator. As frightening as the possibility of a data exposure is, there are going to be customers that ignore the warnings. You should therefore do your best to convey this security best practice to customers as their trusted advisor. This is the reason for the “Auto-Manage” field in the configuration for the services. It creates non- overlapping, internal, non-routable address spaces for each service and encourages customers to protect them. If disabling this, and changing the inline service IP spaces, take care to protect these. Known Limitations It is recommended to consult with the Knowledge ArticleK00805840for current known issues and workarounds. Strictness Underlying the BIG-IP SSL Orchestrator is the BIG-IP TMOS infrastructure.The BIG-IP SSL Orchestrator provides a guided configuration front-end to configure TMOS to build the services, and different policies to make the solution work as designed. If the administrator were to dive deeper in the low-level configuration of the device after building a topology, the configuration would contain BIG-IP configuration items – refer to Appendix C for sample configuration for a simple SSL Orchestrator configuration. The notion of strictness is introduced to “lock” the configured items in such a manner as to prevent the administrator to be able to modify the underlying object without using the SSL Orchestrator interface. Any attempt to modify an object that is strictly managed will result in an error message as shown below: Gossip For BIG-IP SSL Orchestrator deployed in redundant pairs, BIG-IP should be in manual/full sync mode (Device Management>>Device Group).For more information related to BIG-IP high availability (HA) configuration, refer to theuser guide. The traditional synchronization mechanism is augmented with “gossip” for the SSL Orchestrator.This synchronization mechanism is independent from typical LTM config sync.For the version of BIG-IP SSL Orchestrator used for this deployment, ”gossip” config sync is two-way automatic."gossip" uses REST API calls for the SSL Orchestrator related part of the config sync.If a change is made to one device in the pair, it is automatically propagated to the other device.This differs from the traditional BIG-IP sync behavior. SSL Orchestrator config needs to be re-deployed if a STANDALONE environment is changed to an HA environment. Also, you need to make sure VLAN configuration is the same on both the box for successful re-deployment. Summary This article should be a good starting point for planning your initial SSL Orchestrator deployment. The network topology and architecture was explained with the help of diagrams. We also went over the software requirements, best practices and known limitations. Next Steps Click Next to proceed to the next article in the series.
