Packet Analysis with Scapy and tcpdump: Checking Compatibility with F5 SSL Orchestrator
In this guide I want to demonstrate how you can use Scapy (https://scapy.net/) and tcpdump for reproducing and troubleshooting layer 2 issues with F5 BIG-IP devices. Just in case you get into a finger-pointing situation... Starting situation This is a quite recent story from the trenches: My customer uses a Bypass Tap to forward or mirror data traffic to inline tools such as IDS/IPS, WAF or threat intelligence systems. This ByPass Tap offers a feature called Network Failsafe (also known as Fail-to-Wire). This is a fault tolerance feature that protects the flow of data in the event of a power outage and/or system failure. It allows traffic to be rerouted while the inline tools (IDS/IPS, WAF or threat intelligence systems) are shutting down, restarting, or unexpectedly losing power (see red line named Fallback in the picture below). Since the ByPass Tap itself does not have support for SSL decryption and re-encryption, an F5 BIG-IP SSL Orchestrator shall be introduced as an inline tool in a Layer 2 inbound topology. Tools directly connected to the Bypass Tap will be connected to the SSL Orchestrator for better visibility. To check the status of the inline tools, the Bypass Tap sends health checks through the inline tools. What is sent on one interface must be seen on the other interface and vice versa. So if all is OK (health check is green), traffic will be forwarded to the SSL Orchestrator, decrypted and sent to the IDS/IPS and the TAP, and then re-encrypted and sent back to the Bypass Tap. If the Bypass Tap detects that the SSL Orchestrator is in a failure state, it will just forward the traffic to the switch. This is the traffic flow of the health checks: Target topology This results in the following topology: Problem description During commissioning of the new topology, it turned out that the health check packets are not forwarded through the vWire configured on the BIG-IP. A packet analysis with Wireshark revealed that the manufacturer uses ARP-like packets with opcode 512 (HEX 02 00). This opcode is not defined in the RFC that describes ARP (https://datatracker.ietf.org/doc/html/rfc826), the RFC only describes the opcodes Request (1 or HEX 00 01) and Reply (2 or HEX 00 02). NOTE: Don't get confused that you see ARP packets on port 1.1 and 1.2. They are not passing through, the Bypass Tap is just send those packets from both sides of the vWire, as explained above. The source MAC on port 1.1 and 1.2 are different. Since the Bypass Tap is located right behind the customer's edge firewall, lengthy and time-consuming tests on the live system are not an option, since it would result in a massive service interruption. Therefore, a BIG-IP i5800 (the same model as the customer's) was set up as SSL Orchestrator and a vWire configuration was build in my employers lab. The vWire configuration can be found in this guide (https://clouddocs.f5.com/sslo-deployment-guide/chapter2/page2.7.html). INFO: For those not familiar with vWire: "Virtual wire … creates a layer 2 bridge across the defined interfaces. Any traffic that does not match a topology listener will pass across this bridge." Lab Topology The following topology was used for the lab: I build a vWire configuration on the SSL Orchestrator, as in the customer's environment. A Linux system with Scapy installed was connected to Interface 1.1. With Scapy TCP, UDP and ARP packets can be crafted or sent like a replay from a Wireshark capture. Interface 1.3 was connected to another Linux system that should receive the ARP packets. All tcpdumps were captured on the F5 and analyzed on the admin system (not plotted). Validating vWire Configuration To check the functionality of the F5 and the vWire configuration, two tests were performed. A replay of the Healthcheck packets from the Bypass Tap and a test with RFC-compliant ARP requests. Use Scapy to resend the faulty packets First, I used Wireshark to extract a single packet from packet analysis we took in the customer environment and saved it to a pcap file. I replayed this pcap file to the F5 with Scapy. The sendp() function will work at layer 2, it requires the parameters rdpcap (location of the pcap file for replay) and iface (which interface it shall use for sending). webserverdude@tux480:~$ sudo scapy -H WARNING: IPython not available. Using standard Python shell instead. AutoCompletion, History are disabled. Welcome to Scapy (2.5.0) >>> sendp(rdpcap("/home/webserverdude/cusomter-case/bad-example.pcap"),iface="enp0s31f6") . Sent 1 packets. This test confirmed the behavior that was observed in the customer's environment. The F5 BIG-IP does not forward this packet. Use PING and Scapy to send RFC-compliant ARP packets To create RFC-compliant ARP requests, I first sent an ARP request (opcode 1) through the vWire via PING command. As expected, this was sent through the vWire. To ensure that this also works with Scapy, I also resent this packet with Scapy. >>> sendp(rdpcap("/home/webserverdude/cusomter-case/good-example.pcap"),iface="enp0s31f6") . Sent 1 packets. In the Wireshark analysis it can be seen that this packet is incoming on port 1.1 and then forwarded to port 1.3 through the vWire. Solving the issue with the help of the vendor It became evident that the BIG-IP was dropping ARP packets that failed to meet RFC compliance, rendering the Bypass Tap from this particular vendor seemingly incompatible with the BIG-IP. Following my analysis, the vendor was able to develop and provide a new firmware release addressing this issue. To verify that the issue was resolved in this firmware release, my customer's setup, the exact same model of the Bypass Tap and a BIG-IP i5800, were deployed in my lab, where the new firmware underwent thorough testing. With this approach I could test the functionality and compatibility of the systems under controlled conditions. In this Wireshark analysis it can be seen that the Healthcheck packets are incoming on port 1.1 and then forwarded to port 1.3 through the vWire (marked in green) and also the other way round, coming in on port 1.3 and then forwarded to port 1.1 (marked in pink). Also now you can see that the packet is a proper gratuitous ARP reply (https://wiki.wireshark.org/Gratuitous_ARP). Because the Healthcheck packets were not longer dropped by the BIG-IP, but were forwarded through the vWire the Bypass Tap subsequently marked the BIG-IP as healthy and available. The new firmware resolved the issue. Consequently, my customer could confidently proceed with this project, free from the constraints imposed by the compatibility issue.
Office 365 Data Group builder using F5 Python SDK
Problem this snippet solves: Python script which pulls O365 URL's/IP's from microsoft published XML formatted list, parses into python dictionaries formatted for F5 data group, and uses F5 Python SDK to create or update data groups on BIG-IP for each MS product. The script will check if the data group already exists, and if so update based on changes. Once the data groups are installed they can used/referenced by any F5 feature/configuration that can reference a data group. How to use this snippet: Python version: 3.6 Python Modules Required: * f5-sdk * requests * f5-icontrol-rest * xmltodict Update script variables for your environment: BIGIP_ADDRESS = ' ' BIGIP_USER = ' ' BIGIP_PASS = ' ' to execute: python3 o365_dg_builder.py Code : import requests import xmltodict from collections import OrderedDict from f5.bigip import ManagementRoot from icontrol.exceptions import iControlUnexpectedHTTPError import argparse MS_URL = 'https://support.content.office.net/en-us/static/O365IPAddresses.xml' BIGIP_ADDRESS = '192.168.15.165' BIGIP_USER = 'admin' BIGIP_PASS = 'admin' def load_bigip(f5_host, f5_user, f5_pass): bigip = ManagementRoot(f5_host, f5_user, f5_pass) return bigip def get_o365_data(): o365data = requests.get(MS_URL) records = [] pdata = xmltodict.parse(o365data.content) for product in pdata['products']['product']: prod_name = product['@name'] if isinstance(product['addresslist'],list): for addrlist in product['addresslist']: record = {} if 'address' in addrlist.keys(): record['name'] = "{}_{}_{}".format('MS',prod_name, addrlist['@type']) if '@type' in addrlist.keys(): if addrlist['@type'] == 'URL': record['type'] = 'string' elif (addrlist['@type'] == 'IPv4' or addrlist['@type'] == 'IPv6'): record['type'] = 'ip' # MS XML has duplicate entries, unify the list before adding to record new_addr_l = list(set(addrlist['address'])) record['records'] = new_addr_l records.append(record) if isinstance(product['addresslist'],OrderedDict): if len(product['addresslist']['address']) > 0: record = {} record['name'] = "{}_{}_{}".format('MS', prod_name, product['addresslist']['@type']) if product['addresslist']['@type'] == 'URL': record['type'] = 'string' elif product['addresslist']['@type'] == 'IPv4' or product['addresslist']['@type'] == 'IPv6': record['type'] = 'ip' record['records'] = list(set(product['addresslist']['address'])) records.append(record) return records def compare_dg_entries(record, old_dg_l): # let's get the values to compare, and see what updates we need to make #compare lists addrs_to_rem = [x for x in old_dg_l if x not in record] addrs_to_add = [x for x in record if x not in old_dg_l] return addrs_to_rem, addrs_to_add def create_dg(bigip, record): if not bigip.tm.ltm.data_group.internals.internal.exists(name=record['name']): bigip.tm.ltm.data_group.internals.internal.create(**record) create_status = 'SUCCESS' else: old_dg_entries = bigip.tm.ltm.data_group.internals.internal.load(name=record['name']) # pull into a list old_dg_l = [] for entry in old_dg_entries.records: old_dg_l.append(entry['name']) addrs_to_rem, addrs_to_add = compare_dg_entries(record['records'], old_dg_l) if len(addrs_to_add)==0 and len(addrs_to_rem)==0: create_status = 'SKIPPED' else: if len(addrs_to_rem) > 0: for addr in addrs_to_rem: old_dg_entries.records.remove(next(x for x in old_dg_entries.records if x['name'] == addr)) if len(addrs_to_add) > 0: for addr in addrs_to_add: old_dg_entries.records.append({'name':addr}) old_dg_entries.update() # if you get time create log of updated entries create_status = 'UPDATED' return create_status # ----- try: bigip = load_bigip(BIGIP_ADDRESS, BIGIP_USER, BIGIP_PASS) except iControlUnexpectedHTTPError as e: print(e) exit(1) if bigip: records = get_o365_data() for record in records: try: status = create_dg(bigip, record) print("BIGIP: {}, Data Group: {}, Create Status {}".format(BIGIP_ADDRESS, record['name'], status)) except iControlUnexpectedHTTPError as e: status = 'FAILED' print("BIGIP: {}, Data Group: {}, Create Status {}, Error {}".format(BIGIP_ADDRESS, record['name'], status, e)) Tested this on version: 13.0
