Packet Analysis with Scapy and tcpdump: Checking Compatibility with F5 SSL Orchestrator
In this guide I want to demonstrate how you can use Scapy (https://scapy.net/) and tcpdump for reproducing and troubleshooting layer 2 issues with F5 BIG-IP devices. Just in case you get into a finger-pointing situation... Starting situation This is a quite recent story from the trenches: My customer uses a Bypass Tap to forward or mirror data traffic to inline tools such as IDS/IPS, WAF or threat intelligence systems. This ByPass Tap offers a feature called Network Failsafe (also known as Fail-to-Wire). This is a fault tolerance feature that protects the flow of data in the event of a power outage and/or system failure. It allows traffic to be rerouted while the inline tools (IDS/IPS, WAF or threat intelligence systems) are shutting down, restarting, or unexpectedly losing power (see red line namedFallbackin the picture below). Since the ByPass Tap itself does not have support for SSL decryption and re-encryption, an F5 BIG-IP SSL Orchestrator shall be introduced as an inline tool in a Layer 2 inbound topology. Tools directly connected to the Bypass Tap will be connected to the SSL Orchestrator for better visibility. To check the status of the inline tools, the Bypass Tap sends health checks through the inline tools. What is sent on one interface must be seen on the other interface and vice versa. So if all is OK (health check is green), traffic will be forwarded to the SSL Orchestrator, decrypted and sent to the IDS/IPS and the TAP, and then re-encrypted and sent back to the Bypass Tap. If the Bypass Tap detects that the SSL Orchestrator is in a failure state, it will just forward the traffic to the switch. This is the traffic flow of the health checks: Target topology This results in the following topology: Problem description During commissioning of the new topology, it turned out that the health check packets are not forwarded through the vWire configured on the BIG-IP. A packet analysis with Wireshark revealed that the manufacturer uses ARP-like packets with opcode 512 (HEX 02 00). This opcode is not defined in the RFC that describes ARP (https://datatracker.ietf.org/doc/html/rfc826), the RFC only describes the opcodes Request (1 or HEX 00 01) and Reply (2 or HEX 00 02). NOTE:Don't get confused that you see ARP packets on port 1.1 and 1.2. They are not passing through, the Bypass Tap is just send those packets from both sides of the vWire, as explained above. The source MAC on port 1.1 and 1.2 are different. Since the Bypass Tap is located right behind the customer's edge firewall, lengthy and time-consuming tests on the live system are not an option, since it would result in a massive service interruption. Therefore, a BIG-IP i5800 (the same model as the customer's) was set up as SSL Orchestrator and a vWire configuration was build in my employers lab. The vWire configuration can be found in this guide (https://clouddocs.f5.com/sslo-deployment-guide/chapter2/page2.7.html). INFO:For those not familiar with vWire: "Virtual wire … creates a layer 2 bridge across the defined interfaces. Any traffic that does not match a topology listener will pass across this bridge." Lab Topology The following topology was used for the lab: I build a vWire configuration on the SSL Orchestrator, as in the customer's environment. A Linux system with Scapy installed was connected to Interface 1.1. With Scapy TCP, UDP and ARP packets can be crafted or sent like a replay from a Wireshark capture. Interface 1.3 was connected to another Linux system that should receive the ARP packets. All tcpdumps were captured on the F5 and analyzed on the admin system (not plotted). Validating vWire Configuration To check the functionality of the F5 and the vWire configuration, two tests were performed. A replay of the Healthcheck packets from the Bypass Tap and a test with RFC-compliant ARP requests. Use Scapy to resend the faulty packets First, I used Wireshark to extract a single packet from packet analysis we took in the customer environment and saved it to a pcap file. I replayed this pcap file to the F5 with Scapy. The sendp() function will work at layer 2, it requires the parametersrdpcap(location of the pcap file for replay) andiface(which interface it shall use for sending). webserverdude@tux480:~$ sudo scapy -H WARNING: IPython not available. Using standard Python shell instead. AutoCompletion, History are disabled. Welcome to Scapy (2.5.0) >>> sendp(rdpcap("/home/webserverdude/cusomter-case/bad-example.pcap"),iface="enp0s31f6") . Sent 1 packets. This test confirmed the behavior that was observed in the customer's environment. The F5 BIG-IP does not forward this packet. Use PING and Scapy to send RFC-compliant ARP packets To create RFC-compliant ARP requests, I first sent an ARP request (opcode 1) through the vWire via PING command. As expected, this was sent through the vWire. To ensure that this also works with Scapy, I also resent this packet with Scapy. >>> sendp(rdpcap("/home/webserverdude/cusomter-case/good-example.pcap"),iface="enp0s31f6") . Sent 1 packets. In the Wireshark analysis it can be seen that this packet is incoming on port 1.1 and then forwarded to port 1.3 through the vWire. Solving the issue with the help of the vendor It became evident that the BIG-IP was dropping ARP packets that failed to meet RFC compliance, rendering the Bypass Tap from this particular vendor seemingly incompatible with the BIG-IP. Following my analysis, the vendor was able to develop and provide a new firmware release addressing this issue. To verify that the issue was resolved in this firmware release, my customer's setup, the exact same model of the Bypass Tap and a BIG-IP i5800, were deployed in my lab, where the new firmware underwent thorough testing. With this approach I could test the functionality and compatibility of the systems under controlled conditions. In this Wireshark analysis it can be seen that the Healthcheck packets are incoming on port 1.1 and then forwarded to port 1.3 through the vWire (marked in green) and also the other way round, coming in on port 1.3 and then forwarded to port 1.1 (marked in pink). Also now you can see that the packet is a proper gratuitous ARP reply (https://wiki.wireshark.org/Gratuitous_ARP). Because the Healthcheck packets were not longer dropped by the BIG-IP, but were forwarded through the vWire the Bypass Tap subsequently marked the BIG-IP as healthy and available. The new firmware resolved the issue. Consequently, my customer could confidently proceed with this project, free from the constraints imposed by the compatibility issue.SSL Orchestrator Advanced Use Cases: Outbound SNAT Persistence
Introduction F5 BIG-IP is synonymous with "flexibility". You likely have few other devices in your architecture that provide the breadth of capabilities that come native with the BIG-IP platform. And for each and every BIG-IP product module, the opportunities to expand functionality are almost limitless. In this article series we examine the flexibility options of the F5 SSL Orchestrator in a set of "advanced" use cases. If you haven't noticed, the world has been steadily moving toward encrypted communications. Everything from web, email, voice, video, chat, and IoT is now wrapped in TLS, and that's a good thing. The problem is, malware - that thing that creates havoc in your organization, that exfiltrates personnel records to the Dark Web - isn't stopped by encryption. TLS 1.3 and multi-factor authentication don't eradicate malware. The only reasonable way to defend against it is to catch it in the act, and an entire industry of security products are designed for just this task. But ironically, encryption makes this hard. You can't protect against what you can't see. F5 SSL Orchestrator simplifies traffic decryption and malware inspection, and dynamically orchestrates traffic to your security stack. But it does much more than that. SSL Orchestrator is built on top of F5's BIG-IP platform, and as stated earlier, is abound with flexibility. SSL Orchestrator Use Case: Outbound SNAT Persistence It may not be the most obvious thing to think about persistence in the vein of outbound traffic. We are all groomed to accept that any given load balancer can handle persistence (or "affinity", or "stickiness") to backend servers. This is an important characteristic for sure. But in an outbound scenario, you don't load balance remote servers, so why on Earth would you need persistence? Well, I'm glad you asked. There indeed happens to be a somewhat unique, albeit infrequent use case where two different servers need to persist on YOUR IP address. The classic example is a site that requires federated authentication, where the service provider (SP) generates a token (perhaps a SAML auth request) and inside of that request the SP has embedded the client IP. The client receives this message and is redirected to the IdP to authenticate. But in this case the client is talking to the outside world through a forward proxy, and outbound source NAT (SNAT) could be required in this environment. That means there's a potential that the client IP address as seen from the two remote servers could be different. So if the IdP needs to verify the client IP based on what's embedded in the authentication request token, that could possibly fail. The good news here is that federated authentication doesn't normally require client IP verification, and there aren't many other similar use cases, but it can happen. The F5 BIG-IP, as with ANY proxy server, load balancer, or ADC device, clearly supports server affinity, and in a highly flexible way. But, as with ANY proxy server, load balancer, or ADC device, that doesn't apply to SNAT addresses. Nevertheless, the F5 BIG-IP can be configured to do this, which is exactly what this article is about. We're going to flex some BIG-IP muscle to derive a unique and innovative way to enable outbound SNAT persistence. What we're basically talking about is ensuring that a single internal client persists a single outbound SNAT IP address, when and where needed, and as long as possible. It's important to note here that we're not really talking about persistence in the same way you think about load balanced server affinity. With affinity, you're stapling a single (remote) client "session" to a single load balanced server. With SNAT persistence, you're stapling a single outbound SNAT IP to a single internal client so that all remote servers see that same source address. Same-same but different-different. To do this we'll need a SNAT pool and an iRule. We need the SNAT pool to define the SNAT addresses we can use. And since SNAT pools don't provide a persistence option like regular pools do, we'll use an iRule to provide the stickiness. It's also worth noting here, again since we're not really talking about load balancing stickiness, that the IP persistence mechanism in the iRule may not (likely will not) evenly distribute the IPs in the SNAT pool. Your best bet is to provide as many SNAT pool IPs as possible and reasonable. The good news here is that, because you're using a BIG-IP, you can define exactly how you assert that IP stickiness. In most cases, you'll probably just want to persist on the internal client IP, but you could also persist on: Client source address and remote server port Client source address and remote destination addresses Client source, day of the week, the year+month+day % mod 2, a hash of the word-of-the-day...and hopefully you get the idea. Lot's of options. To make this work, let's start with the SNAT pool. Navigate to Local Traffic -> Address Translation -> SNAT Pool List in the BIG-IP and click Create. In the Member List section, add as many SNAT IPs as you can afford. Remember, these are going to be IPs on your outbound VLAN, so in the same subnet as your outbound VLAN self-IP. Figure: SNAT pool list You don't need to assign the SNAT pool to anything directly. The iRule will handle that. And now onto the iRule. Navigate to Local Traffic -> iRules -> iRule List in the BIG-IP, and click Create. Copy the following into the iRule editor: when RULE_INIT { ## This iRule should be applied to your SSLO intercaption rule ending with in-t-4. catch { unset -nocomplain static::snat_ips } ## For each SNAT IP needed define the IP versus dynamically looking it up. ## These need to be in the real SNAT pool as well so ARP works. set static::snat_ips(1) 10.1.20.50 set static::snat_ips(2) 10.1.20.51 set static::snat_ips(3) 10.1.20.52 set static::snat_ips(4) 10.1.20.53 set static::snat_ips(5) 10.1.20.54 ## Set to how many SNAT IPs were added set static::array_size 5 } when CLIENT_ACCEPTED priority 100 { ## Select and uncomment only ONE of the below SNAT persistence options ## Persist SNAT based on client address only snat $static::snat_ips([expr {[crc32 [IP::client_addr]] % $static::array_size}]) ## Persist SNAT based on client address and remote port #snat $static::snat_ips([expr {[crc32 [IP::client_addr] [TCP::remote_port]] % $static::array_size}]) ## Persist SNAT based on client address and remote address #snat $static::snat_ips([expr {[crc32 [IP::client_addr] [IP::local_addr]] % $static::array_size}]) } Let's take a moment to explain what this iRule is actually doing, and it is fairly straightforward. In RULE_INIT, which fires ONCE when you update the iRule, the members of the defined SNAT pool are read into an array. Then a second static variable is created to store the size of the array. These values are stored as static, global variables. In CLIENT_ACCEPTED we set a priority of 100 to control the order of execution under SSL Orchestrator as there is already a CLIENT_ACCEPTED iRule event on the topology (we want our new event to run first). Below that you're provided with three choices for persistence: persist on source IP only, source IP and destination port, or source IP and destination IP. You'll want to uncomment only ONE of these. Each basically performs a quick CRC hash on the selected value, then calculates a modulus based on the array size. This returns a number within the size of the array, that is then applied as the index to the array to extract one of the array values. This calculation is always the same for the same input value(s), so effectively persisting on that value. The selected SNAT IP is then fed to the 'snat' command, and there you have it. As stated, you're probably only going to need the source-only persistence option. Using either of the others will pin a SNAT IP to a client IP and protocol port (ex. client IP:443 or client IP:80), or pin a SNAT IP to a specific host (ex. client IP:www.example.com), respectively. At the end of the day, you can insert any reasonable expression that will result in the selection of one of the values in the SNAT pool array, so the sky is really the limit here. The last step is easiest of all. You need to attach this iRule to your SSL Orchestrator topology. To do that. navigate to SSL Orchestrator -> Configuration in the UI, select the Interception Rules tab, and click to edit the respective outbound interception rule. Scroll to the bottom of this page, and under Resources, add the new iRule to the Selected column. The order doesn't matter. Click Deploy to complete the change, and you're done. You can do a packet capture on your outbound VLAN to see what is happening. tcpdump -lnni [outbound vlan] host 93.184.216.34 And then access https://www.example.com to test. For your IP address you should see a consistent outgoing SNAT IP. If you have access to a Linux client, you can add multiple IP addresses to an interface and test with each: ifconfig eth0:1 10.1.10.51 ifconfig eth0:2 10.1.10.52 ifconfig eth0:3 10.1.10.53 ifconfig eth0:4 10.1.10.54 ifconfig eth0:5 10.1.10.55 curl -vk https://www.example.com --interface 10.1.10.51 curl -vk https://www.example.com --interface 10.1.10.52 curl -vk https://www.example.com --interface 10.1.10.53 curl -vk https://www.example.com --interface 10.1.10.54 curl -vk https://www.example.com --interface 10.1.10.55 And again there you have it. In just a few steps you've been able to enable outbound SNAT persistence, and along the way you have hopefully recognized the immense flexibility at your command.SSL Orchestrator Advanced Use Cases: Client Certificate Constrained Delegation (C3D) Support
Introduction F5 BIG-IP is synonymous with "flexibility". You likely have few other devices in your architecture that provide the breadth of capabilities that come native with the BIG-IP platform. And for each and every BIG-IP product module, the opportunities to expand functionality are almost limitless. In this article series we examine the flexibility options of the F5 SSL Orchestrator in a set of "advanced" use cases. If you haven't noticed, the world has been steadily moving toward encrypted communications. Everything from web, email, voice, video, chat, and IoT is now wrapped in TLS, and that's a good thing. The problem is, malware - that thing that creates havoc in your organization, that exfiltrates personnel records to the Dark Web - isn't stopped by encryption. TLS 1.3 and multi-factor authentication don't eradicate malware. The only reasonable way to defend against it is to catch it in the act, and an entire industry of security products are designed for just this task. But ironically, encryption makes this hard. You can't protect against what you can't see. F5 SSL Orchestrator simplifies traffic decryption and malware inspection, and dynamically orchestrates traffic to your security stack. But it does much more than that. SSL Orchestrator is built on top of F5's BIG-IP platform, and as stated earlier, is abound with flexibility. SSL Orchestrator Use Case: Client Certificate Constrained Delegation (C3D) Using certificates to authenticate is one of the oldest and most reliable forms of authentication. While not every application supports modern federated access or multi-factor schemes, you'll be hard-pressed to find something that doesn't minimally support authentication over TLS with certificates. And coupled with hardware tokens like smart cards, certificates can enable one of the most secure multi-factor methods available. But certificate-based authentication has always presented a unique challenge to security architectures. Certificate "mutual TLS" authentication requires an end-to-end TLS handshake. When a server indicates a requirement for the client to submit its certificate, the client must send both its certificate, and a digitally-signed hash value. This hash value is signed (i.e. encrypted) with the client's private key. Should a device between the client and server attempt to decrypt and re-encrypt, it would be unable to satisfy the server's authentication request by virtue of not having access to the client's private key (to create the signed hash). This makes encrypted malware inspection complicated, often requiring a total bypass of inspection to sites that require mutual TLS authentication. Fortunately, F5 has an elegant solution to this challenge in the form of Client Certificate Constrained Delegation, affectionally referred to as "C3D". The concept is reasonably straightforward. In very much the same way that SSL forward proxy re-issues a remote server certificate to local clients, C3D can re-issue a remote client certificate to local servers. A local server can continue to enforce secure mutual TLS authentication, while allowing the BIG-IP to explicitly decrypt and re-encrypt in the middle. This presents an immediate advantage in basic load balancing, where access to the unencrypted data allows the BIG-IP greater control over persistence. In the absence of this, persistence would typically be limited to IP address affinity. But of course, access to the unencrypted data also allows the content to be scanned for malicious malware. C3D actually takes this concept of certificate re-signing to a higher level though. The "constrained delegation" portion of the name implies a characteristic much like Kerberos constrained delegation, where (arbitrary) attributes can be inserted into the re-signed token, like the PAC attributes in a Kerberos ticket, to inform the server about the client. Servers for their part can then simply filter on client certificates issued by the BIG-IP (to prevent direct access), and consume any additional attributes in the certificate to understand how better to handle the client. With C3D you can maintain strong mutual TLS authentication all the way through to your servers, while allowing the BIG-IP to more effectively manage availability. And combined with SSL Orchestrator, C3D can enable decryption and inspection of content for malware inspection. This article describes how to configure SSL Orchestrator to enable C3D for inbound decrypted inspection. Arguably, most of what follows is the C3D configuration itself, as the integration with SSL Orchestrator is pretty simple. Note that Client Certificate Constrained Delegation (C3D) is included with Local Traffic Manager (LTM) 13.0 and beyond, but for integration with SSL Orchestrator you should be running 14.1 or later.To learn more about C3D, please see the following resources: K14065425: Configuring Client Certificate Constrained Delegation (C3D):https://support.f5.com/csp/article/K14065425 Manual Chapter: SSL Traffic Management:https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-system-ssl-administration/ssl-traffic-management.html#GUID-B4D2529E-D1B0-4FE2-8C7F-C3774ADE1ED2 SSL::c3d iRule reference - not required to use C3D, but adds powerful functionalityhttps://clouddocs.f5.com/api/irules/SSL__c3d.html The integration of C3D with SSL Orchestrator involves effectively replacing the client and server SSL profiles that the SSL Orchestrator topology creates, with C3D SSL profiles. This is done programmatically with an iRule, so no "non-strict" customization is required at the topology. Also note that an inbound (reverse proxy) SSL Orchestrator topology will take the form of a "gateway mode" deployment (a routed path to multiple applications), or "application mode" deployment (a single application instance hosted at the BIG-IP). See section 2.5 of the SSL Orchestrator deployment guide for a deeper examination of gateway and application modes: https://clouddocs.f5.com/sslo-deployment-guide/ The C3D integration is only applicable to application mode deployments. Configuration C3D itself involves the creation of client and server SSL profiles: Create a new Client SSL profile: Configuration Certificate Key Chain: public-facing server certificate and private key. This will be the certificate and key presented to the client on inbound request. It will likely be the same certificate and key defined in the SSL Orchestrator inbound topology. Client Authentication Client Certificate: require Trusted Certificate Authorities: bundle that can validate client certificate. This is a certificate bundle used to verify the client's certificate, and will contain all of the client certificate issuer CAs. Advertised Certificate Authorities: optional CA hints bundle. Not expressly required, but this certificate bundle is forwarded to the client during the TLS handshake to "hint" at the correct certificate, based on issuer. Client Certificate Constrained Delegation Client Certificate Constrained Delegation: Enabled Client Fallback Certificate (new in 15.1): option to select a default client certificate if client does not send one. This option was introduced in 15.1 and provides the means to select an alternate (local) certificate if the client does not present one. The primary use case here might be to select a "template" certificate, and use an iRule function to insert arbitrary attributes. OCSP: optional client certificate revocation control. This option defines an OCSP revocation provider for the client certificate. Unknown OCSP Response Control (new in 15.1): determines what happens when OCSP returns Unknown. If an OCSP revocation provider is selected, this option defines what to do if the response to the OCSP query is "unknown". Create a new Server SSL profile: Configuration Certificate: default.crt. The certificate and key here are used as "templates" for the re-signed client certificate. Key: default.key Client Certificate Configuration Delegation Client Certificate Constrained Delegation: Enabled CA Certificate: local forging CA cert. This is the CA certificate used to re-sign the client certificate. This CA must be trusted by the local servers. CA Key: local forging CA key CA Passphrase: optional CA passphrase Certificate Extensions: extensions from the real client cert to be included in the forged cert. This is the list of certificate extensions to be copied from the original certificate to the re-issued certificate. Custom Extension: additional extensions to copy to forged cert from real cert (OID). This option allows you to insert additional extensions to be copied, as OID values. Additional considerations: Under normal conditions, the F5 and backend server attempt to resume existing SSL sessions, whereby the server doesn’t send a Certificate Request message. The effect is that all connections to the backend server use the same forged client cert. There are two ways to get around this: Set a zero-length cache value in the server SSL profile, or Set server authentication frequency to ‘always’ in the server SSL profile CA certificate considerations: A valid signing CA certificate should possess the following attributes. While it can work in some limited scenarios, a self-signed server certificate is generally not an adequate option for the signing CA. keyUsage: certificate extension containing "keyCertSign" and "digitalSignature" attributes basicConstraints: certificate extension containing "CA = true" (for Yes), marked as "Critical" With the client and server SSL profiles built, the C3D configuration is basically done. To integrate with an inbound SSL Orchestrator topology, create a simple iRule and add it to the topology's Interception Rule configuration. Modify the SSL profile paths below to reflect the profiles you created earlier. ### Modify the SSL profile paths below to match real C3D SSL profiles when CLIENT_ACCEPTED priority 250 { ## set clientssl set cmd1 "SSL::profile /Common/c3d-clientssl" ; eval $cmd1 } when SERVER_CONNECTED priority 250 { ## set serverssl SSL::profile "/Common/c3d-serverssl" } In the SSL Orchestrator UI, either from topology workflow, or directly from the corresponding Interception Rule configuration, add the above iRule and deploy. The above iRule programmatically overrides the SSL profiles applied to the Interception Rule (virtual server), effectively enabling C3D support. At this point, the virtual server will request a client certificate, perform revocation checks if defined, and then mint a local copy of the client certificate to pass to the backend server. Optionally, you can insert additional certificate attributes via the server SSL profile configuration, or more dynamically through additional iRule logic: ### Added in 15.1 - allows you to send a forged cert to the server ### irrespective of the client side authentication (ex. APM SSO), ### and insert arbitrary values when SERVERSSL_SERVERCERT { ### The following options allow you to override/replace a submitted ### client cert. For example, a minted client certificate can be sent ### to the server irrespective of the client side authentication method. ### This certificate "template" could be defined locally in the iRule ### (Base64-encoded), pulled from an iFile, or some other certificate source. # set cert1 [b64decode "LS0tLS1a67f7e226f..."] # set cert1 [ifile get template-cert] ### In order to use a template cert, it must first be converted to DER format # SSL::c3d cert [X509::pem2der $cert1] ### Insert arbitrary attributes (OID:value) SSL::c3d extension 1.3.6.1.4.1.3375.3.1 "TEST" } If you've configured the above, a server behind SSL Orchestrator that requires mutual TLS authentication can receive minted client certificates from external users, and SSL Orchestrator can explicitly decrypt and pass traffic to the set of malware inspection tools. You can look at the certificate sent to the server by injecting a tcpdump packet between the BIG-IP and server, then open in Wireshark. tcpdump -lnni [VLAN] -Xs0 -w capture.pcap [additional filters] Finally, you might be asking what to do with certificate attributes injected by C3D, and really it depends on what the server can support. The below is a basic example in an Apache config file to block a client certificate that doesn't contain your defined attribute. <Directory /> SSLRequire "HTTP/%h" in PeerExtList("1.3.6.1.4.1.3375.3.1") RewriteEngine on RewriteCond %{SSL::SSL_CLIENT_VERIFY} !=SUCCESS RewriteRule .? - [F] ErrorDocument 403 "Delegation to SPN HTTP/%h failed. Please pass a valid client certificate" </Directory> And there you have it. In just a few steps you've configured your SSL Orchestrator to integrate with Client Certificate Constrained Delegation to support mutual TLS authentication, and along the way you have hopefully recognized the immense flexibility at your command. Updates As of F5 BIG-IP 16.1.3, there are some new C3D capabilities: C3D has been updated to encode and return the commonName (CN) found in the client certificate subject field in printableString format if possible, otherwise the value will be encoded as UTF8. C3D has been updated to support inserting a subject commonName (CN) via 'SSL::c3d subject commonName' command: when CLIENTSSL_HANDSHAKE { if {[SSL::cert count] > 0} { SSL::c3d subject commonName [X509::subject [SSL::cert 0] commonName] } } C3D has been updated to support inserting a Subject Alternative Name (SAN) via 'SSL::c3d extention SAN' command: when CLIENTSSL_HANDSHAKE { SSL::c3d extension SAN "DNS:*.test-client.com, IP:1.1.1.1" } C3D has been updated to add the Authority Key Identifier (AKI) extension to the client certificate if the CA certificate has a Subject Key Identifier (SKI) extension. Another interesting use case is copying the real client certificate Subject Key Identifier (SKI) to the minted client certificate. By default, the minted client certificate will not contain an SKI value, but it's easy to configure C3D to copy the origin cert's SKI by modifying the C3D server SSL profile. In the "Custom extension" field of the C3D section, add 2.5.29.14 as an available extension. As of F5 BIG-IP 17.1.0 (SSL Orchestrator 11.0), C3D has been integrated natively.Now, for a deployed Inbound topology, the C3D SSL profiles are listed in the Protocol Settings section of the Interception Rules tab. You can replace the client and server SSL profiles created by SSL Orchestrator, with C3D SSL profiles in the Interception Rules tab to support C3D. The C3D support is now extended to both Gateway and Application modes.SSL Orchestrator Advanced Use Cases: Integrating SOCKS Proxy
Introduction When we talk about security, and in particular, malware defenses, we spend a lot of time focusing on the capabilities of a product or an entire security stack. We want to know what types of malware a solution covers to get a sense of how effective that solution is. But beyond that, and something that’s not always obvious, is exactly how a solution “sees” the traffic. Now I’m not talking about decryption. Clearly that needs to happen somewhere in the stack for the malware solutions to actually be useful. What I’m referring to though is how traffic actually gets TO the security stack. For an Internet-facing web application that’s usually obvious – it’s routed to an IP address, it's TCP-based, it's wrapped in TLS encryption, and it's application layer HTTP. Client browsers send a request that’s routed across the Internet, arriving at a firewall, and then transported internally to a web server, maybe passing through a load balancer, proxy server, WAF and/or DDoS solution to get there. In any case, this describes a common “reverse proxy” scenario. But this is not the ONLY way that malware security solutions are employed. For example, in an enterprise scenario, it is just as important to apply defenses to the outbound flow for traffic leaving the network to the Internet (or perhaps other business units). But in a forward proxy, the how isn’t always as obvious. To get traffic to the security stack, SSL Orchestrator natively supports inbound and outbound flows, layer 3 (standard routing), layer 2 (bump-in-the-wire), and explicit HTTP proxy. You can also turn on WCCP, BGP, OSPF and others to enable dynamic routing, and WPAD and Proxy PAC files to enable dynamic proxying. You can deploy SSL Orchestrator inline of the traffic flow, or “on-a-stick”. You can even deploy SSL Orchestrator as a load balanced solution for greater scale, using another BIG-IP or even ECMP*. Truth be told, most other TLS visibility products only handle a small subset of those just listed. But there is something I subtly didn’t mention. SSL Orchestrator supports explicit proxy for HTTP/HTTPS traffic. An explicit proxy is a type of proxy that the client is aware of and must target directly. The proxy creates a TCP connection to another server behind the firewall on the client’s behalf, usually employed when the client is not permitted to establish TCP connections to outside servers directly. This would be the proxy server settings in your browser configuration. The native HTTP explicit proxy support is fine for anything a browser needs to talk to, but there may be scenarios where a proxy server is required in an environment for other protocols. For that, the “Socket Secure” (SOCKS) protocol could be a reasonable option. SOCKS isn’t built into the SSL Orchestrator configuration, but BIG-IP has had it forever, and as you’ve no doubt heard me say a few times the flexibility of the BIG-IP platform prevails. In this article I’m going to show just how easy it is to enable a SOCKS proxy listener for SSL Orchestrator. Let’s go see what that looks like! SSL Orchestrator Use Case: Integrating SOCKS Proxy To start, it’s important to understand that SSL Orchestrator creates an HTTP explicit forward proxy with two virtual servers: The first is the actual proxy virtual server. This contains the IP address and port that the client will configure in its proxy settings. All proxied traffic from clients will enter through this virtual server. The second is the TCP tunnel virtual server. This is where SSL Orchestrator decryption and service chaining magic happens. This virtual server listens on an internal “tunnel” VLAN provided by the proxy virtual. Traffic enters through the proxy virtual and is forwarded (wrapped) through the TCP tunnel virtual. The TCP tunnel virtual server is basically identical to a transparent proxy, but that’s just listening internally for traffic. To create a SOCKS proxy implementation, we are simply going to: Create the SOCKS proxy virtual server configuration manually Create an SSL Orchestrator transparent proxy topology Add the internal proxy tunnel VLAN to the resulting transparent proxy virtual server(s) In the same way that the native HTTP explicit proxy works, traffic will enter the SOCKS proxy VIP and wrap around to the TCP tunnel (transparent proxy) virtual server that the SSL Orchestrator created for us. Licensing and Version Requirements The beauty here is that you simply need SSL Orchestrator to do this. The SOCKS proxy profile is a function of Local Traffic Manager, which is also included in the standalone SSL Orchestrator license. As for versions, your best bet is any SSL Orchestrator version 9.0 or higher. It will certainly work with earlier versions, but in those you’ll need to disable configuration strictness. Versions 9.0 and above remove this requirement. Configuring a SOCKS Proxy To build a SOCKS proxy virtual server, you’ll need a DNS resolver, TCP tunnel, SOCKS profile, and a virtual server: DNS resolver: In the BIG-IP UI, under Network – DNS Resolvers – DNS Resolver List, click Create. Give this DNS resolver a name and then click Finished. Now click to edit this resolver, move to the Forward Zones tab, and click Add. Name: enter a single period (“.”, without the quotes) Address: enter the IP address of a DNS nameserver Service Port: enter the listening port of that DNS nameserver (almost always 53) Click Add to complete. TCP tunnel: In the BIG-IP UI, under Network – Tunnels – Tunnel List, click Create. Give this tunnel a name. Profile: tcp-forward Click Finished to complete. SOCKS profile: In the BIG-IP UI, under Local Traffic – Profiles – Services – SOCKS, click Create. Give the SOCKS profile a name. Protocol Versions: select the required versions to support, usually just Socks5 DNS Resolver: select the previously created DNS resolver IPv6: enable this if IPv6 is required Route Domain: set this if a unique (non-zero) route domain is required Tunnel Name: select the previously create TCP tunnel Default Connect Handling: set to Deny Click Finished to complete SOCKS virtual server: In the BIG-IP UI, under Local Traffic – Virtual Servers – Virtual Server List, click Create. Give the virtual server a name. Source: 0.0.0.0/0 Destination Address/Mask: enter the proxy IP address that clients will access Service Port: enter the port for this SOCKS proxy instance SOCKS Profile: select the previously created SOCKS profile VLANs and Tunnels: select the client facing VLAN Source Address Translation: set to None Address Translation: set to enabled Port Translation: set to enabled Click Finished to complete. This is all that you really must do to create the SOCKS proxy configuration. The next step is to create the TCP tunnel virtual server(s), and for that, we’ll let SSL Orchestrator do the work. Configuring SSL Orchestrator You’re going to create an L3 Outbound SSL Orchestrator topology. This will minimally create one TCP forward proxy listener, but you also can create separate listeners for other StartTLS protocols (i.e., SMTP, IMAP, POP3, FTPS). In the interest of brevity, we won’t go through the entire topology configuration in detail. For deeper insights into all of the available settings, hop on over to the SSL Orchestrator deployment guide: https://clouddocs.f5.com/sslo-deployment-guide/. The following will serve as a skeleton for the full topology workflow. In the SSL Orchestrator UI, under Configuration, start a new topology configuration. Topology Properties: provide a name and select L3 Outbound. SSL Configurations: set your CA certificate key chain. Services List: create the security services Service Chain List: create the service chain(s) and attach the services as required Security Policy: adjust the security policy as required Interception Rule: select any VLAN (we will modify this later).Optionally select addition L7 Interception Rules for FTP, IMAP, POP3, and SMTP. Egress Settings: select SNAT as required, andselect gateway settings as required Review the settings and then deploy this configuration. If you’re on a version of SSL Orchestrator older than 9.0, you will need to disable strictness on the new topology. This is the little lock icon to the far right of the listed topology. Note that you’ll need to leave this unlocked, and any changes you make outside of the SSL Orchestrator configuration could be overwritten if you re-deploy the topology. In versions 9.0 and higher, the strictness lock has been removed, so you can freely make the required SOCKS proxy modifications. Once the SSL Orchestrator topology is deployed, head over to Local Traffic – Virtual Servers – Virtual Server List in the BIG-IP UI. You will potentially see a lot of new virtual servers here. Most will have names that correspond to the security services you created, prefixed with “ssloS_”. But you’re looking for virtual servers that are prefixed with “sslo_” and then the name of the topology you just created. For example, if you named the topology “test”, you would see a virtual server named “sslo_test-in-t-4”. This is the standard transparent proxy (TCP tunnel) virtual. If you opted to create the additional L7 interception rules, you’ll see separate virtual servers for each of these. Again, for example, “sslo_test-pop3-4”, would represent the POP3 tunnel listener. To enable the SOCKS proxy, edit each of these “sslo_” virtual servers and replace the existing selected VLAN with the TCP tunnel you created in the above SOCKS proxy configuration steps. That should be it. Configure a client to point to your SOCKS proxy and test. That traffic should arrive at the SOCKS proxy virtual server, and then wrap back through the respective TCP tunnel virtual. The SSL Orchestrator configuration on that tunnel will handle decryption and service chaining to the security stack. Testing your SOCKS Proxy Now, if you’re reading this article, you’re either the highly motivated, super inquisitive type, which is awesome by the way, and/or have a real need to implement a SOCKS proxy for your decrypted malware security stack. For the latter, you probably already have a set of tools that needs a proxy to get to the Internet, so you can use those to test. But if you don’t, I’ve taken the liberty to add a few below to get you started. Nothing fancy here, just some simple command line utilities to test your new SSL Orchestrator SOCKS proxy implementation. Let’s also assume for examples below that the SOCKS proxy is configured to listen at 10.1.10.150:2222. We’ll use the set of test services at rebex.net to try these out. When a password is required, it’s ‘password’. SSH: ssh -o ProxyCommand='nc -X5 -x 10.1.10.150:2222 %h %p' demo@test.rebex.net SFTP sftp -o ProxyCommand='nc -v -x10.1.10.150:2222 %h %p' demo@test.rebex.net HTTP and HTTPS curl -vk --socks5 10.1.10.150:2222 https://www.example.com FTP and FTPS (implicit) curl --socks5 10.1.10.150:2222 -1 -v --disable-epsv --ftp-skip-pasv-ip -u demo --ftp-ssl -k ftps://test.rebex.net:990/pub/example/readme.txt -o readme.txt Caveats As you can tell, we’re just swapping out the HTTP explicit proxy front end for a SOCKS proxy front end. This layer’s job is primarily to establish the client’s server-side TCP socket (on the client’s behalf), and then forward everything through the established tunnel. All of the SSL Orchestrator magic happens inside that tunnel. The SOCKS proxy itself does not introduce any new constraints. It’ll work with mostly any protocol that an SSL Orchestrator L3 Outbound topology will handle. The only significant caveats are: BIG-IP cannot decrypt outbound SSH (and SFTP). It can, however, be service chained. BIG-IP's FTP profile does not support passive mode FTP through a SOCKS proxy configuration. To get FTP traffic to flow, remove the FTP virtual server configuration from the SSL Orchestrator topology (sslo_xxxx-ftp-4). This removes the FTP profile and allows the use of SSL Orchestrator as a SOCKS proxy for FTP. Note also that for a standard explicit outbound SSL Orchestrator topology, passive FTP works correctly with the FTP virtual server and FTP profile. Active mode FTP is not supported in any outbound proxy configuration. Resources Below is a small list of resources to further your SSL Orchestrator and SOCKS proxy education: Not that you’ll need any of this to implement the above solution, but if you just want to get nerdy with details on how SOCKS works, try this: https://en.wikipedia.org/wiki/SOCKS. If you have any questions about setting up SSL Orchestrator, the official deployment guide is your friend: https://clouddocs.f5.com/sslo-deployment-guide/. To see a full list of test services available at rebex.net, see the following: https://test.rebex.net. Summary And there you have it. In just a few steps you’ve configured your SSL Orchestrator solution to work as a SOCKS proxy, and along the way you have hopefully recognized some of the immense flexibility at your command.SSL Orchestrator Advanced Use Cases: DNS-over-HTTPS Detection
Introduction F5 BIG-IP is synonymous with "flexibility". You likely have few other devices in your architecture that provide the breadth of capabilities that come native with the BIG-IP platform. And for each and every BIG-IP product module, the opportunities to expand functionality are almost limitless. In this article series we examine the flexibility options of the F5 SSL Orchestrator in a set of "advanced" use cases. If you haven't noticed, the world has been steadily moving toward encrypted communications. Everything from web, email, voice, video, chat, and IoT is now wrapped in TLS, and that's a good thing. The problem is, malware - that thing that creates havoc in your organization, that exfiltrates personnel records to the Dark Web - isn't stopped by encryption. TLS 1.3 and multi-factor authentication don't eradicate malware. The only reasonable way to defend against it is to catch it in the act, and an entire industry of security products are designed for just this task. But ironically, encryption makes this hard. You can't protect against what you can't see. F5 SSL Orchestrator simplifies traffic decryption and malware inspection, and dynamically orchestrates traffic to your security stack. But it does much more than that. SSL Orchestrator is built on top of F5's BIG-IP platform, and as stated earlier, is abound with flexibility. Let us now explore one perspective of that flexibility and how SSL Orchestrator can be used to handle DNS-over-HTTPS and DNS-over-TLS. SSL Orchestrator Use Case: DNS-over-HTTPS and DNS-over-TLS Handling Despite the rapid evolution of Internet standards, and increasing amount of encryption, there's one aspect of our daily online world that hasn't really changed that much in its nearly 4 decades of breath. That of course is DNS. We don't tend to think about it often, which is probably why it hasn't evolved as much as other things, but it truly is the heart and backbone of everything we do online. That is, unless you want to memorize "2607:f8b0:400a:0804:0000:0000:0000:2004" as the way to get to Google, you had better have a working DNS. But DNS is inherently insecure. It's been shown to be vulnerable to all manner of attacks, and for the purposes of this discussion specifically, also exposes where you're going. That is, while the HTTP payload may be encrypted, there's still that (visible) DNS request that goes out first. That's not to say that there haven't been any improvements though. DNSSEC was developed to help secure DNS and prevent spoofing, but in the many years since its introduction it still isn't as widespread as we'd have hoped. DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) are more recent developments that focus mainly on the privacy aspect of DNS communications (or lack thereof). With DoH and DoT, clients and servers forego the typical DNS protocol request over UDP or TCP port 53 and embed the request inside an encrypted HTTPS or pure TLS connection. But...while that sounds pretty cool, there can be additional consequences to encrypting DNS, both good and bad: [Good] DoH and DoT can protect against ISPs that use your DNS information for targeted advertising. [Bad] DoH and DoT can effectively blind local DNS security and filtering controls. DNS monitoring is often an effective defense against spam and malware infections. [Bad] DoH and DoT providers are public services (ex. Cloudflare and Google), so while DNS is being protected from eavesdropping along the way, the providers have access to your DNS requests. The US National Security Agency (NSA) actually published a warning on this: https://media.defense.gov/2021/Jan/14/2002564889/-1/-1/0/CSI_ADOPTING_ENCRYPTED_DNS_U_OO_102904_21.P... The basic idea in this document is that while your DNS privacy is a good thing, encrypted DoH and DoT can also mask malware command-and-control. If you strictly follow the NSA guidance, you could either block DoH and/or DoT altogether, or force users to only use a local enterprise DoH resolver. If you don't have access to a local DoH resolver though, what else could you do? Well, I'm so glad you asked. DoH and DoT are of course encrypted, so unless you disable either protocol at the client, set up a local DoH resolver and force clients to it, or attempt to do encrypted analysis to find DoH traffic, you need to decrypt and inspect. And as it turns out, F5 has an elegant solution to do just that. In this blog post I am going to present a few solutions on how to handle these protocols through SSL Orchestrator decrypted analysis. The protocol implementations of DoH and DoT are a little different, so I will address them separately. Handling DNS-over-HTTPS For all intents and purposes, DoH looks and feels like normal HTTPS traffic. There are some semi-unique patterns you might be able to follow to infer that something is DoH rather than regular HTTPS (i.e., packet size, frequency), but it’s never going to be completely accurate. You could also potentially filter on the IP addresses of the known public DoH services, but that list is rather large and growing: https://github.com/curl/curl/wiki/DNS-over-HTTPS, and clients can ultimately choose the service they point to. If you’re serious about actually detecting DoH traffic with reasonable accuracy, then there’s no better way than through decrypted analysis. Now, once you've decrypted HTTPS and detected that it's DoH, there are a number of things you can do: Simply detect and block: there are actually three forms of DoH requests – Wireformat GET and POST, and JSON (detailed here:https://developers.cloudflare.com/1.1.1.1/dns-over-https), though the vast majority of clients use the Wireformat GET and POST methods. In any case, this option follows the NSA's first recommendation to simply detect and block DoH requests. A browser that receives a reject on a DoH request will natively revert to normal TCP/UDP DNS to allow your local DNS security tools to do what they do best. Detect and log: you may simply want to log that DoH is passing through your network, who's asking, and what they're asking for as an extension to your current DNS monitoring protocols. Detect and blackhole: also, as a possible extension to your current DNS protection protocols, you may need to block some DoH requests from happening. As mentioned earlier, if you simply block DoH, the client will revert to DNS. One of the best ways to block a DoH/DNS request is to provide a good but fake response instead, a technique called "blackholing". In this approach, you can flag specific hostnames (matching on URL categories) to provide a blackhole response. Let’s now look at each of the above options in turn. The first option is actually pretty straightforward. You just need to add the below iRule to an SSL Orchestrator outbound layer 3 topology*, on the “-in-t” TCP tunnel virtual server. On any decrypted HTTPS traffic, the iRule examines the HTTP request methods for the three signature DoH methods and sends a reject on a match. Easy peasy. when HTTP_REQUEST priority 750 { if { ( [HTTP::method] equals "GET" and [HTTP::header exists "accept"] and [HTTP::header "accept"] equals "application/dns-json" ) or \ ( [HTTP::method] equals "GET" and [HTTP::header exists "accept"] and [HTTP::header "accept"] equals "application/dns-message" ) or \ ( [HTTP::method] equals "POST" and [HTTP::header exists "content-type"] and [HTTP::header "content-type"] equals "application/dns-message" ) } { reject } } The second two options above are available in a single iRule here:https://github.com/f5devcentral/sslo-script-tools/tree/main/sslo-dns-over-https-detection. As with the previous, import the iRule to the BIG-IP, then associate that iRule with an SSL Orchestrator outbound layer 3 topology*. You'll need to make a few adjustments in the iRule to suit for your environment, and those are all presented as simple static variable assignments: static::LOCAL_LOG: enable or disable local Syslog logging (0 = off, 1 = on). static::HSL: enable or disable remote high-speed logging (0 = off, 1 = on). static::URLDB_LICENSED: set to 1 (on) if you've licensed and provisioned the subscription based URLDB, or 0 (off) if you have not and want to just use custom URL categories for DoH blackholing. static::BLACKHOLE_URLCAT: leave empty to disable or add a list of URL categories to search for URL-based DoH blackholing (ex. /Common/block-doh-urls) static::BLACKHOLE_RTYPE_A: enable or disable blackholing matched A record requests (0 = off, 1 = on). static::BLACKHOLE_RTYPE_AAAA: enable or disable blackholing matched AAAA record requests (0 = off, 1 = on). static::BLACKHOLE_RTYPE_TXT: enable or disable blackholing matched TXT record requests (0 = off, 1 = on). You can quickly test the solution by enabling DoH support in a browser. Please refer to the following instructions for each: https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-https/encrypted-dns-browsers/. With the static::LOCAL_LOG value set to 1 (enabled), you can tail the BIG-IP LTM log (tail -f /var/log/ltm) and watch the DoH traffic flow across the SSL Orchestrator topology. 10.1.10.50:59804-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:www.google.com 10.1.10.50:59804-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:encrypted-tbn0.gstatic.com 10.1.10.50:59804-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:fonts.gstatic.com 10.1.10.50:59812-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:www.gstatic.com 10.1.10.50:59812-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:id.google.com 10.1.10.50:59812-104.18.42.171:443 :: DoH (WireFormat POST) Request: A:play.google.com Handling DNS-over-TLS DNS-over-TLS (DoT) is essentially plain DNS wrapped in TLS. If you decrypt it, you’ll see a regular DNS request. DoT by standard, travels on TCP port 853, so that’s one very simple way to block DoT traffic, by just blocking that port. But if you wanted to actually log the DoT requests flowing through the BIG-IP, or blackhole some URLs, you again have to decrypt. The aforementioned DoH logging iRule can also handle DoT requests, inspecting any decrypted TCP:853 traffic: https://github.com/f5devcentral/sslo-script-tools/tree/main/sslo-dns-over-https-detection. The same static variable flags apply. Considering DoH/DoT proxy options I would be remiss in not mentioning options to actually proxy DoH and DoT traffic, and BIG-IP 16.1 introduced two new capabilities: DoH proxy – where the BIG-IP proxies client DoH requests to external DoH resources. You get to explicitly decrypt here, so you get another opportunity to log and control the DNS. DoH server – where the BIG-IP proxies client DoH requests to external TPC or UDP DNS resources. As with the proxy option, you get to decrypt the client’s DoH request. You can find out more about both of these capabilities here: https://support.f5.com/csp/article/K05451012 To proxy DoT to DoT, you simply need a TCP:853 virtual server with client and server SSL profiles. There are also various programmatic options to convert DoT to DNS, DNS to DoT, and DNS to DoH. But an important consideration here is that to proxy you would need clients to direct their DNS/DoT/DoH requests to your local resource. In some cases, you can do that through DHCP, or through enterprise browser policy management, but other than simply blocking outbound access to these protocols, it can be non-trivial to prevent clients from trying to use any of the external providers. That is specifically where decrypted analysis can be beneficial. Summary With an iRule and just a few configuration changes we have been able to implement a capability on top of SSL Orchestrator to log and actively control encrypted DNS. Decrypted inspection of DoH and DoT is just one of the many interesting benefits of and SSL Orchestrator solution. A demo of the above is available here:SSL Orchestrator Advanced Use Case: DNS over HTTPS * The iRules will also technically work for outbound layer 2 topologies.
SSL Orchestrator and ASM application policy om BigIP
Hello. I met a problem while trying combine SSL Orcestrator and ASM application policy together. Is anyone met the same? Or have any ideas about such deployment. I need to inspect unencrypted HTTP trafiic on inline antivirus and same time I need provide web application security on F5 Big IP. Thank you in advance.
SNI Routing with DNS Lookup
SNI (Server Name Indication) is an extension of the TLS protocol that is used by the client to indicate the hostname it is attempting to connect to at the start of the SSLhandshake. In the case of the BIG-IP the SNI can be used to select which client SSL profile should be applied to the ingress traffic but it requires at least one client SSL profile to be attached to the virtual server. In other words the SSL data must still be decrypted. This iRule which is suitable for some SSLO use cases does not require SSL traffic to be terminated (i.e. decrypted) on the BIG-IP. It only needs a generic SSL persistence profile to be attached to the virtual server and this just to circumvent MCP validation issues. Once the CLIENTSSL_CLIENTHELLO iRule event is triggered, the SNI can then be determined and used to steer traffic. In this example below, once the SNI is found, an attempt is made to resolve it using DNS and then route the request using the BIG-IP to a gateway which could (in the case of SSLO) be a Secure Web Gateway device. Note SSL::extensions -type <extension type value> returns the opaque extension byte array corresponding to the specified extension type value, or an empty string if not found. Extension Type 0 is the server name. The "node" command is useful if you want to send traffic to a specific IP/port combination that is not defined as a pool member iRule [root@sslo:Active:Standalone] config # tmsh list ltm rule bluecoat ltm rule bluecoat { when RULE_INIT { set static::resolver "8.8.8.8" } when CLIENT_ACCEPTED { HTTP::disable } when CLIENTSSL_CLIENTHELLO { set s_sni "NULL" set sni_exists [SSL::extensions exists -type 0] if {$sni_exists} { binary scan [SSL::extensions -type 0] @9a* s_sni log local0. "sni: $s_sni" if { [catch "RESOLV::lookup @$static::resolver -a ${s_sni}" addrs] } { log local0. "DNS resolution error" reject } else { log local0. "Resolved: [lindex ${addrs} 0]" node [lindex ${addrs} 0] [TCP::local_port] } } } }SSL Orchestrator Advanced Use Cases: Reducing Complexity with Internal Layered Architecture
Introduction Sir Isaac Newton said, "Truth is ever to be found in the simplicity, and not in the multiplicity and confusion of things". The world we live in is...complex. No getting around that. But at the very least, we should strive for simplicity where we can achieve it. As IT folk, we often find ourselves mired in the complexity of things until we lose sight of the big picture, the goal. How many times have you created an additional route table entry, or firewall exception, or virtual server, because the alternative meant having a deeper understanding of the existing (complex) architecture? Sure, sometimes it's unavoidable, but this article describes at least one way that you can achieve simplicity in your architecture. SSL Orchestrator sits as an inline point of presence in the network to decrypt, re-encrypt, and dynamically orchestrate that traffic to the security stack. You need rules to govern how to handle specific types of traffic, so you create security policy rules in the SSL Orchestrator configuration to describe and take action on these traffic patterns. It's definitely easy to create a multitude of traffic rules to match discrete conditions, but if you step back and look at the big picture, you may notice that the different traffic patterns basically all perform the same core actions. They allow or deny traffic, intercept or bypass TLS (decrypt/not-decrypt), and send to one or a few service chains. If you were to write down all of the combinations of these actions, you'd very likely discover a small subset of discrete "functions". As usual, F5 BIG-IP and SSL Orchestrator provide some innovative and unique ways to optimize this. And so in this article we will explore SSL Orchestrator topologies "as functions" to reduce complexity. Specifically, you can reduce the complexity of security policy rules, and in doing so, quite likely increase the performance of your SSL Orchestrator architecture. SSL Orchestrator Use Case: Reducing Complexity with Internal Layered Architectures The idea is simple. Instead of a single topology with a multitude of complex traffic pattern matching rules, create small atomic topologies as static functions and steer traffic to the topologies by virtue of "layered" traffic pattern matching. Granted, if your SSL Orchestrator configuration is already pretty simple, then please keep doing what you're doing. You've got this, Tiger. But if your environment is getting complex, and you're not quite convinced yet that topologies as functions is a good idea, here are a few additional benefits you'll get from this topology layering: Dynamic egress selection: topologies as functions can define different egress paths. Dynamic CA selection: topologies as functions can use different local issuing CAs for different traffic flows. Dynamic traffic bypass: certain types of traffic can be challenging to handle internally. For example, mutual TLS traffic can be bypassed globally with the "Bypass on client cert failure" option in the SSL configuration, but bypassing mutual TLS sites by hostname is more complex. A layered architecture can steer traffic (by SNI) through a bypass topology, with a service chain. More flexible pattern recognition: for all of its flexibility, SSL Orchestrator security policy rules cannot catch every possible use case. External traffic pattern recognition, via iRules or CPM (LTM policies) offer near infinite pattern matching options. You could, for example, steer traffic based on incoming tenant VLAN or route domain for multi-tenancy configurations. More flexible automation strategies: as iRules, data groups, and CPM policies are fully automate-able across many AnO platforms (ex. AS3, Ansible, Terraform, etc.), it becomes exceedingly easy to automate SSL Orchestrator traffic processing, and removes the need to manage individual topology security policy rules. Hopefully these benefits give you a pretty clear indication of the value in this architecture strategy. So without further ado, let's get started. Configuration Before we begin, I'd like to make note of the following caveats: While every effort has been made to simplify the layered architecture, there is still a small element of complexity. If you are at all averse to creating virtual servers or modifying iRules, then maybe this isn't for you. But as you are reading this in a forum dedicated to programmability, I'm guessing you the reader are ready for a challenge. This is a "field contributed" solution, so not officially supported by F5. This topology layering architecture is applicable to all modern versions of SSL Orchestrator, from 5.0 to 8.0. While topology layering can be used for inbound topologies, it is most appropriate for outbound. The configuration below also only describes the layer 3 implementation. But layer 2 layering is also possible. With this said, there are just a few core concepts to understand: Basic layered architecture configuration - how the pieces fit together The iRules - how traffic moves through the architecture Or the CPM policies - an alternative to iRules Note again that this is primarily useful in outbound topologies. Inbound topologies are typically more atomic on their own already. I will cover both transparent and explicit forward proxy configurations below. Basic layered architecture configuration A layered architecture takes advantage of a powerful feature of the BIG-IP called "VIP targeting". The idea is that one virtual server calls another with negligible latency between the two VIPs. The "external" virtual server is client-facing. The SSL Orchestrator topology virtual servers are thus "internal". Traffic enters the external VIP and traffic rules pass control to any of a number of internal "topology function" VIPs. You certainly don't have to use the iRule implementation presented here. You just need a client-facing virtual server with an iRule that VIP-targets to one or more SSL Orchestrator topologies. Each outbound topology is represented by a virtual server that includes the application server name. You can see these if you navigate to Local Traffic -> Virtual Servers in the BIG-IP UI. So then the most basic topology layering architecture might just look like this: when CLIENT_ACCEPTED { virtual "/Common/sslo_my_topology.app/sslo_my_topology-in-t-4" } This iRule doesn't do anything interesting, except get traffic flowing across your layered architecture. To be truly useful you'll want to include conditions and evaluations to steer different types of traffic to different topologies (as functions). As the majority of security policy rules are meant to define TLS traffic patterns, the provided iRules match on TLS traffic and pass any non-TLS traffic to a default (intercept/inspection) topology. These iRules are intended to simplify topology switching by moving all of the complexity of traffic pattern matching to a library iRule. You should then only need to modify the "switching" iRule to use the functions in the library, which all return Boolean true or false results. Here are the simple steps to create your layered architecture: Step 1: Build a set of "dummy" VLANs. A topology must be bound to a VLAN. But since the topologies in this architecture won't be listening on client-facing VLANs, you will need to create a separate VLAN for each topology you intend to create. A dummy VLAN is a VLAN with no interface assigned. In the BIG-IP UI, under Network -> VLANs, click Create. Give your VLAN a name and click Finished. It will ask you to confirm since you're not attaching an interface. Repeat this step by creating unique VLAN names for each topology you are planning to use. Step 2: Build a set of static topologies as functions. You'll want to create a normal "intercept" topology and a separate "bypass" topology, though you can create as many as you need to encompass the unique topology functions. Your intercept topology is configured as such: L3 outbound topology configuration, normal topology settings, SSL configuration, services, service chains No security policy rules - just the ALL rule with TLS intercept action (and service chain), and optionally remove the built-in Pinners rule Attach to a dummy VLAN (a VLAN with no assigned interfaces) Your bypass topology should then look like this: L3 outbound topology configuration, skip the SSL Configuration settings, optionally re-use services and service chains No security policy rules - just the ALL rule with TLS bypass action (and service chain) Attach to a separate dummy VLAN (a VLAN with no assigned interfaces) Note the name you use for each topology, as this will be called explicitly in the iRule. For example, if you name the topology "myTopology", that's the name you will use in each "call SSLOLIB::target" function (more on this in a moment) . If you look in the SSL Orchestrator UI, you will see that it prepends "sslo_" (ex. sslo_myTopology). Don't include the "sslo_" portion in the iRule. Step 3: Import the SSLOLIB iRule (attached here). Name it "SSLOLIB". This is the library rule, so no modifications are needed. The functions within (as described below) will return a true or false, so you can mix these together in your switching rule as needed. Step 4: Import the traffic switching iRule (attached here). You will modify this iRule as required, but the SSLOLIB library rule makes this very simple. Step 5: Create your external layered virtual server. This is the client-facing virtual server that will catch the user traffic and pass control to one of the internal SSL Orchestrator topology listeners. Type: Standard Source: 0.0.0.0/0 Destination: 0.0.0.0/0 Service Port: 0 Protocol: TCP VLAN: client-facing VLAN Address Translation: disabled Port Translation: disabled Default Persistence Profile: ssl iRule: the traffic switching iRule Note that the ssl persistence profile is enabled here to allow the iRules to handle client side SSL traffic without SSL profiles attached. Also make sure that Address and Port Translation are disabled before clicking Finished. Step 6: Modify the traffic switching iRule to meet your traffic matching requirements (see below). You have the basic layered architecture created. The only remaining step is to modify the traffic switching iRule as required, and that's pretty easy too. The iRules I'll repeat, there are near infinite options here. At the very least you need to VIP target from the external layered VIP to at least one of the SSL Orchestrator topology VIPs. The iRules provided here have been cultivated to make traffic selection and steering as easy as possible by pushing all of the pattern functions to a library iRule (SSLOLIB). The idea is that you will call a library function for a specific traffic pattern and if true, call a separate library function to steer that flow to the desired topology. All of the build instructions are contained inside the SSLOLIB iRule, with examples. SSLOLIB iRule: https://github.com/f5devcentral/sslo-script-tools/blob/main/internal-layered-architecture/transparent-proxy/SSLOLIB Switching iRule: https://github.com/f5devcentral/sslo-script-tools/blob/main/internal-layered-architecture/transparent-proxy/sslo-layering-rule The function to steer to a topology (SSLOLIB::target) has three parameters: <topology name>: this is the name of the desired topology. Use the basic topology name as defined in the SSL Orchestrator configuration (ex. "intercept"). ${sni}: this is static and should be left alone. It's used to convey the SNI information for logging. <message>: this is a message to send to the logs. In the examples, the message indicates the pattern matched (ex. "SRCIP"). Note, include an optional 'return' statement at the end to cancel any further matching. Without the 'return', the iRule will continue to process matches and settle on the value from the last evaluation. Example (sending to a topology named "bypass"): call SSLOLIB::target "bypass" ${sni} "DSTIP" ; return There are separate traffic matching functions for each pattern: SRCIP IP:<ip/subnet> SRCIP DG:<data group name> (address-type data group) SRCPORT PORT:<port/port-range> SRCPORT DG:<data group name> (integer-type data group) DSTIP IP:<ip/subnet> DSTIP DG:<data group name> (address-type data group) DSTPORT PORT:<port/port-range> DSTPORT DG:<data group name> (integer-type data group) SNI URL:<static url> SNI URLGLOB:<glob match url> (ends_with match) SNI CAT:<category name or list of categories> SNI DG:<data group name> (string-type data group) SNI DGGLOB:<data group name> (ends_with match) Examples: # SOURCE IP if { [call SSLOLIB::SRCIP IP:10.1.0.0/16] } { call SSLOLIB::target "bypass" ${sni} "SRCIP" ; return } if { [call SSLOLIB::SRCIP DG:my-srcip-dg] } { call SSLOLIB::target "bypass" ${sni} "SRCIP" ; return } # SOURCE PORT if { [call SSLOLIB::SRCPORT PORT:5000] } { call SSLOLIB::target "bypass" ${sni} "SRCPORT" ; return } if { [call SSLOLIB::SRCPORT PORT:1000-60000] } { call SSLOLIB::target "bypass" ${sni} "SRCPORT" ; return } # DESTINATION IP if { [call SSLOLIB::DSTIP IP:93.184.216.34] } { call SSLOLIB::target "bypass" ${sni} "DSTIP" ; return } if { [call SSLOLIB::DSTIP DG:my-destip-dg] } { call SSLOLIB::target "bypass" ${sni} "DSTIP" ; return } # DESTINATION PORT if { [call SSLOLIB::DSTPORT PORT:443] } { call SSLOLIB::target "bypass" ${sni} "DSTPORT" ; return } if { [call SSLOLIB::DSTPORT PORT:443-9999] } { call SSLOLIB::target "bypass" ${sni} "DSTPORT" ; return } # SNI URL match if { [call SSLOLIB::SNI URL:www.example.com] } { call SSLOLIB::target "bypass" ${sni} "SNIURLGLOB" ; return } if { [call SSLOLIB::SNI URLGLOB:.example.com] } { call SSLOLIB::target "bypass" ${sni} "SNIURLGLOB" ; return } # SNI CATEGORY match if { [call SSLOLIB::SNI CAT:$static::URLCAT_list] } { call SSLOLIB::target "bypass" ${sni} "SNICAT" ; return } if { [call SSLOLIB::SNI CAT:/Common/Government] } { call SSLOLIB::target "bypass" ${sni} "SNICAT" ; return } # SNI URL DATAGROUP match if { [call SSLOLIB::SNI DG:my-sni-dg] } { call SSLOLIB::target "bypass" ${sni} "SNIDGGLOB" ; return } if { [call SSLOLIB::SNI DGGLOB:my-sniglob-dg] } { call SSLOLIB::target "bypass" ${sni} "SNIDGGLOB" ; return } To combine these, you can use simple AND|OR logic. Example: if { ( [call SSLOLIB::DSTIP DG:my-destip-dg] ) and ( [call SSLOLIB::SRCIP DG:my-srcip-dg] ) } Finally, adjust the static configuration variables in the traffic switching iRule RULE_INIT event: ## User-defined: Default topology if no rules match (the topology name as defined in SSLO) set static::default_topology "intercept" ## User-defined: DEBUG logging flag (1=on, 0=off) set static::SSLODEBUG 0 ## User-defined: URL category list (create as many lists as required) set static::URLCAT_list { /Common/Financial_Data_and_Services /Common/Health_and_Medicine } CPM policies LTM policies (CPM) can work here too, but with the caveat that LTM policies do not support URL category lookups. You'll probably want to either keep the Pinners rule in your intercept topologies, or convert the Pinners URL category to a data group. A "url-to-dg-convert.sh" Bash script can do that for you. url-to-dg-convert.sh: https://github.com/f5devcentral/sslo-script-tools/blob/main/misc-tools/url-to-dg-convert.sh As with iRules, infinite options exist. But again for simplicity here is a good CPM configuration. For this you'll still need a "helper" iRule, but this requires minimal one-time updates. when RULE_INIT { ## Default SSLO topology if no rules match. Enter the name of the topology here set static::SSLO_DEFAULT "intercept" ## Debug flag set static::SSLODEBUG 0 } when CLIENT_ACCEPTED { ## Set default topology (if no rules match) virtual "/Common/sslo_${static::SSLO_DEFAULT}.app/sslo_${static::SSLO_DEFAULT}-in-t-4" } when CLIENTSSL_CLIENTHELLO { if { ( [POLICY::names matched] ne "" ) and ( [info exists ACTION] ) and ( ${ACTION} ne "" ) } { if { $static::SSLODEBUG } { log -noname local0. "SSLO Switch Log :: [IP::client_addr]:[TCP::client_port] -> [IP::local_addr]:[TCP::local_port] :: [POLICY::rules matched [POLICY::names matched]] :: Sending to $ACTION" } virtual "/Common/sslo_${ACTION}.app/sslo_${ACTION}-in-t-4" } } The only thing you need to do here is update the static::SSLO_DEFAULT variable to indicate the name of the default topology, for any traffic that does not match a traffic rule. For the comparable set of CPM rules, navigate to Local Traffic -> Policies in the BIG-IP UI and create a new CPM policy. Set the strategy as "Execute First matching rule", and give each rule a useful name as the iRule can send this name in the logs. For source IP matches, use the "TCP address" condition at ssl client hello time. For source port matches, use the "TCP port" condition at ssl client hello time. For destination IP matches the "TCP address" condition at ssl client hello time. Click on the Options icon and select "Local" and "External". For destination port matches the "TCP port" condition at ssl client hello time. Click on the Options icon and select "Local" and "External". For SNI matches, use the "SSL Extension server name" condition at ssl client hello time. For each of the conditions, add a simple "Set variable" action as ssl client hello time. Name the variable "ACTION" and give it the name of the desired topology. Apply the helper iRule and CPM policy to the external traffic steering virtual server. The "first" matching rule strategy is applied here, and all rules trigger on ssl client hello, so you can drag them around and re-order as necessary. Note again that all of the above only evaluates TLS traffic. Any non-TLS traffic will flow through the "default" topology that you identify in the iRule. It is possible to re-configure the above to evaluate HTTP traffic, but honestly the only significant use case here might be to allow or drop traffic at the policy. Layered architecture for an explicit forward proxy You can use the same logic to support an explicit proxy configuration. The only difference will be that the frontend layered virtual server will perform the explicit proxy functions. The backend SSL Orchestrator topologies will continue to be in layer 3 outbound (transparent proxy) mode. Normally SSL Orchestrator would build this for you, but it's pretty easy and I'll show you how. You could technically configure all of the SSL Orchestrator topologies as explicit proxies, and configure the client facing virtual server as a layer 3 pass-through, but that adds unnecessary complexity. If you also need to add explicit proxy authentication, that is done in the one frontend explicit proxy configuration. Use the settings below to create an explicit proxy LTM configuration. If not mentioned, settings can be left as defaults. Under SSL Orchestrator -> Configuration in the UI, click on the gear icon in the top right corner. This will expose the DNS resolver configuration. The easiest option here is to select "Local Forwarding Nameserver" and then enter the IP address of the local DNS service. Click "Save & Next" and then "Deploy" when you're done. Under Network -> Tunnels in the UI, click Create. This will create a TCP tunnel for the explicit proxy traffic. Profile: select tcp-forward Under Local Traffic -> Profiles -> Services -> HTTP in the UI, click Create. This will create the HTTP explicit proxy profile. Proxy Mode: Explicit Explicit Proxy: DNS Resolver: select the ssloGS-net-resolver Explicit Proxy: Tunnel Name: select the TCP tunnel created earlier Under Local Traffic -> Virtual Servers, click Create. This will create the client-facing explicit proxy virtual server. Type: Standard Source: 0.0.0.0/0 Destination: enter an IP the client can use to access the explicit proxy interface Service Port: enter the explicit proxy listener port (ex. 3128, 8080) HTTP Profile: HTTP explicit profile created earlier VLANs and Tunnel Traffic: set to "Enable on..." and select the client-facing VLAN Address Translation: enabled Port Translation: enabled Under Local Traffic -> Virtual Servers, click Create again. This will create the TCP tunnel virtual server. Type: Standard Source: 0.0.0.0/0 Destination: 0.0.0.0/0 Service Port: * VLANs and Tunnel Traffic: set to "Enable on..." and select the TCP tunnel created earlier Address Translation: disabled Port Translation: disabled iRule: select the SSLO switching iRule Default Persistence Profile: select ssl Note, make sure that Address and Port Translation are disabled before clicking Finished. Under Local Traffic -> iRules, click Create. This will create a small iRule for the explicit proxy VIP to forward non-HTTPS traffic through the TCP tunnel. Change "<name-of-TCP-tunnel-VIP>" to reflect the name of the TCP tunnel virtual server created in the previous step. when HTTP_REQUEST { virtual "/Common/<name-of-TCP-tunnel-VIP>" [HTTP::proxy addr] [HTTP::proxy port] } Add this new iRule to the explicit proxy virtual server. To test, point your explicit proxy client at the IP and port defined IP:port and give it a go. HTTP and HTTPS explicit proxy traffic arriving at the explicit proxy VIP will flow into the TCP tunnel VIP, where the SSLO switching rule will process traffic patterns and send to the appropriate backend SSL Orchestrator topology-as-function. Testing and Considerations Assuming you have the default topology defined in the switching iRule's RULE_INIT, and no traffic matching rules defined, all traffic from the client should pass effortlessly through that topology. If it does not, Ensure the named defined in the static::default_topology variable is the actual name of the topology, without the prepended "sslo_". Enable debug logging in the iRule and observe the LTM log (/var/log/ltm) for any anomalies. Worst case, remove the client facing VLAN from the frontend switching virtual server and attach it to one of your topologies, effectively bypassing the layered architecture. If traffic does not pass in this configuration, then it cannot in the layered architecture. You need to troubleshoot the SSL Orchestrator topology itself. Once you have that working, put the dummy VLAN back on the topology and add the client facing VLAN to the switching virtual server. Considerations The above provides a unique way to solve for complex architectures. There are however a few minor considerations: This is defined outside of SSL Orchestrator, so would not be included in the internal HA sync process. However, this architecture places very little administrative burden on the topologies directly. It is recommended that you create and sync all of the topologies first, then create the layered virtual server and iRules, and then manually sync the boxes. If you make any changes to the switching iRule (or CPM policy), that should not affect the topologies. You can initiate a manual BIG-IP HA sync to copy the changes to the peer. If upgrading to a new version of SSL Orchestrator (only), no additional changes are required. If upgrading to a new BIG-IP version, it is recommended to break HA (SSL Orchestrator 8.0 and below) before performing the upgrade. The external switching virtual server and iRules should migrate natively. Summary And there you have it. In just a few steps you've been able to reduce complexity and add capabilities, and along the way you have hopefully recognized the immense flexibility at your command.
