SSL Orchestrator Advanced Use Cases: Client Certificate Constrained Delegation (C3D) Support
Introduction F5 BIG-IP is synonymous with "flexibility". You likely have few other devices in your architecture that provide the breadth of capabilities that come native with the BIG-IP platform. And for each and every BIG-IP product module, the opportunities to expand functionality are almost limitless. In this article series we examine the flexibility options of the F5 SSL Orchestrator in a set of "advanced" use cases. If you haven't noticed, the world has been steadily moving toward encrypted communications. Everything from web, email, voice, video, chat, and IoT is now wrapped in TLS, and that's a good thing. The problem is, malware - that thing that creates havoc in your organization, that exfiltrates personnel records to the Dark Web - isn't stopped by encryption. TLS 1.3 and multi-factor authentication don't eradicate malware. The only reasonable way to defend against it is to catch it in the act, and an entire industry of security products are designed for just this task. But ironically, encryption makes this hard. You can't protect against what you can't see. F5 SSL Orchestrator simplifies traffic decryption and malware inspection, and dynamically orchestrates traffic to your security stack. But it does much more than that. SSL Orchestrator is built on top of F5's BIG-IP platform, and as stated earlier, is abound with flexibility. SSL Orchestrator Use Case: Client Certificate Constrained Delegation (C3D) Using certificates to authenticate is one of the oldest and most reliable forms of authentication. While not every application supports modern federated access or multi-factor schemes, you'll be hard-pressed to find something that doesn't minimally support authentication over TLS with certificates. And coupled with hardware tokens like smart cards, certificates can enable one of the most secure multi-factor methods available. But certificate-based authentication has always presented a unique challenge to security architectures. Certificate "mutual TLS" authentication requires an end-to-end TLS handshake. When a server indicates a requirement for the client to submit its certificate, the client must send both its certificate, and a digitally-signed hash value. This hash value is signed (i.e. encrypted) with the client's private key. Should a device between the client and server attempt to decrypt and re-encrypt, it would be unable to satisfy the server's authentication request by virtue of not having access to the client's private key (to create the signed hash). This makes encrypted malware inspection complicated, often requiring a total bypass of inspection to sites that require mutual TLS authentication. Fortunately, F5 has an elegant solution to this challenge in the form of Client Certificate Constrained Delegation, affectionally referred to as "C3D". The concept is reasonably straightforward. In very much the same way that SSL forward proxy re-issues a remote server certificate to local clients, C3D can re-issue a remote client certificate to local servers. A local server can continue to enforce secure mutual TLS authentication, while allowing the BIG-IP to explicitly decrypt and re-encrypt in the middle. This presents an immediate advantage in basic load balancing, where access to the unencrypted data allows the BIG-IP greater control over persistence. In the absence of this, persistence would typically be limited to IP address affinity. But of course, access to the unencrypted data also allows the content to be scanned for malicious malware. C3D actually takes this concept of certificate re-signing to a higher level though. The "constrained delegation" portion of the name implies a characteristic much like Kerberos constrained delegation, where (arbitrary) attributes can be inserted into the re-signed token, like the PAC attributes in a Kerberos ticket, to inform the server about the client. Servers for their part can then simply filter on client certificates issued by the BIG-IP (to prevent direct access), and consume any additional attributes in the certificate to understand how better to handle the client. With C3D you can maintain strong mutual TLS authentication all the way through to your servers, while allowing the BIG-IP to more effectively manage availability. And combined with SSL Orchestrator, C3D can enable decryption and inspection of content for malware inspection. This article describes how to configure SSL Orchestrator to enable C3D for inbound decrypted inspection. Arguably, most of what follows is the C3D configuration itself, as the integration with SSL Orchestrator is pretty simple. Note that Client Certificate Constrained Delegation (C3D) is included with Local Traffic Manager (LTM) 13.0 and beyond, but for integration with SSL Orchestrator you should be running 14.1 or later.To learn more about C3D, please see the following resources: K14065425: Configuring Client Certificate Constrained Delegation (C3D): https://support.f5.com/csp/article/K14065425 Manual Chapter: SSL Traffic Management: https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-system-ssl-administration/ssl-traffic-management.html#GUID-B4D2529E-D1B0-4FE2-8C7F-C3774ADE1ED2 SSL::c3d iRule reference - not required to use C3D, but adds powerful functionality https://clouddocs.f5.com/api/irules/SSL__c3d.html The integration of C3D with SSL Orchestrator involves effectively replacing the client and server SSL profiles that the SSL Orchestrator topology creates, with C3D SSL profiles. This is done programmatically with an iRule, so no "non-strict" customization is required at the topology. Also note that an inbound (reverse proxy) SSL Orchestrator topology will take the form of a "gateway mode" deployment (a routed path to multiple applications), or "application mode" deployment (a single application instance hosted at the BIG-IP). See section 2.5 of the SSL Orchestrator deployment guide for a deeper examination of gateway and application modes: https://clouddocs.f5.com/sslo-deployment-guide/ The C3D integration is only applicable to application mode deployments. Configuration C3D itself involves the creation of client and server SSL profiles: Create a new Client SSL profile: Configuration Certificate Key Chain: public-facing server certificate and private key. This will be the certificate and key presented to the client on inbound request. It will likely be the same certificate and key defined in the SSL Orchestrator inbound topology. Client Authentication Client Certificate: require Trusted Certificate Authorities: bundle that can validate client certificate. This is a certificate bundle used to verify the client's certificate, and will contain all of the client certificate issuer CAs. Advertised Certificate Authorities: optional CA hints bundle. Not expressly required, but this certificate bundle is forwarded to the client during the TLS handshake to "hint" at the correct certificate, based on issuer. Client Certificate Constrained Delegation Client Certificate Constrained Delegation: Enabled Client Fallback Certificate (new in 15.1): option to select a default client certificate if client does not send one. This option was introduced in 15.1 and provides the means to select an alternate (local) certificate if the client does not present one. The primary use case here might be to select a "template" certificate, and use an iRule function to insert arbitrary attributes. OCSP: optional client certificate revocation control. This option defines an OCSP revocation provider for the client certificate. Unknown OCSP Response Control (new in 15.1): determines what happens when OCSP returns Unknown. If an OCSP revocation provider is selected, this option defines what to do if the response to the OCSP query is "unknown". Create a new Server SSL profile: Configuration Certificate: default.crt. The certificate and key here are used as "templates" for the re-signed client certificate. Key: default.key Client Certificate Configuration Delegation Client Certificate Constrained Delegation: Enabled CA Certificate: local forging CA cert. This is the CA certificate used to re-sign the client certificate. This CA must be trusted by the local servers. CA Key: local forging CA key CA Passphrase: optional CA passphrase Certificate Extensions: extensions from the real client cert to be included in the forged cert. This is the list of certificate extensions to be copied from the original certificate to the re-issued certificate. Custom Extension: additional extensions to copy to forged cert from real cert (OID). This option allows you to insert additional extensions to be copied, as OID values. Additional considerations: Under normal conditions, the F5 and backend server attempt to resume existing SSL sessions, whereby the server doesn’t send a Certificate Request message. The effect is that all connections to the backend server use the same forged client cert. There are two ways to get around this: Set a zero-length cache value in the server SSL profile, or Set server authentication frequency to ‘always’ in the server SSL profile CA certificate considerations: A valid signing CA certificate should possess the following attributes. While it can work in some limited scenarios, a self-signed server certificate is generally not an adequate option for the signing CA. keyUsage: certificate extension containing "keyCertSign" and "digitalSignature" attributes basicConstraints: certificate extension containing "CA = true" (for Yes), marked as "Critical" With the client and server SSL profiles built, the C3D configuration is basically done. To integrate with an inbound SSL Orchestrator topology, create a simple iRule and add it to the topology's Interception Rule configuration. Modify the SSL profile paths below to reflect the profiles you created earlier. ### Modify the SSL profile paths below to match real C3D SSL profiles when CLIENT_ACCEPTED priority 250 { ## set clientssl set cmd1 "SSL::profile /Common/c3d-clientssl" ; eval $cmd1 } when SERVER_CONNECTED priority 250 { ## set serverssl SSL::profile "/Common/c3d-serverssl" } In the SSL Orchestrator UI, either from topology workflow, or directly from the corresponding Interception Rule configuration, add the above iRule and deploy. The above iRule programmatically overrides the SSL profiles applied to the Interception Rule (virtual server), effectively enabling C3D support. At this point, the virtual server will request a client certificate, perform revocation checks if defined, and then mint a local copy of the client certificate to pass to the backend server. Optionally, you can insert additional certificate attributes via the server SSL profile configuration, or more dynamically through additional iRule logic: ### Added in 15.1 - allows you to send a forged cert to the server ### irrespective of the client side authentication (ex. APM SSO), ### and insert arbitrary values when SERVERSSL_SERVERCERT { ### The following options allow you to override/replace a submitted ### client cert. For example, a minted client certificate can be sent ### to the server irrespective of the client side authentication method. ### This certificate "template" could be defined locally in the iRule ### (Base64-encoded), pulled from an iFile, or some other certificate source. # set cert1 [b64decode "LS0tLS1a67f7e226f..."] # set cert1 [ifile get template-cert] ### In order to use a template cert, it must first be converted to DER format # SSL::c3d cert [X509::pem2der $cert1] ### Insert arbitrary attributes (OID:value) SSL::c3d extension 1.3.6.1.4.1.3375.3.1 "TEST" } If you've configured the above, a server behind SSL Orchestrator that requires mutual TLS authentication can receive minted client certificates from external users, and SSL Orchestrator can explicitly decrypt and pass traffic to the set of malware inspection tools. You can look at the certificate sent to the server by injecting a tcpdump packet between the BIG-IP and server, then open in Wireshark. tcpdump -lnni [VLAN] -Xs0 -w capture.pcap [additional filters] Finally, you might be asking what to do with certificate attributes injected by C3D, and really it depends on what the server can support. The below is a basic example in an Apache config file to block a client certificate that doesn't contain your defined attribute. <Directory /> SSLRequire "HTTP/%h" in PeerExtList("1.3.6.1.4.1.3375.3.1") RewriteEngine on RewriteCond %{SSL::SSL_CLIENT_VERIFY} !=SUCCESS RewriteRule .? - [F] ErrorDocument 403 "Delegation to SPN HTTP/%h failed. Please pass a valid client certificate" </Directory> And there you have it. In just a few steps you've configured your SSL Orchestrator to integrate with Client Certificate Constrained Delegation to support mutual TLS authentication, and along the way you have hopefully recognized the immense flexibility at your command. Updates As of F5 BIG-IP 16.1.3, there are some new C3D capabilities: C3D has been updated to encode and return the commonName (CN) found in the client certificate subject field in printableString format if possible, otherwise the value will be encoded as UTF8. C3D has been updated to support inserting a subject commonName (CN) via 'SSL::c3d subject commonName' command: when CLIENTSSL_HANDSHAKE { if {[SSL::cert count] > 0} { SSL::c3d subject commonName [X509::subject [SSL::cert 0] commonName] } } C3D has been updated to support inserting a Subject Alternative Name (SAN) and Authority Info Access (AIA) via 'SSL::c3d extension' commands: when CLIENTSSL_HANDSHAKE { ## Insert Subject Alternative Name (SAN) value SSL::c3d extension SAN "DNS:*.test-client.com, IP:1.1.1.1" ### Insert Authority Info Access (AIA) value SSL::c3d extension AIA "ocsp,https://ocsp.entrust.net.com; caIssuer, https://aia.entrust.net/l1m-chain256.cer" } C3D has been updated to add the Authority Key Identifier (AKI) extension to the client certificate if the CA certificate has a Subject Key Identifier (SKI) extension. Another interesting use case is copying the real client certificate Subject Key Identifier (SKI) to the minted client certificate. By default, the minted client certificate will not contain an SKI value, but it's easy to configure C3D to copy the origin cert's SKI by modifying the C3D server SSL profile. In the "Custom extension" field of the C3D section, add 2.5.29.14 as an available extension. __________________________________________ As of F5 BIG-IP 17.1.0 (SSL Orchestrator 11.0), C3D has been integrated natively. Now, for a deployed Inbound topology, the C3D SSL profiles are listed in the Protocol Settings section of the Interception Rules tab. You can replace the client and server SSL profiles created by SSL Orchestrator, with C3D SSL profiles in the Interception Rules tab to support C3D. The C3D support is now extended to both Gateway and Application modes. __________________________________________ As of F5 BIG-IP 21.1, there are some new C3D capabilities: C3D adds a new option in the server SSL profile to enable overriding the certificate start date. This overrides the notBefore (valid-from) attribute of forged certificates to dynamically set the certificate's start time to align with the exact time of certificate forging. C3D adds a new option in the server SSL profile to enable caching of forged certificates. This reduces the need to repeatedly generate them for similar transactions, thereby improving efficiency during high-traffic scenarios, and reducing computational overhead in TLS proxy flows. C3D adds a new iRule command to keep or override the notBefore date in the forged certificate. when CLIENTSSL_HANDSHAKE { ## Update forged certificate lifespan SSL::c3d cert_lifespan [1-8760] } C3D adds a new iRule command to override the forged certificate lifespan. when CLIENTSSL_HANDSHAKE { ## Override the forged certificate notBefore date SSL::c3d cert_start_date [ override | original ] } C3D adds a new iRule command to inject X.509 keyUsage attributes into the forged certificate. when CLIENTSSL_HANDSHAKE { ## Injects KeyUsage constraints into the forged certificate SSL::c3d extension KU "digitalSignature, keyEncipherment" } C3D adds a new iRule command to inject X.509 extendedKeyUsage attributes into the forged certificate. when CLIENTSSL_HANDSHAKE { ## Injects extendedKeyUsage constraints into the forged certificate SSL::c3d extension EKU "serverAuth, clientAuth" } C3D adds a new iRule command to inject X.509 subjectDirectoryAttributes into the forged certificate. when CLIENTSSL_HANDSHAKE { ## Injects subjectDirectoryAttributes constraints into the forged certificate SSL::c3d extension SDA "dateOfBirth:20000101000000Z, countryOfCitizenship:US" } Note that certificate extensions enabled in the C3D server SSL profile configuration are copy operations. When enabled they copy the selected attributes from the origin certificate. The iRule commands are write operations. They allow you to specify new attribute values.
SSL Forward Proxy, iRules and Client Hello
Hi all, I am seeing odd behaviour using SSL fwd proxy (SSLO): My intention is to use the client hello (SNI) to influence SSSL profile selection. I have 2 SSSL profiles setup, let call them A and B For trusted connections (i.e. certs issuers in SSSL CA bundle) is am unable to extract the SNI from the initial CH, using the CLIENTSSL_CLIENTHELLO event and [SSL::extensions -type 0]. These are send to profile A based on SNI. I have pcaps showing the CH incoming to the F5. I assume this may have something to do with the 'verified handshake' functionality. It appears the test client browser keeps attempting connection and I see inconsistent results (some connections are reset, some succeed). In irule logs its apparent the SNI does eventually become available in the CLIENTSSL_CLIENTHELLO event. For untrusted/self signed etc this doesn't appear to happen, these are sent to Profile B (identical to A for testing purposes) so my assumption is the F5 is doing some kind of SNI processing (compare to CN's in trust store?) and then connecting to the server for 'verified handshake' before releasing the SNI into the CLIENTSSL_CLIENTHELLO event? I have seen an iRule that effectively disables SSL then parses the raw client hello for SNI, I expect this may work as it would intercept the raw CH so the F5 cannot interfere or do any server-side preamble, but I'd rather do this within the realms of defined events if possible... :-) Any suggestions or comments welcome! thanksWhat's new in BIG-IP v21.0?
Introduction In November of 2025 F5 released the latest version of BIG-IP software, v21.0. This release is packed with fixes and new features that enhance the F5 Application Delivery and Security Platform (ADSP). These changes complement the Delivery, Security and Deployment aspects of the ADSP. Demo Video: New SSL Orchestrator Features SNI Preservation SNI (Server Name Indication) Preservation is now supported for Inbound Gateway Mode. This preserves the client’s original SNI information as traffic passes through the reverse proxy, allowing backend TLS servers to access and use this information. This enables accurate application routing and supports security workflows like threat detection and compliance enforcement. Previous software versions required custom iRules to enable this functionality. Note: SNI preservation is enabled by default. However, if you have existing Inbound Gateway Topologies, you must redeploy them for the change to take effect. iRule Control for Service Entry and Return Previously, iRules were only available on the entry (ingress) side, limiting customization to traffic entering the Inspection Service. iRule control is now extended to the return-side traffic of Inspection Services. You can now apply iRules on both sides of an Inspection Service (L2, L3, HTTP). This enhancement provides full control over traffic entering and leaving the Inspection Service, enabling more flexible, powerful, and fine-grained traffic handling. The Services page will now include configuration for iRules on service entry and iRules on service return. A typical use-case for this feature is what we call Header Enrichment. In this case, iRules are used to add headers to the payload before sending it to the Inspection Service. The headers could contain the authenticated username/group membership of the person who initiated the connection. This information can be useful for Inspection Services for either logging, policy enforcement, or both. The benefit of this feature is that the authenticated username/group membership header can be removed from the payload on egress, preventing it from being leaked to origin servers. New Access Policy Manager (APM) Features Expanded Exclusion Support for Locked Client Mode Previously, APM-locked client mode allowed a maximum of 10 exclusions, preventing administrators from adding more than 10 destinations. This limitation has now been removed, and the exclusion list can contain more than 10 entries. OAuth Authorization Server Max Claims Data Support The max claim data size is set to 8kb by default, but a large claim size can lead to excessive memory consumption. You must allocate the right amount of memory dynamically as required based on claims configuration. New Features in BIG-IP v21.0.0 Control Plane Performance and Scalability Improvements The BIG-IP 21.0.0 release introduces significant improvements to the BIG-IP control plane, including better scalability and support for large-scale configurations (up to 1 million objects). This includes MCPD efficiency enhancements and eXtremeDB scale improvements. AI Data Delivery Optimize performance and simplify configuration with new S3 data storage integrations. Use cases include secure ingestion for fine-tuning and batch inference, high-throughput retrieval for RAG and embeddings generation, policy-driven model artifact distribution with observability, and controlled egress with consistent security and compliance. F5 BIG-IP optimizes and secures S3 data ingress and egress for AI workloads. Model Context Protocol (MCP) support for AI traffic Accelerate and scale AI workloads with support for MCP that enables seamless communication between AI models, applications, and data sources. This enhances performance, secures connections, and streamlines deployment for AI workloads. F5 BIG-IP optimizes and secures S3 data ingress and egress for AI workloads. Migrating BIG-IP from Entrust to Alternative Certificate Authorities Entrust is soon to be delisted as a certificate authority by many major browsers. Following a variety of compliance failures with industry standards in recent years, browsers like Google Chrome and Mozilla made their distrust for Entrust certificates public last year. As such, Entrust certificates issued on or after November 12, 2024, are deemed insecure by most browsers. Conclusion Upgrade your BIG-IP to version 21.0 today to take advantage of these fixes and new features that enhance the F5 Application Delivery and Security Platform (ADSP). These changes complement the Delivery, Security and Deployment aspects of the ADSP. Related Content SSL Orchestrator Release Notes BIG-IP Release Notes BLOG F5 BIG-IP v21.0: Control plane, AI data delivery and security enhancements Press Release F5 launches BIG-IP v21.0 Introduction to BIG-IP SSL Orchestrator
F5 BIG-IP SSL Orchestrator Layer 2 Services with rSeries & VELOS
Introduction F5 rSeries & VELOS are rearchitected, next-generation hardware platforms that scale application delivery performance and automate application services to address many of today’s most critical business challenges. F5 rSeries & VELOS are key components of the F5 Application Delivery and Security Platform (ADSP). rSeries & VELOS rely on a Kubernetes-based platform layer (F5OS) that is tightly integrated with F5 TMOS software. Going to a microservice-based platform layer allows rSeries & VELOS to provide additional functionality that was not possible in previous generations of F5 BIG-IP platforms. The introduction of a new tenant-based architecture changes many things, including how you configure BIG-IP. Some of these changes affect the network configuration for Inline Layer 2 Services. By default, BIG-IP tenants only have a small set of internal MAC addresses available to them. However, Layer 2 Services (or Bridging) require additional MAC addresses. You must assign an adequate number of MAC addresses to what is called a “MAC pool”. A single Layer 2 Service requires two unique MAC addresses. The MAC Pool must have sufficient MAC addresses based on the number of Layer 2 Services you need. The following KB articles contain additional information on configuring MAC Pools on a BIG-IP rSeries or VELOS platform: K000133655: MAC address assignment in VELOS and rSeries systems K000135389: Configure the MAC Block Size for an existing BIG-IP tenant on the VELOS and rSeries systems Demo Video F5OS Configuration Let’s review the Network configuration on F5OS for a BIG-IP Tenant. From Network Settings select VLANs. Here you can see I have 6 Interfaces configured with VLANs. There’s a Lan VLAN for connectivity from the internal network to the BIG-IP. A Wan VLAN for connectivity from the BIG-IP to the internet. Then there are 4 “L2” VLANs configured to support two Inline Layer 2 Services with SSL Orchestrator. From the Interfaces screen you can associate the VLANs with the physical Interfaces. Next, allocate the VLANs to your BIG-IP Tenant. This is also where you configure the MAC Pool Size for your current BIG-IP Tenant. The MAC Pool can only be changed when the Tenant is not running. From Tenant Management > Tenant Deployments, you can stop the current Tenant if it is already running. Do this with caution during a change window or prior to deployment. Check the box next to the name of the Tenant you wish to configure, “big-ip-kevin” in this example. Then click Configure. Click OK to stop the Tenant When it’s stopped click the name of the Tenant to edit the configuration. Note the VLANs that are allocated to this BIG-IP Tenant: Find the section on MAC Data/MAC Block Size. Set the allocation to Small (8), Medium (16), or Large (32) depending upon your needs. I set mine to Medium. A Small allocation would be sufficient for this deployment but I want to leave room to add more Layer 2 Services in the future. Click Save & Close Click OK to update the configuration You can Deploy the Tenant now that the changes have been made Click OK to Deploy F5 BIG-IP Configuration Minimal configuration is needed on the BIG-IP since F5OS handles the underlying physical interfaces and VLANs. Check the status of the VLANs from Network > VLANs. From here we can see the VLAN configuration from F5OS is reflected in the BIG-IP. Define any Self IPs from Network > Self IPs Now we’re ready to configure SSL Orchestrator. In the interest of time, I will skip to the Network and Services configuration. From Services List click Add Service Double-click on Generic Inline Layer 2 Under Network Configuration click Add Select the L2 VLANs for this Inline L2 Service. Click Done. Click Add again and select the L2 VLANs for this Inline L2 Service. Click Done. It should look like the following: Click Save at the bottom For the Interception Rule select the Lan VLAN under Ingress Network and move it to the right. Click Save & Next at the bottom The Network configuration is now complete. SSL Orchestrator is configured with a Generic Inline Layer 2 Service that contains two Layer 2 “servers” Conclusion F5 rSeries & VELOS are hardware platforms that scale application delivery performance and automate application services to address many of today’s most critical business challenges. They are key components of the F5 Application Delivery and Security Platform (ADSP). In this article, you learned how to configure MAC Pools on rSeries and VELOS in order to create Layer 2 Inline Services with SSL orchestrator. Related Content K000133655: MAC address assignment in VELOS and rSeries systems K000135389: Configure the MAC Block Size for an existing BIG-IP tenant on the VELOS and rSeries systems SSL Orchestrator CloudDocs: Creating an Inline Layer 2 Service F5 rSeries: Next-Generation Fully Automatable Hardware F5 VELOS: A Next-Generation Fully Automatable Platform
wccp configuration for SSL Orchestrator
Web Cache Communication Protocol (WCCP) is a Cisco-developed content-routing protocol that provides a mechanism to redirect traffic flows in real-time. It has built-in load balancing, scaling, fault tolerance, and service-assurance (failsafe) mechanisms. Cisco IOS Release 12.1 and later releases allow using either Version 1 (WCCPv1) or Version 2 (WCCPv2) of the protocol. Well that's a mouthful; to say basically that WCCP is a Cisco developed protocol designed to load balance traffic among proxy web cache servers. The beauty of it is it’s really easy to set up on a router (and Cisco Firepower) and can intercept outbound traffic and redirect it to the proxies. The proxies do not need to be in the network path. It’s basically a form of policy based routing. And if the proxy servers are down, the router will just continue to forward the traffic down the default route. This makes it relatively easy for SSL Orchestrator to receive traffic. But I can say there’s not exactly the greatest documentation from either organization. But it's really pretty simple. The fun fact is, once your device is registered with the WCCP group on the router, it just works. As in, it just starts sending any traffic that matches the ACL off to the router. Now as for HA. WCCP was designed to handle the HA. Right? I have a pool of web caches and I’m distributing the traffic among them. But if I set up the BIG-IP using its standard Active/Standby HA and configuration sync, there’s some additional thought that comes in to play. With a configuration where both devices in the BIG-IP HA pair and each designates its local self-ip as the local tunnel address. There can be a delay while the newly active device registers with the WCCP group on the router. It’s a short blip. But a blip nonetheless. But what about using the floating IP address? Isn't that used to provide a movable HA address? Yes. Yes, it is on a normal network segment. Similar to VRRP. 01070734:3: Configuration error: In wccp /Common/wccpsg service tunnel local address (192.168.8.222) cannot be a floating self IP So you’re denied from configuring the floating self-ip from being the target. The reason is, these are treated as tunnel interfaces and in the case of using the GRE configuration for WCCP, it is a tunnel! So, peering is done between each device individually in the HA group. That means, though, that only the active device will register. What that means for failover then is that the active device registers and the standby does not. When a failover event happens, the newly active device registers and the inactive device drops out. modify net wccp wccpsg { services add {90{ hash-fields {src-ip} port-type dest ports add { 443 } redirection-method l2 return-method l2 routers add { 192.168.8.128 } tunnel-local-address 192.168.8.105 tunnel-remote-addresses add { 192.168.8.128 } } } } The most important thing is that the service group number matches. In this case, I used 90 ip wccp 90 redirect-list wccp-redirect ! interface GigabitEthernet1 ip address dhcp no ip redirects ip wccp 90 redirect in negotiation auto ! interface GigabitEthernet2 ip address dhcp negotiation auto ! interface GigabitEthernet3 ip address 192.168.1.209 255.255.255.0 no ip redirects negotiation auto ! ip route 0.0.0.0 0.0.0.0 192.168.1.1 ! ip access-list extended wccp-redirect 10 permit tcp any any eq www 20 permit tcp any any eq 443 30 deny ip any any ! ip access-list extended 110 10 permit ip 192.168.1.0 0.0.0.255 any 20 deny ip any any ip access-list extended 120 10 permit ip any any ip access-list extended 130 10 deny ip any 10.0.0.0 0.255.255.255 20 deny ip any 172.16.0.0 0.15.255.255 30 deny ip any 192.168.0.0 0.0.255.255 40 permit ip any any ! end Verifying on the Cisco router: router#sh ip wccp Global WCCP information: Router information: Router Identifier: 192.168.153.128 Service Identifier: 90 Protocol Version: 2.00 Number of Service Group Clients: 1 Number of Service Group Routers: 1 Total Packets Redirected: 34390 Process: 0 CEF: 0 Platform: 34390 Service mode: Open Service Access-list: -none- Total Packets Dropped Closed: 0 Redirect access-list: wccp-redirect Total Packets Denied Redirect: 0 Total Packets Unassigned: 0 Group access-list: -none- Total Messages Denied to Group: 0 Total Authentication failures: 0 Total GRE Bypassed Packets Received: 0 Process: 0 CEF: 0 Platform: 0 Validating which client is registered router#sh ip wccp 90 clients WCCP Client information: WCCP Client ID: 192.168.8.105 Protocol Version: 2.00 State: Usable Redirection: L2 Packet Return: L2 Assignment: HASH Connect Time: 03:23:47 Redirected Packets: Process: 0 CEF: 0 Platform: 35918 GRE Bypassed Packets: Process: 0 CEF: 0 Hash Allotment: 256 of 256 (100.00%) Initiating the failover on the BIG-IP root@(bip1)(cfg-sync In Sync)(Active)(/Common)(tmos)# run sys failover standby router#sh ip wccp 90 clients WCCP Client information: WCCP Client ID: 192.168.8.59 Protocol Version: 2.00 State: Usable Redirection: L2 Packet Return: L2 Assignment: HASH Connect Time: 00:03:26 Redirected Packets: Process: 0 CEF: 0 Platform: 6522 GRE Bypassed Packets: Process: 0 CEF: 0 Hash Allotment: 256 of 256 (100.00%) Activity on the newly active device tail -f /var/log/wccpd.log <13> Mar 27 11:56:27 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:388 : <<< Got: I See You <13> Mar 27 11:56:29 bip1.local notice wccpd-1[17bb:f59af340] WccpApp.cpp:208 : Failover status active 0 <13> Mar 27 12:22:37 bip1.local notice wccpd-1[17bb:f59af340] WccpApp.cpp:208 : Failover status active 1 <13> Mar 27 12:22:37 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:503 : <<< Got: Removal Query ! <13> Mar 27 12:22:37 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:212 : >>> Sending Here I Am ::: Service 90 Protocol 6 ::: SecurityInfo: Opt: 0x0 <13> Mar 27 12:22:37 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:388 : <<< Got: I See You <13> Mar 27 12:22:38 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:212 : >>> Sending Here I Am ::: Service 90 Protocol 6 ::: SecurityInfo: Opt: 0x0 <13> Mar 27 12:22:38 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:388 : <<< Got: I See You <13> Mar 27 12:22:39 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:212 : >>> Sending Here I Am ::: Service 90 Protocol 6 ::: SecurityInfo: Opt: 0x0 Activity on the newly standby device tail -f /var/log/wccpd.log <13> Mar 27 11:56:07 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:388 : <<< Got: I See You <13> Mar 27 11:56:17 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:212 : >>> Sending Here I Am ::: Service 90 Protocol 6 ::: SecurityInfo: Opt: 0x0 <13> Mar 27 11:56:17 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:388 : <<< Got: I See You <13> Mar 27 11:56:27 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:212 : >>> Sending Here I Am ::: Service 90 Protocol 6 ::: SecurityInfo: Opt: 0x0 <13> Mar 27 11:56:27 bip1.local notice wccpd-1[17bb:f59af340] ServiceGroup.cpp:388 : <<< Got: I See You <13> Mar 27 11:56:29 bip1.local notice wccpd-1[17bb:f59af340] WccpApp.cpp:208 : Failover status active 0SSL Orchestrator Service Extensions: DoH Guardian
SSL Orchestrator Service Extensions provide a powerful new way to add a programmable inspection service object directly inside the service chain enabling enormous possibilities for additional security value without additional, external tooling. DoH Guardian is one implementation of Service Extensions that logs and manages decrypted outbound DNS-over-HTTPS traffic, and detects potentially malicious DoH anomalies include data exfiltration attacks.SSL Orchestrator Service Extensions: User Coaching
SSL Orchestrator Service Extensions provide a powerful new way to add a programmable inspection service object directly inside the service chain enabling enormous possibilities for additional security value without additional, external tooling. User Coaching is one implementation of Service Extensions that notifies and "coaches" a user when they attempt to access something that may violate local security policy.Packet Analysis with Scapy and tcpdump: Checking Compatibility with F5 SSL Orchestrator
In this guide I want to demonstrate how you can use Scapy (https://scapy.net/) and tcpdump for reproducing and troubleshooting layer 2 issues with F5 BIG-IP devices. Just in case you get into a finger-pointing situation... Starting situation This is a quite recent story from the trenches: My customer uses a Bypass Tap to forward or mirror data traffic to inline tools such as IDS/IPS, WAF or threat intelligence systems. This ByPass Tap offers a feature called Network Failsafe (also known as Fail-to-Wire). This is a fault tolerance feature that protects the flow of data in the event of a power outage and/or system failure. It allows traffic to be rerouted while the inline tools (IDS/IPS, WAF or threat intelligence systems) are shutting down, restarting, or unexpectedly losing power (see red line named Fallback in the picture below). Since the ByPass Tap itself does not have support for SSL decryption and re-encryption, an F5 BIG-IP SSL Orchestrator shall be introduced as an inline tool in a Layer 2 inbound topology. Tools directly connected to the Bypass Tap will be connected to the SSL Orchestrator for better visibility. To check the status of the inline tools, the Bypass Tap sends health checks through the inline tools. What is sent on one interface must be seen on the other interface and vice versa. So if all is OK (health check is green), traffic will be forwarded to the SSL Orchestrator, decrypted and sent to the IDS/IPS and the TAP, and then re-encrypted and sent back to the Bypass Tap. If the Bypass Tap detects that the SSL Orchestrator is in a failure state, it will just forward the traffic to the switch. This is the traffic flow of the health checks: Target topology This results in the following topology: Problem description During commissioning of the new topology, it turned out that the health check packets are not forwarded through the vWire configured on the BIG-IP. A packet analysis with Wireshark revealed that the manufacturer uses ARP-like packets with opcode 512 (HEX 02 00). This opcode is not defined in the RFC that describes ARP (https://datatracker.ietf.org/doc/html/rfc826), the RFC only describes the opcodes Request (1 or HEX 00 01) and Reply (2 or HEX 00 02). NOTE: Don't get confused that you see ARP packets on port 1.1 and 1.2. They are not passing through, the Bypass Tap is just send those packets from both sides of the vWire, as explained above. The source MAC on port 1.1 and 1.2 are different. Since the Bypass Tap is located right behind the customer's edge firewall, lengthy and time-consuming tests on the live system are not an option, since it would result in a massive service interruption. Therefore, a BIG-IP i5800 (the same model as the customer's) was set up as SSL Orchestrator and a vWire configuration was build in my employers lab. The vWire configuration can be found in this guide (https://clouddocs.f5.com/sslo-deployment-guide/chapter2/page2.7.html). INFO: For those not familiar with vWire: "Virtual wire … creates a layer 2 bridge across the defined interfaces. Any traffic that does not match a topology listener will pass across this bridge." Lab Topology The following topology was used for the lab: I build a vWire configuration on the SSL Orchestrator, as in the customer's environment. A Linux system with Scapy installed was connected to Interface 1.1. With Scapy TCP, UDP and ARP packets can be crafted or sent like a replay from a Wireshark capture. Interface 1.3 was connected to another Linux system that should receive the ARP packets. All tcpdumps were captured on the F5 and analyzed on the admin system (not plotted). Validating vWire Configuration To check the functionality of the F5 and the vWire configuration, two tests were performed. A replay of the Healthcheck packets from the Bypass Tap and a test with RFC-compliant ARP requests. Use Scapy to resend the faulty packets First, I used Wireshark to extract a single packet from packet analysis we took in the customer environment and saved it to a pcap file. I replayed this pcap file to the F5 with Scapy. The sendp() function will work at layer 2, it requires the parameters rdpcap (location of the pcap file for replay) and iface (which interface it shall use for sending). webserverdude@tux480:~$ sudo scapy -H WARNING: IPython not available. Using standard Python shell instead. AutoCompletion, History are disabled. Welcome to Scapy (2.5.0) >>> sendp(rdpcap("/home/webserverdude/cusomter-case/bad-example.pcap"),iface="enp0s31f6") . Sent 1 packets. This test confirmed the behavior that was observed in the customer's environment. The F5 BIG-IP does not forward this packet. Use PING and Scapy to send RFC-compliant ARP packets To create RFC-compliant ARP requests, I first sent an ARP request (opcode 1) through the vWire via PING command. As expected, this was sent through the vWire. To ensure that this also works with Scapy, I also resent this packet with Scapy. >>> sendp(rdpcap("/home/webserverdude/cusomter-case/good-example.pcap"),iface="enp0s31f6") . Sent 1 packets. In the Wireshark analysis it can be seen that this packet is incoming on port 1.1 and then forwarded to port 1.3 through the vWire. Solving the issue with the help of the vendor It became evident that the BIG-IP was dropping ARP packets that failed to meet RFC compliance, rendering the Bypass Tap from this particular vendor seemingly incompatible with the BIG-IP. Following my analysis, the vendor was able to develop and provide a new firmware release addressing this issue. To verify that the issue was resolved in this firmware release, my customer's setup, the exact same model of the Bypass Tap and a BIG-IP i5800, were deployed in my lab, where the new firmware underwent thorough testing. With this approach I could test the functionality and compatibility of the systems under controlled conditions. In this Wireshark analysis it can be seen that the Healthcheck packets are incoming on port 1.1 and then forwarded to port 1.3 through the vWire (marked in green) and also the other way round, coming in on port 1.3 and then forwarded to port 1.1 (marked in pink). Also now you can see that the packet is a proper gratuitous ARP reply (https://wiki.wireshark.org/Gratuitous_ARP). Because the Healthcheck packets were not longer dropped by the BIG-IP, but were forwarded through the vWire the Bypass Tap subsequently marked the BIG-IP as healthy and available. The new firmware resolved the issue. Consequently, my customer could confidently proceed with this project, free from the constraints imposed by the compatibility issue.
