SNI Routing with BIG-IP
In the previous article, The Three HTTP Routing Patterns, Lori MacVittie covers 3 methods of routing. Today we will look at Server Name Indication (SNI) routing as an additional method of routing HTTPS or any protocol that uses TLS and SNI. Using SNI we can route traffic to a destination without having to terminate the SSL connection. This enables several benefits including: Reduced number of Public IPs Simplified configuration More intelligent routing of TLS traffic Terminating SSL Connections When you have a SSL certificate and key you can perform the cryptographic actions required to encrypt traffic using TLS. This is what I refer to as “terminating the SSL connection” throughout this article. When you want to route traffic this is a chicken and an egg problem, because for TLS traffic you want to be able to route the traffic by being able to inspect the contents, but this normally requires being able to “terminate the SSL connection”. The goal of this article is to layer in traffic routing for TLS traffic without having to require having/knowing the original SSL certificate and key. Server Name Indication (SNI) SNI is a TLS extension that makes it possible to "share" certificates on a single IP address. This is possible due to a client using a TLS extension that requests a specific name before the server responds with a SSL certificate. Prior to SNI, the other options would be a wildcard certificate or Subject Alternative Name (SAN) that allows you to specify multiple names with a single certificate. SNI with Virtual Servers It has been possible to use SNI on F5 BIG-IP since TMOS 11.3.0. The following KB13452 outlines how it can be configured. In this scenario (from the KB article) the BIG-IP is terminating the SSL connection. Not all clients support SNI and you will always need to specify a “fallback” profile that will be used if a SNI name is not used or matched. The next example will look at how to use SNI without terminating the SSL connection. SNI Routing Occasionally you may have the need to have a hybrid configuration of terminating SSL connections on the BIG-IP and sending connections without terminating SSL. One method is to create two separate virtual servers, one for SSL connections that the BIG-IP will handle (using clientssl profile) and one that it will not handle SSL (just TCP). This works OK for a small number of backends, but does not scale well if you have many backends (run out of Public IP addresses). Using SNI Routing we can handle everything on a single virtual server / Public IP address. There are 3 methods for performing SNI Routing with BIG-IP iRule with binary scan a. Article by Colin Walker code attribute to Joel Moses b. Code Share by Stanislas Piron iRule with SSL::extensions Local Traffic Policy Option #1 is for folks that prefer complete control of the TLS protocol. It only requires the use of a TCP profile. Options #2 and #3 only require the use of a SSL persistence profile and TCP profile. SNI Routing with Local Traffic Policy We will skip option #1 and #2 in this article and look at using a Local Traffic Policy for SNI Routing. For a review of Local Traffic Policies check out the following DevCentral articles: LTM Policy Jan 2015 Simplifying Local Traffic Policies in BIG-IP 12.1 June 2016 In previous articles about Local Traffic Policies the focus was on routing HTTP traffic, but today we will use it to route SSL connections using SNI. In the following example, using a Local Traffic Policy named “sni_routing”, we are setting a condition on the SSL Extension “servername” and sending the traffic to a pool without terminating the SSL connection. The pool member could be another server or another BIG-IP device. The next example will forward the traffic to another virtual server that is configured with a clientssl profile. This uses VIP targeting to send traffic to another virtual server on the same device. In both examples it is important to note that the “condition”/“action” has been changed from occurring on “request” (that maps to a HTTP L7 request) to “ssl client hello”. By performing the action prior to any L7 functions occurring, we can forward the traffic without terminating the SSL connection. The previous example policy, “sni_routing”, can be attached to a Virtual Server that only has a TCP profile and SSL persistence profile. No HTTP or clientssl profile is required! This method can also be used to solve the issue of how to consolidate multiple SSL virtual servers behind a single virtual server that have different APM and/or ASM policies. This is similar to the architecture that is used by the Container Connector for Cloud Foundry; in creating a two-tier load balancing solution on a single device. Routed Correctly? TLS 1.3 has interesting proposals on how to obscure the servername (TLS in TLS?), but for now this is a useful and practical method of handling multiple SSL certs on a single IP. In the future this may still be possible as well with TLS 1.3. For example the use of a HTTP Fronting service could be a tier 1 virtual server (this is just my personal speculation, I have not tried, at the time of publishing this was still a draft proposal). In other news it has been demonstrated that a combination of using SNI and a different host header can be used for “domain fronting”. A method to enforce consistent policy (prevent domain fronting) would be to layer in additional conditions that match requested SNI servername (TLS extension) with requested HOST header (L7 HTTP header). This would help enforce that a tenant is using a certificate that is associated with their application and not “borrowing” the name and certificate that is being used by an adjacent service. We don’t think of a TLS extension as an attribute that can be used to route application traffic, but it is useful and possible on BIG-IP.SSL Orchestrator Advanced Use Cases: Client Certificate Constrained Delegation (C3D) Support
Introduction F5 BIG-IP is synonymous with "flexibility". You likely have few other devices in your architecture that provide the breadth of capabilities that come native with the BIG-IP platform. And for each and every BIG-IP product module, the opportunities to expand functionality are almost limitless. In this article series we examine the flexibility options of the F5 SSL Orchestrator in a set of "advanced" use cases. If you haven't noticed, the world has been steadily moving toward encrypted communications. Everything from web, email, voice, video, chat, and IoT is now wrapped in TLS, and that's a good thing. The problem is, malware - that thing that creates havoc in your organization, that exfiltrates personnel records to the Dark Web - isn't stopped by encryption. TLS 1.3 and multi-factor authentication don't eradicate malware. The only reasonable way to defend against it is to catch it in the act, and an entire industry of security products are designed for just this task. But ironically, encryption makes this hard. You can't protect against what you can't see. F5 SSL Orchestrator simplifies traffic decryption and malware inspection, and dynamically orchestrates traffic to your security stack. But it does much more than that. SSL Orchestrator is built on top of F5's BIG-IP platform, and as stated earlier, is abound with flexibility. SSL Orchestrator Use Case: Client Certificate Constrained Delegation (C3D) Using certificates to authenticate is one of the oldest and most reliable forms of authentication. While not every application supports modern federated access or multi-factor schemes, you'll be hard-pressed to find something that doesn't minimally support authentication over TLS with certificates. And coupled with hardware tokens like smart cards, certificates can enable one of the most secure multi-factor methods available. But certificate-based authentication has always presented a unique challenge to security architectures. Certificate "mutual TLS" authentication requires an end-to-end TLS handshake. When a server indicates a requirement for the client to submit its certificate, the client must send both its certificate, and a digitally-signed hash value. This hash value is signed (i.e. encrypted) with the client's private key. Should a device between the client and server attempt to decrypt and re-encrypt, it would be unable to satisfy the server's authentication request by virtue of not having access to the client's private key (to create the signed hash). This makes encrypted malware inspection complicated, often requiring a total bypass of inspection to sites that require mutual TLS authentication. Fortunately, F5 has an elegant solution to this challenge in the form of Client Certificate Constrained Delegation, affectionally referred to as "C3D". The concept is reasonably straightforward. In very much the same way that SSL forward proxy re-issues a remote server certificate to local clients, C3D can re-issue a remote client certificate to local servers. A local server can continue to enforce secure mutual TLS authentication, while allowing the BIG-IP to explicitly decrypt and re-encrypt in the middle. This presents an immediate advantage in basic load balancing, where access to the unencrypted data allows the BIG-IP greater control over persistence. In the absence of this, persistence would typically be limited to IP address affinity. But of course, access to the unencrypted data also allows the content to be scanned for malicious malware. C3D actually takes this concept of certificate re-signing to a higher level though. The "constrained delegation" portion of the name implies a characteristic much like Kerberos constrained delegation, where (arbitrary) attributes can be inserted into the re-signed token, like the PAC attributes in a Kerberos ticket, to inform the server about the client. Servers for their part can then simply filter on client certificates issued by the BIG-IP (to prevent direct access), and consume any additional attributes in the certificate to understand how better to handle the client. With C3D you can maintain strong mutual TLS authentication all the way through to your servers, while allowing the BIG-IP to more effectively manage availability. And combined with SSL Orchestrator, C3D can enable decryption and inspection of content for malware inspection. This article describes how to configure SSL Orchestrator to enable C3D for inbound decrypted inspection. Arguably, most of what follows is the C3D configuration itself, as the integration with SSL Orchestrator is pretty simple. Note that Client Certificate Constrained Delegation (C3D) is included with Local Traffic Manager (LTM) 13.0 and beyond, but for integration with SSL Orchestrator you should be running 14.1 or later.To learn more about C3D, please see the following resources: K14065425: Configuring Client Certificate Constrained Delegation (C3D): https://support.f5.com/csp/article/K14065425 Manual Chapter: SSL Traffic Management: https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-system-ssl-administration/ssl-traffic-management.html#GUID-B4D2529E-D1B0-4FE2-8C7F-C3774ADE1ED2 SSL::c3d iRule reference - not required to use C3D, but adds powerful functionality https://clouddocs.f5.com/api/irules/SSL__c3d.html The integration of C3D with SSL Orchestrator involves effectively replacing the client and server SSL profiles that the SSL Orchestrator topology creates, with C3D SSL profiles. This is done programmatically with an iRule, so no "non-strict" customization is required at the topology. Also note that an inbound (reverse proxy) SSL Orchestrator topology will take the form of a "gateway mode" deployment (a routed path to multiple applications), or "application mode" deployment (a single application instance hosted at the BIG-IP). See section 2.5 of the SSL Orchestrator deployment guide for a deeper examination of gateway and application modes: https://clouddocs.f5.com/sslo-deployment-guide/ The C3D integration is only applicable to application mode deployments. Configuration C3D itself involves the creation of client and server SSL profiles: Create a new Client SSL profile: Configuration Certificate Key Chain: public-facing server certificate and private key. This will be the certificate and key presented to the client on inbound request. It will likely be the same certificate and key defined in the SSL Orchestrator inbound topology. Client Authentication Client Certificate: require Trusted Certificate Authorities: bundle that can validate client certificate. This is a certificate bundle used to verify the client's certificate, and will contain all of the client certificate issuer CAs. Advertised Certificate Authorities: optional CA hints bundle. Not expressly required, but this certificate bundle is forwarded to the client during the TLS handshake to "hint" at the correct certificate, based on issuer. Client Certificate Constrained Delegation Client Certificate Constrained Delegation: Enabled Client Fallback Certificate (new in 15.1): option to select a default client certificate if client does not send one. This option was introduced in 15.1 and provides the means to select an alternate (local) certificate if the client does not present one. The primary use case here might be to select a "template" certificate, and use an iRule function to insert arbitrary attributes. OCSP: optional client certificate revocation control. This option defines an OCSP revocation provider for the client certificate. Unknown OCSP Response Control (new in 15.1): determines what happens when OCSP returns Unknown. If an OCSP revocation provider is selected, this option defines what to do if the response to the OCSP query is "unknown". Create a new Server SSL profile: Configuration Certificate: default.crt. The certificate and key here are used as "templates" for the re-signed client certificate. Key: default.key Client Certificate Configuration Delegation Client Certificate Constrained Delegation: Enabled CA Certificate: local forging CA cert. This is the CA certificate used to re-sign the client certificate. This CA must be trusted by the local servers. CA Key: local forging CA key CA Passphrase: optional CA passphrase Certificate Extensions: extensions from the real client cert to be included in the forged cert. This is the list of certificate extensions to be copied from the original certificate to the re-issued certificate. Custom Extension: additional extensions to copy to forged cert from real cert (OID). This option allows you to insert additional extensions to be copied, as OID values. Additional considerations: Under normal conditions, the F5 and backend server attempt to resume existing SSL sessions, whereby the server doesn’t send a Certificate Request message. The effect is that all connections to the backend server use the same forged client cert. There are two ways to get around this: Set a zero-length cache value in the server SSL profile, or Set server authentication frequency to ‘always’ in the server SSL profile CA certificate considerations: A valid signing CA certificate should possess the following attributes. While it can work in some limited scenarios, a self-signed server certificate is generally not an adequate option for the signing CA. keyUsage: certificate extension containing "keyCertSign" and "digitalSignature" attributes basicConstraints: certificate extension containing "CA = true" (for Yes), marked as "Critical" With the client and server SSL profiles built, the C3D configuration is basically done. To integrate with an inbound SSL Orchestrator topology, create a simple iRule and add it to the topology's Interception Rule configuration. Modify the SSL profile paths below to reflect the profiles you created earlier. ### Modify the SSL profile paths below to match real C3D SSL profiles when CLIENT_ACCEPTED priority 250 { ## set clientssl set cmd1 "SSL::profile /Common/c3d-clientssl" ; eval $cmd1 } when SERVER_CONNECTED priority 250 { ## set serverssl SSL::profile "/Common/c3d-serverssl" } In the SSL Orchestrator UI, either from topology workflow, or directly from the corresponding Interception Rule configuration, add the above iRule and deploy. The above iRule programmatically overrides the SSL profiles applied to the Interception Rule (virtual server), effectively enabling C3D support. At this point, the virtual server will request a client certificate, perform revocation checks if defined, and then mint a local copy of the client certificate to pass to the backend server. Optionally, you can insert additional certificate attributes via the server SSL profile configuration, or more dynamically through additional iRule logic: ### Added in 15.1 - allows you to send a forged cert to the server ### irrespective of the client side authentication (ex. APM SSO), ### and insert arbitrary values when SERVERSSL_SERVERCERT { ### The following options allow you to override/replace a submitted ### client cert. For example, a minted client certificate can be sent ### to the server irrespective of the client side authentication method. ### This certificate "template" could be defined locally in the iRule ### (Base64-encoded), pulled from an iFile, or some other certificate source. # set cert1 [b64decode "LS0tLS1a67f7e226f..."] # set cert1 [ifile get template-cert] ### In order to use a template cert, it must first be converted to DER format # SSL::c3d cert [X509::pem2der $cert1] ### Insert arbitrary attributes (OID:value) SSL::c3d extension 1.3.6.1.4.1.3375.3.1 "TEST" } If you've configured the above, a server behind SSL Orchestrator that requires mutual TLS authentication can receive minted client certificates from external users, and SSL Orchestrator can explicitly decrypt and pass traffic to the set of malware inspection tools. You can look at the certificate sent to the server by injecting a tcpdump packet between the BIG-IP and server, then open in Wireshark. tcpdump -lnni [VLAN] -Xs0 -w capture.pcap [additional filters] Finally, you might be asking what to do with certificate attributes injected by C3D, and really it depends on what the server can support. The below is a basic example in an Apache config file to block a client certificate that doesn't contain your defined attribute. <Directory /> SSLRequire "HTTP/%h" in PeerExtList("1.3.6.1.4.1.3375.3.1") RewriteEngine on RewriteCond %{SSL::SSL_CLIENT_VERIFY} !=SUCCESS RewriteRule .? - [F] ErrorDocument 403 "Delegation to SPN HTTP/%h failed. Please pass a valid client certificate" </Directory> And there you have it. In just a few steps you've configured your SSL Orchestrator to integrate with Client Certificate Constrained Delegation to support mutual TLS authentication, and along the way you have hopefully recognized the immense flexibility at your command. Updates As of F5 BIG-IP 16.1.3, there are some new C3D capabilities: C3D has been updated to encode and return the commonName (CN) found in the client certificate subject field in printableString format if possible, otherwise the value will be encoded as UTF8. C3D has been updated to support inserting a subject commonName (CN) via 'SSL::c3d subject commonName' command: when CLIENTSSL_HANDSHAKE { if {[SSL::cert count] > 0} { SSL::c3d subject commonName [X509::subject [SSL::cert 0] commonName] } } C3D has been updated to support inserting a Subject Alternative Name (SAN) and Authority Info Access (AIA) via 'SSL::c3d extension' commands: when CLIENTSSL_HANDSHAKE { ## Insert Subject Alternative Name (SAN) value SSL::c3d extension SAN "DNS:*.test-client.com, IP:1.1.1.1" ### Insert Authority Info Access (AIA) value SSL::c3d extension AIA "ocsp,https://ocsp.entrust.net.com; caIssuer, https://aia.entrust.net/l1m-chain256.cer" } C3D has been updated to add the Authority Key Identifier (AKI) extension to the client certificate if the CA certificate has a Subject Key Identifier (SKI) extension. Another interesting use case is copying the real client certificate Subject Key Identifier (SKI) to the minted client certificate. By default, the minted client certificate will not contain an SKI value, but it's easy to configure C3D to copy the origin cert's SKI by modifying the C3D server SSL profile. In the "Custom extension" field of the C3D section, add 2.5.29.14 as an available extension. As of F5 BIG-IP 17.1.0 (SSL Orchestrator 11.0), C3D has been integrated natively. Now, for a deployed Inbound topology, the C3D SSL profiles are listed in the Protocol Settings section of the Interception Rules tab. You can replace the client and server SSL profiles created by SSL Orchestrator, with C3D SSL profiles in the Interception Rules tab to support C3D. The C3D support is now extended to both Gateway and Application modes.
Any approach to encrypting HSL traffic such that no plain text is ever sniffable?
Hi - we have been asked to integrate a vendor logging solution on our F5s that uses HSL to send information about requests to and responses from our HTTP/HTTPS VIPs on our LTMs. (I describe this configuration in more detail in a post yesterday, with the title "Pool used with HSL::open - what are the requirements? Any way to make it send using TLS?" ... but in summary, they use an iRule with HSL:open against a manually created pool, that uses as members an HTTP VIP created on the same F5; that in turn requires us to use a static route to send the traffic over the management interface, so it can reach the VIP on the same F5). Though we have a solution for the flow (see other post), it results in the logged content being plain-text on our network, whilst going from the management port to the VIP used as a destination of the HSL pool. Is there any general solution or design that would permit us to send out HSL logging such that it is TLS encrypted right from the moment it is sent by the F5, never appearing in plain text on our network? Thank you!A Closer Look at mTLS and the Default Server in F5 NGINX
When you connect to an HTTPS site, your browser (or other client) typically sends a Server Name Indication (SNI)—the hostname it wants to reach—during the TLS handshake. This lets F5 NGINX figure out which cryptographic settings to use for that specific hostname. In other words, SNI influences: Which x509 certificate is sent to the client Which cryptographic algorithms are offered Which session ticket key is used for encrypting and decrypting session tickets Which session cache is active, if you’re using caching Which Certificate Authority (CA) is checked when you require mutual TLS (mTLS) If the client doesn’t provide SNI or if the hostname doesn’t match any of your configured server_name directives, F5 NGINX defaults to a “fallback” setup—usually called the default server. That means: The default server’s certificate, ciphers, ticket key, and session cache get used automatically. If you haven’t explicitly marked any server block as default_server, F5 NGINX chooses the first block for that listen socket in your configuration as the fallback. Here’s the crucial detail: once the TLS handshake finishes and a certificate has been selected (for example, from the default server), F5 NGINX will still examine the HTTP Host header for request routing. If it specifies a different domain matching another server block, the request is forwarded there at the HTTP layer. From the client’s perspective, however, the originally cryptographic settings remain in effect, because the TLS negotiation is already complete. Single Server Block In the simplest configuration, only one server block listens for TLS connections on a given IP address and port: server { listen 443 ssl; server_name example.com; ssl_certificate /etc/ssl/certs/example.com.crt; ssl_certificate_key /etc/ssl/private/example.com.key; ssl_client_certificate /etc/ssl/certs/ca.crt; ssl_verify_client on; # Additional configuration, such as locations and logging } In this example: If the client provides SNI matching example.com, F5 NGINX presents /etc/ssl/certs/example.com.crt and verifies the client certificate against /etc/ssl/certs/ca.crt. If the client does not provide SNI, this same server block still handles the request because there are no other blocks to consider; the same certificate and CA (ca.crt) apply. Once authenticated, the client proceeds under the cryptographic settings of this single server block. With only one server block present, there is no additional routing or fallback scenario to manage. Multiple Server Blocks on the Same IP/Port When multiple server blocks listen on the same IP address and port, F5 NGINX uses SNI to determine which server block should handle the request. If no matching SNI is found, requests fall back to the server marked with default_server. As previously stated, if the default_server is not explicitly defined, F5 NGINX will use the first server block in the configuration as the fallback. # example.com and the default server (first in config) server { listen 443 ssl; server_name example.com; ssl_certificate /etc/ssl/certs/example.com.crt; ssl_certificate_key /etc/ssl/private/example.com.key; ssl_client_certificate /etc/ssl/certs/ca_A.crt; ssl_verify_client on; # Additional configuration, such as locations and logging } # www.example.com server { listen 443 ssl; server_name www.example.com; ssl_certificate /etc/ssl/certs/www.example.com.crt; ssl_certificate_key /etc/ssl/private/www.example.com.key; ssl_client_certificate /etc/ssl/certs/ca_B.crt; ssl_verify_client on; # Additional configuration, such as locations and logging } In this example: If the client provides SNI matching example.com, the first server block’s certificate (example.com.crt) and CA settings (ca_A.crt) are used. If the client provides SNI matching www.example.com, the second server block’s certificate (www.example.com.crt) and CA settings (ca_B.crt) are used. If the client does not provide SNI (or provides an unmatched server name), the first server block (example.com) acts as the default. Its certificate (example.com.crt) and CA (ca_A.crt) apply for the TLS handshake. After TLS is established under the default server, if the HTTP Host header is www.example.com, F5 NGINX routes the request to the second server block for application-level processing. However, the TLS session—including which certificate and CA were used—remains with the default server’s settings. This means the second server’s client certificate configuration (ca_B.crt) is not involved in re-validating the client, since no new TLS handshake occurs. Recommendations The fallback behavior mentioned above might not fit all use cases. If it poses a risk or doesn’t align with your security needs, consider reconfiguring F5 NGINX (e.g., setting up a stub default server or applying tighter mTLS rules) to restrict or eliminate this fallback path. Defining a Default Server (or Stub Default Server) It is highly recommended to define a default server in F5 NGINX. If you do not want to allow fallback for clients without valid SNI, you can set up a stub default server (configuration example below). A stub default server, as shown below, ensures that unmatched SNI (or no SNI) connections are rejected at the handshake level, preventing unintended fallback to a less restrictive configuration. Perhaps most importantly, it does not contain any client authentication configuration directives, forcing client authentication to occur in the most specific server blocks. In the example below I have added `ssl_verify_client off;` for illustrative purposes, however the setting of `off` is the default. Note: ssl_reject_handshake appeared in nginx version 1.19.4. For versions prior to that, one can define a server that simply returns the special 444 HTTP response code. Authorization Checks in All Server Blocks Even with a stub default server, all server blocks should implement authorization checks if they serve sensitive content. Because requests may be forwarded from the default server to a non-default server after decryption, every server block must enforce its own mTLS policies and check variables such as $ssl_client_s_dn or $ssl_client_i_dn (if you rely on client certificates) to ensure consistent and robust security across your deployment. # explicit default server server { listen 443 ssl default_server; ssl_reject_handshake on; ssl_verify_client off; ssl_protocols TLSv1.2 TLSv1.3; } # example.com server { listen 443 ssl; server_name example.com; ssl_certificate /etc/ssl/certs/example.com.crt; ssl_certificate_key /etc/ssl/private/example.com.key; ssl_client_certificate /etc/ssl/certs/ca_A.crt; ssl_verify_client on; # Check subject DN if ($ssl_client_s_dn !~ "CN=TrustedClient_A,O=MyOrg") { return 403; } # Check issuer DN (this may not be necessary for all deployments) if ($ssl_client_i_dn !~ "CN=TrustedCA_A,O=MyOrg") { return 403; } # Additional configuration, such as locations and logging } # This server block handles requests for www.example.com server { listen 443 ssl; server_name www.example.com; ssl_certificate /etc/ssl/certs/www.example.com.crt; ssl_certificate_key /etc/ssl/private/www.example.com.key; ssl_client_certificate /etc/ssl/certs/ca_B.crt; ssl_verify_client on; # Check subject DN if ($ssl_client_s_dn !~ "CN=TrustedClient_B,O=MyOrg") { return 403; } # Check issuer DN (this may not be necessary for all deployments) if ($ssl_client_i_dn !~ "CN=TrustedCA_B,O=MyOrg") { return 403; } # Additional configuration, such as locations and logging } Conclusion In summary, working with multiple server blocks, SNI, and mTLS can make F5 NGINX setups more complex. Knowing precisely which server block handles the TLS handshake—particularly when there is no SNI or an unmatched name—helps maintain the desired security posture. Careful attention to these details keeps certificates and policies consistent for all client connections.
Question on configuring SNI clientSSL Profile
Hi Experts , I have a question on configuring the SNI SSL profile .Suppose say I have 3 different certificate and 3 SSL profile to be attached to the VIP to configure SNI . https://www.securesite1.com ClientSSL1 > Default SSL Profile for SNI https://www.securesite2.com ClientSSL2 https://www.securesite3.com ClientSSL3 To enable SNI, we configure the Server Name and Default SSL Profile for SNI will be checked on an SSL profile of ClientSSL1, and then assign the profile to a virtual server. How about on other 2 SSL profiles ClientSSL2 & ClientSSL3 ? For other SSL profiles do I need to type the name for the HTTPS site in the Server Name box ? or it can be left blank ?INFORM: Entrust CA will be untrusted in Chrome after Oct 31, 2024
If you manage certs from Entrust in your environment, this will impact your Google Chrome users, so intermediate certs will likely need to be bundled to handle this in your clientssl profiles OR if you control all the clients you can assure that explicit trust in the clients is enabled for Entrust CAs. Google details on the situation
failed to find TLS certificate objects, no entry finder provided
I'm trying to setup TLS @ the Cluster Level on the backend connection to the Origin Endpoint. The front end between the client and the HTTP load balancer works fine, but when i attempt to configure TLS @ the cluster level for a route, i get this error: failed to find TLS certificate objects, no entry finder provided when saving changes to the cluster in the gui. I'm quite certain i have the proper CA cert uploaded..and i've tried adding the trust chain into the same cert--both with or without the intermediate chain in the CA cert, i get the same error. Does anyone have any thoughts on what might be wrong here?JA4 Part 2: Detecting and Mitigating Based on Dynamic JA4 Reputation
In my previous article on JA4 I provided a brief introduction to what is JA4 and JA4+, and I shared an iRule that enables you to generate a JA4 client TLS fingerprint. But having a JA4 fingerprint (or any "identifier") is only valuable if you can take some action on it. It is even more valuable when you can take immediate action on it. In this article, I'll explain how I integrated F5 BIG-IP Advanced WAF with a third-party solution that allowed me to identify JA4s that were consistently doing "bad" things, build a list of those JA4s that have a "bad" reputation, pull that list into the F5 BIG-IP, and finally, make F5 Advanced WAF blocking decisions based on that reputation. Understanding JA4 Fingerprints It is important to understand that a JA4 TLS fingerprint, or any TLS fingerprint for that matter, is NOT a fingerprint of an individual instance of a device or browser. Rather, it is a fingerprint of a TLS "stack" or application. For example, all Chrome browsers of the same version and the same operating system will generate the same JA4 fingerprint*. Similarly, all Go HTTP clients with the same version and operating system will generate an identical JA4 fingerprint. Because of this, we have to be careful when taking action based on JA4 fingerprints. We cannot simply block in our various security devices based on JA4 fingerprint alone UNLESS we can be certain that ALL (or nearly all) requests with that JA4 are malicious. To make this determination, we need to watch requests over time. TLDR; I used CrowdSec Security Engine to build a JA4 real-time reputation database; and 3 irules, an iCall script, and a custom WAF violation to integrate that JA4 reputation into F5 BIG-IP Advanced WAF. CrowdSec and John Althouse - Serendipity While at Black Hat each year, I frequently browse the showroom floor (when I'm not working the F5 booth) looking for cool new technology, particularly cool new technology that can potentially be integrated with F5 security solutions. Last year I was browsing the floor and came across CrowdSec. As the name suggests, CrowdSec provides a crowd-sourced IP reputation service. I know, I know. On the surface this doesn't sound that exciting — there are hundreds of IP reputation services out there AND IP address, as an identifier of a malicious entity, is becoming (has become?) less and less valuable. So what makes CrowdSec any different? Two things jumped out at me as I looked at their solution. First, while they do provide a central crowd-sourced IP reputation service like everyone else, they also have "Security Engines". A security engine is an agent/application that you can install on-premises that can consume logs from your various security devices, process those logs based on "scenarios" that you define, and produce a reputation database based on those scenarios. This enables you to create an IP reputation feed that is based on your own traffic/logs and based on your own conditions and criteria for what constitutes "malicious" for your organization. I refer to this as "organizationally-significant" reputation. AND, because this list can be updated very frequently (every few seconds if you wanted) and pushed/pulled into your various security devices very frequently (again, within seconds), you are afforded the ability to block for much shorter periods of time and, possibly, more liberally. Inherent in such an architecture, as well, is the ability for your various security tools to share intelligence in near real-time. i.e. If your firewall identifies a bad actor, your WAF can know about that too. Within seconds! At this point you're probably wondering, "How does this have anything to do with JA4?" Second, while the CrowdSec architecture was built to provide IP reputation feeds, I discovered that it can actually create a reputation feed based on ANY "identifier". In the weeks leading up to Black Hat last year, I had been working with John Althouse on the JA4+ spec and was actually meeting him in person for the first time while there. So JA4 was at the forefront of my mind. I wondered if I could use CrowdSec to generate a reputation based on a JA4 fingerprint. Yes! You can! Deploying CrowdSec As soon as I got home from Black Hat, I started playing. I already had my BIG-IP deployed, generating JA4s, and including those in the WAF logs. Following the very good documentation on their site, I created an account on CrowdSec's site and deployed a CrowdSec Security Engine on an Ubuntu box that I deployed next to my BIG-IP. It is beyond the scope of this article to detail the complete deployment process but, I will include details relevant to this article. After getting the CrowdSec Security Engine deployed I needed to configure a parser so that the CrowdSec Security Engine (hereafter referred to simply as "SE") could properly parse the WAF logs from F5. Following their documentation, I created a YAML file at /etc/crowdsec/parsers/s01-parse/f5-waf-logs.yaml: onsuccess: next_stage debug: false filter: "evt.Parsed.program == 'ASM'" name: f5/waf-logs description: "Parse F5 ASM/AWAF logs" pattern_syntax: F5WAF: 'unit_hostname="%{DATA:unit_hostname}",management_ip_address="%{DATA:management_ip_address}",management_ip_address_2="%{DATA:management_ip_address_2}",http_class_name="%{DATA:http_class_name}",web_application_name="%{DATA:web_application_name}",policy_name="%{DATA:policy_name}",policy_apply_date="%{DATA:policy_apply_date}",violations="%{DATA:violations}",support_id="%{DATA:support_id}",request_status="%{DATA:request_status}",response_code="%{DATA:response_code}",ip_client="%{IP:ip_client}",route_domain="%{DATA:route_domain}",method="%{DATA:method}",protocol="%{DATA:protocol}",query_string="%{DATA:query_string}",x_forwarded_for_header_value="%{DATA:x_forwarded_for_header_value}",sig_ids="%{DATA:sig_ids}",sig_names="%{DATA:sig_names}",date_time="%{DATA:date_time}",severity="%{DATA:severity}",attack_type="%{DATA:attack_type}",geo_location="%{DATA:geo_location}",ip_address_intelligence="%{DATA:ip_address_intelligence}",username="%{DATA:username}",session_id="%{DATA:session_id}",src_port="%{DATA:src_port}",dest_port="%{DATA:dest_port}",dest_ip="%{DATA:dest_ip}",sub_violations="%{DATA:sub_violations}",virus_name="%{DATA:virus_name}",violation_rating="%{DATA:violation_rating}",websocket_direction="%{DATA:websocket_direction}",websocket_message_type="%{DATA:websocket_message_type}",device_id="%{DATA:device_id}",staged_sig_ids="%{DATA:staged_sig_ids}",staged_sig_names="%{DATA:staged_sig_names}",threat_campaign_names="%{DATA:threat_campaign_names}",staged_threat_campaign_names="%{DATA:staged_threat_campaign_names}",blocking_exception_reason="%{DATA:blocking_exception_reason}",captcha_result="%{DATA:captcha_result}",microservice="%{DATA:microservice}",tap_event_id="%{DATA:tap_event_id}",tap_vid="%{DATA:tap_vid}",vs_name="%{DATA:vs_name}",sig_cves="%{DATA:sig_cves}",staged_sig_cves="%{DATA:staged_sig_cves}",uri="%{DATA:uri}",fragment="%{DATA:fragment}",request="%{DATA:request}",response="%{DATA:response}"' nodes: - grok: name: "F5WAF" apply_on: message statics: - meta: log_type value: f5waf - meta: user expression: "evt.Parsed.username" - meta: source_ip expression: "evt.Parsed.ip_client" - meta:violation_rating expression:"evt.Parsed.violation_rating" - meta:request_status expression:"evt.Parsed.request_status" - meta:attack_type expression:"evt.Parsed.attack_type" - meta:support_id expression:"evt.Parsed.support_id" - meta:violations expression:"evt.Parsed.violations" - meta:sub_violations expression:"evt.Parsed.sub_violations" - meta:session_id expression:"evt.Parsed.session_id" - meta:sig_ids expression:"evt.Parsed.sig_ids" - meta:sig_names expression:"evt.Parsed.sig_names" - meta:method expression:"evt.Parsed.method" - meta:device_id expression:"evt.Parsed.device_id" - meta:uri expression:"evt.Parsed.uri" nodes: - grok: pattern: '%{GREEDYDATA}X-JA4: %{DATA:ja4_fp}\\r\\n%{GREEDYDATA}' apply_on: request statics: - meta: ja4_fp expression:"evt.Parsed.ja4_fp" Sending WAF Logs On the F5 BIG-IP, I created a logging profile to send the WAF logs to the CrowdSec Security Engine IP address and port. Defining "Scenarios" At this point, I had the WAF logs being sent to the SE and properly being parsed. Now I needed to define the "scenarios" or the conditions under which I wanted to trigger and alert for an IP address or, in this case, a JA4 fingerprint. For testing purposes, I initially created a very simple scenario that flagged a JA4 as malicious as soon as I saw 5 violations in a sliding 30 second window but only if the violation rating was 3 or higher. That worked great! But that would never be practical in the real world (see the Understanding JA4 Fingerprints section above). I created a more practical "scenario" that only flags a JA4 as malicious if we have seen at least X number of requests AND more than 90% of requests from that JA4 have triggered some WAF violation. The premise with this scenario is that there should be enough legitimate traffic from popular browsers and other client types to keep the percentage of malicious traffic from any of those JA4s below 90%. Again, following the CrowdSec documentation, I created a YAML file at /etc/crowdsec/scenarios/f5-waf-ja4-viol-percent.yaml: type: conditional name: f5/waf-ja4-viol-percent description: "Raise an alert if the percentage of requests from a ja4 finerprint is above X percent" filter: "evt.Meta.violations != 'JA4 Fingerprint Reputation'" blackhole: 300s leakspeed: 5m capacity: -1 condition: | len(queue.Queue) > 10 and (count(queue.Queue, Atof(#.Meta.violation_rating) > 1) / len(queue.Queue)) > 0.9 groupby: "evt.Meta.ja4_fp" scope: type: ja4_fp expression: evt.Meta.ja4_fp labels: service: f5_waf type: waf_ja4 remediation: true debug: false There are a few key lines to call out from this configuration file. leakspeed: This is the "sliding window" within which we are looking for our "scenarios". i.e. events "leak" out of the bucket after 5 minutes. condition: The conditions under which I want to trigger this bucket. For my scenario, I have defined a condition of at least 10 events (with in that 5 minute window) AND where the total number of events, divided by the number of events where the violation rating is above 1, is greater than 0.9. in other words, if more than 90% of the requests have triggered a WAF violation with a rating higher than 1. filter: used to filter out events that you don't want to include in this scenario. In my case, I do not want to include requests where the only violation is the "JA4 Fingerprint Reputation" violation. groupby: this defines how I want to group requests. Typiiccally, in most CrowdSec scenarios this wil be some IP address field from the logs. In my scenario, I wanted to group by the JA4 fingerprint parsed out of the WAF logs. blackhole: this defines how long I want to "silence" alerts per JA4 fingerprint after this scenario has triggered. This prevents the same scenario from triggering repeatedly every time a new request comes into the bucket. scope: the scope is used by the reputation service to "categorize" alerts triggered by scenarios. the type field is used to define the type of data that is being reported. In most CrowdSec scenarios the type is "ip". In my case, I defined a custom type of "ja4_fp" with an "expression" (or value) of the JA4 fingerprint extracted from the WAF logs. Defining "Profiles" In the CrowdSec configuration "profiles" are used to define the remediation that should be taken when a scenario is triggered. I edited the /etc/crowdsec/profiles.yaml file to include the new profile for my JA4 scenario. name: ban_ja4_fp filters: - Alert.Remediation == true && Alert.GetScope() == "ja4_fp" decisions: - type: ban scope: "ja4_fp" duration: 5m debug: true on_success: continue --- ##### Everything below this point was already in the profiles.yaml file. Truncated here for brevity. name: default_ip_remediation #debug: true filters: - Alert.Remediation == true && Alert.GetScope() == "Ip" decisions: ... on_success: break Again, there are a few key lines from this configuration file. First, I only added a new profile named "ban_ja4_fp" with lines 1 through 9 in the file above. filters: Used to define which triggered scenarios should be included in this profile. In my case, all scenarios with the "remediation" label AND the "ja4_fp" scope. decisions: Used to define what type of remediation should be taken, for which "scope", and for how long. In my case, I chose the default of "ban", for the "ja4_fp" scope, and for 5 minutes. With this configuration in place I sent several malicious requests from my browser to my test application protected by the F5 Advanced WAF. I then checked the CrowdSec decisions list and voila! I had my browser's JA4 fingerprint listed! This was great but I wanted to be able to take action based on this intelligence in the F5 WAF. CrowdSec has the concept of "bouncers". Bouncers are devices the can take action on the remediation decisions generated by the SEs. Technically, anything that can call the local CrowdSec API and take some remediating action can be a bouncer. So, using the CLI on the CrowdSec SE, I defined a new "bouncer" for the F5 BIG-IP. ubuntu@xxxxxxxx:~$ sudo cscli bouncer add f5-bigip Api key for 'f5-bigip': xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Please keep this key since you will not be able to retrieve it! I knew that I could write an iRule that could call the SE API. However, the latency introduced by a sideband API call on EVERY HTTP request would just be completely untenable. I wanted a way to download the entire reputation list at a regular interval and store it on the F5 BIG-IP in a way that would be easily and efficiently accessible from the data plane. This sounded like a perfect job for an iCall script. Customizing the F5 BIG-IP Configuration If you are not familiar with iCall scripts, they are a programmatic way of checking or altering the F5 configuration based on some trigger; they are to the F5 BIG-IP management plane what iRules are to the data plane. The trigger can be some event, condition, log message, time interval, etc. I needed my iCall script to do two things. First, pull the reputation list from the CrowdSec SE. Second, store that list somewhere accessible to the F5 data plane. Like many of you, my first thought was either an iFile or a data group. Both of these are easily configurable components accessible via iCall scripts that are also accessible via iRules. For several reasons that I will not bother to detail here, I did not want to use either of these solutions, primarily for performance reasons (this reputation lookup needs to be very performant). And the most performant place to store information like this is the session table. The session table is accessible to iRules via "table" commands. However, the session table is not accessible via iCall scripts. At least not directly. I realized that I could send an HTTP request using the iCall script, AND that HTTP request could be to a local virtual server on the same BIG-IP where I could use an iRule to populate the session table with the JA4 reputation list pulled from the CrowdSec SE. The iCall Script From the F5 BIG-IP CLI I created the following iCall script using the tmsh command 'tmsh create sys icall script crowdsec_ja4_rep_update': sys icall script crowdsec_ja4_rep_update { app-service none definition { package require http set csapi_resp [http::geturl http://10.0.2.240:8080/v1/decisions/stream?startup=true&scopes=ja4_fp -headers "X-api-Key 1a234xxxxxxxxxxxxxxe56fab7"] #tmsh::log "[http::data ${csapi_resp}]" set payload [http::data ${csapi_resp}] http::cleanup ${csapi_resp} set tupdate_resp [http::geturl http://10.0.1.110/updatetables -type "application/json" -query ${payload}] tmsh::log "[http::data ${tupdate_resp}]" http::cleanup ${tupdate_resp} } description none events none } Let's dig through this iCall script line by line: 4. Used to "require" or "include" the TCL http library. 5. HTTP request to the CrowdSec API to get the JA4 reputation list. 10.0.2.240:8080 is the IP:port of the CrowdSec SE API /v1/decisions/stream is the API endpoint used to grab an entire reputation list (rather than just query for the status of an individual IP/JA4) startup=true tells the API to send the entire list, not just additions/deletions since the last API call scopes=ja4_fp limits the returned results to just JA4 fingerprint-type decisions -headers "X-api-Key xxxxxxxxxxxxxxxxxxxxxxxxxx" includes the API key generated previously to authenticate the F5 BIG-IP as a "bouncer" 7. Store just the body of the API response in a variable called "payload" 8. free up memory used by the HTTP request tot eh CrowdSec API 9. HTTP Request to a local virtual server (on the same F5 BIG-IP) including the contents of the "payload" variable as the POST body. The IP address needs to be the IP address of the virtual server defined in the next step. An iRule will be created and placed on this virtual server that parses the "payload" and inserts the JA4 reputation list into the session table. An iCall script will not run unless an iCall handler is created that defines when that iCall script should run. I call handlers can be "triggered", "perpetual", or "periodic". I created the following periodic iCall handler to run this iCall script at regular intervals. sys icall handler periodic crowdsec-api-ja4 { interval 30 script crowdsec_ja4_list } This iCall handler is very simple; it has an "interval" for how often you want to run the script and the script that you want to run. I chose to run the iCall script every 30 seconds so that the BIGIP session table would be updated with any new malicious JA4 fingerprints very quickly. But you could choose to run the iCall script every 1 minute, 5 minutes, etc. The Table Updater Virtual Server and iRule I then created a HTTP virtual server with no pool associated to it. This virtual server exists solely to accept and process the HTTP requests from the iCall script. I then created the following iRule to process the requests and payload from the iCall script: proc duration2seconds {durstr} { set h 0 set m 0 set s 0 regexp {(\d+)h} ${durstr} junk h regexp {(\d+)m} ${durstr} junk m regexp {(\d+)\.} ${durstr} junk s set seconds [expr "(${h}*3600) + (${m}*60) + ${s}"] return $seconds } when HTTP_REQUEST { if { ([HTTP::uri] eq "/updatetables" || [HTTP::uri] eq "/lookuptables") && [HTTP::method] eq "POST"} { HTTP::collect [HTTP::header value "content-length"] } else { HTTP::respond 404 } } when HTTP_REQUEST_DATA { #log local0. "PAYLOAD: '[HTTP::payload]'" regexp {"deleted":\[([^\]]+)\]} [HTTP::payload] junk cs_deletes regexp {"new":\[([^\]]+)\]} [HTTP::payload] junk cs_adds if { ![info exists cs_adds] } { HTTP::respond 200 content "NO NEW ENTRIES" return } log local0. "CS Additions: '${cs_adds}'" set records [regexp -all -inline -- {\{([^\}]+)\},?} ${cs_adds}] set update_list [list] foreach {junk record} $records { set urec "" foreach k {scope value type scenario duration} { set v "" regexp -- "\"${k}\":\"?(\[^\",\]+)\"?,?" ${record} junk v log local0. "'${k}': '${v}'" if { ${k} eq "duration" } { set v [call duration2seconds ${v}] } append urec "${v}:" } set urec [string trimright ${urec} ":"] #log local0. "$urec" lappend update_list ${urec} } set response "" foreach entry $update_list { scan $entry {%[^:]:%[^:]:%[^:]:%[^:]:%s} scope entity type scenario duration if { [HTTP::uri] eq "/updatetables" } { table set "${scope}:${entity}" "${type}:${scenario}" indefinite $duration append response "ADDED ${scope}:${entity} FOR ${duration} -- " } elseif { [HTTP::uri] eq "/lookuptables" } { set remaining "" set action "" if { [set action [table lookup ${scope}:${entity}]] ne "" } { set remaining [table lifetime -remaining ${scope}:${entity}] append response "${scope}:${entity} - ${action} - ${remaining}s remaining\r\n" } else { append response "${scope}:${entity} - NOT IN TABLE\r\n" } } } HTTP::respond 200 content "${response}" } I have attempted to include sufficient inline comments so that the iRule is self-explanatory. If you have any questions or comments on this iRule please feel free to DM me. It is important to note here that the iRule is storing not only each JA4 fingerprint in the session table as a key but also the metadata passed back from the CrowdSec API about each JA4 reputation as the value for each key. This metadata includes the scenario name, the "type" or action, and the duration. So at this point I had a JA4 reputation list, updated continuously based on the WAF violation logs and CrowdSec scenarios. I also had an iCall script on the F5 BIG-IP that was pulling that reputation list via the local CrowdSec API every 30 seconds and pushing that reputation list into the local session table on the BIG-IP. Now I just needed to take some action based on that reputation list. Integrating JA4 Reputation into F5 WAF To integrate the JA4 reputation into the F5 Advanced WAF we only need two things: a custom violation defined in the WAF an iRule to lookup the JA4 in the local session table and raise that violation Creating a Custom Violation Creating a custom violation in F5 Advanced WAF (or ASM) will vary slightly depending on which version of the TMOS software you are running. In version 17.1 it is at Security ›› Options : Application Security : Advanced Configuration : Violations List. Select the User-Defined Violations tab and click Create. Give the Violation a Title and define the Type, Severity, and Attack Type. Finally, I modified the Learning and Blocking Settings of my policy to ensure that the new custom violation was set to Alarm and Block. F5 iRule for Custom Violation I then created the following iRule to raise this new custom WAF violation if the JA4 fingerprint is found in the reputation list in the local session table. when ASM_REQUEST_DONE { # Grab JA4 fingerprint from x-ja4 header # This header is inserted by the JA4 irule set ja4_fp [HTTP::header value "x-ja4"] # Lookup JA4 fingerprint in session table if { [set result [table lookup "ja4_fp:${ja4_fp}"]] ne "" } { # JA4 was found in session table, scan the value to get "category" and "action" scan ${result} {%[^:]:%s} action category # Initialize all the nested list of lists format required for the # violation details of the ASM::raise command set viol [] set viol_det1 [] set viol_det2 [] set viol_det3 [] # Populate the variables with values parsed from the session table for this JA4 lappend viol_det1 "JA4 FP" "${ja4_fp}" lappend viol_det2 "CrowdSec Category" "${category}" lappend viol_det3 "CrowdSec Action" "${action}" lappend viol ${viol_det1} ${viol_det2} ${viol_det3} # Raise custom ASM violation with violation details ASM::raise VIOL_JA4_REPUTATION ${viol} } } Again, I tried to include enough inline documentation for the iRule to be self-explanatory. Seeing It All In Action With everything in place, I sent several requests, most malicious and some benign, to the application protected by the F5 Advanced WAF. Initially, only the malicious requests were blocked. After about 60 seconds, ALL of the requests were being blocked due to the new custom violation based on JA4 reputation. Below is a screenshot from one of my honeypot WAF instances blocking real "in-the-wild" traffic based on JA4 reputation. Note that the WAF violation includes (1) the JA4 fingerprint, (2) the "category" (or scenario), and (3) the "action" (or type). Things to Note The API communication between the F5 BIG-IP and the CrowdSec SE is over HTTP. This is obviously insecure; for this proof-of-concept deployment I was just too lazy to spend the extra time to get signed certs on all the devices involved and alter the iCall script to use the TCL SSL library.
Decrypting SSL traffic - PMS and egress
Hi - two questions combined. Background - trying to catch and decipher tcpdump both for Client -> VIP and F5-> Pool Members traffic I'm following this tutorial: Decrypt with tcpdump --f5 ssl I managed to catch the frontend traffic, but I'm struggling with creating the PMS key. I want to automate it using the provided wireshark cmd command, but I get the error: C:\Program Files\Wireshark: invalid option -- 'T' C:\Program Files\Wireshark: invalid option -- 'e' I'm using Wireshark 3.4.8 - what would be the equivalent options for my version? Unfortunately using a Linux in this environment is out of the question. I can only work on Windows stepping stone and can't send the captures to my PC Second issue: Catching the backend traffic does not produce the F5 TLS in the pcap capture... The server ssl profile is present, but I have no idea how to force the --f5 ssl option in tcpdump to catch the keys. Will appreciate any advice - It is my second day struggling with the issue
