Technical Challenge

Recently I needed to deploy the SSL Forward Proxy functionality on a BIG-IP so that I could inspect HTTPS traffic on the fly. The goal was to detect malicious traffic hidden inside the SSL/TLS payload and drop those connection before they reached the client.

After completing the Implementing SSL Forward Proxy deployment guide everything worked great and I was able to inspect SSL/TLS requests.

Now to put the final touches on my deployment I wanted to handle untrusted/expired certificates differently than trusted ones.

Inside the ServerSSL profile that was created enable Server Certificate validation and set Trusted Certificate Authorities to the default ca-bundle.

And this is where things got challenging. The SSL Serverside profile that is used for SSL Forward Proxy only supports drop or ignore for untrusted/expired certificates. but that was a little more user impacting than I was looking for.

Drop would generate more service-desk calls than I wanted and ignore would improperly mark untrusted/expired certificates as valid.

So how do you gracefully handle untrusted/expired certificates when leverage SSL Forward Proxy?

The Solution

There is an iRule for it

It took a little bit of work and a couple rants on how it can’t be done before I identified a clean workaround for the issue above.

The first part was identifying the difference between a valid and untrusted/expired certificate, and to do this I used my favorite debugging iRule

when FLOW_INIT priority 1 { log local0. "EVENT FIRED" }
when CLIENT_ACCEPTED priority 1 { log local0. "EVENT FIRED" }
when SERVER_CONNECTED priority 1 { log local0. "EVENT FIRED" }
when CLIENTSSL_CLIENTCERT priority 1 { log local0. "EVENT FIRED" }
when CLIENTSSL_CLIENTHELLO priority 1 { log local0. "EVENT FIRED" }
when CLIENTSSL_HANDSHAKE priority 1 { log local0. "EVENT FIRED" } 
when CLIENTSSL_SERVERHELLO_SEND priority 1 { log local0. "EVENT FIRED" } 
when SERVERSSL_CLIENTHELLO_SEND priority 1 { log local0. "EVENT FIRED" } 
when SERVERSSL_HANDSHAKE priority 1 { log local0. "EVENT FIRED" } 
when SERVERSSL_SERVERHELLO priority 1 { log local0. "EVENT FIRED" } 
when LB_FAILED priority 1 { log local0. "EVENT FIRED" } 
when LB_SELECTED priority 1 { log local0. "EVENT FIRED" } 
when CLIENT_CLOSED priority 1 { log local0. "EVENT FIRED" } 
when SERVER_CLOSED priority 1 { log local0. "EVENT FIRED" } 

Now before I generate any traffic it is important to clear out the any SSL Certs that were cached

tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [Client-SSL Profile Name] virtual [Virtual Server Name]

After clearing the cert cache generate a SSL/TLS request to a valid site and collect the debug information from /var/log/ltm and then repeat the steps for a site that you know will have an untrusted/expired certificate

You should end up with something similar to this for your valid request

Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <FLOW_INIT>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_ACCEPTED>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_CLIENTHELLO>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: EVENT FIRED
Oct 22 23:50:54 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_HANDSHAKE>: EVENT FIRED
Oct 22 23:50:54 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_HANDSHAKE>: EVENT FIRED
Oct 22 23:50:54 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_CLOSED>: EVENT FIRED
Oct 22 23:50:54 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>: EVENT FIRED

And something like this for your untrusted request

Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <FLOW_INIT>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_ACCEPTED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_CLIENTHELLO>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <LB_FAILED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_CLOSED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>: EVENT FIRED

Now if we load this information into our favorite diff tool we should see a couple major differences

1. The SERVERSSL_HANDSHAKE event never fires for the untrusted request
2. The LB_FAILED event fires during the untrusted request
3. Before both of these events are fired the SERVERSSL_SERVERHELLO event fires

Now what does this mean? It tells me two things, first that I need to identify the state of a certificate before the SERVERSSL_HANDSHAKE event will fire and that an untrusted/expired certificate will trigger the LB_FAILED event.

If we take a look at SSL iRule options we will see a bunch of different functions but which one will do what we need? The first thing we need to do is disable Certificate Verification so let’s take a look at SSL::cert, and it looks like SSL::cert mode can handle this part.

Earlier we identified that the last event to fire in both trusted and untrusted was SERVERSSL_SERVERHELLO so let’s see what happens when we add this command to that event

when SERVERSSL_SERVERHELLO {
log local0. "CERT MODE BEFORE => [SSL::cert mode]"
SSL::cert mode ignore
log local0. "CERT MODE AFTER => [SSL::cert mode]"
}

And if we re-test our untrusted website we will see that it now works. And /var/log/ltm looks like this.

Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <FLOW_INIT>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_ACCEPTED>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_CLIENTHELLO>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>: => EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: => EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE BEFORE => require
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE AFTER => ignore
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_HANDSHAKE>: => EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_SERVERHELLO_SEND>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_HANDSHAKE>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_CLOSED>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>: => EVENT FIRED

But if we leave our iRule like this it will disable certificate validation for all websites. So let’s go back to the event comparison earlier and see what else we have to work with. It looks like the LB_FAILED event only fires for untrusted websites so let’s add an iRule snippet to detect untrusted certificates.

Replace the SERVERSSL_SERVERHELLO logic we created earlier with this. This iRule will store the SSL::cert mode as a variable that can be used during the LB_FAILED event. If the variable s_certmode exists and doesn’t equal ignore it will trigger the LB::reselect function within LB::failed and set the cert mode to ignore.

when SERVERSSL_SERVERHELLO {
log local0. "CERT MODE BEFORE => [SSL::cert mode]"
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
SSL::cert mode ignore
}
log local0. "CERT MODE AFTER => [SSL::cert mode]"

set s_certmode [SSL::cert mode]
}
when LB_FAILED {
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
LB::reselect
}
}

Now if we re-test our untrusted website and watch /var/log/ltm should look similar to this

Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <FLOW_INIT>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENT_ACCEPTED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_CLIENTHELLO>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>:  EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE BEFORE => require
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE AFTER => require
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <LB_FAILED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>:  EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE BEFORE => require
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE AFTER => ignore
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_HANDSHAKE>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_SERVERHELLO_SEND>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_HANDSHAKE>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENT_CLOSED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>:  => EVENT FIRED

Here we can see that the first request to the untrusted website fails and the certificate verification mode is set to require, but on the second request it get’s updated to ignore.

Perfect this exactly what we wanted, a graceful failure for untrusted/expired certificates.

Next we want to prevent trusted certificates from being generated for untrusted websites. This part is easy

Create a new ClientSSL Profile with SSL Forward Proxy enabled that has an untrusted RootCA applied

tmsh create ltm profile client-ssl clientssl_proxy-untrusted proxy-ca-cert untrusted.crt proxy-ca-key untrusted.key ssl-forward-proxy enabled ssl-forward-proxy-bypass enabled
tmsh create ltm profile server-ssl serverssl_proxy-untrusted ssl-forward-proxy enabled ssl-forward-proxy-bypass enabled 

After creating the new ClientSSL Profile you will need to update the LB_FAILED section of our iRule from earlier

when LB_FAILED {
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
SSL::profile "/Common/clientssl_proxy-untrusted"
LB::reselect
}
}

What we have configured so far will work great for curl and any other non-interactive browsers, but will get stuck in a certificate validation loop for most modern browsers.

To work around this we need to add SSL::forward_proxy bypass logic to our iRule and create a second virtual server that will be targeted with the virtual to virtual iRule function.

The layered virtual server should be a wildcard virtual and have no vlans assigned to it

tmsh create ltm virtual lvs_CertificateError profiles add { clientssl_proxy-untrusted serverssl_proxy-untrusted tcp} vlans none vlans-enabled

The last bit of iRule logic needs to execute within the CLIENTSSL_SERVERHELLO_SEND event so that we can enable forward_proxy bypass and forward it to the layered virtual server. If we look at the event list above we can see that CLIENTSSL_SERVERHELLO_SEND event executes after we disabled certificate verification. This means we will need to change our logic will need to expect the s_certmode variable to be equal to ignore. This will prevent us from generating untrusted certificates for valid websites.

when CLIENTSSL_SERVERHELLO_SEND {
if {[info exists s_certmode] && $s_certmode eq "ignore"}
{
SSL::forward_proxy policy bypass
}
}
when LB_SELECTED {
if {[info exists s_certmode] && $s_certmode eq "ignore"}
{
virtual lvs_CertificateError
LB::reselect
}
}

Now retest and you should get a valid certificate for trusted websites and an untrusted certificate error message for untrusted websites.

Putting Everything Together

This section will make the following assumptions

1. You have already created a trusted Root Certificate and installed it on end user workstations.
2. You have completed Implementing SSL Forward Proxy and have a working environment
3. That you have enabled SSL Forward Proxy Bypass on both the clientssl and serverssl profiles

Step 1 – Use OpenSSL to create an untrusted

## This will create the root key that will be used to for your untrusted RootCA
openssl genrsa -des3 -out UntrustedCA.key 2048 
## For the purposes of this demo and to make it easier to move the file around I will decrypt the PEM file
openssl rsa -in UntrustedCA.key -out UntrustedCA.pem 
## Next you will create the certificate that will be used for your untrusted RootCA
openssl req -x509 -new -nodes -key UntrustedCA.key -days 3650 -out UntrustedCA.cert
## And then you will be prompted for additional information
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:Washington
Locality Name (eg, city) [Default City]:Seattle
Organization Name (eg, company) [Default Company Ltd]:F5 Networks
Organizational Unit Name (eg, section) []:DO NOT TRUST ME
Common Name (eg, your name or your server's hostname) []:DO NOT TRUST ME
Email Address []:

Step 2 – Create the ClientSSL & ServerSSL profile to handle Certificate Errors

This profile will be used to generate certificates for any website that is responds with an expired or untrusted certificate.

tmsh create ltm profile client-ssl clientssl_proxy-untrusted proxy-ca-cert untrusted.crt proxy-ca-key untrusted.key ssl-forward-proxy enabled ssl-forward-proxy-bypass enabled
tmsh create ltm profile server-ssl serverssl_proxy-untrusted ssl-forward-proxy enabled ssl-forward-proxy-bypass enabled 

Step 3 – Create the Layered Virtual Server that untrusted requests will be sent

This virtual server shouldn’t have any vlans assigned to it, this will prevent unexpected traffic from hitting it.

tmsh create ltm virtual lvs_CertificateError profiles add { clientssl_proxy-untrusted serverssl_proxy-untrusted tcp} vlans none vlans-enabled

Step 4 – Create our Graceful Failure iRule and apply it the appropriate Virtual Server

when SERVERSSL_SERVERHELLO{
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
SSL::cert mode ignore
}
set s_certmode [SSL::cert mode]
}
when LB_FAILED {
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
SSL::profile "/Common/clientssl_proxy-untrusted"
LB::reselect
}
}
when CLIENTSSL_SERVERHELLO_SEND {
if {[info exists s_certmode] && $s_certmode eq "ignore"}
{
SSL::forward_proxy policy bypass
}
}
when LB_SELECTED {
if {[info exists s_certmode] && $s_certmode eq "ignore"}
{
virtual lvs_CertificateError
LB::reselect
}
}

Step 5 - Clear the cached-certs

Now before I generate any traffic it is important to clear out the any SSL Certs that were cached

tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [Client-SSL Profile Name] virtual [Virtual Server Name]