SSL Forward Proxy – Certificate Error Graceful Failure

Technical Challenge

Recently I needed to deploy the SSL Forward Proxy functionality on a BIG-IP so that I could inspect HTTPS traffic on the fly. The goal was to detect malicious traffic hidden inside the SSL/TLS payload and drop those connection before they reached the client.

After completing the Implementing SSL Forward Proxy deployment guide everything worked great and I was able to inspect SSL/TLS requests.

Now to put the final touches on my deployment I wanted to handle untrusted/expired certificates differently than trusted ones.

Inside the ServerSSL profile that was created enable Server Certificate validation and set Trusted Certificate Authorities to the default ca-bundle.

And this is where things got challenging. The SSL Serverside profile that is used for SSL Forward Proxy only supports drop or ignore for untrusted/expired certificates. but that was a little more user impacting than I was looking for.

Drop would generate more service-desk calls than I wanted and ignore would improperly mark untrusted/expired certificates as valid.

So how do you gracefully handle untrusted/expired certificates when leverage SSL Forward Proxy?

The Solution

There is an iRule for it

It took a little bit of work and a couple rants on how it can’t be done before I identified a clean workaround for the issue above.

The first part was identifying the difference between a valid and untrusted/expired certificate, and to do this I used my favorite debugging iRule

when FLOW_INIT priority 1 { log local0. "EVENT FIRED" }
when CLIENT_ACCEPTED priority 1 { log local0. "EVENT FIRED" }
when SERVER_CONNECTED priority 1 { log local0. "EVENT FIRED" }
when CLIENTSSL_CLIENTCERT priority 1 { log local0. "EVENT FIRED" }
when CLIENTSSL_CLIENTHELLO priority 1 { log local0. "EVENT FIRED" }
when CLIENTSSL_HANDSHAKE priority 1 { log local0. "EVENT FIRED" } 
when CLIENTSSL_SERVERHELLO_SEND priority 1 { log local0. "EVENT FIRED" } 
when SERVERSSL_CLIENTHELLO_SEND priority 1 { log local0. "EVENT FIRED" } 
when SERVERSSL_HANDSHAKE priority 1 { log local0. "EVENT FIRED" } 
when SERVERSSL_SERVERHELLO priority 1 { log local0. "EVENT FIRED" } 
when LB_FAILED priority 1 { log local0. "EVENT FIRED" } 
when LB_SELECTED priority 1 { log local0. "EVENT FIRED" } 
when CLIENT_CLOSED priority 1 { log local0. "EVENT FIRED" } 
when SERVER_CLOSED priority 1 { log local0. "EVENT FIRED" }

Now before I generate any traffic it is important to clear out the any SSL Certs that were cached

tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [Client-SSL Profile Name] virtual [Virtual Server Name]

After clearing the cert cache generate a SSL/TLS request to a valid site and collect the debug information from /var/log/ltm and then repeat the steps for a site that you know will have an untrusted/expired certificate

You should end up with something similar to this for your valid request

Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <FLOW_INIT>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_ACCEPTED>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_CLIENTHELLO>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>: EVENT FIRED
Oct 22 23:50:53 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: EVENT FIRED
Oct 22 23:50:54 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_HANDSHAKE>: EVENT FIRED
Oct 22 23:50:54 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_HANDSHAKE>: EVENT FIRED
Oct 22 23:50:54 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_CLOSED>: EVENT FIRED
Oct 22 23:50:54 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>: EVENT FIRED

And something like this for your untrusted request

Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <FLOW_INIT>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_ACCEPTED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_CLIENTHELLO>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <LB_FAILED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_CLOSED>: EVENT FIRED
Oct 22 23:51:23 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>: EVENT FIRED

Now if we load this information into our favorite diff tool we should see a couple major differences


  1. The SERVERSSL_HANDSHAKE event never fires for the untrusted request
  2. The LB_FAILED event fires during the untrusted request
  3. Before both of these events are fired the SERVERSSL_SERVERHELLO event fires

Now what does this mean? It tells me two things, first that I need to identify the state of a certificate before the SERVERSSL_HANDSHAKE event will fire and that an untrusted/expired certificate will trigger the LB_FAILED event.

If we take a look at SSL iRule options we will see a bunch of different functions but which one will do what we need? The first thing we need to do is disable Certificate Verification so let’s take a look at SSL::cert, and it looks like SSL::cert mode can handle this part.

Earlier we identified that the last event to fire in both trusted and untrusted was SERVERSSL_SERVERHELLO so let’s see what happens when we add this command to that event

when SERVERSSL_SERVERHELLO {
log local0. "CERT MODE BEFORE => [SSL::cert mode]"
SSL::cert mode ignore
log local0. "CERT MODE AFTER => [SSL::cert mode]"
}

And if we re-test our untrusted website we will see that it now works. And /var/log/ltm looks like this.

Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <FLOW_INIT>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_ACCEPTED>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_CLIENTHELLO>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>: => EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: => EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE BEFORE => require
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE AFTER => ignore
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVERSSL_HANDSHAKE>: => EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_SERVERHELLO_SEND>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_HANDSHAKE>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <CLIENT_CLOSED>: EVENT FIRED
Oct 23 03:55:04 slot1/mybigip info tmm1[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>: => EVENT FIRED

But if we leave our iRule like this it will disable certificate validation for all websites. So let’s go back to the event comparison earlier and see what else we have to work with. It looks like the LB_FAILED event only fires for untrusted websites so let’s add an iRule snippet to detect untrusted certificates.

Replace the SERVERSSL_SERVERHELLO logic we created earlier with this. This iRule will store the SSL::cert mode as a variable that can be used during the LB_FAILED event. If the variable s_certmode exists and doesn’t equal ignore it will trigger the LB::reselect function within LB::failed and set the cert mode to ignore.

when SERVERSSL_SERVERHELLO {
log local0. "CERT MODE BEFORE => [SSL::cert mode]"
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
SSL::cert mode ignore
}
log local0. "CERT MODE AFTER => [SSL::cert mode]"

set s_certmode [SSL::cert mode]
}
when LB_FAILED {
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
LB::reselect
}
}

Now if we re-test our untrusted website and watch /var/log/ltm should look similar to this

Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <FLOW_INIT>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENT_ACCEPTED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_CLIENTHELLO>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>:  EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE BEFORE => require
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE AFTER => require
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <LB_FAILED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <LB_SELECTED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVER_CONNECTED>:  EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_CLIENTHELLO_SEND>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE BEFORE => require
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_SERVERHELLO>: CERT MODE AFTER => ignore
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVERSSL_HANDSHAKE>:  => EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_SERVERHELLO_SEND>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENTSSL_HANDSHAKE>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <CLIENT_CLOSED>: EVENT FIRED
Oct 23 03:57:46 slot1/mybigip info tmm[21809]: Rule /Common/sslForwardProxy <SERVER_CLOSED>:  => EVENT FIRED

Here we can see that the first request to the untrusted website fails and the certificate verification mode is set to require, but on the second request it get’s updated to ignore.

Perfect this exactly what we wanted, a graceful failure for untrusted/expired certificates.

Next we want to prevent trusted certificates from being generated for untrusted websites. This part is easy

Create a new ClientSSL Profile with SSL Forward Proxy enabled that has an untrusted RootCA applied

tmsh create ltm profile client-ssl clientssl_proxy-untrusted proxy-ca-cert untrusted.crt proxy-ca-key untrusted.key ssl-forward-proxy enabled ssl-forward-proxy-bypass enabled
tmsh create ltm profile server-ssl serverssl_proxy-untrusted ssl-forward-proxy enabled ssl-forward-proxy-bypass enabled

After creating the new ClientSSL Profile you will need to update the LB_FAILED section of our iRule from earlier

when LB_FAILED {
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
SSL::profile "/Common/clientssl_proxy-untrusted"
LB::reselect
}
}

What we have configured so far will work great for curl and any other non-interactive browsers, but will get stuck in a certificate validation loop for most modern browsers.

To work around this we need to add SSL::forward_proxy bypass logic to our iRule and create a second virtual server that will be targeted with the virtual to virtual iRule function.

The layered virtual server should be a wildcard virtual and have no vlans assigned to it

tmsh create ltm virtual lvs_CertificateError profiles add { clientssl_proxy-untrusted serverssl_proxy-untrusted tcp} vlans none vlans-enabled

The last bit of iRule logic needs to execute within the CLIENTSSL_SERVERHELLO_SEND event so that we can enable forward_proxy bypass and forward it to the layered virtual server. If we look at the event list above we can see that CLIENTSSL_SERVERHELLO_SEND event executes after we disabled certificate verification. This means we will need to change our logic will need to expect the s_certmode variable to be equal to ignore. This will prevent us from generating untrusted certificates for valid websites.

when CLIENTSSL_SERVERHELLO_SEND {
if {[info exists s_certmode] && $s_certmode eq "ignore"}
{
SSL::forward_proxy policy bypass
}
}
when LB_SELECTED {
if {[info exists s_certmode] && $s_certmode eq "ignore"}
{
virtual lvs_CertificateError
LB::reselect
}
}

Now retest and you should get a valid certificate for trusted websites and an untrusted certificate error message for untrusted websites.

Putting Everything Together

This section will make the following assumptions

  1. You have already created a trusted Root Certificate and installed it on end user workstations.
  2. You have completed Implementing SSL Forward Proxy and have a working environment
  3. That you have enabled SSL Forward Proxy Bypass on both the clientssl and serverssl profiles

Step 1 – Use OpenSSL to create an untrusted

## This will create the root key that will be used to for your untrusted RootCA
openssl genrsa -des3 -out UntrustedCA.key 2048 
## For the purposes of this demo and to make it easier to move the file around I will decrypt the PEM file
openssl rsa -in UntrustedCA.key -out UntrustedCA.pem 
## Next you will create the certificate that will be used for your untrusted RootCA
openssl req -x509 -new -nodes -key UntrustedCA.key -days 3650 -out UntrustedCA.cert
## And then you will be prompted for additional information
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:Washington
Locality Name (eg, city) [Default City]:Seattle
Organization Name (eg, company) [Default Company Ltd]:F5 Networks
Organizational Unit Name (eg, section) []:DO NOT TRUST ME
Common Name (eg, your name or your server's hostname) []:DO NOT TRUST ME
Email Address []:

Step 2 – Create the ClientSSL & ServerSSL profile to handle Certificate Errors

This profile will be used to generate certificates for any website that is responds with an expired or untrusted certificate.

tmsh create ltm profile client-ssl clientssl_proxy-untrusted proxy-ca-cert untrusted.crt proxy-ca-key untrusted.key ssl-forward-proxy enabled ssl-forward-proxy-bypass enabled
tmsh create ltm profile server-ssl serverssl_proxy-untrusted ssl-forward-proxy enabled ssl-forward-proxy-bypass enabled

Step 3 – Create the Layered Virtual Server that untrusted requests will be sent

This virtual server shouldn’t have any vlans assigned to it, this will prevent unexpected traffic from hitting it.

tmsh create ltm virtual lvs_CertificateError profiles add { clientssl_proxy-untrusted serverssl_proxy-untrusted tcp} vlans none vlans-enabled

Step 4 – Create our Graceful Failure iRule and apply it the appropriate Virtual Server

when SERVERSSL_SERVERHELLO{
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
SSL::cert mode ignore
}
set s_certmode [SSL::cert mode]
}
when LB_FAILED {
if {[info exists s_certmode] && $s_certmode ne "ignore"}
{
SSL::profile "/Common/clientssl_proxy-untrusted"
LB::reselect
}
}
when CLIENTSSL_SERVERHELLO_SEND {
if {[info exists s_certmode] && $s_certmode eq "ignore"}
{
SSL::forward_proxy policy bypass
}
}
when LB_SELECTED {
if {[info exists s_certmode] && $s_certmode eq "ignore"}
{
virtual lvs_CertificateError
LB::reselect
}
}

Step 5 - Clear the cached-certs

Now before I generate any traffic it is important to clear out the any SSL Certs that were cached

tmsh delete ltm clientssl-proxy cached-certs clientssl-profile [Client-SSL Profile Name] virtual [Virtual Server Name]
Published Oct 22, 2015
Version 1.0

Was this article helpful?

7 Comments

Sort By
  • rob_carr's avatar
    rob_carr
    Icon for Cirrostratus rankCirrostratus
    Oct 26, 2015
    Your log snippets and what you fed into diff don't match - you should update your rule to include the event name and then repost the log snippets for clarity.
  • Aurel's avatar
    Aurel
    Icon for Cirrus rankCirrus
    Nov 02, 2015
    Hi Robert, I'm seeing CERT MODE AFTER 2 times in the logs you show, but not CERT MODE BEFORE as written in the iRule. Does that really mean the event is matched 2 times ? or is it a simple info typo in the rule ? thank you
  • Robert_Teller_7's avatar
    Robert_Teller_7
    Historic F5 Account
    Nov 02, 2015
    Aurel, the iRule logic is collect the logging snippet i included was from a previous run and didn't correctly reflect what you should see. I have updated the example logging output to what it should be. Robert
  • Aurel's avatar
    Aurel
    Icon for Cirrus rankCirrus
    Nov 04, 2015
    Thank you for the update. I got everything now, if i'm not mistaken. This is a very interesting job. A question : if the server cert is expired (trusted or not), then the client will have an untrusted error message, right ? It should be easy to update it with a relevant ssl client profile for an expired cert message to the client, but i don't know if browsers behaviour may block without warning, whereas still displaying warning with untrusted.
  • Robert_Teller_7's avatar
    Robert_Teller_7
    Historic F5 Account
    Nov 05, 2015
    That is correct if the certificate is valid but expired an untrusted cert will still be generated. At this time there isn't a way to generate an expired trusted cert.
  • ShivaNaidu's avatar
    ShivaNaidu
    Icon for Nimbostratus rankNimbostratus
    Jan 30, 2024

    Thank you for your update. If I recall correctly, I now have everything. This is an extremely intriguing profession. A question: if the server certificate has expired (trusted or not), the client will receive an untrusted error message, correct? It should be simple to update it with a relevant SSL client profile for an expired certificate message to the client, however I'm not sure if browser behaviour will block without warning while still displaying the warning with untrusted.

    If want to know more about SSL Certificate Error Check The Blogs >> https://www.cheapsslshop.com/blog/category/ssl-error