Understanding Server SSL Trusted Certificate Authorities on BIG-IP

1 Quick Intro

This article is about how this Server SSL option works on BIG-IP:

We'll create a certificate chain for 2 back-end servers using OpenSSL and copy the root CA of each server to Trusted Certificate Authorities on Server SSL profile:

This way, when BIG-IP is configured to validate back-end server certificates, i.e. when Server Certificate (peer-cert-mode in tmsh) is set to require, BIG-IP can trust back-end certificates that were signed by certificates in the file we added to Trusted Certificate Authorities (ca-file in tmsh).

Also, because Trusted Certificate Authorities file only contain the Root CAs (and not the intermediate ones), back-end servers should also send the intermediate certificates so that the whole chain can be verified by BIG-IP.

Note: for production environment, you'll probably going to use the same chain on both servers. The reason I create 2 separate chains here was just to add 2 concatenated CA's public keys to Trusted Certificate Authorities for demonstration purposes.

For the purpose of this article, here's what we're going to be doing if you follow along:

Creating Certificate Chain using OpenSSL
Adding Certificate Chain to Apache and NGINX
Adding Root CAs to Trusted Certificate Authorities on BIG-IP
Confirming it all works with curl command

2 Creating Certificate Chain using OpenSSL

First we create the Root Certificate Authority that is going to sign our intermediate certificate.

Note that the same steps are done for server2 bundle so I'm not repeating it here.

I first created a directory to keep our certificate/key files in:

The next steps would be to create the CAs and the end entity certificate along with corresponding keys.

2.1 Creating server1_rootCA.key

The private key (server1_rootCA.key) is the first one to be created since public key is generated from private key:

Note: Are you wondering why I added -aes256 which is symmetric key whilst creating a private key, which is asymmetric key? That's only to protect the private key's contents in disk. For example, if an attacker gets hold of server1_rootCA.key, they can create a public key with such a key or decrypt any TLS traffic protected by server1_rootCA.crt. This is because our keys here are PEM encoded by default, but not encrypted. To avoid this, we can encrypt it at disk (known as encryption at rest) using symmetric key encryption with AES 256. By the way, openssl command to verify contents of private key is openssl rsa -text -in <key>.key

2.2 Creating server1_rootCA.crt

Now we create our public key:

Note: The required fields were filled for illustration purposes.

2.3 Creating server1_intermediateCA.key

As always, we also create the private key first:

2.4 Creating server1_intermediateCA.csr

Wow, wait! Why .csr? Why not .crt like we did before?

The reason is because our intermediate CA will be signed by our Root CA, remember?

We'd only create a .crt directly if we were creating a self-signed certificate like our Root CA.

The idea of the .csr is that we'll set up all the information we want our certificate to have and we'll use our Root CA to turn it into a .crt file.

Does it make sense?

2.5 Creating server1_intermediateCA.crt

Based on the information from server1_intermediateCA.csr, we can now use server1_rootCA.key to create server1_intermediateCA.crt.

However, before we do that, we may want to add some capabilities to our intermediate certificate which is recommended by man x509v3_config command:

Note: the flags above are out of the scope of this article but setting basicConstraints to critical means that for validation purposes, the flags we're setting have to be taken into consideration. For example, CA:true means that this is a CA capable of signing other certificates such as our end-of-chain certificate that we'll create after that. We'd set this flag to false when creating end of chain client or server certificates as they're not supposed to sign other certificates. The keyUsage indicates what this CA should be used for.

We're now using these extensions to create and sign server1_intermediateCA.crt using Root CA's private key:

This way, if BIG-IP has Root CA's public key, it can confirm that this certificate was indeed signed by Root CA's private key.

2.6 Creating server1_endofchain.key

Just like Root and Intermediate CAs, we create private key first:

2.6 Creating server1_endofchain.csr

We now type the info we want our .crt file to have once it's signed by our intermediate CA:

2.6 Creating server1_endofchain.crt

We also have an extension file here with the typical back-end server flags and constraints that can be found in man page as well:

And we now create server1_intermediateCA.crt:

We can also verify csr/crt/key as they should all output the same md5 hash:

Note: For server2, we use the same steps but replacing server1 with server2 when appropriate.

3 Adding Certificate Chain to WebServer

3.1 Apache

If not enabled, activate ssl module using a2enmod command:

Don't restart Apache just now because we'll first add our end entity certificate and intermediate certificate to Apache's config file first.

First I'll the default ssl file from sites-available to sites-enabled because only files in sites-enabled are actually active on the webserver.

As Apache already has a default template, I'll create a symbolic link as this is just for demonstration purposes:

And the only thing I did was to edit SSLCertificateFile, SSLCertificateKeyFile and SSLCertificateChainFile and add our end entity certificate, end entity key and intermediate CA certificate respectively:

Once we try to restart Apache, we need to type in our passphrase for the certificates encrypted with AES256 at rest, remember? Otherwise, Apache won't be able to read it.

We now confirm Apache is running properly:

And most importantly, that we can reach our SSL protected website locally:

And from BIG-IP:

Note: the k flag on curl command skips certificate validation and for the moment that's what we want. Even if we add certificate validation to BIG-IP configuration, we need to make sure we understand that our curl command above is executed from BIG-IP's control plane (Linux) so a certificate added to BIG-IP's configuration file only applies to BIG-IP's forwarding plane (tmm). Therefore, even after we add the correct valid certificate to Trusted Certificate Authorities, the curl command above will not automatically reference it.

3.2 NGINX

If we're using NGINX webserver, we should bundle our end entity certificate together with intermediate CA's certificate:

And add it to your webserver's configuration on /etc/nginx/conf.d/ directory:

In this case, NGINX will send 2 certificates (end entity and intermediate one) back to BIG-IP.

FYI, a bundled chain is just one PEM certificate concatenated with the other:

Not that hard to picture, is it?

And lastly, we reload NGINX:

For more details, please have a look at NGINX documentation.

4 Adding Root CAs to Trusted Certificate Authorities on BIG-IP

4.1 What do we add to Trusted Certificate Authorities?

One important concept to grasp is that we don't need to copy the whole certificate chain to BIG-IP.

The chain belongs to your back-end server.

The only thing we need from the chain to add to BIG-IP is the top-level Root certificate(s) that signed the chain BIG-IP will verify.

From the point of view of BIG-IP, we'll make it trust whichever certificate is signed by CA list added to Trusted Certificate Authorities.

Therefore, we only need to add server1_rootCA.crt and server2_rootCA.crt in our lab test.

I have copied both Root CAs from Server1 and Server2 to a temporary directory on BIG-IP:

I'll make it a bundle named servers_rootCA.crt:

4.2 Importing Root CA to BIG-IP

4.2.1 Via GUI

We can now import it to BIG-IP.

In the GUI on v15.x, you'd go to Certificate Management → Traffic Certificate Management → SSL Certificate List → Import:

Select Import Type → Certificate:

And upload (or copy/paste it if you wish) the contents of servers_rootCA.crt:

4.2.2 Via TMSH

As BIG-IP versions change, GUI is more susceptible to changes.

If that is the case and this article ever gets outdated, here's the tmsh command equivalent to import certificate chain:

4.3 Adding Chain to Trusted Certificate Authorities

If we go to our Server SSL profile we should now see servers_rootCA.crt as option to add to Trusted Certificate Authorities:

And we just click on Update.

Note: notice that Server Certificate is set to require! Don't forget that!

4.4 Final test with Curl command

For this test I disabled Generic Alert on Server SSL profile:

Otherwise, BIG-IP would not tell us the real reason why handshake failed.

4.4.1 Trusted Certificate Authorities set to None

When I issued curl command from my client it failed as expected because Server Certificate is set to require and we added nothing to Trusted Certificate Authorities:

And I took the opportunity to take a tcpdump capture in parallel:

And on Wireshark we can clearly see the reason why our curl command failed:

BIG-IP doesn't recognise the CA as trusted because it's not on Trusted Certificate Authorities.

In fact, there's nothing there so any back-end certificate would fail anyway.

4.4.2 Trusted Certificate Authorities set to servers_rootCA.crt

Now our curl command works:

I also took a tcpdump capture here:

In fact, one thing that is interesting to note is that we can see that Certificate message sent from back-end server only has the intermediate CA and the end entity certificate:

If we wanted, we could've configured Apache or NGINX to add Root CA to the bundle but it's not necessary.

The reason why is because BIG-IP already has the Root CA in the bundle we added to Trusted Certificate Authorities, remember?

That's it for now.

Appendix 1 When your certificate is signed by a well-known Root CA

Web browsers have somewhere close to a thousand trusted Root CAs by default.

These were added by the developers of your web browser.

BIG-IP has a default file called ca-bundle.crt with the usual browser's trusted CA repository and on v15.x it contains 857 trusted CAs:

If your certificate was signed by one of the major CAs out there, it's likely the only thing you'll need to do in production would be to use ca-bundle.crt: