Automating ACMEv2 Certificate Management on BIG-IP

Introduction

Back in March of 2023, Google proposed a significant reduction in the lifespan of Internet certificates, from the average 13 months down to just 90 days. No firm date was set for this proposal, though it should at least give a clue into where the industry is headed. Put plainly, the longer a certificate is valid, the less reliable it becomes. But perhaps the most important takeaway from all of this is, that while shorter-lived certificates are a good thing in principle, the current, common, manual approach to managing these 90-day certificates will become all but untenable. There is practically no scenario where an enterprise would or should employ, nor rely, on any human person, to manage certificates that live for 3 months. This is a recipe for disaster, and all the more reason to embrace certificate automation.

In this article, I’m specifically going to address this need for the F5 BIG-IP using the most well-known of the certificate automation protocols, ACMEv2. For what it’s worth, certificate automation in principle isn’t that complicated. There are a number of protocols (i.e., ACME, SCEP, CMP, EST), a variety of implementations for each of these, and lots of documentation that makes it seem like rocket science. But in truth, these protocols and processes all do more or less three things:

• Make a request to a certificate authority for a new certificate
• Satisfy a challenge to verify ownership of the certificate/domain
• Retrieve and install the new certificate

The “client” in this case is some utility, typically within arms-reach of the TLS server, that will perform the automation functions and push the new certificate(s) to the server. 

I will propose here, an ACMEv2 certificate automation solution that runs ON the BIG-IP, that performs all of the necessary and ancillary functions needed to support an enterprise BIG-IP certificate renewal strategy, and supports most of the ACMEv2 8555 RFC functionality, not just what is required for Let’s Encrypt. The project is called Kojot Acme and is basically two things:

• A wrapper for the Dehydrated ACMEv2 client.
• A utility to support all of the additional, ancillary functions, including installation, configuration, authentication, revocation, scheduling, and reporting.

Now, in lieu of repeating everything that is thoroughly documented on the project page, I thought it more fun to come at this from a different angle. In the below, we’ll set up a local ACMEv2 server, install and configure Kojot ACME on the BIG-IP, and start generating certificates. Once you see how it works, it’s a super simple manipulation of a config file to point to a public ACMEv2 server to get real certificates (ex. Let’s Encrypt, ZeroSSL, Buypass, etc.). But also, for those of you in military bunkers miles below the surface, where no sunlight or Internet is accessible to you, you’ll likely have your own ACMEv2 servers installed somewhere in a cave nearby. Kojot Acme will work for that too. Kojot is heavily influenced by Jason Rahm's pivotal ACMEv2 client work. Let’s get started!

A Quick Review of the ACMEv2 Protocol

Before we go stumbling through this experience, it is perhaps best to step back and take another look at the ACMEv2 protocol as a whole so that we know what’s happening. I’ll start with a ridiculously simple flow diagram, as described in the introduction.

The client asks for a new certificate, the server asks the client to prove ownership, and then the server issues a new certificate. That’s basic ACME protocol flow. Now let’s overlay the above with the TLS server, the thing that actually needs the cert.

Let’s unpack this.

1. Some process needs to know when to renew the certificate(s). This isn’t expressly required of the ACME client, but it’s not uncommon for the ACME client to poll the TLS server’s certificate status.
2. When the ACME client decides that it needs to renew a certificate, it contacts the ACME server.
3. In turn, two things need to happen:
   • The ACME server needs to challenge the requestor to prove that that requestor owns this domain. Without getting too deep into the weeds, ACMEv2 defines multiple challenge types: http-01, dns-01, and tls-01. The latter two are useful, but the http-01 type is predominant, so we’ll focus on this one. In this case, the ACME server sends the client a unique token.
   • As a function of the http-01 challenge, the ACME server will use public DNS to resolve the IP of the TLS server stated in the original new certificate request, then make an HTTP request to that IP at a specifically defined URL. The ACME server will expect the HTTP server to respond with the token that was provided in step 3a. So then, the ACME client will use a “hook” function to push that token to the HTTP server so that it can be ready for the ACME server request. If we were talking about an Apache or NGINX web server, the hook would imply placing the token in a folder on the TLS server and potentially configuring the web server to listen on the ACME challenge URL.
4. If the ACME challenge was successful, the ACME server will let the ACME client know that it’s working on generating a new certificate, and eventually, where to download the new certificate.
5. And finally, once the ACME client has the new certificate, it’ll use another hook function to update that certificate on the TLS server. Again, if talking about an Apache or NGINX web server, this might also mean restarting the web server domain to pick up the new information.

The above is of course by no means exhaustive. You can dig much deeper in the ACME Protocol Flow section on the project page or go directly to RFC 8555. In any case, this should give you a pretty good idea of how ACMEv2 works, so let’s move on to building stuff!

ACMEv2 Server Setup

The only requirements you need to satisfy for this local ACMEv2 server installations are:

• Docker and Docker Compose
• Network connectivity to the BIG-IP

The latter requirement implies that a) the ACME client running on the BIG-IP can reach the ACME server, and b) that the ACME server can reach the BIG-IP virtual server hosting the challenge response (token). We’ll use internal static Host information to simulate public DNS. To get started, log into the terminal of the Docker/Docker Compose server you’re going to use, and either:

• Git clone the project and then change directory to the acme-servers folder.
• Navigate to the acme-servers folder in the project page and copy one of the YAML files to a file.

You’ll have two ACMEv2 server options. Pebble is an open-source derivative managed by Let’s Encrypt, so will have similar functionality. Smallstep CA is an ACMEv2 server managed by Smallstep. Whichever one you choose, edit the corresponding YAML file. The ONLY thing you need to change inside this file is the set of “echo” statements near the bottom. These are setting up /etc/hosts entries in the ACME server container and should point your hostnames to the corresponding BIG-IP virtual server IPs. Recall in earlier review, this will be step 3b, where the ACME server resolves the IP of the TLS server and makes an ACME challenge request. That IP needs to be hosted by a BIG-IP virtual server. In lieu of pasting all of that YAML here, you can follow either of these links to show the example YAML:

• Pebble - Docker Compose YAML
• Smallstep CA - Docker Compose YAML

In both cases you'll see a section near the bottom that looks like this:

## Insert local hosts DNS entries (if needed for internal testing)
echo "172.16.1.151   foo.f5labs.com" >> /etc/hosts
echo "172.16.1.152   bar.f5labs.com" >> /etc/hosts
echo "172.16.1.153   www.f5labs.com" >> /etc/hosts

Update the echo statements to match the IP address of the BIG-IP port 80 VIP that's handling the ACME challenge, and the corresponding certificate subject name that this VIP is going to manage. In a real world scenario, you might have multiple HTTPS applications hosted on the BIG-IP, each with a unique IP address. You'll need a port 80 HTTP VIP for each of those unique applications. Again, the ACME server will do a DNS lookup for the site based on the certificate name, and then make its challenge to that IP address, on HTTP port 80. Everything else in the YAML file is configured to generate certificates from a private CA. Save the YAML file and launch the container with Docker Compose:

docker-compose -f docker-compose-smallstep-ca.yaml up -d

or

docker-compose -f docker-compose-pebble-ca.yaml up -d

From any machine in network range of this server, you can test to see if the ACME server is running:

[Pebble] curl -vk https://<server-ip>:14000/dir
[SmallStep] curl -vk https://<server-ip>:9000/acme/acme/directory

If it’s working, you’ll get JSON response with a list of ACMEv2 protocol message types and associated URLs. Example:

{
  "newNonce": "https://172.16.1.112:9000/acme/acme/new-nonce",
  "newAccount": "https://172.16.1.112:9000/acme/acme/new-account",
  "newOrder": "https://172.16.1.112:9000/acme/acme/new-order",
  "revokeCert": "https://172.16.1.112:9000/acme/acme/revoke-cert",
  "keyChange": "https://172.16.1.112:9000/acme/acme/key-change"
}

That’s it, we’re done here. Let’s move on to the BIG-IP.

Kojot Acme Installation

I don’t want to scare you, but this process is only going to take THREE quick steps:

1. Install Kojot Acme – SSH to the BIG-IP console and run the following command:

curl -s https://raw.githubusercontent.com/f5devcentral/kojot-acme/main/install.sh | bash

This is going to install a few BIG-IP objects and create a working folder at /shared/acme. I realize at this phase some secret squirrel types might not be able to access this script from the BIG-IP. One option would be to download the project to ZIP file, expand on the BIG-IP, then run the install.sh script locally.

2. Modify the global configuration data group – The installation creates a data group named dg_acme_config. The purpose of this data group is to store “global” configuration information. Basically, it’s a list of all of the managed certificate domain names and corresponding ACME server information. Let’s say you own a domain foo.com, you want to manage a certificate for www.foo.com, and you’re using the local Pebble ACME server to provide renewable certificates. Add the following information to the data group: 

www.foo.com := --ca https://<pebble-server-ip>:14000/dir

For the sake of this demonstration, you can rely on the defaults, but you can add additional parameters here, like a specific “client” configuration file (--config), and preferred certificate algorithm (-a rsa, -a prime256v1, -a secp384r1). The default client configuration file is at /shared/acme/config, and also explained in detail in the project README page under Configuration Details.

3. Create an HTTP virtual server – As previously stated, in a real world scenario you’d have a “TLS server” represented by a BIG-IP HTTPS virtual server. But for the ACMEv2 challenge to work, you only really need an HTTP VIP for now. The IP of this VIP needs to match the IP added in the YAML file, and the hostname in the global configuration data group needs to match the corresponding hostname in the YAML file. Once that’s created, you might want to test that you can ping the HTTP virtual server from the ACME server. For example:

docker exec -it pebble /bin/sh
ping www.foo.com

If you can ping the BIG-IP port 80 VIP from the ACME server, you're good to go. At this point you can trigger an ACMEv2 certificate fetch from the BIG-IP SSH console:

cd /shared/acme
./f5acmehandler.sh --verbose

The --verbose flag will let you see all of the communications between the BIG-IP and the ACME server. And when that’s done, you should have a new certificate installed on the BIG-IP. If this were a production environment where the certificate already exists and is associated with a client SSL profile, the utility uses a transaction to make sure the certificate, and potentially a new private key, are successfully applied. Finally, when you’re convinced that the ACME client is working as needed, you can modify the global configuration data group to point to production ACME servers. Additional information on scheduling, authentication, revocation, and reporting are all documented in the project page.

Both of the provided ACME servers adhere to RFC 8555 functionality, so what works for these will also work for the Internet-based ACMEv2 providers.