Automating certificate lifecycle management with HashiCorp Vault

One of the challenges many enterprises face today, is keeping track of various certificates and ensuring those which are associated with critical applications deployed across multi-cloud are current and valid. This integration helps you to improve your security poster with short lived dynamic SSL certificates using HashiCorp Vault and AS3 on BIG-IP.

First, a bit about AS3…

Application Services 3 Extension (referred to as AS3 Extension or more often simply AS3) is a flexible, low-overhead mechanism for managing application-specific configurations on a BIG-IP system. AS3 uses a declarative model, meaning you provide a JSON declaration rather than a set of imperative commands. The declaration represents the configuration which AS3 is responsible for creating on a BIG-IP system. AS3 is well-defined according to the rules of JSON Schema, and declarations validate according to JSON Schema. AS3 accepts declaration updates via REST (push), reference (pull), or CLI (flat file editing).

What is Vault?

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log. A modern system requires access to a multitude of secrets: database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. Understanding who is accessing what secrets is already very difficult and platform specific. Adding on key rolling, secure storage, and detailed audit logs is almost impossible without a custom solution. This is where Vault steps in.

Public Key Infrastructure (PKI) provides a way to verify authenticity and guarantee secure communication between applications. Setting up your own PKI infrastructure can be a complex and very manual process. Vault PKI allows users to dynamically generate X.509 certificates quickly and on demand. Vault PKI can streamline distributing TLS certificates and allows users to create PKI certificates with a single command. Vault PKI reduces overhead around the usual manual process of generating a private key and CSR, submitting to a CA, and waiting for a verification and signing process to complete, while additionally providing an authentication and authorization mechanism to validate as well.

Benefits of using Vault automation for BIG-IP

Cloud and platform independent solution for your application anywhere (public cloud or private cloud)
Uses vault agent and Leverages AS3 Templating to update expiring certificates
No application downtime - Dynamically update configuration without affecting traffic

Configuration:

1. Setting up the environment - deploy instances of BIG-IP VE and Vault in cloud or on-premises

You can create instances in the cloud for Vault & BIG-IP using terraform. The repo https://github.com/f5devcentral/f5-certificate-rotate

This will pretty much download Vault binary and start the Vault server. Also, it will deploy the F5 BIG-IP instance on the AWS Cloud. Once we have the instances ready, you can SSH into the Vault ubuntu server and change directory to /tmp and execute below commands.

# Point to the Vault Server
export VAULT_ADDR=http://127.0.0.1:8200 

# Export the Vault Token
export VAULT_TOKEN=root 

# Create roles and define allowed domains with TTL for Certificate
vault write pki/roles/web-certs allowed_domains=demof5.com ttl=160s max_ttl=30m allow_subdomains=true

# Enable the app role
vault auth enable approle

# Create a app policy and apply https://github.com/f5devcentral/f5-certificate-rotate/blob/master/templates/app-pol.hcl
vault policy write app-pol app-pol.hcl

# Apply the app policy using app role
vault write auth/approle/role/web-certs policies="app-pol"

# Read the Role id from the Vault
vault read -format=json auth/approle/role/web-certs/role-id | jq -r '.data.role_id' > roleID

# Using the role id use the secret id to authenticate vault server
vault write -f -format=json auth/approle/role/web-certs/secret-id | jq -r '.data.secret_id' > secretID

# Finally run the Vault agent using the config file
vault agent -config=agent-config.hcl -log-level=debug

2. Use AS3 Template file certs.tmpl with the values as shown

The template file shown below will be automatically uploaded to the Vault instance, the ubuntu server in the /tmp directory

Here I am using an AS3 file called certs.tmpl which is templatized as shown below.

{{ with secret "pki/issue/web-certs" "common_name=www.demof5.com" }}
[
    {
        "op": "replace",
        "path": "/Demof5/HTTPS/webcert/remark",
        "value": "Updated on {{ timestamp }}"
    },

    {
        "op": "replace",
        "path": "/Demof5/HTTPS/webcert/certificate",
        "value": "{{ .Data.certificate | toJSON | replaceAll "\"" "" }}"
    },
    {
        "op": "replace",
        "path": "/Demof5/HTTPS/webcert/privateKey",
        "value": "{{ .Data.private_key | toJSON | replaceAll "\"" "" }}"

    },
    {
        "op": "replace",
        "path": "/Demof5/HTTPS/webcert/chainCA",
        "value": "{{ .Data.issuing_ca | toJSON | replaceAll "\"" "" }}"
    }
]
{{ end }}

3. Vault will render a new JSON payload file called certs.json whenever the SSL Certs expires

When the Certificate expires, Vault generates a new Certificate which we can use to update the BIG-IP using ssh script, below shows the certs.json created automatically.

Snippet of certs.json being created

[
    {
        "op": "replace",
        "path": "/Demof5/HTTPS/webcert/remark",
        "value": "Updated on 2020-10-02T19:05:53Z"
    },
    {
        "op": "replace",
        "path": "/Demof5/HTTPS/webcert/certificate",
        "value": "-----BEGIN CERTIFICATE-----\nMIIDSDCCAjCgAwIBAgIUaMgYXdERwzwU+tnFsSFld3DYrkEwDQYJKoZIhvcNAQEL\nBQAwEzERMA8GA1UEAxMIZGVtby5jb20wHhcNMjAxMDAyMTkwNTIzWhcNMj

4. Use Vault Agent file to run the integration forever without application traffic getting affected

Example Vault Agent file

pid_file = "./pidfile"

vault {
   address = "http://127.0.0.1:8200"
}

auto_auth {

   method "approle" {
       mount_path = "auth/approle"
       config = {
           role_id_file_path = "roleID"
           secret_id_file_path = "secretID"
           remove_secret_id_file_after_reading = false
       }
   }

   sink "file" {
       config = {
           path = "approleToken"
       }
   }
}

template {
  source      = "./certs.tmpl"
  destination = "./certs.json"
  #command = "bash updt.sh"
}
template {
    source = "./https.tmpl"
    destination = "./https.json"
}

5. For Integration with HCP Vault

If you are using HashiCorp hosted Vault solution instead of standalone Vault you can still use this solution with making few changes in the vault agent file. Detail documentation when using HCP vault is here at README. You can map tenant application objects on BIG-IP to Namespace on HCP Vault which provides islotation. More details how to create this solution at https://github.com/f5businessdevelopment/f5-hcp-vault