NGINX is commonly used to perform TLS termination, securing client-facing traffic through TLS encryption. To set up the TLS connection, the client must show server certificates to the server. These certificates are usually stored in files on the NGINX instance. The certificates have expiration dates on them and need to be rotated periodically. When managing more than a handful of NGINX instances, each potentially hosting a large number of domains, it presents significant operational overheads.

Additionally, not many organizations manage their certificates as static files on their NGINX instances. Certificates are typically issued via a public or central PKI, then stored in a common repository. 

If many of your services and/or infrastructure are running on AWS, it may be smart to manage the TLS certificates directly in AWS. One service that does this is AWS Secrets Manager (ASM).

In this article, I will show two ways to connect AWS Secrets Manager and NGINX to help with TLS certificate rotation in a scalable. The goal is to reduce the effort required to make changes to TLS certificates available for NGINX to consume, particularly when it comes to introducing new certificates or rotating existing ones.

 

Polling

With the TLS certificates stored as secrets on AWS Secrets Manager, they can be accessed by calling the AWS Secrets Manager API. The values containing the TLS certificates (stored in base64 encoding under the cert_b64 and key_b64 keys in the sample screenshot below) can then be stored locally on the NGINX instance. 

The script below shows an implementation example with use of the AWS CLI:

#! /usr/bin/env bash set -e # Fetch a list of secrets of interest. Using filter to find secrets with certain tags as AWS Secrets Manager does not support multi-tenancy secrets=$(aws secretsmanager list-secrets --filters "Key=tag-key,Values=nginx-cert-rotation" | jq -r '[.SecretList[].Name]|join(" ")') echo "Found $secrets" # Loop through each secret name, fetch the the key pair values and store them locally in /etc/nginx/certs for secret_id in $secrets; do echo "Processing $secret_id"; # certificate public key is stored in the cert_b64 key, base64 encoded aws secretsmanager get-secret-value --secret-id "$secret_id" | jq -r .SecretString | jq -r .cert_b64 | base64 -d > "/etc/nginx/certs/$secret_id.crt"; # certificate public key is stored in the key_b64 key, base64 encoded aws secretsmanager get-secret-value --secret-id "$secret_id" | jq -r .SecretString | jq -r .key_b64 | base64 -d > "/etc/nginx/certs/$secret_id.key"; echo "Success"; done # reload nginx if is running systemctl is-active --quiet nginx.service && nginx -s reload || true

When executed at regular intervals, using cron or something similar, admins can update the TLS certificates on AWS Secrets Manager and expect the changes to be reflected on the NGINX instances within a reasonable time. This is similar to how NGINXaaS for Azure enables certificate management via Azure Key Vault, where it automatically rotates certificates every 4 hours.

This is simple to set up and allows you to control how frequently the checks for new certificates/changes are performed. The difficulty is in fine-tuning the polling rate, as

• Too long, you run the risk of using outdated certificates longer than expected,
• Too short, it gets expensive as each API call costs money, particularly as the number of NGINX instances and certificates to be managed grows.

 

Event Driven

Another way to approach this is via an event-driven design. We should only have to kick off the certificate rotation process if there are changes to the certificates. AWS provides a serverless way of detecting such changes and triggering the certificate rotation process.

AWS Secrets Manager writes entries to AWS CloudTrail logs for all Secrets Manager operations, particularly of interests are the CreateSecret, UpdateSecret and PutSecretValue operations. AWS EventBridge can then be configured with rules to match the aforementioned entries/events and forward them to a target to take specific actions, which in this case would be telling the NGINX instances to fetch the latest certificates from AWS Secrets Manager.

One way of linking AWS EventBridge to NGINX instances would be via a webhook. By setting up an HTTP endpoint on the NGINX instance (Python-based FastAPI is a good candidate), AWS EventBridge can trigger AWS Lambda to call said HTTP endpoint whenever an AWS Secrets Manager operation of interest occurs. 

When the HTTP endpoint is called, it kicks off the process on the NGINX instances to fetch the new certificates from AWS Secrets Manager directly. This step is similar to the polling example shown above, except it is no longer triggered unnecessarily at regular intervals, but only when there is an actual change on AWS Secrets Manager.

For a deployment example, please see this GitHub repository - leonseng/nginx-cert-rotation-aws which uses Terraform to deploy the environment.

 

Conclusion

Two integration options for automating certificate rotation on NGINX deployed in AWS have been presented in this article - one based on polling for changes and the other being event-driven, each with its own pros and cons:

 

Let me know in the comments below if you are employing other patterns to achieve a similar outcome.