F5 App Study Tool with passwords stored in Vault

Summary

The App Study Tool is a popular tool for observability into BIG-IP metrics. This is a simple introduction to using Hashicorp Vault, aimed at network admins who may be unfamiliar with secrets management.

Hashicorp Vault is a popular approach for secret management. We will cover an example using Vault to store BIG-IP passwords and use them with the F5 Application Study Tool (AST).

 

Overview of architecture

In this example, we’ll follow the official documentation to run F5’s Application Study Tool. Then we will modify our configuration by removing the .env.device-secrets file that contains the passwords for BIG-IP devices, and provides the value with an environment variable from the Docker Host.

Vault Agent pulls secret from Vault and provides it to host operating system.

Deploy AST with defaults

Prerequisites:

• Hashicorp Vault server, running and accessible, with:
  • a secret called bigip_password_1 stored at /secrets/bigip_password_1/config
  • an AppRole with access to this secret
    • role_id and secret_id values will be expected in files
• Docker Engine and Docker Compose

Install and configure F5 AST

Follow the official documentation to install the F5 AST.

Once you have configured settings for device defaults and individual devices, you’ll configure device secrets in the file .env.device-secrets. When you run the configuration helper, configuration files are created that will be mounted to your containers.

Validate AST with .env.device-secrets

Before we create environment variables with dynamically-pulled values, run the AST to ensure that you are starting with a working configuration. Typically this means running docker compose up.

After following all default installation steps, my docker-compose.yaml file looks as follows:

version: '3'

volumes:
  prometheus:
  grafana:

services:
  prometheus:
    image: prom/prometheus:v2.54.1
    container_name: prometheus
    restart: unless-stopped
    stop_grace_period: 5m
    volumes:
      - ./services/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
      - '--enable-feature=otlp-write-receiver'
      - '--storage.tsdb.retention.time=1y'
    ports:
      - 9090:9090
    networks:
      - 7lc_network

  otel-collector:
    image: ghcr.io/f5devcentral/application-study-tool/otel_custom_collector:v0.9.3
    restart: unless-stopped
    volumes:
      - ./services/otel_collector:/etc/otel-collector-config
    command:
      - "--config=/etc/otel-collector-config/defaults/bigip-scraper-config.yaml"
    env_file:
      - ".env"
      - ".env.device-secrets"
    networks:
      - 7lc_network

  grafana:
    image: grafana/grafana:11.2.0
    container_name: grafana
    restart: unless-stopped
    ports:
      - 3000:3000
    volumes:
      - grafana:/var/lib/grafana
      - ./services/grafana/provisioning/:/etc/grafana/provisioning
    env_file: ".env"
    networks:
      - 7lc_network

networks:
  7lc_network:

 

Use Vault Agent running locally to provide a secret

Now let’s move away from using .env.device-secrets and install Vault Agent.

Install Vault Agent on the Docker Host

These instructions are for Ubuntu 22.04, but Vault supports major Linux distributions:

sudo apt update
sudo apt install -y wget unzip

wget https://releases.hashicorp.com/vault/1.15.4/vault_1.15.4_linux_amd64.zip
unzip vault_1.15.4_linux_amd64.zip
sudo mv vault /usr/local/bin/
vault --version

Create a tmpfs volume

Create a tmpfs mount [1]. This is where we will write any files on the host that contain secrets.

sudo mkdir -p /mnt/vault-secrets

Let’s now edit /etc/fstab and add this line (you can adjust the size).

tmpfs /mnt/vault-secrets tmpfs nodev,nosuid,noexec,nodiratime,size=10m 0 0

Mount it immediately:

sudo mount /mnt/vault-secrets

Create config files for vault agent

Now create a Vault Agent config file. Save this file to /etc/vault-agent.d/vault-agent.hcl

Notice: In lines 24-34, we're creating two files with Vault Agent. One can be used to set a host env var, and the other in case we want to set container env vars with a file. You may use one, and remove the other, if you choose. 

pid_file = "/run/vault-agent.pid"

vault {
  address = "https://<VAULT_SERVER>:8200"
}

auto_auth {
  method "approle" {
    mount_path = "auth/approle"
    config = {
      role_id_file_path = "/etc/vault-agent.d/role_id"
      secret_id_file_path = "/etc/vault-agent.d/secret_id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = {
      path = "/mnt/vault-secrets/vault-token"
    }
  }
}

template {
  source      = "/etc/vault-agent.d/secrets.ctmpl"
  destination = "/mnt/vault-secrets/config.txt"
  perms       = "0640"
}

template {
  source      = "/etc/vault-agent.d/secrets-env-vars.ctmpl"
  destination = "/mnt/vault-secrets/source-env-vars.sh"
  perms       = "0640"
}

Create the Vault Agent template file at /etc/vault-agent.d/secrets.ctmpl with the following content:

BIGIP_PASSWORD_1={{ with secret "secret/bigip_password_1/config" }}{{ .Data.data.password }}{{ end }

Create a second template file at /etc/vault-agent.d/secrets-env-vars.ctmpl with the following content: 

#!/bin/bash
export BIGIP_PASSWORD_1="{{ with secret "secret/bigip_password_1/config" }}{{ .Data.data.password }}{{ end }}"

Create a systemd Unit for Vault Agent

This will make Vault Agent run at startup as a service. Create a file at /etc/systemd/system/vault-agent.service with the following content:

[Unit]
Description=Vault Agent
Requires=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/vault agent -config=/etc/vault-agent.d/vault-agent.hcl
Restart=on-failure
User=root
Group=root

[Install]
WantedBy=multi-user.target

And now, enable and start Vault Agent with the following commands:

sudo systemctl daemon-reexec 
sudo systemctl daemon-reload 
sudo systemctl enable vault-agent.service 
sudo systemctl start vault-agent.service

Verify your Vault Agent service

1. Verify the service is running: 

   sudo systemctl status vault-agent.service

2. Verify our secret is pulled from vault.
   1. Check the contents of /mnt/vault-secrets/config.txt. It should look something like this:

      BIGIP_PASSWORD_1="SuperSecretPassword"
      

   2. Check the contents of /mnt/vault-secrets/source-env-vars.sh. It should look like:

      #!/bin/bash
      export BIGIP_PASSWORD_1="SuperSecretPassword"

3. If you want to use a host env var, you must source the environment variables by running this command on your system: 

   $ source <(sudo cat /mnt/vault-secrets/source-env-vars.sh)
   $ echo "password is: $BIGIP_PASSWORD_1" #this line is optional

We now have two options for providing the password to our AST container:

• a file containing the value, or
• an environment variable from our Docker Host

Update docker-compose.yaml

If the above verification was successful, we’re almost done. Just edit your docker-compose.yaml file:

#if you want to use an environment variable from the Host
...
    environment: #<--- add this line
      BIGIP_PASSWORD_1: $BIGIP_PASSWORD_1 #<--- add this line
...
#or if you want to use a file
...
     env_file:
       - ".env"
       - ".env.device-secrets" #<--- remove this line
       - "/mnt/vault-secrets/config.txt" #<--- add this line
...

What is preferable about this new file compared to our original .env.device-secrets file?

• The secrets are only ever in memory, thanks to tempfs
• Vault Agent can handle authentication, token renewal, and template rendering to keep the file up to date

Conclusion:

Vault and Vault Agent allow us to store secrets on a separate server and use them in our applications. This applies to all applications, but we've used F5 AST as an example application here. Remember, this solution:

• Never stores passwords in code or commits them in git
• Never writes password to disk (thanks to tmpfs)

Thanks for reading! Please reach out if you have any questions or feedback.

Related Articles

• Introducing the F5 Application Study Tool (AST)
• Displaying Application Study Tool (AST) Dashboards in Your Own Grafana Instance

Footnotes

1. Tmpfs is a file system which keeps all of its files in virtual memory. Everything in tmpfs is temporary in the sense that no files will be created on your hard drive. If you unmount a tmpfs instance, everything stored therein is lost. This is a good place to put secrets if they are in files, because they will not be written to disk.