Manage F5 BIG-IP Advanced WAF Configuration Drift with webhooks and GitOps

We typically think of our repos as THE source of truth. This allows us to confidently employ the "nuke and pave" philosophy common in the modern DevOps world; knowing that the repo contains a representation of the running configuration of our application deployment (and possibly even the adjacent supporting application infrastructure). And if/when we need to scale or "nuke" that application we can confidently re-deploy from that "source of truth". However, with a Web Application Firewall (WAF), policy tuning is necessary to tighten or loosen various settings in the policy in order to provide the optimal security for the protected application. This "requirement", that the running configuration drift from the "source of truth" represented in the repo appears at odds with the "repo-is-the-source-of-truth" philosophy. So how do we reconcile this contradiction?

Using F5 BIG-IP Advanced WAF webhooks, we can configure our WAF policy to keep the repo in-sync with the running configuration. This is done by configuring a webhook that fires anytime a change is applied to the WAF policy. This webhook triggers a Gitlab pipeline which calls the F5 BIG-IP Rest API, exports the policy in JSON format, and pushes that into the Gitlab repo.

When using GitOps to deploy services, users still want the ability to observe and adjust Web Application Firewall (WAF) policies but have that new policy managed within the same version control system. This solution allows the most up to date WAF policy to be deployed anywhere with the same AS3 declaration.

Process walk-through:

This deployment uses an AS3 declaration to deliver service configuration to the BIG-IP. The AS3 policy also references an external Declarative WAF policy:

"policyWAF": {
               "use": "f5app_demo"
              },
"f5app_demo": {
               "class": "WAF_Policy",
               "url": "http://gitlab.com/cwise/trigger_waf/-/raw/main/f5app_demo.json",
               "ignoreChanges": false
            },

WAF Policy Syncronization

In order to keep the WAF policy up to date, you’ll need to configure some extra settings within your repository. For this demo, I’m using GitLab as my Git and CI/CD platform, if you'd rather use Github you'll need to set up a webhook proxy service. A link to a demo repository showing that set-up utilizing one of many webhook proxy services (I used Webhooks.io) can be found here. Back to Gitlab!

First, you’ll need to set up a pipeline trigger:

What this pipeline trigger allows you to do is execute pipeline jobs based on an API endpoint and token. Typically pipeline jobs would be executed on a change to the repository but in our case, we want to begin the pipeline from an external change.

Next, you’ll need to create some ssh keys to create a deploy key. This key pair will be used by gitlab-runner (our container doing the pipeline jobs) to complete all of our GitOps on the WAF repository. In the Deploy Keys section you’ll add your ssh public key just like you would any system you’d like to access. Don’t forget to check “Grant write permissions to this key”.

For the private key you’ll need to set up CI/CD Variables. As you can see from the example below, I have put mine into a variable called `SSH_PUSH_KEY`. Now the gitlab-runner will have access to push changes to the existing WAF policy file in the repository. The other two variables are referenced in my bash script to access the BIG-IP.

Note* You can also use Personal Access Tokens (PAT) to be used instead of SSH keys. One thing to remember is that PAT's have an expiration time that must be set, whereas SSH keys do not.

Webhooks

With BIG-IP Advanced WAF you also have the ability to utilize webhooks. In the example we are using 3 webhooks. One to fire the GitOps pipeline to keep the WAF policy in sync and two allowing us to enable ChatOps. When updates are made to the policy or malicious traffic is detected BIG-IP Advanced WAF can deliver messages to your team.

Here are optiions available when setting up trigger reasons from BIG-IP Advanced WAF:

Here is the example webook configuration as configured from BIG-IP Advanced WAF user interface.

Here is an example of our webhooks being defined in the Declarative WAF policy. There are some predefined variables (i.e. {{policy.name}} ) that can be used but equally as important we can send custom variables such as F5_IP. Allowing repository scripts to be as generic as possible.

 "webhooks": [
            {
              "body": "payload={\"channel\": \"#slackChannel\", \"text\": \"Thwarting the Rebel scum! An apply has been made:\n on Device: [{{device.hostname}}] Policy: [{{policy.name}}]\"}",
              "contentType": "application/x-www-form-urlencoded",
              "name": "chat_hook",
              "triggerEvent": "apply-policy",
              "url": "https://hooks.slack.com/services/<token from slack>"
            },
            {
              "body": "payload={\"channel\": \"#slackChannel\", \"text\": \"Malicious Request:\n on Device: [{{device.hostname}}] Policy: [{{policy.name}}], Client IP: [{{request.clientIp}}], Method: [{{request.method}}], Viol Rating [{{request.rating}}], Event ID [{{request.id]]}\"}",
              "contentType": "application/x-www-form-urlencoded",
              "name": "mal_chat",
              "triggerEvent": "http-request-likely-malicious",
              "url": "https://hooks.slack.com/services/<token from slack>"
            },
            {
              "body": "{\"token\": \"<insert token>\",\"ref\": \"main\",\"variables\": {\"POLICY_NAME\": \"{{policy.name}}\", \"HOST_NAME\": \"{{device.hostname}}\", \"F5_IP\": \"10.0.0.1\"}}",
              "contentType": "application/json",
              "name": "waf_hook",
              "triggerEvent": "apply-policy",
              "url": "http://gitlab.fqdn/api/v4/projects/1/trigger/pipeline"
            }
          ],