Advanced WAF v16.0 - Declarative API

Matt_Dierick · ‎30-Jul-2020

Since v15.1 (in draft), F5® BIG-IP® Advanced WAF™ can import Declarative WAF policy in JSON format.

The F5® BIG-IP® Advanced Web Application Firewall (Advanced WAF) security policies can be deployed using the declarative JSON format, facilitating easy integration into a CI/CD pipeline. The declarative policies are extracted from a source control system, for example Git, and imported into the BIG-IP.

Using the provided declarative policy templates, you can modify the necessary parameters, save the JSON file, and import the updated security policy into your BIG-IP devices. The declarative policy copies the content of the template and adds the adjustments and modifications on to it. The templates therefore allow you to concentrate only on the specific settings that need to be adapted for the specific application that the policy protects.

This Declarative WAF JSON policy is similar to NGINX App Protect policy. You can find more information on the Declarative Policy here :

Audience

This guide is written for IT professionals who need to automate their WAF policy and are familiar with Advanced WAF configuration. These IT professionals can fill a variety of roles:

SecOps deploying and maintaining WAF policy in Advanced WAF
DevOps deploying applications in modern environment and willing to integrate Advanced WAF in their CI/CD pipeline
F5 partners who sell technology or create implementation documentation

This article covers how to PUSH/PULL a declarative WAF policy in Advanced WAF:

With Postman
With AS3

{
    "policy": {
        "name": "policy-api-arcadia",
        "description": "Arcadia API",
        "template": {
            "name": "POLICY_TEMPLATE_API_SECURITY"
        },
        "enforcementMode": "blocking",
        "server-technologies": [
            {
                "serverTechnologyName": "MySQL"
            },
            {
                "serverTechnologyName": "Unix/Linux"
            },
            {
                "serverTechnologyName": "MongoDB"
            }
        ],
        "signature-settings": {
            "signatureStaging": false
        },
        "policy-builder": {
            "learnOnlyFromNonBotTraffic": false
        }
    }
}

1. Upload Policy in BIG-IP

There are 2 options to upload a JSON file into the BIG-IP:

1.1 Either you PUSH the file into the BIG-IP and you IMPORT IT

OR

1.2 the BIG-IP PULL the file from a repository (and the IMPORT is included) <- BEST option

1.1 PUSH JSON file into the BIG-IP

The call is below. As you can notice, it requires a 'Content-Range' header. And the value is 0-(filesize-1)/filesize. In the example below, the file size is 662 bytes. This is not easy to integrate in a CICD pipeline, so we created the PULL method instead of the PUSH (in v16.0)

curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/file-transfer/uploads/policy-api.json' \
--header 'Content-Range: 0-661/662' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--header 'Content-Type: application/json' \
--data-binary '@/C:/Users/user/Desktop/policy-api.json'

At this stage, the policy is still a file in the BIG-IP file system. We need to import it into Adv. WAF. To do so, the next call is required.

This call import the file "policy-api.json" uploaded previously. An CREATE the policy /Common/policy-api-arcadia

curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \
--header 'Content-Type: application/javascript' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{
    "filename":"policy-api.json",
    "policy":
        {
        "fullPath":"/Common/policy-api-arcadia"
        }
}'

1.2 PULL JSON file from a repository

Here, the JSON file is hosted somewhere (in Gitlab or Github ...). And the BIG-IP will pull it.

The call is below. As you can notice, the call refers to the remote repo and the body is a JSON payload. Just change the link value with your JSON policy URL.

With one call, the policy is PULLED and IMPORTED.

curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{
    "fileReference": { 
        "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json"
    }
}'

A second version of this call exists, and refer to the fullPath of the policy. This will allow you to update the policy, from a second version of the JSON file, easily. One call for the creation and the update.

As you can notice below, we add the "policy":"fullPath" directive. The value of the "fullPath" is the partition and the name of the policy set in the JSON policy file.

This method is VERY USEFUL for CI/CD integrations.

curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{
    "fileReference": { 
        "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json"
    },
    "policy":
        {
            "fullPath":"/Common/policy-api-arcadia"
        }
}'

2. Check the IMPORT

Check if the IMPORT worked. To do so, run the next call.

curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \

You should see a 200 OK, with the content below (truncated in this example).
Please notice the "status":"COMPLETED".

{
    "kind": "tm:asm:tasks:import-policy:import-policy-taskcollectionstate",
    "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy?ver=16.0.0",
    "totalItems": 11,
    "items": [
        {
            "isBase64": false,
            "executionStartTime": "2020-07-21T15:50:22Z",
            "status": "COMPLETED",
            "lastUpdateMicros": 1.595346627e+15,
            "getPolicyAttributesOnly": false,
...

From now, your policy is imported and created in the BIG-IP. You can assign it to a VS as usual (Imperative Call or AS3 Call). But in the next session, I will show you how to create a Service with AS3 including the WAF policy.

3. APPLY the policy

As you may know, a WAF policy needs to be applied after each change. This is the call.

curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/apply-policy/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{"policy":{"fullPath":"/Common/policy-api-arcadia"}}'

4. OpenAPI spec file IMPORT

As you know, Adv. WAF supports OpenAPI spec (2.0 and 3.0). Now, with the declarative WAF, we can import the OAS file as well. The BEST solution, is to PULL the OAS file from a repo. And in most of the customer' projects, it will be the case.

In the example below, the OAS file is hosted in SwaggerHub (Github for Swagger files). But the file could reside in a private Gitlab repo for instance.

The URL of the project is : https://app.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3
The URL of the OAS file is : https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3

This swagger file (OpenAPI 3.0 Spec file) includes all the application URL and parameters. What's more, it includes the documentation (for NGINX APIm Dev Portal).

Now, it is pretty easy to create a WAF JSON Policy with API Security template, referring to the OAS file.

Below, you can notice the new section "open-api-files" with the link reference to SwaggerHub. And the new template POLICY_TEMPLATE_API_SECURITY.

Now, when I upload / import and apply the policy, Adv. WAF will download the OAS file from SwaggerHub and create the policy based on API_Security template.

{
    "policy": {
        "name": "policy-api-arcadia",
        "description": "Arcadia API",
        "template": {
            "name": "POLICY_TEMPLATE_API_SECURITY"
        },
        "enforcementMode": "blocking",
        "server-technologies": [
            {
                "serverTechnologyName": "MySQL"
            },
            {
                "serverTechnologyName": "Unix/Linux"
            },
            {
                "serverTechnologyName": "MongoDB"
            }
        ],
        "signature-settings": {
            "signatureStaging": false
        },
        "policy-builder": {
            "learnOnlyFromNonBotTraffic": false
        },
        "open-api-files": [
            {
            "link": "https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3"
            }
        ]
    }
}

5. AS3 declaration

Now, it is time to learn how we can do all of these steps in one call with AS3 (3.18 minimum).

The documentation is here : https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/application-se...

With this AS3 declaration, we:

Import the WAF policy from a external repo
Import the Swagger file (if the WAF policy refers to an OAS file) from an external repo
Create the service

{
    "class": "AS3",
    "action": "deploy",
    "persist": true,
    "declaration": {
        "class": "ADC",
        "schemaVersion": "3.2.0",
        "id": "Prod_API_AS3",
        "API-Prod": {
            "class": "Tenant",
            "defaultRouteDomain": 0,
            "API": {
                "class": "Application",
                "template": "generic",
                "VS_API": {
                    "class": "Service_HTTPS",
                    "remark": "Accepts HTTPS/TLS connections on port 443",
                    "virtualAddresses": ["10.1.10.27"],
                    "redirect80": false,
                    "pool": "pool_NGINX_API_AS3",
                    "policyWAF": {
                        "use": "Arcadia_WAF_API_policy"
                    },
                    "securityLogProfiles": [{
                        "bigip": "/Common/Log all requests"
                    }],
                    "profileTCP": {
                        "egress": "wan",
                        "ingress": { "use": "TCP_Profile" } },
                    "profileHTTP": { "use": "custom_http_profile" },
                    "serverTLS": { "bigip": "/Common/arcadia_client_ssl" }
                },
                "Arcadia_WAF_API_policy": {
                    "class": "WAF_Policy",
                    "url": "http://10.1.20.4/root/as3-waf-api/-/raw/master/policy-api.json",
                    "ignoreChanges": true
                },
                "pool_NGINX_API_AS3": {
                    "class": "Pool",
                    "monitors": ["http"],
                    "members": [{
                        "servicePort": 8080,
                        "serverAddresses": ["10.1.20.9"]
                    }]
                },
                "custom_http_profile": {
                    "class": "HTTP_Profile",
                    "xForwardedFor": true
                },
                "TCP_Profile": {
                    "class": "TCP_Profile",
                    "idleTimeout": 60 }
            }
        }
    }
}

6. CI/CID integration

As you can notice, it is very easy to create a service with a WAF policy pulled from an external repo. So, it is easy to integrate these calls (or the AS3 call) into a CI/CD pipeline.

Below, an Ansible playbook example. This playbook run the AS3 call above.
That's it 🙂

---

    - hosts: bigip
      connection: local
      gather_facts: false
      vars:
        my_admin: "admin"
        my_password: "admin"
        bigip: "10.1.1.12"

      tasks:
      - name: Deploy AS3 WebApp
        uri:
          url: "https://{{ bigip }}/mgmt/shared/appsvcs/declare"
          method: POST
          headers:
            "Content-Type": "application/json"
            "Authorization": "Basic YWRtaW46YWRtaW4="
          body: "{{ lookup('file','as3.json') }}"
          body_format: json
          validate_certs: no
          status_code: 200

7. FIND the Policy-ID

When the policy is created, a Policy-ID is assigned. By default, this ID doesn't appear anywhere. Neither in the GUI, nor in the response after the creation.

You have to calculate it or ask for it. This ID is required for several actions in a CI/CD pipeline.

7.1 Calculate the Policy-ID

We created this python script to calculate the Policy-ID. It is an hash from the Policy name (including the partition). For the previous created policy named "/Common/policy-api-arcadia", the policy ID is "Ar5wrwmFRroUYsMA6DuxlQ"

Paste this python code in a new waf-policy-id.py file, and run the command python waf-policy-id.py "/Common/policy-api-arcadia"

Outcome will be The Policy-ID for /Common/policy-api-arcadia is: Ar5wrwmFRroUYsMA6DuxlQ

#!/usr/bin/python

from hashlib import md5
import base64
import sys
pname = sys.argv[1]
print 'The Policy-ID for', sys.argv[1], 'is:', base64.b64encode(md5(pname.encode()).digest()).replace("=", "")

7.2 Retrieve the Policy-ID and fullPath with a REST API call

Make this call below, and you will see in the response, all the policy creations. Find yours and collect the PolicyReference directive. The Policy-ID is in the link value "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0"

You can see as well, at the end of the definition, the "fileReference" referring to the JSON file pulled by the BIG-IP.

And please notice the "fullPath", required if you want to update your policy

curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \
--header 'Content-Range: 0-601/601' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \

{
    "isBase64": false,
    "executionStartTime": "2020-07-22T11:23:42Z",
    "status": "COMPLETED",
    "lastUpdateMicros": 1.595417027e+15,
    "getPolicyAttributesOnly": false,
    "kind": "tm:asm:tasks:import-policy:import-policy-taskstate",
    "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy/B45J0ySjSJ9y9fsPZ2JNvA?ver=16.0.0",
    "filename": "",
    "policyReference": {
        "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0",
        "fullPath": "/Common/policy-api-arcadia"
    },
    "endTime": "2020-07-22T11:23:47Z",
    "startTime": "2020-07-22T11:23:42Z",
    "id": "B45J0ySjSJ9y9fsPZ2JNvA",
    "retainInheritanceSettings": false,
    "result": {
        "policyReference": {
            "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0",
            "fullPath": "/Common/policy-api-arcadia"
        },
        "message": "The operation was completed successfully. The security policy name is '/Common/policy-api-arcadia'. "
    },
    "fileReference": {
        "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json"
    }
},

8 UPDATE an existing policy

It is pretty easy to update the WAF policy from a new JSON file version. To do so, collect from the previous call 7.2 Retrieve the Policy-ID and fullPath with a REST API call the "Policy" and "fullPath" directive. This is the path of the Policy in the BIG-IP.

Then run the call below, same as 1.2 PULL JSON file from a repository, but add the Policy and fullPath directives

Don't forget to APPLY this new version of the policy 3. APPLY the policy

curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--data-raw '{
    "fileReference": { 
        "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json"
    },
    "policy":
        {
            "fullPath":"/Common/policy-api-arcadia"
        }
}'

TIP : this call, above, can be used in place of the FIRST call when we created the policy "1.2 PULL JSON file from a repository". But be careful, the fullPath is the name set in the JSON policy file.

The 2 values need to match:

"name": "policy-api-arcadia" in the JSON Policy file pulled by the BIG-IP
"policy":"fullPath" in the POST call

9 Video demonstration

In order to help you to understand how it looks with the BIG-IP, I created this video covering 4 topics explained in this article :

The JSON WAF policy
Pull the policy from a remote repository
Update the WAF policy with a new version of the declarative JSON file
Deploy a full service with AS3 and Declarative WAF policy

At the end of this video, you will be able to adapt the REST Declarative API calls to your infrastructure, in order to deploy protected services with your CI/CD pipelines.

Direct link to the video on DevCentral YouTube channel : https://youtu.be/EDvVwlwEFRw

Nikoolayy1 · ‎06-Feb-2023

Thanks! Is there a way to push JSON file with F5 Ansible module like "bigip_asm_policy_import". I know that probably the URI Ansbile module will do the trick but I am looking for a native way to push the ASM policy and then use the new " bigip_as3_deploy" module (the new way to push AS3 Declarations and I made an article about it https://community.f5.com/t5/codeshare/comparison-between-deploying-as3-or-fast-iapp-declarations-wit... ) to deploy the AS3 declaration that will reference the pushed policy.

I tried to replace "url": "http://10.1.20.4/root/as3-waf-api/-/raw/master/policy-api.json" with the ASM policy but if I do not want to host the ASM polcy on git then I see the option I mentioned as pushing the ASM policy to the F5 and then referencing it in the AS3.

Edit:

I think with the Ansible URI module I can script the file upload commands (the bigip_asm_policy_import seems to have issues so the URI module seems the better option for file upload and then making the asm policy from the file upload) as have for other stuff but still a native Ansible module that can also auto calculate the "Content-Range" header for file upload will really be helpfull but as you mentioned using the "Pull" could be an option.

Managed to make it work with the URI module, just for some reason the F5 thinks the files are the wrong falue and this seems like a bug.

"message": "Chunk byte count 402 in Content-Range header different from received buffer length 394",

└─# stat -c%s asm_policy.json
665

tasks:

- name: PUSH FILE WAF

uri:

url: "https://{{ ansible_host }}/mgmt/tm/asm/file-transfer/uploads/asm_policy.json"

method: POST

body: "{{ lookup('template','asm_policy.json', split_lines=False) }}"

status_code: 200

timeout: 300

body_format: json

force_basic_auth: yes

user: "{{ ansible_user }}"

password: "{{ ansible_httpapi_password }}"

validate_certs: no

headers:
Content-Range: 0-393/394
Content-Type: application/octet-stream

- name: PUSH WAF