Making WAF Monitoring Mode Powerful
In a nutshell, the current state of the F5 Distributed Cloud WAF considers several factors before making a block/allow decision. The default configuration does the following: Operates in a monitoring mode, which logs WAF hits but doesn't block matches requests against high and medium attack signature accuracy classes Automatic Attack Signature tuning automatically suppresses likely false positives Bot Signature fill drop requests from clients that match known malicious bot signatures The focus here is that the default is to match based on signature accuracy, an approach that has been leveraged for years until the introduction of the AI-enhanced WAF. This deserves its own article, but to summarize here it pivots to a risk-based enforcement model instead of a pure signature match model. This decision is based on a custom model trained by F5 to identify requests that were riskier regardless of signature accuracy which improves both the block rate while lowering false positives further. But this isn't why you came here. Stepping outside of AI functions that have reduced false positives and better blocking, let's pivot to the challenge that many security teams face in actual businesses: you get stuck with a WAF forever in monitoring mode. Many on the security side of this equation might feel that watching malicious actors walk through the wide open front door has very little utility before the fact. There is a growing stash of events that could have been blocked or maybe get analyzed downstream to enforce a blocking mechanism (perhaps a SOC engineer maintaining a deny list?) in a purely reactive fashion. What if there was a better way? This is how we recently helped a customer find the best balance: Enter Malicious User Detection/Mitigation When Malicious User Detection is enabled, instead of looking at each request in isolation like a WAF or Firewall does, we look at the behavior of a particular client over time. This includes detecting anomalous behavioral changes, failed logins, unauthorized access, and most importantly for this article, all other Distributed Cloud security engine detections including but not limited to a WAF in monitoring mode. Why does this matter? An Application Firewall looks at the requests in their own context. We can assess risk inside the context of a given request, across all accuracy classes of signatures, but that context exists constrained to the request. Malicious User Detection adds the context of the user behavior over time. For instance, we will look at things that may never trigger a WAF block but because they are anomalous in time we can change the risk level of that user; with Malicious User Mitigation we can make a subtler decision to challenge a user with proportional responses. For instance, when a user transitions from no risk to low risk behavior, we can ensure they are running a browser with a javascript challenge. When they transition from low to medium risk we can present a captcha challenge. When they transition into high risk we can temporarily block that user. We can even identify users with a wide set of criteria than simple source IP. We can use XFF source IP, TLS fingerprints, headers, JWT claims, and other things, in a cascading list, to provide those mitigations in a very targeted way at an individual level. This means that one user that sends a request that triggers a WAF violation through a monitoring policy will be allowed through while another attacker that commits multiple WAF violations (or even just anomalous requests) will have mitigation applied in a progressive manner proportional to their risk. This is more subtle and targeted than a pure WAF block. An example Malicious User log from an Apache Bench test against an application with WAF monitoring mode and MUM: Configuration: App Firewall in Monitoring mode Enable Malicious User Detection and Mitigation in the HTTP Load Balancer 3. Save the HTTP Load Balancer Summary and Conclusions Pros: Mitigation are applied per-user instead of per signature A true false positive will be allowed through for a well-behaved user because where it might otherwise be blocked with a blocking WAF policy A malicious user can be stopped with mitigations based on anomalous behavior before they ever trigger a WAF block Cons: It takes a little more time to come to a decision to mitigate because behavior is a time-dependent variable Using behavioral analysis costs more than using WAF alone Key Point: Malicious User Detection/Mitigation with a monitoring WAF policy means you shift a block decision from a request context to a user-behavioral context which dramatically narrows the scope of impact from an application-related false positive. It is a safer middle ground to start doing something with all that WAF logging when you aren't allowed to turn on a blocking policy.AI-Enabled Risk Scoring Helps Reduce Risks
Risk Categories AI-enabled Risk Scoring for F5 Distributed Cloud WAF reduces key risk categories: Security, Business/availability, and Operational. Security risk (missed attacks / false negatives): F5 Distributed Cloud's AI-Powered WAF Risk Scoring improves detection by combining multiple signals per request so you don't miss attacks that traditional WAFs may not catch: High-confidence signatures Curated signature combinations (with LLM labeling to improve precision) Attack indicators (e.g., SQLi signals, libinjection, multiple signatures) A real-time ML model—to catch attacks that traditional WAFs may miss Business/availability risk (false positives blocking real users) By assigning High/Medium/Low risk outcomes using layered analysis, teams can enforce blocking with more confidence and keep false positives low, reducing accidental customer impact such as blocking legitimate users. Staged workflows are enabled, such as: Block High Review Medium (implicitly allow Low while continuing to observe) Operational risk (slow time-to-protection and heavy tuning burden) F5 Distributed Cloud's AI-Powered WAF Risk Scoring reduces manual exceptions and case-by-case policy tuning, enabling teams to deploy the WAF in blocking mode sooner, with less ongoing friction across SecOps, dev, and platform teams. Outcome-based scoring enables: Improved consistency of enforcement across distributed apps/APIs Standardization of protection by reducing bespoke tuning per app How the system makes a risk decision Risk level is computed from layering multiple complementary analyses: High Risk or High Accuracy Signature matches Heuristics – such as injection attacks, multiple attack signatures detected, predictable resource exploitation, other risk indicators Neural network - Signatures can sometimes lead to false positives. To address that, a neural network acts as a secondary classifier to determine whether attack fragments flagged by signatures signal an attack, improving accuracy while maintaining real-time performance. Key system scope The ML model analyzes behavioral patterns to refine risk assessment, ensuring accurate classification and enabling effective threat prioritization. Calling the ML model will adhere to the following scope: The ML model is called only if at least one enabled (not excluded/disabled) signature triggers in these categories: Server-Side Code Injection, SQL Injection, XSS, Command Execution, Path Traversal, LDAP Injection, XPath Injection The model analyzes only HTTP request fragments that trigger signatures (not full raw requests). If signatures are excluded or disabled, they are not considered for invoking the model. Model output: 1 = malicious → request risk level set to High 0 = benign → request risk level set to False Positive A primer on Signature Accuracy vs Signature Risk Accuracy Indicates the ability of the attack signature to identify the attack including susceptibility to false-positive alarms: Low: Indicates a high likelihood of false positives. Medium: Indicates some likelihood of false positives. High: Indicates a low likelihood of false positives. Risk Indicates the level of potential damage this attack might cause if it is successful: Low: Indicates the attack does not cause direct damage or reveal highly sensitive data. Medium: Indicates the attack may reveal sensitive data or cause moderate damage. High: Indicates the attack may cause a full system compromise. Does AI-enabled Risk Scoring add latency? AI-enabled Risk Scoring works in line with F5 Distributed Cloud WAF, inspecting real-time traffic without adding noticeable latency in our tests.Implementing Risk-Based Actions with AI-Powered WAF: Customer Policy Paths
Why Custom policy is where risk-based actions matter most The default policy is straightforward: it applies a broad mix of signatures, threat campaigns, and violations; “Enhance with AI” is an optional add-on. Custom policies are where customers can accidentally recreate the same problems Risk Scoring is designed to solve—usually by combining: Overly broad/noisy signature selection (especially low-accuracy signatures) Aggressive enforcement (blocking Medium too early) Disabling/excluding key signatures and unintentionally reducing ML invocation So the rest of this blog is a tight, configuration-oriented walkthrough of the Custom path. Custom policy: configuration walkthrough (decision points → operational outcomes) Baseline: Navigate to the Custom controls LB Config → Web Application Firewall Create/edit the WAF object (Metadata `Name`, etc.) Set Security Policy = Custom Choose Signature Selection by Accuracy Optionally enable Enhance with AI (Risk Scoring) If enabled, optionally configure Action by Risk Score (risk-based enforcement) Step 1: Signature Selection by Accuracy (choose your baseline level) Accuracy indicates susceptibility to false positives: Low: high likelihood of false positives Medium: some likelihood of false positives High: low likelihood of false positives Note: This setting is foundational: it determines which signatures are active, and therefore the quality and volume of detection signals that feed into downstream risk evaluation. Operationally: High accuracy tends to support faster, safer enforcement. Medium/Low accuracy can expand coverage but increases the chance you’ll need exceptions, investigations, or staged rollout discipline. Step 2: Enhance with AI (turn on Risk Scoring) Enhance with AI = On enables AI-powered risk scoring and assigns each request a High/Medium/Low risk score using layered signals. Two implementation details to make explicit in your blog because they affect customer expectations: ML invocation depends on enabled signatures firing in the specified injection/execution categories. If teams disable/exclude those signatures, they may reduce when the model runs—changing practical behavior of risk evaluation. Step 3: Action by Risk Score (map risk levels to enforcement) When Action by Risk Score is enabled: By default, high-risk requests are blocked Users can choose whether Medium-risk requests are blocked (via dropdown) This is the primary knob that determines how quickly a user decides to move from “safe enforcement” to “broad enforcement.” Recommended rollout path: Day 0 → Day 7 → Steady state This is the most common and safest operational progression for customers Day 0 (safe enforcement baseline) Custom → Signature Selection by Accuracy = High (or High + Medium if you need broader coverage immediately) Enhance with AI = On Action by Risk Score = High Outcome Gets to blocking quickly while minimizing availability risk. High is blocked. This is the “prove safety while stopping obvious bad” posture. Day 7 (controlled expansion) Keep Custom + Enhance with AI + Action by Risk Score Optionally widen Signature Selection from High → High + Medium if coverage is insufficient Enhance with AI = On Action by Risk Score = High + Medium Outcome Expands detection inputs without immediately expanding enforcement. Teams focus on what’s landing in Medium and whether exclusions/disabled signatures are reducing ML invocation in key categories Steady state (mature enforcement) Custom → signature selection set to the broadest set Widen Signature Selection from High + Medium → High + Medium + Low Action by Risk Score = High + Medium Enhance with AI = On Action by Risk Score = High + Medium Outcome Risk outcomes become the enforcement interface. Broad, consistent blocking across apps/APIs with reduced per-app tuning and fewer signature-level decisions Common Pitfalls: Avoid Block Medium on Day 0 when including low-accuracy signatures—this is the fastest way to recreate false-positive outages. If you disable/exclude signatures in the key injection/execution categories, you can reduce ML invocation and change risk evaluation behavior. Summary Custom policies traditionally scale poorly because every app ends up with bespoke signature decisions and exception handling. Risk Scoring is designed to invert that: keep signatures as key signals but standardize enforcement via risk outcomes. If you implement Custom with the Day 0 → Day 7 → Steady state progression above, you get a predictable path from “block safely now” to “enforce broadly later” without returning to signature-by-signature tuning as your primary operating model.Securing Applications using mTLS Supported by F5 Distributed Cloud
Introduction Mutual Transport Layer Security (mTLS) is a process that establishes encrypted and secure TLS connection between the parties and ensures both parties use X.509 digital certificates to authenticate each other. It helps to prevent the malicious third-party attacks which will imitate the genuine applications. This authentication method helps when a server needs to ensure the authenticity and validity of either a specific user or device. As the SSL became outdated several companies like Skype, Cloudfare are now using mTLS to secure business servers. Not using TLS or other encryption tools without secure authentication leads to ‘man in the middle attacks.’ Using mTLS we can provide an identity to a server that can be cryptographically verified and makes your resources more flexible. mTLS with XFCC Header Not only supporting the mTLS process, F5 Distributed Cloud WAF is giving the feasibility to forward the Client certificate attributes (subject, issuer, root CA etc..) to origin server via x-forwarded-client-cert header which provides additional level of security when the origin server ensures to authenticate the client by receiving multiple requests from different clients. This XFCC header contains the following attributes by supporting multiple load balancer types like HTTPS with Automatic Certificate and HTTPS with Custom Certificate. Cert Chain Subject URI DNS How to Configure mTLS In this Demo we are using httpbin as an origin server which is associated through F5 XC Load Balancer. Here is the procedure to deploy the httpbin application, creating the custom certificates and step-by-step process of configuring mTLS with different LB (Load Balancer) types using F5 XC. Deploying HttpBin Application Here is the link to deploy the application using docker commands. Signing server/leaf cert with locally created Root CA Commands to generate CA Key and Cert: openssl genrsa -out root-key.pem 4096 openssl req -new -x509 -days 3650 -key root-key.pem -out root-crt.pem Commands to generate Server Certificate: openssl genrsa -out cert-key2.pem 4096 openssl req -new -sha256 -subj "/CN=test-domain1.local" -key cert-key2.pem -out cert2.csr echo "subjectAltName=DNS:test-domain1.local" >> extfile.cnf openssl x509 -req -sha256 -days 501 -in cert2.csr -CA root-crt.pem -CAkey root-key.pem -out cert2.pem -extfile extfile.cnf -CAcreateserial Note: Add the TLS Certificate to XC console, create a LB(HTTP/TCP) and attach origin pools and TLS certificates to it. In Ubuntu: Move above created CA certificate (ca-crt.pem) to /usr/local/share/ca-certificates/ca-crt.pem and modify "/etc/hosts" file by mapping the VIP(you can get this from your configured LB -> DNS info -> IP Addr) with domain, in this case the (test-domain1.local). mTLS with HTTPS Custom Certificate Log in the F5 Distributed Cloud Console and navigate to “Web APP & API Protection” module. Go to Load Balancers and Click on ‘Add HTTP Load Balancer’. Give the LB Name (test-mtls-cust-cert), Domain name (mtlscusttest.f5-hyd-demo.com), LB Type as HTTPS with Custom Certificate, Select the TLS configuration as Single Certificate and configure the certificate details. Click in ‘Add Item’ under TLS Certificates and upload the cert and key files by clicking on import from files. Click on apply and enable the mutual TLS, import the root cert info, and add the XFCC header value. Configure the origin pool by clicking on ‘Add Item’ under Origins. Select the created origin pool for httpbin. Click on ‘Apply’ and then save the LB configuration with ‘Save and Exit’. Now, we have created the Load Balancer with mTLS parameters. Let us verify the same with the origin server. mTLS with HTTPS with Automatic Certificate Log in the F5 Distributed Cloud Console and navigate to “Web APP & API Protection” module. Goto Load Balancers and Click on ‘Add HTTP Load Balancer’. Give the LB Name(mtls-auto-cert), Domain name (mtlstest.f5-hyd-demo.com), LB Type as HTTPS with Automatic Certificate, enable the mutual TLS and add the root certificate. Also, enable x-forwarded-client-cert header to add the parameters. Configure the origin pool by clicking on ‘Add Item’ under Origins. Select the created origin pool for httpbin. Click on ‘Apply’ and then save the LB configuration with ‘Save and Exit’. Now, we have created the HTTPS Auto Cert Load Balancer with mTLS parameters. Let us verify the same with the origin server. Conclusion As you can see from the demonstration, F5 Distributed Cloud WAF is providing the additional security to the origin servers by forwarding the client certificate info using mTLS XFCC header. Reference Links mTLS Insights Create root cert pair F5 Distributed Cloud WAFAdvanced WAF v16.0 - Declarative API
Since v15.1 (in draft), F5® BIG-IP® Advanced WAF™ can import Declarative WAF policy in JSON format. The F5® BIG-IP® Advanced Web Application Firewall (Advanced WAF) security policies can be deployed using the declarative JSON format, facilitating easy integration into a CI/CD pipeline. The declarative policies are extracted from a source control system, for example Git, and imported into the BIG-IP. Using the provided declarative policy templates, you can modify the necessary parameters, save the JSON file, and import the updated security policy into your BIG-IP devices. The declarative policy copies the content of the template and adds the adjustments and modifications on to it. The templates therefore allow you to concentrate only on the specific settings that need to be adapted for the specific application that the policy protects. This Declarative WAF JSON policy is similar to NGINX App Protect policy. You can find more information on the Declarative Policy here : NAP : https://docs.nginx.com/nginx-app-protect/policy/ Adv. WAF : https://techdocs.f5.com/en-us/bigip-15-1-0/big-ip-declarative-security-policy.html Audience This guide is written for IT professionals who need to automate their WAF policy and are familiar with Advanced WAF configuration. These IT professionals can fill a variety of roles: SecOps deploying and maintaining WAF policy in Advanced WAF DevOps deploying applications in modern environment and willing to integrate Advanced WAF in their CI/CD pipeline F5 partners who sell technology or create implementation documentation This article covers how to PUSH/PULL a declarative WAF policy in Advanced WAF: With Postman With AS3 Table of contents Upload Policy in BIG-IP Check the import Apply the policy OpenAPI Spec File import AS3 declaration CI/CD integration Find the Policy-ID Update an existing policy Video demonstration First of all, you need a JSON WAF policy, as below : { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false } } } 1. Upload Policy in BIG-IP There are 2 options to upload a JSON file into the BIG-IP: 1.1 Either you PUSH the file into the BIG-IP and you IMPORT IT OR 1.2 the BIG-IP PULL the file from a repository (and the IMPORT is included) <- BEST option 1.1 PUSH JSON file into the BIG-IP The call is below. As you can notice, it requires a 'Content-Range' header. And the value is 0-(filesize-1)/filesize. In the example below, the file size is 662 bytes. This is not easy to integrate in a CICD pipeline, so we created the PULL method instead of the PUSH (in v16.0) curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/file-transfer/uploads/policy-api.json' \ --header 'Content-Range: 0-661/662' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --header 'Content-Type: application/json' \ --data-binary '@/C:/Users/user/Desktop/policy-api.json' At this stage, the policy is still a file in the BIG-IP file system. We need to import it into Adv. WAF. To do so, the next call is required. This call import the file "policy-api.json" uploaded previously. An CREATE the policy /Common/policy-api-arcadia curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/javascript' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "filename":"policy-api.json", "policy": { "fullPath":"/Common/policy-api-arcadia" } }' 1.2 PULL JSON file from a repository Here, the JSON file is hosted somewhere (in Gitlab or Github ...). And the BIG-IP will pull it. The call is below. As you can notice, the call refers to the remote repo and the body is a JSON payload. Just change the link value with your JSON policy URL. With one call, the policy is PULLED and IMPORTED. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" } }' A second version of this call exists, and refer to the fullPath of the policy. This will allow you to update the policy, from a second version of the JSON file, easily. One call for the creation and the update. As you can notice below, we add the "policy":"fullPath" directive. The value of the "fullPath" is the partition and the name of the policy set in the JSON policy file. This method is VERY USEFUL for CI/CD integrations. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" }, "policy": { "fullPath":"/Common/policy-api-arcadia" } }' 2. Check the IMPORT Check if the IMPORT worked. To do so, run the next call. curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ You should see a 200 OK, with the content below (truncated in this example). Please notice the "status":"COMPLETED". { "kind": "tm:asm:tasks:import-policy:import-policy-taskcollectionstate", "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy?ver=16.0.0", "totalItems": 11, "items": [ { "isBase64": false, "executionStartTime": "2020-07-21T15:50:22Z", "status": "COMPLETED", "lastUpdateMicros": 1.595346627e+15, "getPolicyAttributesOnly": false, ... From now, your policy is imported and created in the BIG-IP. You can assign it to a VS as usual (Imperative Call or AS3 Call). But in the next session, I will show you how to create a Service with AS3 including the WAF policy. 3. APPLY the policy As you may know, a WAF policy needs to be applied after each change. This is the call. curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/apply-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{"policy":{"fullPath":"/Common/policy-api-arcadia"}}' 4. OpenAPI spec file IMPORT As you know, Adv. WAF supports OpenAPI spec (2.0 and 3.0). Now, with the declarative WAF, we can import the OAS file as well. The BEST solution, is to PULL the OAS file from a repo. And in most of the customer' projects, it will be the case. In the example below, the OAS file is hosted in SwaggerHub (Github for Swagger files). But the file could reside in a private Gitlab repo for instance. The URL of the project is : https://app.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3 The URL of the OAS file is : https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3 This swagger file (OpenAPI 3.0 Spec file) includes all the application URL and parameters. What's more, it includes the documentation (for NGINX APIm Dev Portal). Now, it is pretty easy to create a WAF JSON Policy with API Security template, referring to the OAS file. Below, you can notice the new section "open-api-files" with the link reference to SwaggerHub. And the new template POLICY_TEMPLATE_API_SECURITY. Now, when I upload / import and apply the policy, Adv. WAF will download the OAS file from SwaggerHub and create the policy based on API_Security template. { "policy": { "name": "policy-api-arcadia", "description": "Arcadia API", "template": { "name": "POLICY_TEMPLATE_API_SECURITY" }, "enforcementMode": "blocking", "server-technologies": [ { "serverTechnologyName": "MySQL" }, { "serverTechnologyName": "Unix/Linux" }, { "serverTechnologyName": "MongoDB" } ], "signature-settings": { "signatureStaging": false }, "policy-builder": { "learnOnlyFromNonBotTraffic": false }, "open-api-files": [ { "link": "https://api.swaggerhub.com/apis/F5EMEASSA/Arcadia-OAS3/1.0.0-oas3" } ] } } 5. AS3 declaration Now, it is time to learn how we can do all of these steps in one call with AS3 (3.18 minimum). The documentation is here : https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/declarations/application-security.html?highlight=waf_policy#virtual-service-referencing-an-external-security-policy With this AS3 declaration, we: Import the WAF policy from a external repo Import the Swagger file (if the WAF policy refers to an OAS file) from an external repo Create the service { "class": "AS3", "action": "deploy", "persist": true, "declaration": { "class": "ADC", "schemaVersion": "3.2.0", "id": "Prod_API_AS3", "API-Prod": { "class": "Tenant", "defaultRouteDomain": 0, "API": { "class": "Application", "template": "generic", "VS_API": { "class": "Service_HTTPS", "remark": "Accepts HTTPS/TLS connections on port 443", "virtualAddresses": ["10.1.10.27"], "redirect80": false, "pool": "pool_NGINX_API_AS3", "policyWAF": { "use": "Arcadia_WAF_API_policy" }, "securityLogProfiles": [{ "bigip": "/Common/Log all requests" }], "profileTCP": { "egress": "wan", "ingress": { "use": "TCP_Profile" } }, "profileHTTP": { "use": "custom_http_profile" }, "serverTLS": { "bigip": "/Common/arcadia_client_ssl" } }, "Arcadia_WAF_API_policy": { "class": "WAF_Policy", "url": "http://10.1.20.4/root/as3-waf-api/-/raw/master/policy-api.json", "ignoreChanges": true }, "pool_NGINX_API_AS3": { "class": "Pool", "monitors": ["http"], "members": [{ "servicePort": 8080, "serverAddresses": ["10.1.20.9"] }] }, "custom_http_profile": { "class": "HTTP_Profile", "xForwardedFor": true }, "TCP_Profile": { "class": "TCP_Profile", "idleTimeout": 60 } } } } } 6. CI/CID integration As you can notice, it is very easy to create a service with a WAF policy pulled from an external repo. So, it is easy to integrate these calls (or the AS3 call) into a CI/CD pipeline. Below, an Ansible playbook example. This playbook run the AS3 call above. That's it :) --- - hosts: bigip connection: local gather_facts: false vars: my_admin: "admin" my_password: "admin" bigip: "10.1.1.12" tasks: - name: Deploy AS3 WebApp uri: url: "https://{{ bigip }}/mgmt/shared/appsvcs/declare" method: POST headers: "Content-Type": "application/json" "Authorization": "Basic YWRtaW46YWRtaW4=" body: "{{ lookup('file','as3.json') }}" body_format: json validate_certs: no status_code: 200 7. FIND the Policy-ID When the policy is created, a Policy-ID is assigned. By default, this ID doesn't appear anywhere. Neither in the GUI, nor in the response after the creation. You have to calculate it or ask for it. This ID is required for several actions in a CI/CD pipeline. 7.1 Calculate the Policy-ID We created this python script to calculate the Policy-ID. It is an hash from the Policy name (including the partition). For the previous created policy named "/Common/policy-api-arcadia", the policy ID is "Ar5wrwmFRroUYsMA6DuxlQ" Paste this python code in a new waf-policy-id.py file, and run the command python waf-policy-id.py "/Common/policy-api-arcadia" Outcome will be The Policy-ID for /Common/policy-api-arcadia is: Ar5wrwmFRroUYsMA6DuxlQ #!/usr/bin/python from hashlib import md5 import base64 import sys pname = sys.argv[1] print 'The Policy-ID for', sys.argv[1], 'is:', base64.b64encode(md5(pname.encode()).digest()).replace("=", "") 7.2 Retrieve the Policy-ID and fullPath with a REST API call Make this call below, and you will see in the response, all the policy creations. Find yours and collect the PolicyReference directive. The Policy-ID is in the link value "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0" You can see as well, at the end of the definition, the "fileReference" referring to the JSON file pulled by the BIG-IP. And please notice the "fullPath", required if you want to update your policy curl --location --request GET 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Range: 0-601/601' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ { "isBase64": false, "executionStartTime": "2020-07-22T11:23:42Z", "status": "COMPLETED", "lastUpdateMicros": 1.595417027e+15, "getPolicyAttributesOnly": false, "kind": "tm:asm:tasks:import-policy:import-policy-taskstate", "selfLink": "https://localhost/mgmt/tm/asm/tasks/import-policy/B45J0ySjSJ9y9fsPZ2JNvA?ver=16.0.0", "filename": "", "policyReference": { "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0", "fullPath": "/Common/policy-api-arcadia" }, "endTime": "2020-07-22T11:23:47Z", "startTime": "2020-07-22T11:23:42Z", "id": "B45J0ySjSJ9y9fsPZ2JNvA", "retainInheritanceSettings": false, "result": { "policyReference": { "link": "https://localhost/mgmt/tm/asm/policies/Ar5wrwmFRroUYsMA6DuxlQ?ver=16.0.0", "fullPath": "/Common/policy-api-arcadia" }, "message": "The operation was completed successfully. The security policy name is '/Common/policy-api-arcadia'. " }, "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" } }, 8 UPDATE an existing policy It is pretty easy to update the WAF policy from a new JSON file version. To do so, collect from the previous call 7.2 Retrieve the Policy-ID and fullPath with a REST API call the "Policy" and "fullPath" directive. This is the path of the Policy in the BIG-IP. Then run the call below, same as 1.2 PULL JSON file from a repository, but add the Policy and fullPath directives Don't forget to APPLY this new version of the policy 3. APPLY the policy curl --location --request POST 'https://10.1.1.12/mgmt/tm/asm/tasks/import-policy/' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic YWRtaW46YWRtaW4=' \ --data-raw '{ "fileReference": { "link": "http://10.1.20.4/root/as3-waf/-/raw/master/policy-api.json" }, "policy": { "fullPath":"/Common/policy-api-arcadia" } }' TIP : this call, above, can be used in place of the FIRST call when we created the policy "1.2 PULL JSON file from a repository". But be careful, the fullPath is the name set in the JSON policy file. The 2 values need to match: "name": "policy-api-arcadia" in the JSON Policy file pulled by the BIG-IP "policy":"fullPath" in the POST call 9 Video demonstration In order to help you to understand how it looks with the BIG-IP, I created this video covering 4 topics explained in this article : The JSON WAF policy Pull the policy from a remote repository Update the WAF policy with a new version of the declarative JSON file Deploy a full service with AS3 and Declarative WAF policy At the end of this video, you will be able to adapt the REST Declarative API calls to your infrastructure, in order to deploy protected services with your CI/CD pipelines. Direct link to the video on DevCentral YouTube channel : https://youtu.be/EDvVwlwEFRwMitigating OWASP 2019 API Security Top 10 risks using F5 NGINX App Protect
This 2019 API Security article provides a valuable summary of the OWASP API Security Top 10 risks identified for that year, outlining key vulnerabilities. We will deep-dive into some of those common risks and how we can protect our applications against these vulnerabilities using F5 NGINX App Protect. API2:2019 - Broken User Authentication Problem Statement: A critical API security risk, Broken Authentication occurs when weaknesses in the API's identity verification process permit attackers to circumvent authentication mechanisms. Successful exploitation leads attackers to impersonate legitimate users, gain unauthorized access to sensitive data, perform actions on behalf of victims, and potentially take over accounts or systems. This demonstration utilizes the Damn Vulnerable Web Application (DVWA) to illustrate the exploitability of Broken Authentication. We will execute a brute-force attack against the login interface, iterating through potential credential pairs to achieve unauthorized authentication. Below is the selenium automated script to execute brute-force attack, submitting multiple credential combinations to attempt authentication. The brute-force attack successfully compromised the authentication controls by iterating through multiple credential pairs, ultimately granting access. Solution: To mitigate the above vulnerability, NGINX App Protect is deployed and configured as reverse proxy in front of the application and requests are first validated by NAP for the vulnerabilities. The NGINX App Protect Brute Force WAF policy is utilized as shown below. Re-attempt to gain access to the application using the brute force approach is rejected and blocked. Support ID verification in the Security logs shows request is blocked because of Brute Force Policy. Request captured in NGINX App Protect security log API3:2019 - Excessive Data Exposure Problem Statement: As shown below in one of the demo application API’s, Personal Identifiable Information (PII) data, like Credit Card Numbers (CCN) and U.S. Social Security Numbers (SSN), are visible in responses that are highly sensitive. So, we must hide these details to prevent personal data exploits. Solution: To prevent this vulnerability, we will use the DataGuard feature in NGINX App Protect, which validates all response data for sensitive details and will either mask the data or block those requests, as per the configured settings. First, we will configure DataGuard to mask the PII data as shown below and will apply this configuration. Next, if we resend the same request, we can see that the CCN/SSN numbers are masked, thereby preventing data breaches. If needed, we can update configurations to block this vulnerability after which all incoming requests for this endpoint will be blocked. If you open the security log and filter with this support ID, we can see that the request is either blocked or PII data is masked, as per the DataGuard configuration applied in the above section. Request captured in NGINX App Protect security log API4:2019 - Lack of Resources & Rate Limiting Problem Statement: APIs do not have any restrictions on the size or number of resources that can be requested by the end user. Above mentioned scenarios sometimes lead to poor API server performance, Denial of Service (DoS), and brute force attacks. Solution: NGINX App Protect provides different ways to rate limit the requests as per user requirements. A simple rate limiting use case configuration is able to block requests after reaching the limit, which is demonstrated below. API6:2019 - Mass Assignment Problem Statement: API Mass Assignment vulnerability arises when clients can modify immutable internal object properties via crafted requests, bypassing API Endpoint restrictions. Attackers exploit this by sending malicious HTTP requests to escalate privileges, bypass security mechanisms, or manipulate the API Endpoint's functionality. Placing an order with quantity as 1: Bypassing API Endpoint restrictions and placing the order with quantity as -1 is also successful. Solution: To overcome this vulnerability, we will use the WAF API Security Policy in NGINX App Protect which validates all the API Security event triggered and based on the enforcement mode set in the validation rules, the request will either get reported or blocked, as shown below. Restricted/updated swagger file with .json extension is added as below: Policy used: App Protect API Security Re-attempting to place the order with quantity as -1 is getting blocked. Validating the support ID in Security log as below: Request captured in NGINX App Protect security log API7:2019 - Security Misconfiguration Problem Statement: Security misconfiguration occurs when security best practices are neglected, leading to vulnerabilities like exposed debug logs, outdated security patches, improper CORS settings, unnecessary allowed HTTP methods, etc. To prevent this, systems must stay up to date with security patches, employ continuous hardening, ensure API communications use secure channels (TLS), etc. Example: Unnecessary HTTP methods/verbs represent a significant security misconfiguration under the OWASP API Top 10. APIs often expose a range of HTTP methods (such as PUT, DELETE, PATCH) that are not required for the application's functionality. These unused methods, if not properly disabled, can provide attackers with additional attack surfaces, increasing the risk of unauthorized access or unintended actions on the server. Properly limiting and configuring allowed HTTP methods is essential for reducing the potential impact of such security vulnerabilities. Let’s dive into a demo application which has exposed “PUT” method., this method is not required as per the design and attackers can make use of this insecure unintended method to modify the original content. Solution: NGINX App Protect makes it easy to block unnecessary or risky HTTP methods by letting you customize which methods are allowed. By easily configuring a policy to block unauthorized methods, like disabling the PUT method by setting "$action": "delete", you can reduce potential security risks and strengthen your API protection with minimal effort. As shown below the attack request is captured in security log which conveys the request was successfully blocked, because of “Illegal method” violation. Request captured in NGINX App Protect security log API8:2019 - Injection Problem Statement: Customer login pages without secure coding practices may have flaws. Intruders could use those flaws to exploit credential validation using different types of injections, like SQLi, command injections, etc. In our demo application, we have found an exploit which allows us to bypass credential validation using SQL injection (by using username as “' OR true --” and any password), thereby getting administrative access, as below: Solution: NGINX App Protect has a database of signatures that match this type of SQLi attacks. By configuring the WAF policy in blocking mode, NGINX App Protect can identify and block this attack, as shown below. App Protect WAF Policy If you check in the security log with this support ID, we can see that request is blocked because of SQL injection risk, as below. Request captured in NGINX App Protect security log API9:2019 - Improper Assets Management Problem Statement: Improper Asset Management in API security signifies the crucial risk stemming from an incomplete awareness and tracking of an organization's full API landscape, including all environments like development and staging, different versions, both internal and external endpoints, and undocumented or "shadow" APIs. This lack of comprehensive inventory leads to an expanded and often unprotected attack surface, as security measures cannot be consistently applied to unknown or unmanaged assets. Consequently, attackers can exploit these overlooked endpoints, potentially find older, less secure versions or access sensitive data inadvertently exposed in non-production environments, thereby undermining overall security posture because you simply cannot protect assets you don't know exist. We’re using a flask database application with multiple API endpoints for demonstration. As part of managing API assets, the “/v1/admin/users” endpoint in the demo Flask application has been identified as obsolete. The continued exposure of the deprecated “/v1/admin/users” endpoint constitutes an Improper Asset Management vulnerability, creating an unnecessary security exposure that could be leveraged for exploitation. <public_ip>/v1/admin/users The current endpoint for user listing is “/v2/users”. <public_ip>/v2/users with user as admin1 Solution: To mitigate the above vulnerability, we are using NGINX as an API Gateway. The API Gateway acts as a filtering gateway for API incoming traffic, controlling, securing, and routing requests before they reach the backend services. The server’s name used for the above case is “f1-api” which is listening to the public IP where our application is running. To query the “/v1/admin/users” endpoint, use the curl command as shown below. Below is the configuration for NGINX as API Gateway, in “api_gateway.conf”, where “/v1/admin/users” endpoint is deprecated. The “api_json_errors.conf” is configured with error responses as shown below and included in the above “api_gateway.conf”. Executing the curl command against the endpoint yields an “HTTP 301 Moved Permanently” response. https://f1-api/v1/admin/users is deprecated API10:2019 - Insufficient Logging & Monitoring Problem Statement: Appropriate logging and monitoring solutions play a pivotal role in identifying attacks and also in finding the root cause for any security issues. Without these solutions, applications are fully exposed to attackers and SecOps is completely blind to identifying details of users and resources being accessed. Solution: NGINX provides different options to track logging details of applications for end-to-end visibility of every request both from a security and performance perspective. Users can change configurations as per their requirements and can also configure different logging mechanisms with different levels. Check the links below for more details on logging: https://www.nginx.com/blog/logging-upstream-nginx-traffic-cdn77/ https://www.nginx.com/blog/modsecurity-logging-and-debugging/ https://www.nginx.com/blog/using-nginx-logging-for-application-performance-monitoring/ https://docs.nginx.com/nginx/admin-guide/monitoring/logging/ https://docs.nginx.com/nginx-app-protect-waf/logging-overview/logs-overview/ Conclusion: In short, this article covered some common API vulnerabilities and shows how NGINX App Protect can be used as a mitigation solution to prevent these OWASP API security risks. Related resources for more information or to get started: F5 NGINX App Protect OWASP API Security Top 10 2019 OWASP API Security Top 10 2023Mitigating OWASP Web Application Security Top 10 risks using F5 NGINX App Protect
The OWASP Web Application Security Top 10 outlines the most critical security risks to web applications, serving as a global standard for understanding and mitigating vulnerabilities. Based on data from over 500,000 real-world applications, the list highlights prevalent security issues. The 2021 edition introduces new categories such as "Insecure Design" and "Software and Data Integrity Failures" emphasizing secure design principles and proactive security throughout the software development lifecycle. For more information please visit: OWASP Web Application Security Top 10 - 2021 F5 products provide controls to secure applications against these risks. F5 NGINX App Protect offers security controls using both positive and negative security models to protect applications from OWASP Top 10 risks. The positive security model combines validated user sessions, user input, and application response, while the negative security model uses attack signatures to detect and block OWASP Top 10 application security threats. This guide outlines how to implement effective protection based on the specific needs of your application. Note - The OWASP Web Application Security Top 10 risks listed below are tested on both F5 NGINX App Protect versions 4.x and 5.x A01:2021-Broken Access Control Problem statement: As the risk name suggests, Broken Access Control refers to failures in access control mechanisms that lead to a vulnerable application. In this demonstration, the application is susceptible to “Directory/Path Traversal” via the URL, which allows unauthorized access to sensitive information stored on the server. Solution: F5 NGINX App Protect WAF(Web Application Firewall) offers an inherent solution to the “Directory/Path Traversal” vulnerability discussed, through its “app_protect_default_policy” bundle. This policy, which will be active by default when “App Protect” is enabled in the nginx configuration, helps prevent Directory/Path Traversal attacks by validating the values provided to the “page” key in URL. The attack request is recorded in the security log, indicating that the attack type is Predictable Resource Location, Path Traversal. The request was blocked, and the signatures responsible for detecting the attack are also visible. Note: The security log shown in the image below is not the default log configuration but has been customized by following the instructions provided in the link. A02:2021-Cryptographic Failures Problem statement: Earlier this attack was known as “Sensitive Data Exposure”, focusing on cryptographic failures that often result in the exposure of sensitive data. The “Juice Shop” demo application, as demonstrated below, is vulnerable to sensitive information disclosure due to the insecure storage of data, which is displayed in plain text to end users. Solution: F5 NGINX App Protect WAF provides best in class “Data Guard” policy, which can block as well as mask (based on policy configuration) sensitive information displayed to the end users. After applying the policy to mask the sensitive data, it’s observed the sensitive information which was visible(Fig. 2.1) is masked now. The attack request is recorded in the security log, indicating that the dataguard_mask policy is triggered, and the request was alerted. . 2.4 – Request captured in NGINX App Protect security log A03:2021-Injection Problem statement: An injection vulnerability arises when an application fails to properly handle user-supplied data, sending it to an interpreter (e.g., a database or operating system) as part of a query or command. Without proper validation, filtering, or sanitization, attackers can inject malicious code, leading to unauthorized access, data breaches, privilege escalation, or system compromise. For example, the DVWA demo application shown below lacks input validation, making it vulnerable to SQL injection attacks that can compromise confidential data. Solution: F5 NGINX App Protect WAF has a robust set of attack signatures which are pre-bundled in default policy. The SQL-Injection vulnerability discussed above can be prevented by enabling App Protect which has around 1000+ signatures related to variety of Injection attacks. The attack request is recorded in the security log, indicating that the attack type is SQL-Injection. The request was blocked, and the signatures responsible for detecting the attack are also visible. A04:2021-Insecure Design Problem statement: The growing reliance on web applications exposes them to security risks, with insecure design being a key concern. For example, a retail chain’s e-commerce website lacks protection against bots used by scalpers to buy high-end video cards in bulk for resale. This causes negative publicity and frustrates genuine customers. Implementing anti-bot measures and domain logic rules can help block fraudulent transactions, with F5 NGINX App Protect providing effective protection against such attacks. Solution: Secure design is an ongoing process that continuously evaluates threats, ensures robust code, and integrates threat modeling into development. It involves constant validation, accurate flow analysis, and thorough documentation. By using F5 NGINX App Protect WAF, which includes bot defense, web applications can effectively prevent bot-driven attacks, identifying and blocking them early to protect against fraudulent transactions. The attack request is recorded in the security log, indicating that the attack type is Non-browser Client. The request was blocked, and the violation stating “VIOL_BOT_CLIENT”. Note: The security log shown in the image below is the default log configuration Request captured in NGINX App Protect security log A05:2021-Security Misconfiguration Problem statement: Security misconfiguration occurs when security settings are improperly configured, exposing web applications to various threats. One such vulnerability is Cross-Site Request Forgery (CSRF), where attackers trick authenticated users into making unauthorized requests. Without proper protection mechanisms, attackers can exploit this misconfiguration to perform malicious actions on behalf of the user. The demonstration using WebGoat below shows how an improperly configured application becomes vulnerable to CSRF, allowing attackers to carry out unauthorized actions. Execute the above malicious script by copying the file path and pasting in new tab of the WebGoat authenticated browser. The script will automatically load the malicious code and redirects to the vulnerable page. Solution: F5 NGINX App Protect WAF provides a comprehensive support against CSRF attack. Users can configure the CSRF policy based on their requirements by following the configuration settings here. In this demonstration, default CSRF policy is used to block the attack. Default CSRF policy used to block CSRF attacks The security log captures the attack request, identifying the type of attack which is CSRF. The request was successfully blocked, and the violations saying “CSRF attack detected” is also visible. A06:2021-Vulnerable and Outdated Components Problem statement: Vulnerable and Outdated Components risk arises when a web application uses third-party libraries or software with known security vulnerabilities that are not updated. Additionally, vulnerable pages like “phpmyadmin.php” that expose sensitive details—such as application versions, user credentials, and database information—further increase the risk. Attackers can use this information to exploit known vulnerabilities or gain unauthorized access, leading to potential data breaches or system compromise. Solution: The vulnerability discussed above can be mitigated using F5 NGINX App Protect WAF Attack Signatures, which includes specific "Signature ID" for various vulnerabilities. These Signature IDs can be incorporated into the policy file to block attacks. For instance, Signature ID 200000014 can be used to block access to phpmyadmin.php page. Attack signatures can be found here. The attack request is recorded in the security log, indicating that the attack type is Predictable Resource Location. The request was blocked, and the signatures responsible for detecting the “/phpmyadmin/ page” attack are also visible. A07:2021-Identification and Authentication Failures Problem statement: Effective authentication and secure session management are crucial in preventing authentication-related vulnerabilities in daily tasks. Applications with weak authentication mechanisms are vulnerable to automated attacks, such as credential stuffing, where attackers use wordlists to perform spray attacks, allowing attackers to determine whether specific credentials are valid, thus increasing the risk of brute-force and other automated attacks. Brute force attacks are attempts to break in to secured areas of a web application by trying exhaustive, systematic, username/password combinations to discover legitimate authentication credentials. Solution: To prevent brute force attacks, F5 NGINX App Protect WAF monitors IP addresses, usernames, and the number of failed login attempts beyond a maximum threshold. When brute force patterns are detected, the F5 NGINX App Protect WAF policy either trigger an alarm or block the attack if the failed login attempts reached a maximum threshold for a specific username or coming from a specific IP address. Note – Brute force attack prevention is supported starting from versions v4.13 and v5.5 The security log captures the attack request, identifying the type of attack as Brute Force Attack. The request was successfully blocked, and the “VIOL_BRUTE_FORCE” violations is also visible. A08:2021-Software and Data Integrity Failures Problem statement: Added as a new entry in the OWASP Top 10 2021, software and data integrity failures, particularly in the context of insecure deserialization, occur when an application deserializes untrusted data without proper validation or security checks. This vulnerability allows attackers to modify or inject malicious data into the deserialization process, potentially leading to remote code execution, privilege escalation, or data manipulation. In this demonstration, a serialized PHP command O:18:"PHPObjectInjection":1:{s:6:"inject";s:18:"system ('ps -ef');";} is passed in the URL to retrieve the running processes. Solution: F5 NGINX App Protect WAF can prevent Serialization Injection PHP attacks by leveraging its default policy bundle, which includes an extensive set of signatures specifically designed to address deserialization vulnerabilities. The security log captures the attack request, identifying the type of attack. The request was successfully blocked, and the signatures used to detect the 'PHP Short Object Serialization Injection' attack are also visible. A09:2021-Security Logging and Monitoring Failures Problem statement: Security logging and monitoring failures occur when critical application activities such as logins, transactions, and user actions are not adequately logged or monitored. This lack of visibility makes it difficult to detect and respond to security breaches, attack attempts, or suspicious user behavior. Without proper logging and monitoring, attackers can exploit vulnerabilities without detection, potentially leading to data loss, revenue impact, or reputational damage. Insufficient logging also hinders the ability to escalate and mitigate security incidents effectively, making the application more vulnerable to exploitation. Solution: F5 NGINX App Protect WAF provides different options to track logging details of applications for end-to-end visibility of every request both from a security and performance perspective. Users can change configurations as per their requirements and can also configure different logging mechanisms with different levels. Check the links below for more details on logging: Version 4 and earlier Version 5 A10:2021-Server-Side Request Forgery Problem statement: Server-Side Request Forgery (SSRF) occurs when a web application fetches a remote resource without properly validating the user-supplied URL. This vulnerability allows attackers to manipulate the application into sending malicious requests to internal systems or external resources, bypassing security measures like firewalls or VPNs. SSRF attacks can expose sensitive internal data or resources that are not meant to be publicly accessible, making them a significant security risk, especially with modern cloud architectures. In this demonstration, patient health records, which should be accessible only within the network, can be retrieved publicly through SSRF. Solution: Server-Side Request Forgery (SSRF) attacks can be prevented by utilizing the default policy bundle of F5 NGINX App Protect WAF, which includes a comprehensive set of signatures designed to detect and mitigate SSRF vulnerabilities. By enabling App Protect, you gain strong defense against SSRF attacks as well as other prevalent security threats, thanks to the default policy's pre-configured signatures that cover a wide range of attack vectors. The security log captures the attack request, identifying the type of attack. The request was successfully blocked, and the signatures used to detect the 'SSRF' attack are also visible. Request captured in NGINX App Protect security log Conclusion: Protecting applications from attacks is simple with F5 NGINX App Protect WAF, a high-performance, lightweight, and platform-agnostic solution that supports diverse deployment options, from edge load balancers to Kubernetes clusters. By leveraging its advanced security controls, organizations can effectively mitigate the OWASP Web Application Security Top 10 risks, ensuring robust protection across distributed architectures and hybrid environments. Ultimately, F5 NGINX App Protect helps strengthen overall security, providing comprehensive defense for modern applications. References: F5 NGINX App Protect WAF OWASP Top 10 - 2021 F5 NGINX App Protect WAF Documentation F5 Attack SignaturesGitOps: Declarative Infrastructure and Application Delivery with NGINX App Protect
First thing first. What is GitOps? In a nutshell, GitOps is a practice (Git Operation) that allows you to use GIT and code repository as your configuration source of truth (Declarative Infrastructure as Code and Application Delivery as Code) couple with various supporting tools. The state of your git repository syncs with your infrastructure and application states. As operation team runs daily operations (CRUD - "Create, Read, Update and Delete") leveraging the goodness and philosophy of DevOps, they no longer required to store configuration manifest onto various configuration systems. THe Git repo will be the source of truth. Typically, the target systems or infrastructures runs on Kubernetes base platform. For further details and better explanation of GitOps, please refer to below or Google search. https://codefresh.io/learn/gitops/ https://www.atlassian.com/git/tutorials/gitops Why practice GitOps? I have been managing my lab environment for many years. I use my lab for research of technologies, customer demos/Proof-of-Concepts, applications testing, and code development. Due to the nature of constant changes to my environment (agile and dynamic nature), especially with my multiple versions of Kubernetes platform, I have been spending too much time updating, changing, building, deploying and testing various cloud native apps. Commands like docker build, kubectl, istioctl and git have been constantly and repetitively used to operate environments. Hence, I practice GitOps for my Kubernetes infrastructure. Of course, task/operation can be automated and orchestrated with tools such as Ansible, Terraform, Chef and Puppet. You may not necessarily need GitOps to achieve similar outcome. I managed with GitOps practice partly to learn the new "language" and to experience first-hand the full benefit of GitOps. Here are some of my learnings and operations experience that I have been using to manage F5's NGINX App Protect and many demo apps protected by it, which may benefit you and give you some insight on how you can run your own GitOps. You may leverage your own GitOps workflow from here. For details and description on F5's NGINX App Protect, please refer to https://www.nginx.com/products/nginx-app-protect/ Key architecture decision of my GitOps Workflow. Modular architecture - allows me to swap in/out technologies without rework. "Lego block" Must reduce my operational works - saves time, no repetitive task, write once and deploy many. Centralize all my configuration manifest - single source of truth. Currently, my configuration exists everywhere - jump hosts, local laptop, cloud storage and etc. I had lost track of which configuration was the latest. Must be simple, modern, easy to understand and as native as possible. Use Case and desirable outcome Build and keep up to date NGINX Plus Ingress Controller with NGINX App Protect in my Kubernetes environment. Build and keep up to date NGINX App Protect's attack signature and Threat Champaign signature. Zero downtime/impact to apps protected by NGINX App Protect with frequent releases, update and patch cycle for NGINX App Protect. My Problem Statement I need to ensure that my infrastructure (Kubernetes Ingress controller) and web application firewall (NGINX App Protect) is kept up to date with ease. For example, when there are new NGINX-ingress and NGINX App Protect updates (e.g., new version, attack signature and threat campaign signature), I would like to seamlessly push changes out to NGINX-ingress and NGINX App Protect (as it protects my backend apps) without impacting applications protected by NGINX App Protect. GitOps Workflow Start small, start with clear workflow. Below is a depiction of the overall GitOps workflow. NGINX App Protect is the target application. Hence, before description of the full GitOps Workflow, let us understand deployment options for NGINX App Protect. NGINX App Protect Deployment model There are four deployment models for NGINX App Protect. A common deployment models are: Edge - external load balancer and proxies (Global enforcement) Dynamic module inside Ingress Controller (per service/URI/ingress resource enforcement) Per-Service Proxy model - Kubernetes service tier (per service enforcement) Per-pod Proxy model - proxy embedded in pod (per endpoint enforcement) Pipeline demonstrated in this article will work with either NGINX App Protect deployed as Ingress Controller (2) or per-service proxy model (3). For the purposes of this article, NGINX App Protect is deployed at the Ingress controller at the entry point to Kubernetes (Kubernetes Edge Proxy). For those who prefer video, Video Demonstration Part 1 /3 – GitOps with NGINX Plus Ingress and NGINX App Protect - Overview Part 2 /3 – GitOps with NGINX Plus Ingress and NGINX App Protect – Demo in Action Part 3/3 - GitOps with NGINX Plus Ingress and NGINX App Protect - WAF Security Policy Management. Description of GitOps workflow Operation (myself) updates nginx-ingress + NAP image repo (e.g. .gitlab-ci.yml in nginx-plus-ingress) via VSCODE and perform code merge and commit changes to repo stored in Gitlab. Gitlab CI/CD pipeline triggered. Build, test and deploy job started. Git clone kubernetes-ingress repo from https://github.com/nginxinc/kubernetes-ingress/. Checkout latest kubernetes-ingress version and build new image with DockerfilewithAppProtectForPlus dockerfile. [ Ensure you have appropriate nginx app protect license in place ] As part of the build process, it triggers trivy container security scanning for vulnerabilities (Open Source version of AquaSec). Upon completion of static binary scanning, pipeline upload scanned report back to repo for continuous security improvement. Pipeline pushes image to private repository. Private image repo has been configured to perform nightly container security scan (Clair Scanner). Leverage multiple scanning tools - check and balance. Pipeline clone nginx-ingress deployment repo (my-kubernetes-apps) and update nginx-ingress deployment manifest with the latest build image tag (refer excerpt of the manifest below). Gitlab triggers a webhook to ArgoCD to refresh/sync the desire application state on ArgoCD with deployment state in Kubernetes. By default, ArgoCD will sync with Kubernetes every 3 mins. Webhook will trigger instance sync. ArgoCD (deployed on independent K3S cluster) fetches new code from the repo and detects code changes. ArgoCD automates the deployment of the desired application states in the specified target environment. It tracks updates to git branches, tags or pinned to a specific version of manifest at a git commit. Kubernetes triggers an image pull from private repo, performs a rolling updates, and ensures zero interruption to existing traffic. New pods (nginx-ingress) will spin up and traffic will move to new pods before terminating the old pod. Depending on environment and organisation maturity, a successful build, test and deployment onto DEV environment can be pushed to production environment. Note: DAST Scanning (ZAP Scanner) is not shown in this demo. Currently, running offline non-automated scanning. ArgoCD and Gitlab are integrated with Slack notifications. Events are reported into Slack channel via webhook. NGINX App Protect events are send to ELK stack for visibility and analytics. Snippet on where Gitlab CI update nginx-plus-ingress deployment manifest (Flow#6). Each new image build will be tagged with <branch>-hash-<version> ... spec: imagePullSecrets: - name: regcred serviceAccountName: nginx-ingress containers: - image: reg.foobz.com.au/apps/nginx-plus-ingress:master-f660306d-1.9.1 imagePullPolicy: IfNotPresent name: nginx-plus-ingress ... Gitlab CI/CD Pipeline Successful run of CI/CD pipeline to build, test, scan and push container image to private repository and execute code commit onto nginx-plus-ingress repo. Note: Trivy scanning report will be uploaded or committed back to the same repo. To prevent Gitlab CI triggering another build process ("pipeline loop"), the code commit is tagged with [skip ci]. ArgoCD continuous deployment ArgoCD constantly (default every 3 mins) syncs desired application state with my Kubernetes cluster. Its ensures configuration manifest stored in Git repository is always synchronised with the target environment. Mytrain-dev apps are protected by nginx-ingress + NGINX App Protect. Specific (per service/URI enforcement). NGINX App Protect policy is applied onto this service. nginx-ingress + NGINX App Protect is deployed as an Ingress Controller in Kubernetes. pod-template-hash=xxxx is labeled and tracked by ArgoCD. $ kubectl -n nginx-ingress get pod --show-labels NAME READY STATUS RESTARTS AGE LABELS nginx-ingress-776b64dc89-pdtv7 1/1 Running 0 8h app=nginx-ingress,pod-template-hash=776b64dc89 nginx-ingress-776b64dc89-rwk8w 1/1 Running 0 8h app=nginx-ingress,pod-template-hash=776b64dc89 Please refer to the attached video links above for full demo in actions. References Tools involved NGINX Plus Ingress Controller - https://www.nginx.com/products/nginx-ingress-controller/ NGINX App Protect - https://www.nginx.com/products/nginx-app-protect/ Gitlab - https://gitlab.com Trivy Scanner - https://github.com/aquasecurity/trivy ArgoCD - https://argoproj.github.io/argo-cd/ Harbor Private Repository - https://goharbor.io/ Clair Scanner - https://github.com/quay/clair Slack - https://slack.com DAST Scanner - https://www.zaproxy.org/ Elasticsearch, Logstash and Kibana (ELK) - https://www.elastic.co/what-is/elk-stack, https://github.com/464d41/f5-waf-elk-dashboards K3S - https://k3s.io/ Source repo used for this demonstration Repo for building nginx-ingress + NGINX App Protect image repo https://github.com/fbchan/nginx-plus-ingress.git Repo use for deployment manifest of nginx-ingress controller with the NGINX App Protect policy. https://github.com/fbchan/my-kubernetes-apps.git Summary GitOps perhaps is a new buzzword. It may or may not make sense in your environment. It definitely makes sense for me. It integrated well with NGINX App Protect and allows me to constantly update and push new code changes into my environment with ease. A few months down the road, when I need to update nginx-ingress and NGINX App Protect, I just need to trigger a CI job, and then everything works like magic. Your mileage may vary. Experience leads me to think along the line of - start small, start simple by "GitOps-ing" on one of your apps that may require frequency changes. Learn, revise and continuously improve from there. The outcome that GitOps provides will ease your operational burden with "do more with less". Ease of integration of nginx-ingress and NGINX App Protect into your declarative infrastructure and application delivery with GitOps and F5's industry leading Web Application firewall protection will definitely alleviate your organisation's risk exposure to external and internal applications threat.
