How to deploy NGINX App Protect WAF on the NGINX Ingress Controller using argoCD

 Overview

The NGINX App Protect WAF can be deployed as an add-on within the NGINX Ingress Controller, making the two function in tandem as a WAF armed with a Kubernetes Ingress Controller.

This repo leverages argoCD as a GitOps continuous delivery tool to showcase an end-to-end example of how to use the combo to frontend a simple Kubernetes application.

As this repo is public facing, I am also using a tool named 'Sealed-Secrets' to encrypt all the secret manifests. However, this is not a requirement for deploying either component.

I will go through argoCD and Sealed-Secrets first as supporting pieces and then go into NGINX App Protect WAF itself.

Please note that this tutorial is applies to the NGINX Plus-based version of NGINX Ingress Controller. If you aren’t sure which version you’re using, read the blog A Guide to Choosing an Ingress Controller, Part 4: NGINX Ingress Controller Options.

argoCD

If you do not know argoCD, I strongly recommend that you check it out. In essence, with argoCD, you create an argoCD application that references a location (e.g., Git repo or folder) containing all your manifests. argoCD applies all the manifests in that location and constantly monitors and syncs changes from that location. e.g., if you made a change to a manifest or added a new one, after you did a Git commit for that change, argoCD picks it up and immediate applies the change within your Kubernetes.

The following screenshot taken from argoCD shows that I have an app named 'cafe'. The 'cafe' app points to a Git repo where all manifests are stored. The status is 'Healthy' and 'Synced'. It means that argoCD has successfully applied all the manifests that it knows about, and these manifests are in sync with the Git repo.

The cafe argoCD application manifest is shown here. For this repo, all argoCD application manifests are stored in the bootstrap folder.

To add the Cafe argoCD application, run the following,

kubectl apply -f cafe.yaml

Sealed Secrets

When Kubernetes manifests that contain secrets such as passwords and private keys, they cannot simply be pushed to a public repo.

With Sealed-Secrets, you can solve this problem by sealing those manifests files offline via the binary and then push them to the public repo. When you apply the sealed secret manifests from the public repo, the sealed-secrets component that sits inside Kubernetes will decrypt the sealed secrets and then apply them on the fly.

To do this, you must upload the encryption key into Kubernetes first.

Please note that I am only using sealed secrets so I can push my secret manifests to a public repo, for the purpose of this demo. It is not a requirement to install NGINX App Protect WAF. If you have a private repo, you can simply push all your secret manifests there and argoCD will then apply them as is.

The following commands set up sealed secrets with my specified certificate/key in its own namespace.

export PRIVATEKEY="dc7.h.l.key"
export PUBLICKEY="dc7.h.l.cer"
export NAMESPACE="sealed-secrets"
export SECRETNAME="dc7.h.l"

kubectl -n "$NAMESPACE" create secret tls "$SECRETNAME" --cert="$PUBLICKEY" --key="$PRIVATEKEY"

kubectl -n "$NAMESPACE" label secret "$SECRETNAME" sealedsecrets.bitnami.com/sealed-secrets-key=active
To create a sealed TLS secret,
kubectl create secret tls wildcard.abbagmbh.de --cert=wildcard.abbagmbh.de.cer --key=wildcard.abbagmbh.de.key -n nginx-ingress --dry-run=client -o yaml | kubeseal \
    --controller-namespace sealed-secrets \
    --format yaml \
    > sealed-wildcard.abbagmbh.de.yaml
​

NGINX App Protect WAF

The NGINX App Protect WAF for Kubernetes is a NGINX Ingress Controller software security module add-on with L7 WAF capabilities. It can be embedded within the NGINX Ingress Controller.

The installation process for NGINX App Protect WAF is identical to NGINX Ingress Controller, with the following additional steps.

• Apply NGINX App Protect WAF specific CRD's to Kubernetes
• Apply NGINX App Protect WAF log configuration (NGINX App Protect WAF logging is different from NGINX Plus)
• Apply NGINX App Protect WAF protection policy

The official installation docs using manifests have great info around the entire process. This repo simply collated all the necessary manifests required for NGINX App Protect WAF in a directory that is then fed to argoCD.

Image Pull Secret

With the NGINX App Protect WAF docker image, you can either pull it from the official NGINX private repo, or from your own repo. In the former case, you would need to create a secret that is generated from the JWT file (part of the NGINX license files). See below for detail.

To create a sealed docker-registry secret,

username=`cat nginx-repo.jwt`

kubectl create secret docker-registry private-registry.nginx.com \
--docker-server=private-registry.nginx.com \
--docker-username=$username \
--docker-password=none \
--namespace nginx-ingress \
--dry-run=client -o yaml | kubeseal \
    --controller-namespace sealed-secrets \
    --format yaml \
    > sealed-docker-registry-secret.yaml

Notice the controller namespace above, it needs to match the namespace where you installed Sealed-Secrets.

NGINX App Protect WAF CRD's

A number of NGINX App Protect WAF specific CRD's (Custom Resource Definition) are required for installation. They are included in the crds directory and should be picked up and applied by argoCD automatically.

NGINX App Protect WAF Configuration

The NGINX App Protect WAF configuration includes the followings in this demo:

• User defined signature
• NGINX App Protect WAF policy
• NGINX App Protect WAF log configuration

This user defined signature shows an example of a custom signature that looks for keyword apple in request traffic.

The NGINX App Protect WAF policy defines violation rules. In this case it blocks traffic caught by the custom signature defined above. This sample policy also enables data guard and all other protection features defined in a base template.

The NGINX App Protect WAF log configuration defines what gets logged and how they look like. e.g., log all traffic versus log illegal traffic.

This repo also includes a manifest for syslog deployment as a log destination used by NGINX App Protect WAF.

Ingress

To use NGINX App Protect WAF, you must create an Ingress resource.

Within the Ingress manifest, you use annotations to apply NGINX App Protect WAF specific settings that were discussed above.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    appprotect.f5.com/app-protect-policy: "nginx-ingress/dataguard-alarm"
    appprotect.f5.com/app-protect-enable: "True"
    appprotect.f5.com/app-protect-security-log-enable: "True"
    appprotect.f5.com/app-protect-security-log: "nginx-ingress/logconf"
    appprotect.f5.com/app-protect-security-log-destination: "syslog:server=syslog-svc.nginx-ingress:514"
The traffic routing logic is done via the followings,
spec:
  ingressClassName: nginx # use only with k8s version >= 1.18.0
  tls:
  - hosts:
    - cafe.abbagmbh.de
    secretName: wildcard.abbagmbh.de
  rules:
  - host: cafe.abbagmbh.de
    http:
      paths:
      - path: /tea
        pathType: Prefix
        backend:
          service:
            name: tea-svc
            port:
              number: 8080
      - path: /coffee
        pathType: Prefix
        backend:
          service:
            name: coffee-svc
            port:
              number: 8080

Testing

Once you added both applications (in bootstrap folder) into argoCD, you should see the followings in argoCD UI.

We can do a test to confirm if NGINX App Protect WAF routes traffic based upon HTTP URI, as well as whether WAF protection is applied.

First get the NGINX Ingress Controller IP.

% kubectl get ingress
NAME           CLASS   HOSTS              ADDRESS          PORTS     AGE
cafe-ingress   nginx   cafe.abbagmbh.de   x.x.x.x   80, 443   1d

Now send traffic to both '/tea' and '/coffee' URI paths.

% curl --resolve cafe.abbagmbh.de:443:x.x.x.x https://cafe.abbagmbh.de/tea
Server address: 10.244.0.22:8080
Server name: tea-6fb46d899f-spvld
Date: 03/May/2022:06:02:24 +0000
URI: /tea
Request ID: 093ed857d28e160b7417bb4746bec774

% curl --resolve cafe.abbagmbh.de:443:x.x.x.x https://cafe.abbagmbh.de/coffee
Server address: 10.244.0.21:8080
Server name: coffee-6f4b79b975-7fwwk
Date: 03/May/2022:06:03:51 +0000
URI: /coffee
Request ID: 0744417d1e2d59329401ed2189067e40

As you can see from above, traffic destined to '/tea' is routed to the tea pod (tea-6fb46d899f-spvld) and traffic destined to '/coffee' is routed to the coffee pod (coffee-6f4b79b975-7fwwk).

Let us trigger a violation based on the user defined signature,

% curl --resolve cafe.abbagmbh.de:443:x.x.x.x https://cafe.abbagmbh.de/apple

<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<br><br>Your support ID is: 10807744421744880061<br><br><a href='javascript&colon;history.back();'>[Go Back]</a></body></html>

Finally, traffic violating the XSS rule.

curl --resolve cafe.abbagmbh.de:443:x.x.x.x 'https://cafe.abbagmbh.de/tea<script>'

<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<br><br>Your support ID is: 10807744421744881081<br><br><a href='javascript&colon;history.back();'>[Go Back]</a></body></html>

 Confirming that logs are received on the syslog pod.

% tail -f /var/log/message

May  3 06:30:32 nginx-ingress-6444787b8-l6fzr ASM:attack_type="Non-browser Client Abuse of Functionality Cross Site Scripting (XSS)"
blocking_exception_reason="N/A"
date_time="2022-05-03 06:30:32"
dest_port="443"
ip_client="x.x.x.x"
is_truncated="false"
method="GET"
policy_name="dataguard-alarm"
protocol="HTTPS"
request_status="blocked"
response_code="0"
severity="Critical"
sig_cves=" "
sig_ids="200000099 200000093"
sig_names="XSS script tag (URI) XSS script tag end (URI)"
sig_set_names="{Cross Site Scripting Signatures;High Accuracy Signatures} {Cross Site Scripting Signatures;High Accuracy Signatures}"
src_port="1478"
sub_violations="N/A"
support_id="10807744421744881591"
threat_campaign_names="N/A"
unit_hostname="nginx-ingress-6444787b8-l6fzr"
uri="/tea<script>"
violation_rating="5"
vs_name="24-cafe.abbagmbh.de:8-/tea"
x_forwarded_for_header_value="N/A"
outcome="REJECTED"
outcome_reason="SECURITY_WAF_VIOLATION"
violations="Illegal meta character in URL
Attack signature detected
Violation Rating Threat detected
Bot Client Detected"
json_log="{violations:[{enforcementState:{isBlocked:false}
violation:{name:VIOL_URL_METACHAR}}
{enforcementState:{isBlocked:true}
violation:{name:VIOL_RATING_THREAT}}
{enforcementState:{isBlocked:true}
violation:{name:VIOL_BOT_CLIENT}}
{enforcementState:{isBlocked:true}
signature:{name:XSS script tag (URI)
signatureId:200000099}
violation:{name:VIOL_ATTACK_SIGNATURE}}
{enforcementState:{isBlocked:true}
signature:{name:XSS script tag end (URI)
signatureId:200000093}
violation:{name:VIOL_ATTACK_SIGNATURE}}]}"

Conclusion

The NGINX App Protect WAF deploys as a software security module add-on to the NGINX Ingress Controller and provides comprehensive application security for your Kubernetes environment.

I hope that you find the deployment simple and straightforward.