NGINX App Protect Deployment in AWS Cloud
Introduction
Official AWS AMI image for NGINX App Protect has been released recently. This fact gives two big benefits for all users. First is that the official image available on the AWS marketplace eliminates the need to manually pre build AMI for your WAF deployment. It contains all the necessary code and packages on top of the OS of your choice. Another benefit that is even more important from my perspective is that the use of official AMI from the AWS marketplace allows you to pay as you go for NGINX App Protect software instead of purchasing a year-long license. Pay as you go licensing model is much more suitable for modern dynamic cloud environments.
The following article proposes an option of how to deploy and automate NGINX App Protect WAF as a motive to play with new AMIs. To make it slightly more useful I'll try to simulate a production-like environment. Here are the requirements:
- Flexibility. A number of instances scale up and down smoothly.
- Redundancy. Loss of an instance or entire datacenter doesn't cause a service outage.
- Automation. All deployment and day to day operations are automated.
Architecture
High-level architecture represents a common deployment pattern for a highly available system. AWS VPC runs an application load balancer and a subset of EC2 instances running NGINX App Protect software behind it. A load balancer is supposed to manage TLS certificates, receive traffic, and distribute it across all EC2 instances. NGINX App Protect VMs inspect traffic and forward it to the application backend. Everything is simple so far.
Diagram 1. High Level Architecture.
Since the system pretends to be production like then redundancy is a must. Deeper dive to AWS architecture on the diagram below reveals more system details.
Diagram 2. VPC Architecture.
AWS VPC has two subnets distributed across two availability zones. Load balancer legs and WAF instances are going to present in each subnet. Such workload distribution provides geographical resiliency. Even if the entire AWS datacenter in one zone goes down WAF instances in other zone keep woking. Therefore WAF deployment keeps handling traffic and applications remain available to the public. Such a scenario reveals the rule of thumb.
Rule: Always keep instances load below fifty percent to prevent overload in case of loss up to half of the instances.
Each tier lives in its security group. The load balancer security group allows access from any IP to HTTPS port for data traffic. WAF security group allows HTTP access from the load balancer and SSH from trusted hosts for administration purposes.
Data traffic enters load balancer public IPs and then reaches one of WAF instances via private IPs. Blocking response pages served right from WAF VMs. Clear traffic departs directly to application backends regardless of their location.
Automation
Automated deployment and operations for modern systems is de facto standard. Similar to any other systems WAF automation should cover deployment and configuration. Deployment automation sets up underlying AWS infrastructure. Configuration automation takes care of WAF policy distribution across all WAF instances. The following diagram represents an option I used to automate the NGINX App Protect instance.
Diagram 3.
Gitlab is used as a CI/CD platform. Gitlab pipeline sets up and configures the entire system from the ground up. The first stage uses terraform to create all necessary AWS resources such as VPC, subnets, load balancer, and EC2 instances out of official NGINX App Protect AMI image. Second stage provisions WAF policy across all instances.
CI/CD Pipeline
Let's take a closer look at the GitLab pipeline listing. The first stage simply uses terraform to create AWS resources as shown in diagram 2.
terraform:
stage: terraform
image:
name: hashicorp/terraform:0.13.5
before_script:
- cd terraform
- terraform init
script:
- terraform plan -out "planfile" && \
- terraform apply -input=false "planfile"
artifacts:
paths:
- terraform/hosts.cfg
The second stage applies WAF policy across all NGINX App Protect instances created by Terraform.
provision: stage: provision image: name: 464d41/ansible before_script: - eval $(ssh-agent -s) && \ - echo $ANSIBLE_PRIVATE_KEY | base64 -d | ssh-add - - export ANSIBLE_REMOTE_USER=ubuntu - cd provision - ansible-galaxy install nginxinc.nginx_config script: - ansible-playbook -i ../terraform/hosts.cfg nap-playbook.yaml only: changes: - "terraform/*" - "provision/**/*" - ".gitlab-ci.yml"
WAF Deployment Automation. Terraform
There are a couple of important code snippets I would like to emphasize from terraform code.
...omitted...
module "nap" {
source = "terraform-aws-modules/ec2-instance/aws"
providers = {
aws = aws.us-west-2
}
version = "~> 2.0"
instance_count = 2
name = "nap.us-west-2c.int"
ami = "ami-045c0c07ba6b04fcc"
instance_type = "t2.medium"
root_block_device = [
{
volume_type = "gp2"
volume_size = 8
}
]
associate_public_ip_address = true
key_name = "aws-f5-nap"
vpc_security_group_ids = [module.nap_sg.this_security_group_id, data.aws_security_group.allow-traffic-from-trusted-sources.id]
subnet_id = data.aws_subnet.public.id
}
resource "local_file" "hosts_cfg" {
content = templatefile("hosts.tmpl",
{
nap_instances = module.nap.public_ip
}
)
filename = "hosts.cfg"
}
...omitted...
A community module to create EC2 instances is in use. It allows to save some time on implementing my own and scale deployment up and down by simply changing "instance_count" or "instance_type" values back and forth. "ami" value represents official NGINX App Protect AMI therefore no need to pre-bake custom images and buy with per-instance licenses. All instances have a public IP address assigned to them for management purposes. Only GitLab IPs are allowed to access those IPs. Data traffic comes from a load balancer through private IPs.
Notice that Terraform creates a "hosts.cfg" local file. This file contains a list of WAF VM IPs which terraform manages. So Ansible in the next stage always knows what instances to provision.
WAF Configuration Automation. Ansible
Ansible generates NGINX and App Protect configuration and applies them across all instances created by Terraform. NGINX team developed a set of Ansible collections that wrap these operations to roles. It allows to avoid dealing with complex Jinja templates but instead define NGINX configuration right as Ansible playbook parameters. Ansible automatically compiles these parameters to the NGINX config file and spreads it across hosts.
The following listing gives an example of a playbook to configure NGINX. First, it copies a custom App Protect policy to all hosts.
--- - name: Converge hosts: all gather_facts: false become: yes tasks: - name: Copy App Protect Policy copy: src: ./app-protect/custom-policy.json dest: /etc/nginx/custom-policy.json
The next task configures general NGINX daemon parameters.
- name: Configure NGINX and App Protect include_role: name: nginxinc.nginx_config vars: nginx_config_debug_output: true nginx_config_main_template_enable: true nginx_config_main_template: template_file: nginx.conf.j2 conf_file_name: nginx.conf conf_file_location: /etc/nginx/ modules: - modules/ngx_http_app_protect_module.so user: nginx worker_processes: auto pid: /var/run/nginx.pid error_log: location: /var/log/nginx/error.log level: warn worker_connections: 1024 http_enable: true http_settings: default_type: application/octet-stream access_log_format: - name: main format: | '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"' access_log_location: - name: main location: /var/log/nginx/access.log keepalive_timeout: 65 cache: false rate_limit: false keyval: false server_tokens: "off" stream_enable: true http_custom_includes: - "/etc/nginx/sites-enabled/*.conf"
The last task of the playbook configures a virtual server with App Protect enabled on it:
nginx_config_http_template_enable: true
nginx_config_http_template:
app:
template_file: http/default.conf.j2
conf_file_name: default.conf
conf_file_location: /etc/nginx/conf.d/
servers:
server1:
listen:
listen_localhost:
ip: 0.0.0.0
port: 80
opts:
- default_server
server_name: localhost
access_log:
- name: main
location: /var/log/nginx/access.log
locations:
frontend:
location: /
proxy_pass: http://app_servers
proxy_set_header:
header_host:
name: Host
value: $host
app_protect:
enable: true
policy_file: /etc/nginx/custom-policy.json
upstreams:
app_upstream:
name: app_servers
servers:
app_server_1:
address: 35.167.144.13
port: 80
Once the pipeline ends successfully the NGINX App Protect WAF cluster is deployed, configured, and ready to inspect traffic.
Conclusion
This is an option for how production grade NGINX App Protect deployment could look like. Simple, redundant architecture automated from the ground up helps to effectively manage WAF deployment and let a team focus on application development and security instead of maintaining a WAF up and running. Official AMIs allow to use pay as you go licensing to easily scale deployment up and down without overpaying for static licenses.
Full listings of configuration files are available at repo. Feel free to reach out with questions and suggestions. Thanks for reading!