Using F5 Application Security and DOS Solutions with AWS Global Accelerator - Part 3: Operations

Welcome back!  Thank you for joining me in part 3 of our journey with F5 security and AWS Global Accelerator!  In part 1 we we covered what we would build and why, in part 2 we built out the test bed, and in this installment we will talk about cloud operations which will get us ready for part 4 - testing your deployment!

Infrastructure as code

There are many ways you can deploy BIG-IP into AWS and there are many different as-code-tools you can use to manage your environments.  This is not a discussion of Terraform vs Ansible vs. organization-built tools.  How you build your platform is up to you, and just know that F5 has answers to and can support all of those approaches.  In this article we are going to focus on how to use AWS AutoScale constructs to scale our environment,  the F5 Automation Tool Chain to update security policies for your environment, and how to change instance types (because maybe you want fewer larger instances, or maybe you want many small instances).

For those that have not worked with the F5 Automation Tool Chain, it is the supported set of APIs that F5 presents to manage BIG-IP systems.  The Tool Chain has 4 components: Declarative Onboarding (DO) which is focused on instantiation of systems, Application Services 3 (AS3) which is how we enable you to manage applications on BIG-IP, Telemetry Streaming (TS) which allows transformation of logs for different systems, and finally, F5 Application Services Templates (FAST) for templated applications.  The F5 Automation Tool Chain is the primary interface for the array of available as-code tools to interface with F5 BIG-IP. 

Review - Our Architecture

Let's take a moment to review what we have built and then we will look how to handle day to day operations in the cloud:

• AWS Global Accelerator acts as our Global ingress disaggregator (DAG) distributing traffic to an array of instances in an array or regions
• BIG-IPs (security enforcement point) are deployed in an AutoScale group in one or more regions
• Application instances deployed and discovered by BIG-IP auto discovery 

What we have deployed is an end-to-end elastic system that can scale in and out at all layers as necessary that does not require manual intervention to adjust for fluctuations in traffic.  This dynamic nature combined with F5 security context makes it much more difficult for attackers to accomplish their goals of resource exhaustion and security bypass. 

So what are the most common things that we will have to address in our environment?  

• Changing the instance type in our AutoScale Group
• Changing the base number of instances we have deployed
• Updating the Security Policy
• Logging and visibility 

Updating CloudFormation Stack Updates

If you recall we instantiated our environment from the CLI and only added in a small set of parameters.

aws cloudformation create-stack --region <REGION> --stack-name <MY NAME> \
--template-url https://f5-cft-v2.s3.amazonaws.com/f5-aws-cloudformation-v2/v2.6.0.0/examples/autoscale/payg/autoscale.yaml  \
--parameters ParameterKey=sshKey,ParameterValue=<KEY PAIR> \
ParameterKey=restrictedSrcAddressMgmt,ParameterValue=<MY IP> \
ParameterKey=restrictedSrcAddressApp,ParameterValue=<0.0.0.0/0 or MY IP> \
ParameterKey=uniqueString,ParameterValue=<MY UNIQUE STRING> \
ParameterKey=provisionExternalBigipLoadBalancer,ParameterValue=false \
ParameterKey=notificationEmail,ParameterValue=<MY EMAIL> \
  --capabilities CAPABILITY_NAMED_IAM

We did not evaluate all of the parameters that are used.  If we look at your example stack you can see these in more detail and note that there is a much larger array of parameters that we can change.  Due to space limitations on the site I have removed text to show some examples. 

[cloudshell-user@ip-10-0-79-206 ~]$ aws cloudformation describe-stacks --stack-name mywaf --output yaml
Stacks:
- Capabilities:
  - CAPABILITY_NAMED_IAM
  - CAPABILITY_AUTO_EXPAND
  CreationTime: '2022-10-25T23:12:32.625000+00:00'
  Description: This template creates BIG-IP PAYG Autoscale WAF solution. The template
    uses nested templates for provisioning network, access, compute resources for
    hosting BIG-IP Autoscale solution.
  DisableRollback: false
  DriftInformation:
    StackDriftStatus: NOT_CHECKED
  EnableTerminationProtection: false
  LastUpdatedTime: '2022-11-23T02:43:23.402000+00:00'
  NotificationARNs: []
  Outputs:
  - OutputKey: bigIpAutoscaleGroupName
 .......
  - ParameterKey: restrictedSrcAddressMgmt
    ParameterValue: <MY IP>
 .......
  - ParameterKey: restrictedSrcAddressApp
    ParameterValue: 0.0.0.0/0
.....
    ParameterValue: 10.0.0.0/16
  - ParameterKey: notificationEmail
    ParameterValue: <MY EMAIL>
.....
  - ParameterKey: bigIpRuntimeInitConfig
    ParameterValue: https://<MY BUCKET>.s3.amazonaws.com/v7as-security-vs-bot.yaml
.....
(END)

If we would like to change something in our environment we should use the Stack update process.  While it is true that we could, in some scenarios, simply change objects directly that would cause configuration drift and remove our stack as the source of truth.  The agility we are looking to gain from using public cloud is rooted in cloud models of templates, automation, and having the platform do the mundane work.   Let's look at several examples. 

Changing the Instance Type

When working in AWS F5 supports an array of instance types.  Different AWS instance types have different network characteristics (varying bandwidth, connections) and different CPU counts (varying network performance, SSL performance, security policy performance) and, in addition, F5 offers different licensed capacity (25MB, 200MB, 1GB/3GB/5GB/10GB/HPVE) in the AWS marketplace.  While extremely flexible, this can create a scenario where you may decide to change to a different instance class.   My general starting point for customers is C5n, M5, C6 or M6.  In the near future I will write a blog post on how to choose an instance. 

The good news is that a stack update operation of your cloud formation template is makes it easy to accomplish this task. If you recall we deployed a very vanilla example stack but we did not specify an instance type and we simply accepted the default. 

Default Instance Type:

- ParameterKey: bigIpInstanceType
ParameterValue: m5.xlarge

 If we would like to change the instance to to a m5.xlarge or other F5 validated instance class you will need to update the parameter value.  To do this you could use the AWS CLI again with a command such as 

aws cloudformation update-stack --stack-name <MY NAME> \
--use-previous-template  \
--parameters ParameterKey=sshKey,ParameterValue=<KEY PAIR> \
ParameterKey=restrictedSrcAddressMgmt,ParameterValue=<MY IP> \
ParameterKey=restrictedSrcAddressApp,ParameterValue=<0.0.0.0/0 or MY IP> \
ParameterKey=uniqueString,ParameterValue=<MY UNIQUE STRING> \
ParameterKey=provisionExternalBigipLoadBalancer,ParameterValue=false \
ParameterKey=notificationEmail,ParameterValue=<MY EMAIL> \
ParameterKey=bigIpInstanceType,ParamaterValue=m5.2xlarge
  --capabilities CAPABILITY_NAMED_IAM

 Or you could initiate it from the AWS Web Console

 

 Reusing the existing template

 Changing the parameters that you would like to change such as instance type

 

 No matter which method you chose you should end up in a state where the stack update is complete

 

Scaling Up and Down the Number of Instances

When we deployed the template there were two parameters that we did not discuss.  These parameters represent the minimum and maximum number of instances in our BIG-IP AutoScale group.  The default values are a minimum of 1 and a maximum of 50.  For many users 1 will not be enough.

By following the same processes as above we can scale our group up and down.

In the graph below I am capturing the number of targets we have in the Global Accelerator target group (remember the custom lambda we deployed to update the group) as our AutoScale group adjusts in size. 

This can take some time to run as new instances are deployed as time is needed to deploy an instance, configure it, and have it be ready to accept traffic and there are cool down periods between each instance change.

Updating the Security Policy

Your application is going to go through changes, security scans may uncover new vulnerabilities and new CVEs will happen.  These are facts that we need to address by updating the security policy, but now we have to do so in an elastic and horizontally scaled system.  Just like changing the instance type this is a stack update operation to our template. 

If we again refer to the template details above you will see the following parameter and you will also note that it is a custom value.

 - ParameterKey: bigIpRuntimeInitConfig
    ParameterValue: https://<MY BUCKET>.s3.amazonaws.com/v7as-security-vs-bot.yaml

You can also see this paramater in the Web Console

It is this file that we use to pass values to the automation tool chain via cloud-init and BIG-IP declared, which in turn passes configuration values to the BIG-IP instances in our autoscale group such as our WAF and telemetry steaming configurations.  Within the file that is passed to our systems via the template we can embed configuration commands and https callouts to objects such as a WAF policy.

 Custom_WAF_Policy:
              class: WAF_Policy
              url: >-
                https://<MY BUCKET>/example_security_policy_comprehensive.xml
              enforcementMode: blocking
              ignoreChanges: false

By updating the stanza in our config file to a new policy and then applying a stack update we will see all of the instances cycled with the new deployments receiving the updated configuration(s).

Non-Default polices

With the BIG-IP Application Security Manager (ASM) policy example above we are passing the entire JSON statement in the file.  What about the other configuration items that are not system defaults such as certificates, BOT polices or DOS polices?  Users have two options.

Option 1: Pass the configuration as part of AS3  

Declarative interfaces make your life easier as you can store the configurations, version them, reuse them... standardize without having to click, click, click, click your way to oblivion.  If we look at a snippet from the configuration below you will see that in the bold lines where I am calling another object created in part of the declaration.  You can see it in the "use: /Tenant_1/value" lines.

Creating objects that are shared in the tenant

  value:
        class: ADC
        schemaVersion: 3.0.0
        label: Autoscale
        remark: Autoscale
        Tenant_1:
          class: Tenant
          Shared:
            class: Application
            template: shared
            shared_pool:
              class: Pool
              ..........
            Custom_HTTP_Profile:
              class: HTTP_Profile
              xForwardedFor: true
            Custom_WAF_Policy:
              class: WAF_Policy
              url: >-
                https://<MY BUCKET>/example_security_policy_comprehensive.xml
              enforcementMode: blocking
              ignoreChanges: false

 Calling the objects that are shared in the tenant: 

 HTTPS_Service:
            class: Application
            template: https
            serviceMain:
              class: Service_HTTPS
              virtualAddresses:
                - 0.0.0.0
              snat: auto
              profileHTTP:
                use: /Tenant_1/Shared/Custom_HTTP_Profile
              policyWAF:
                use: /Tenant_1/Shared/Custom_WAF_Policy
              profileDOS: 
                bigip: /Common/dos
              profileBotDefense: 
                bigip: /Common/bot-defense
              ipIntelligencePolicy: 
                bigip: /Common/ip-intelligence
              pool: /Tenant_1/Shared/shared_pool
              securityLogProfiles:
                - use: /Tenant_1/Shared/telemetry_asm_security_log_profile
              serverTLS:
                bigip: /Common/clientssl
              redirect80: false

Option 2: Refer to an object that exits on the system

Perhaps you want to use a "golden AMI"  for various reasons.  It could be that you have a EHF, or perhaps you do not want to store certificates in a manner that the automation tools can call them or perhaps you just want to use default config options.  If that is the case then you can see where objects could be configured in /Common above with the "bigip: /Common/object" such as:

HTTPS_Service:
            class: Application
            template: https
            serviceMain:
              class: Service_HTTPS
              virtualAddresses:
                - 0.0.0.0
.......
                bigip: /Common/dos
              profileBotDefense: 
                bigip: /Common/bot-defense
              ipIntelligencePolicy: 
                bigip: /Common/ip-intelligence
.......
              serverTLS:
                bigip: /Common/clientssl
              redirect80: false

In this scenario you will need to use a custom AMI.

The good news is that you can create custom AMIs for BYOL or PAY-AS-YOU-GO instances. To do so you need to launch an AMI from the marketplace (with the correct license type) and follow the steps to clone it. To call the AMI you would refer to it as a parameter when you first deploy the stack by referencing the custom image ID.  

Note as of the time of this writing using an IP Intelligence Policy requires the use of a prefigured AMI.  A github issue has been submitted to add the ability to both configure IP Intelligence polices via AS3 and refer to them. 

Seeing is Believing

So what about logging and visibility?  If you are familiar with F5 Telemetry Streaming you know that it is an extremely flexible tool.  Within our configuration statements passed to the system on boot we configured telemetry streaming to send logs to CloudWatch.  From that point we can create dashboards as necessary.  CloudWatch is not your only option, you could use ELK, Splunk, or other. The correct choice depends on how your organization want to visualize the environment.  

These logs are generated from the logstream that is created in AWS CloudWatch from F5 Telemetry Streaming.  Once the data is in CloudWatch (or CloudWatch Metrics) you can build graphs using the AWS query language. 

You can created the graphs by querying the cloud watch data.  For information on how to build these please see the AWS documentation.   The following graph and query are shown as examples

fields sig_ids
| sort @timestamp desc
| limit 20
| display @timestamp,sig_ids
| stats count(*) by sig_ids

You can also experiment with the graph type

 You can add graphs from other parts of the environment and from other regions to build a consolidated view:

Number of active instances in my target group:

Number of New Flows at the Global Accelerator

Bytes process by Global Accelerator 

As you can see that by using the F5 templates, F5 Automation Tool Chain, AWS Global Accelerator and AWS CloudWatch we have created an elastic infrastructure that can span multiple AWS regions allowing use to have a consistent security policy applied to all of our security zones.   Join me in part 4 to investigate some of the security tools I used to test the signatures and a forwarding looking discussion on how NGINX App Protect or F5 Distributed Cloud could also be used.