Increase Security in AWS without Rearchitecting your Applications - Part 2: Wednesday Morning

Welcome back! Yesterday we evaluated what options we have that will increase network and application security in AWS.  Today we will dive in and evaluate the architecture in greater detail.  This morning we will focus on the F5 components in the solution and this afternoon we will dive into the AWS components.  

Yesterday we landed on the following high level architecture to address the security concerns raised by the board. 

Background and Assumptions           

The rest of this article requires background knowledge here are the general assumptions:

• The user has knowledge of networking concepts, AWS Networking, F5 LTM, F5 SSL Orchestrator (SSLO)

To get more understanding of a F5 BIG-IP deployment with AWS Gateway Load balancer please review the topic my colleague  Yossi_Rosenboi1  presented. For further understanding of SSLO please see f5 clouddocs. Kevin Stewart and Kevin Gallaugher have published many articles.

We will begin with a brief discussion of SSLO, followed by how it interoperates with AWS Gateway Load Balancer.  After that we will dive into how to build the solution. 

A Brief Discussion of Forwarding in the BIG-IP SSL Orchestrator Service Insertion

SSLO uses BIG-IP's advanced traffic management capabilities to create polices steering different traffic through different security policies and security service chains. Our capabilities to build dynamic service chains precludes the need to build static daisy chain of security devices. For example, SSLO can be used to configured different security chains for internet in traffic (ingress/north to south), internal to internet (egress/south to north), and internal to internal (east to west) via the same GWLB Endpoint (GWLBE) in AWS. This capability  simplifies route table constructs across the environment while increasing visibility and security.

Conceptual Diagram of SSL Orchestrator and Service Chains

Integrating SSL Orchestrator and AWS Gateway Load Balancer

AWS GWLB allows you to use routing constructs to insert security appliances into a VPC routing table. These appliances reside in their own security VPC and provide services to other VPCs. By using GWLB we will leverage horizontal scale, abstraction, and the ability to run the security stack as a service while protecting cloud native services such as Elastic Load Balancers, Elastic IPs, instances and Kubernetes clusters. If we think of the responsibilities that platform/network engineering has, for example, providing a secure platform to release applications on to, GWLB enables meeting the requirements of securing the infrastructure platform without having interference in how the developers deploy an application.  When you combine SSLO traffic classification capabilities with AWS GWLB we can selectively enable WAFs, NGFWs, DLP, IPS or other security services for north/south, south/north and east/west traffic in the environment. 

The Security VPC – Where SSL Orchestrator and the Security Services Reside

The security VPC does not need direct peering with any of the protected Application VPCs. This security VPC should be isolated because our previously encrypted traffic will not be encrypted here. The security VPC will need a subnet to handle management interfaces, 2 subnets per AZ and per service type, 1 subnet per AZ for the Geneve/Load balancing function. Sometimes VMs require outbound connectivity for CFTs or API actions to complete.  In this scenario you may require a NAT gateway or leverage hosting those objects on an S3 bucket that you can access via a VPC endpoint to minimize risk in the environment. 

You may have unique, different or additional internet access concerns that need to be considered while accounting for the risk that traffic is unencrypted in security VPC.

On my system I have a default route out of the geneve-tunnel VLAN (interface eth1) allowing access AWS to endpoints deployed into my VPC and a NAT gateway.  I have also used VLAN naming to make it easier.

Security Group Rules for SSL Orchestrator and Inspection Instances

When deploying in AWS we will create an array of security groups and network interface behavior (SRC/DST check) to ensure that we can process traffic. Please note that the security VPC does not require inbound access from the internet.

 

Device	Interface Function	SRC / DST Check State	Security Group Rules
SSLO	Management	True	Approved ranges, normally 10.x, 172.x, 192.168.x
SSLO	Tunnel Endpoints	True	ENI subnets for GWLB - VPC address space normally 10.x, 172.x, 192.168.x
SSLO	Ingress / Egress for service chain	False	0.0.0.0/0 or broad network access.
SSLO	Inside Tunnel	N/A	Not managed by AWS SG – broad network access such as 0.0.0.0/0 if applying AFM.
Inspection Device	Management	True	Approved ranges, normally 10.x, 172.x, 192.168.x
Inspection Device	Ingress / Egress for service chain	False	0.0.0.0/0 or broad network access.

Security VPC Traffic Flow 

In the security VPC the GWLB Endpoint uses AWS private link to steer traffic from one or more application VPCs, and load balances the traffic across an array of SSLO systems using the Geneve protocol. Traffic is then moved across the security services in the configured ordered. Traffic is then returned to the GWLB Endpoint and placed back in the protected VPC route table

Single Service in SSL Orchestrator Security Chain

• Internet Client --> AWS Internet Gateway -->
• (1) GWLB Endpoint --> (2) Tunnel Interface --> 
• (3) Out VLAN to security Device --> (4) Out Security Device to SSLO  --> 
• (7) Tunnel Interface -->  (8) GWLB Endpoint --> ENI (Instance, ALB, NLB) 

Two Services in SSL Orchestrator Security Chain 

• Client --> AWS (Internet/VGW/TGW)Gateway -->
• (1) GWLB Endpoint --> (2) Tunnel Interface -->
• (3) Out VLAN 1 to security Device 1 -->  (4) Out Security Device 1  to SSLO -->
• (5) Out VLAN 2 to security Device 2 --> (6) Out Security Device 2  to SSLO -->
• (7) Tunnel Interface --> (8) GWLB Endpoint --> ENI (Instance, ALB, NLB)

Configuring BIG-IP for GWLB

Prior to setting up SSLO we will need to configure BIG-IP to support a Geneve tunnel to use with GWLB, a monitoring virtual server and IPs to use "inside" the tunnel.  Yossi covered this in his article, but it bears repeating since it is critical.

Tunnel Configuration

We will use a basic Geneve tunnel configuration based on the Geneve profile. Geneve uses UDP 6081 as its transport protocol.  In my lab the GENEVE tunnel interface is enabled on the Eth1 interface which is in the same subnet as the GWLB ENIs.  The configuration below can be repeated on any/all SSL Orchestrator deployed.  Please see section on scaling later.

net tunnels tunnel geneve-tunnel { if-index 208 local-address 10.252.20.12 profile geneve remote-address any } 

net self geneve-tunnel-ip { address 10.131.0.1/24 allow-service { none } traffic-group traffic-group-local-only vlan geneve-tunnel }

Returning Traffic into the Tunnel

If you have worked with AWS GWLB you are aware that it does not have an internal IP configuration object similar to how a GRE tunnel works. To make BIG-IP send any traffic back into the tunnel we created a Self IP address on the tunnel interface and will create a pool member in the same subnet matching a fake_arp_entry. These two constructs will ensure that we can use GENEVE tunnel will do so as expected. DO NOT PLACE A MONITOR ON THE POOL. 

ltm pool gwlb-gateway-pool {
    members {
        10.131.0.2:any {
            address 10.131.0.2
        }
    }
}
net arp /Common/fake-arp { ip-address 10.131.0.2 mac-address ff:ff:ff:ff:ff:ff }

SSL Orchestrator and Tunnel Interfaces

The SSLO configuration wizard (v 9.1.70) does not allow the user to select a tunnel interface to be selected. To enable us to process traffic on the Geneve tunnel interface we will need to edit the virtual server enabled on VLANs after we deploy our SSLO configurations, we will also need to disable strictness in our SSL policy.  You will need to navigate to the virtual server that represents the configuration and update the incoming "VLAN Enabled ON" setting adding the tunnel and removing the VLAN that you used in your config selection. You will need to repeat this across all the security chains that you create.  Additionally, all the virtual servers should be set to source-port preserve strict, translate-address disabled, and translate-port disabled.

The SSL Orchestrator Wizard – Building our Topology.

When deploying our SSLO we will be using the L3 Transparent topology, for all of our security chains as we are inserting transparent security services between AWS Public IPs and AWS Elastic Load Balancers, Elastic IPs and instance interfaces or between compute instances.

We will only need to create the services and service chains a single time. You can do this during the creation of the first topology our you can do it out of band and then refer to it during the wizard process. Below we have a screen shot of the first step to create or edit a topology.  The key take ways are we are selecting L3 Inbound/Outbound (depending on which traffic flow).   All topologies will select Gateway.  Please familiarize yourself with the differences between inbound and outboud traffic on SSLO. 

Digging Deeper - Interception Polices and Network Flow

F5 Virtual Servers allow you to configure source address, destination address, port and protocol to apply to virtual server matching. This will be critical in our SSLO GWLB deployment since the listening virtual servers are all on the same tunnel interface.  In a physical deployment the different traffic patterns would commonly arrive on different VLANs that reflected the network topology, not so with this deployment so matching traffic characteristics is key to ensure we select the correct interception rule.  When we configure SSLO you are prompted for which IPs to match in the final step of creating your interception policy.  

The table below provides an example of how we use the interception rules and thus virtual server matching to create the more specific objects (Source IP, Destination IP, Protocol, Port etc) to select the traffic. Please see  this KB for more details on virtual server matching.  In my topology I am using 10.0.0.0 address space in my AWS VPCs.

 

Flow	Source Network Address	Destination Network Address	Port	Protocol
North/South	0.0.0.0/0	0.0.0.0/0	ANY	TCP
North/South	0.0.0.0/0	0.0.0.0/0	ANY	UDP
South/North	10.0.0.0/8	0.0.0.0/0	ANY	TCP
South/North	10.0.0.0/8	0.0.0.0/0	ANY	UDP
East/West	10.0.0.0/8	10.0.0.0/8	ANY	TCP
East/West	10.0.0.0/8	10.0.0.0/8	ANY	UDP
N/S, S/E, E/W Other	0.0.0.0/0	0.0.0.0/0	ANY	ANY

Screen shot - interceptoin rules

You will notice that there is only one interception rule for OTHER protocols: IE non TCP/UDP. This allows protocols such as ICMP to pass which would be required for PMTUD. The reason we only need a single chain for North/South, South/North and East/West is 1. We are not actually inspecting these protocols and 2. The topology is the same in all flows. 

Screen shot - associated servers

At this point it should be evident that network flow characteristics will map to an interception rule and a virtual server object.  This virtual server is the entry into the SSLO processing and service chains enabled on the tunnel interface.

SSL Orchestrator Security Policy Strictness

We will need to apply manual edits to the virtual servers used to process traffic and the gateway pool (Geneve Tunnel) that is used to force traffic back into the tunnel. Prior to making these changes we will need to ensure that all of the inspection polices are "unlocked". This is because we need to edit the virtual servers created for port preservation and which interface they are enabled on.

Conclusion

Join me this afternoon and we will dive deeper into the AWS components required to make this work, how all the pieces of this puzzle fit together allowing traffic to flow and a discussion around scaling and resiliency.