F5 Distributed Cloud - Regional Decryption with Virtual Sites

[MattHarmon]

Introduction:

With the continued growth of multi-cloud infrastructures, enterprises are looking for a "cloud above the clouds" as their new strategic point of delivery and control.  This is exactly what the F5 Distributed Cloud (F5 XC) provides with its SaaS delivered data planes that we call Regional Edges (RE).  The F5 Distributed Cloud enables enterprises to utilize our SaaS platform for Application Proxying, Load Balancing, Web Application Firewall, API Discovery and Security, Bot and Fraud Protection, Multi-Cloud Networking, and much more.  To achieve these multi-cloud delivery and security solutions, we must terminate TLS within our Regional Edges.  However, what if your organization has rules and regulations around encryption, such that inspection points and key storage must reside within specific countries or continents in which the enterprise operates?  Let's look at how you can resolve this challenge with F5 Distributed Cloud's Virtual Site configuration.

F5 Distributed Cloud - Networking Overview

By default, our F5 Distributed Cloud's SaaS data

planes of Regional Edges are connected to two networks, the Internet, and the F5 Global Backbone.  Our Global Backbone is a multi-terrabit, low-latency physical network with a software-defined, distributed overlay network.  We call this the F5 Distributed Cloud Application Delivery Network (ADN).  The ADN provides interconnectivity between our Regional Edges, Customer's Data Centers, Customer's Clouds, and our Customer Edge software.  If you review Hurricane Electric Internet Services BGP Peering Report, you'll see F5 is one of the most peered global networks in the world with almost 5000 IPv4 adjacencies and almost 4000 IPv6 adjacencies.  

These adjacencies are critical to the success of our Anycast route advertisement which utilizes the underlay routing of the internet for BGP multi-path to route clients to the nearest Regional Edge.  Internet standards require organizations advertising network prefixes to the internet to be a /24 or greater.  F5 Distributed Cloud advertises multiple /24 or greater prefixes globally from all Regional Edges.  If you're a networking nerd like me, you can dig deeper into F5's advertisements via Hurricane Electric linked above.  If you're curious about how internet advertisements have changed over the years, you might refer to APNIC's article on BGP in 2021 - The BGP Table where they review how the internet routing tables have changed over the years. 

By now you're probably asking, why all this networking background... well because I like it and it is my article so ha!  Well in reality, this relates to how the F5 Distributed Cloud routes "Virtual Sites" within the platform.  By definition, a Virtual Site is a "tool for indirection. Instead of doing configuration on each Site, it allows for performing a given configuration on set (or group) of Sites. Virtual Site is a configuration object that defines the Sites that are members of the set."  By default, when you sign up for F5 Distributed Cloud, you are provided a public IP address to which you can attach Application Delivery and Security services to.  This IP address is part of a pre-defined Virtual Site, which is configured for ALL Regional Edges to process traffic for any services attached to your tenant's IP address.  This virtual site is used when the "VIP Advertisement" is set to "Internet".

The Solution

As you could probably guess by now, we will utilize Virtual Sites to build logical topologies within the F5 Distributed Cloud.  This configuration allows you to configure which Regional Edges are allowed to store keys, decrypt traffic, proxy, and apply other application delivery and security features of your choosing on a per application basis.  Within the F5 Distributed Cloud you can mix and match Virtual Site configurations for your applications, where some may be using the default of all Regional Edges, and some may be using only specific Regional Edges due to the nature of the application and security requirements.  

Let's step through this diagram and discuss how this solution works.  You can see All Region Edges are advertising a /24 of 185.56.152.0 to the internet.  Now let's imagine we want to have two custom topologies, one for North America Regional Edges, and one for Regional Edges within Europe.  We would configure two new Virtual Sites, which I will show you how to do the steps in the next section.  There is a pre-requisite to setting up decryption topologies via Virtual Sites, we need an additional Virtual IP (VIP) address per topology within the tenant.  Remember, a tenant comes with a single IP by default, but you can add additional IPs.  These IPs can come from the F5 Distributed Cloud blocks, or you can Bring Your Own IPs (BYOIP), but this requires that you bring a full /24 due to the rules of the internet for advertisement.  

Once you have your additional VIPs and your Virtual Sites configured, you're now able to apply these to individual application configurations.  We utilize the Advertise VIP section for the application and select the IP address to advertise associated to the virtual site topolgoy desired.  The IP address that is associated with this virtual site will now be advertised into the F5 Global Backbone as a /32.  Traffic from the internet will still follow its closest path to the /24 prefix, but once traffic reaches the F5 Regional Edge's routing edge, it'll use the F5 Global Backbone to the nearest Regional Edge configured as a Virtual Site for this application.  Simply, the backbone follows its more specific route.  Once traffic reaches the Regional Edge associated with the Virtual Site, the Regional Edge software will decrypt, proxy, and apply any additional configurations to the application traffic flow.  

Configuration Steps:

**Reminder - You must have additional public IP addres(es) within your tenant as a pre-requiste**

1. Create a Virtual Site - From "Select Service" - Choose "Shared Configuration" ==> Manage -> Virtual Site
   1. Use existing labels to choose sites or regions of sites that you would like to be included in your Regional Edge Virtual Site
   2. IMPORTANT - Be sure to include more than 1 Regional Edge for your Virtual Site
   3. Graphic below shows London and Paris Regional Edges configured as part of my EU Virtual Site
2. Update additional Tenant IP Address - From "Select Service" - Choose "Shared Configuration" ==> Manage -> Public IP Addresses
   1. Click the 3 dots under "Actions" and choose "Manage Configuration"
   2. In the top right corner click "Edit Configuration"
   3. Under "Virtual Site" - choose the Virtual Site you created in Step 1
3. Update VIP Advertisement on Load Balancer - From "Select Service" - Choose "Load Balancers" ==> Manage -> Load Balancer -> HTTP Load Balancer 
   1.  Since our Virtual Site and Public IP address are configured within the shared namespace, we can apply this config to any Load Balancer in any Namespace.
   2. Either create a new HTTP Load Balancer, or edit an existing Load Balancer's configuration
   3. Navigate to "Other Settings" and change the VIP Advertisement to "Internet (Specific VIP)"
   4. Choose the public IP Address that you modified for the Virtual Site in step 2.

Verification:

Be sure that DNS has been updated to reflect the FQDN(s) associated with the HTTP Load Balancer and now resolve to the new public IP address.  Verify by running a nslookup or dig from your local machine.  The reason I chose EU Regional Edges for this article was to simplify the verification that this solution is working, as I am based in the United States.  My traffic will typically traverse the NY Regional Edge.  

Navigate: "Select Service" - Choose "Load Balancers" ==> Virtual Host -> Load Balancer -> Click "Performance Monitoring" under your Load Balancer -> Click "Requests" tab and find your request.

You'll see for the client request below, the client is in the United States, specifically Mentor, Ohio.  However, the L5-7 services, TLS Termination, Proxy, Load Balancing, and additional delivery and security features occurred in the Paris Regional Edge (pa4-par).  When reviewing your request, you can click on the JSON tab which provides upwards of 100 attributes of each request.  

For this specific traffic flow, the client traversed the Internet to our NY Regional Edge, then utilized the F5 Global Backbone to the nearest Regional Edge configured as part of the Virtual Site, which in this client's case the BGP multi-path of Paris.  Once the traffic arrives in the Virtual Site Regional Edge (Paris), the F5 XC Software then provides L3-L7 services, most importantly TLS termination and proxying of the traffic.

In the below diagram, we see this traffic flow as red line traversing the internet, yellow line using F5 Global Backbone, and green line within the Virtual Site Regional Edge.  The green line would be the termination point for the client-side traffic flow.  This client-side flow is to the Regional Edge software where L3-4 connection is established, the TLS sesssion setup would begin, and ultimately where application layer services would be proxied to the origin service.

Summary:

I hope you found this article useful and learned how F5 Distributed Cloud allows you to be specific in where your keys are stored and utilized for TLS termination within the F5 Distributed Cloud SaaS architecture.  By utilizing Virtual Sites and additional Public IP Addresses, you can build custom topologies of our Regional Edges over the F5 Global Backbone.  Please leave comments or questions, and I'd be happy to update the article or provide answers directly in the comments.  

Additional Links to Distributed Cloud Content:

Kubernetes Architecture Options with F5 Distributed Cloud 

App Stack, the "iRule" of F5 Distributed Cloud 

Multi-Cluster, Multi-Cloud Networking for K8S with F5 Distributed Cloud – Architecture Pattern 

[MichaelOLeary]

Thanks for the article Matt. I've had customers ask about steering traffic through certain regions. Sometimes it's because they don't want their TLS private key stored outside of their own country/region, but sometimes it's just because they want all traffic flow to go through a given region. I never knew about this possibility before now.

[pkuligowski]

What an amazing article, thanks Matt!