F5 Distributed Cloud: Virtual Sites - Regional Edge (RE)

This article is a part of a mini-series on traffic flows in F5 Distributed Cloud. This first article starts with the Regional Edge (RE) SaaS data plane of F5 Distributed Cloud.  We'll be diving into how you can use Virtual Sites to customize ingress and egress traffic patterns with the Regional Edges.

Background

Related Content:

I've combined two extensive articles for Virtual Sites into this single article.  I wrote an article a little over a year ago about setting which Regional Edge locations can decrypt traffic.  My good friend across the pond (Atlantic Ocean) Philippe_CLOUP wrote a similar article for using Virtual Sites to limit which Regional Edge locations will health-monitor your service.  You can find both articles below for more details:

• F5 Distributed Cloud - Regional Decryption with Virtual Sites
• F5 Distributed Cloud - Regional Edge Health Monitoring Insights

I’d like to bring both articles together, to provide a different perspective of the concept that is Virtual Sites, specifically Regional Edge (RE) Virtual Sites. 

Historical Knowledge:

First, let’s remember that F5 is a proxy company--for Application Delivery and Security. We excel in delivery services such as load-balancing, service discovery, layer 7 routing, rewriting, redirecting, etc. and security services such as Web Application Firewall, Anti-Automation (bot protection), API Security, API Discovery, L7 DDoS, Access Federation, etc.  As a proxy architecture, we create a full OSI stack twice for L7 services.  These two “flows” include the L3/L4 connection, TLS Session, and L7 Request on each side of the proxy:

Connection for Proxy/Reverse Proxy architectures

1. Client-Side or Downstream (orange) – is the connection between the client and server (F5 Proxy).
2. Server-Side or Upstream (green) – is the connection between the client (F5 Proxy) and origin server/application

World of F5

In the F5 BIG-IP world, we’ve always referred to #1 as Client-Side, and #2 as Server-Side.  However, other proxies will refer to #1 as Downstream, and #2 as Upstream.  It is good to know this connection terminology by both standards.

Where BIG-IP is a single data plane, F5 Distributed Cloud has multiple data planes that work together to deliver and secure traffic with an overlay network fabric in-between.  This overlay network fabric allows the Distributed Cloud data planes to split the sides of the proxy across two different data plane instances.  In terms of our Regional Edge data plane, this means we have the ability to ingress the client-side/downstream connection in one Regional Edge, and egress the traffic server-side/upstream out of a completely different Regional Edge.  For example, this occurs naturally when a health monitor fails at the Regional Edge handling the ingress traffic.  This ingress traffic will be sent to the nearest Regional Edge with a healthy health monitor to egress the traffic to origin.  Well, what if we want to statically specify this action always?  Let’s dig in!

F5 Distributed Cloud - Sides of the Proxy

Within F5 Distributed Cloud, you configure Load Balancers.  These Load Balancers are the proxies for application/api delivery and security services.  Within this Load Balancer configuration, you have two configuration objects that are specific to the client-side/downstream and server-side/upstream. 

• Client-Side/Downstream – VIP Advertisement

• Server-Side/Upstream – Origin Pool/Servers

Virtual Site - Overview

So, what is a Virtual Site?  According to our docs site, a Virtual Site is:

“A virtual site provides a mechanism to perform operations on a group of sites, reducing the need to repeat the same set of operations for each site.” 

In my head, I simply think of a Virtual Site as a logical grouping of data-plane instances.  This logical grouping can then be attached to configuration objects. The attachment of the logical grouping of data plane objects tells the F5 Distributed Cloud what components need to be configured and install on which data plane instances.  We support two types of Virtual Sites, which map to the Data Plane types of F5 Distributed Cloud.  You can build a Virtual Site to group Regional Edges (REs) or Customer Edges (CEs).  We're focusing on Regional Edges (REs) for this article.

There is a default Virtual Site applied when choosing “Internet” for your VIP advertisement, or not specifying a site for your Origin Pool, which is “ves-io-all-res”.  There is an additional Virtual Site created by default that is “ves-no-sites”.  This Virtual Site is used when selecting “Do Not Advertise” under the VIP Advertisement selection.  Building your own grouping of sites into a Virtual Site, is easy.  Navigate to the Shared Configuration Tile under Select a Service or from the Home Screen.  Then Manage, Virtual Sites.

Building a Virtual Site:

Shared Configuration -> Manage -> Virtual Sites -> Add Virtual Site

• Metadata
  • Name – This is the name you want to give your Virtual Site and will choose when attaching it to configuration objects.
  • Labels – You can use labels as a selector option
  • Description – Give your Virtual Site a good description and overview so you know why you built it.
• Site Type (Choose One)
  • RE – Regional Edge
  • CE – Customer Edge
• Site Selector Expression
  • Selector Expression (Add Label) – For RE sites, we’ll use the common labels associated with the REs.
    • To find these referred labels per Regional Edge, you can navigate to Multi-Cloud Network Connect -> Overview -> Sites
    • Add Filter to only view “Site Type” “In” “REGIONAL_EDGE”
    • Find the Site you’re looking for and drop down the arrow to the left of the site to review the JSON of the site.
    • Review the “labels” section
      • Example of DC-Ashburn

Example of pre-defined labels for a RE site

• • For CEs, you’d build your own labels and key values

Examples RE Label Selection:

Regional Edge label selection by CountryRegional Edge label selection by SiteName

Applying Virtual Sites

Now, let’s look at how we build and apply these Virtual Sites, both Client-Side/Downstream (VIP Advertisement) and Server-Side/Upstream (Origin).

VIP Advertisement – Client-Side/Downstream

• Attaching your Virtual Site to a Public IP
  • Once you have your virtual site built, you’ll need an IP address that is not your Default Tenant IP. I discuss this in detail here, but ultimately you can bring your enterprise IP space to F5 Distributed Cloud (/24 or larger), or request additional IPs from F5. 
  • You can find your additional IPs for your F5 Distributed Cloud tenant by selecting the “Shared” Tile from the home screen, choose Manage, Choose Public IP Addresses.
  • Select the IP Address you’d like to set your Virtual Site. Drop down the “Virtual Site” option within that IP Address Configuration and select your Virtual Site.
• Using the IP associated to your Virtual Site within a Load Balancer
  • Go to the Load Balancer you want to have this Virtual Site Applied
  • Find “VIP Advertisement”
  • Choose Internet (Specified VIP)
  • Select the IP address you updated for your Virtual Site at the step above
  • Note, you’ll need to update DNS to point at this new IP address.

 

Origin Pool – Server-Side/Upstream

• Building Origin Pool with Virtual Site
  • Navigate from the Home Screen to either the “Multi-Cloud App Connect” or “Web App & API Protection” tile. Choose Manage, Load Balancers, Origin Pools – Add or Edit from here
  • Choose Add Origin Servers
  • You can now choose what way you’d like to enter/discovery the origin pool servers “on given Sites”.
    • IP Address of Origin Server on given Sites
    • DNS Name of Origin Server on given Sites
    • K8s Service Name of Origin Server on given Sites
    • Consul Service Name of Origin Server on given Sites
  • Once you’ve entered the server/service information, be sure to select Virtual Site in the Site or Virtual Site drop down. Choose the Virtual Site you’d like applied to the server-side/upstream.
  • Lastly, for RE Virtual Sites, be sure to always choose the “Outside Network” under Select Network on the site drop down

Understanding Traffic Patterns:

I am going to provide 4 example topologies to give a basic understanding of the traffic flows with Virtual Sites.  These Virtual Site topologies are up to you and your enterprise, and my examples certainly aren't the only possibilities.  Please just remember, we do maintenance every 4-8 weeks that will take REs offline as we roll through locations upgrading the platform.  You never want to have 1 location selected in your virtual site, as it’ll create an outage.

For my diagrams, there are a few items to call out.  I have a subset of the Regional Edges depicted for simplicity’s sake.  I’ve provided a few North America, Europe, and Asia Pacific locations to the diagram.  I’ve randomly selected IP addresses as examples.  I have a single origin located somewhere in the United States called “US Origin”.  I’ve provided two client examples of a “US Client” and “EU Client”.  Pink colors are to identify the client-side/downstream VIP Advertisement side of the proxy, and green is to identify server-side/upstream Origin side of the proxy in the topology diagram.  Lastly, there is an index of the “packet flow” for where the packet is in the process.  You can click on the graphics to view larger and in more detail.

Solution 1: All Regional Edges

• In this solution, we’re maintaining the default Virtual Sites by selecting Internet for VIP Advertisement and selecting an origin with “Public DNS” or “Public IP Address”.
  • This solution is straight-forward and F5 Distributed Cloud’s default topology. The Regional Edge the traffic ingresses into, is likely to be the Regional Edge the traffic egresses. 
  • US Client and EU client follow their local ISP BGP peering over the internet to find closest Regional Edge for the VIP IP address 72.19.3.183. For the US Client that happens to be New York, and the EU Client Paris.
  • Both New York and Paris have Origin Pool configurations, and egress locally back to the internet to route to US Origin. The US Origin will need to allow all Regional Edge SNAT ranges for health checks and traffic.

All Regional Edges - Client & Server Side - Traffic Pattern

 

Solution 2: North America Only (Client-Side/Downstream)

This is a common request or deployment strategy where an enterprise believes to operate only within a certain region or country of the world and prefer to limit the Regional Edges that can process the traffic.

• In this topology, we’d updated the VIP Advertisement with a secondary IP and Virtual Site but didn’t update the Origin to match the Virtual Site.
• US Client doesn’t change in this topology, still routes to/through New York to the US Origin.
• EU Client, however, doesn’t have a regional VIP advertisement for the software stack and routes to Montreal for “Proxy Services”. Since Montreal has an Origin available, the traffic egresses Montreal to the US Origin.
  • You’ll notice the orange-dotted network traffic line still ingresses F5’s Global Backbone in Paris due to local /24 route, but since there isn’t a specific route for the VIP local in Paris, our Global Backbone routes traffic to the next nearest PoP with the VIP advertised.
  • For more information on routing rules of the Internet, see linked article at the top of this article – Regional Edge Decryption
  • Note – Montreal is not necessarily the next PoP to Paris, this is an example.
• In my opinion there are some flaws with this configuration in its current model/topology:
  • EU, APCJ/AUS RE locations will continue to health monitor the origin.
  • If utilizing ACL/NSG security at origin, your origin servers tab will have down origin servers from the regions you don’t allow (showing red), and this will lower your Application Score creating confusion operationally.
  • There is a very rare chance that traffic is routed to one of the EU, APCJ/AUS locations to origin under a failure scenario.

 

Solution 2 - Connection Table:

 

Solution 3: North America Only (Both Sides)

Following along with #2 previously, in this case we’ve updated both VIP Advertisement for Client-Side/Downstream, as well as Origin Server-Side/Upstream with the same Virtual Site.

• The Ingress of traffic doesn’t change from #2 above.
• The Egress to origin is also unlikely to change in typical conditions.
• However, by updating both sides, we’ve:
  • Eliminated the additional traffic from health monitors hitting origin, and likely failing by the enterprise ACL/NSG deny list.
  • Ensured traffic will never source from Regional Edges that are not intended to be used – even in failure scenarios.

 

Solution 3 - Connection Table:

 

Solution 4: North America Only (Server-Side/Upstream)

Again, building on #2 and #3 above, this model utilizes all Regional Edges on the VIP Advertisement, but limits what F5 Regional Edge locations can send to Origin. This is my favorite model, as you’re building a Secure Edge like an “umbrella” or “Funnel”. 

• In this model traffic will ingress any Regional Edge globally based on the clients preferred routing on the Internet.
• Since Proxy Services like LB, WAF, L7 DDoS, API Discovery, API Security, Bot Protections, etc. are applied at the VIP advertisement side, this model provides security close to the client.
• Compared to #2 and #3 – While it is anticipated that the enterprise’s “good clients” are within North America, you cannot predict where you may be attacked from. So having a “Distributed” secure edge takes full advantage of all the compute to serve good clients and block malicious clients.
• Looking closer at the egress, because we are ingressing traffic in a location that doesn’t include that location in the Origin Virtual Site, we need to use our Software Defined Network (SDN) Fabric to get to a location that is included in the Origin Virtual site and has a healthy origin.
• In this case, while the EU Client ingresses through Paris and received proxy services in Paris, it will be routed via the SDN Fabric over to Montreal to egress F5 Distributed Cloud. This means, the US origin will see the source IP address of the Montreal Regional Edge.
• This is an example of a Globally Disperse Proxy, where F5 Distributed Cloud can ingress traffic in one location, and egress in another location.

 

Solution 4 - Connection Table:

 

Summary:

In summary, the topology models are extremely flexible for F5 Distributed Cloud customers utilizing our SaaS data plane via Regional Edges.  There isn’t a one topology that fits all customers, and you should work with your F5 Account Team to collaborate on your business needs.  I’ll be adding another article as it pertains to the Customer Edge data planes with Virtual sites.  Then lastly, adding a combination article to explain how Regional Edge (RE) and Customer Edge (CE) date planes can be used together, extending the Application Aware Fabric capabilities in many ways to serve your enterprise.

Related Resources:

F5 Distributed Cloud Customer Edge Migration Centos to RHEL | DevCentral

F5 Distributed Cloud – CE High Availability Options: A Comparative Exploration | DevCentral