F5 Distributed Cloud for Global Layer 3 Virtual Network Implementation

Introduction

As organizations expand their infrastructure across multiple cloud providers and on-site locations, the need for seamless network connectivity becomes paramount. F5 Distributed Cloud provides a powerful solution for connecting distributed sites while maintaining network isolation and control.

This article walks through implementing a global Layer 3 Virtual Network using segments with Secure Mesh Sites v2 (SMSv2) Customer Edges. It demonstrates connectivity between private data centers and AWS VPCs. We'll explore the configuration steps and BGP peering setup.

 

The Challenge

Organizations need to connect multiple isolated environments—private data centers and cloud VPCs, while maintaining:

• Network segmentation and isolation
• Dynamic routing capabilities
• Consistent connectivity across heterogeneous environments
• Simple management through a unified control plane

 

Solution Architecture

Our implementation consists of three distinct sites:

1. Private Site: Running a Customer Edge (CE) in KVM with BGP peering to the local router for subnet exposure
2. AWS VPC Site 1: Hosting a CE within the VPC
3. AWS VPC Site 2: Another CE deployment with complete isolation (no VPC peering with Site 1)

All sites utilize SMSv2 Customer Edges with dual-NIC configurations, connected through F5 Distributed Cloud's global network fabric.

Figure 1: Global implementation diagram showing all IP subnets across the three sites with CE deployments and network segments

 

Technical Deep Dive

Before diving into the configuration, it's crucial to understand what segments are and how they function within F5 Distributed Cloud:

Segments

In F5 Distributed Cloud, segments can be considered the equivalent of Layer 3 VRFs in traditional networking. Just as VRFs create separate routing table instances in conventional routers, segments provide:

• Routing isolation: Each segment maintains its own routing table, ensuring traffic separation
• Multi-tenancy support: Different segments can overlap IP address spaces without conflict
• Security boundaries: Traffic between segments requires explicit policy configuration
• Simplified network management: Logical separation of different network domains or applications

Key Segment Characteristics

1. Interface Binding Requirements:
   • Segments must be explicitly attached to CE interfaces
   • Each interface can be part of only one segment
   • This one-to-one mapping ensures clear traffic demarcation and prevents routing ambiguity
2. Route Advertisement and Limitations:
   • Supported Route Types:
     • Connected Routes: Routes for subnets directly configured on the segment interface are automatically advertised
     • BGP Learned Routes: Routes received via BGP peering on the segment interface are propagated to other sites in the same segment
   • Current Limitations:
     • No Static Route Support: Static routes cannot currently be advertised through segment interfaces
     • This is an important consideration when planning your routing architecture
     • Workaround: Use BGP to advertise routes that would traditionally be static
3. Traffic Flow:
   • Traffic entering a CE interface flows within the assigned segment
   • Inter-segment communication requires a special configuration
   • Routes learned on one segment remain isolated from other segments unless explicitly shared
   • Only connected and BGP-learned routes are exchanged between sites within a segment
4. Use Cases:
   • Production/Development Separation: Different segments for prod and dev environments
   • Multi-tenant Deployments: Isolated segments per customer or business unit
   • Compliance Requirements: Segmented networks for PCI, HIPAA, or other regulated traffic

This architectural approach provides the flexibility of traditional VRF implementations while leveraging F5 Distributed Cloud's global network capabilities.

 

Customer Edge Interface Architecture

Understanding CE Interface Requirements

F5 Distributed Cloud Customer Edges require careful interface planning to function correctly, especially in SMSv2 deployments with segments. Understanding the interface architecture is crucial for successful implementations.

Interface Capacity and Requirements

• Minimum Requirement: Each CE must be deployed with at least two physical interfaces
• Maximum Capacity: CEs support up to eight physical interfaces
• VLAN Support: Sub-interfaces can be created on top of physical interfaces

 

Interface Types and Roles

Customer Edge interfaces serve distinct purposes within the F5 Distributed Cloud architecture:

1. Site Local Outside (SLO) Interface

The SLO interface is the "management" and control plane interface:

• Primary Functions:
  • Zero-touch provisioning of the Customer Edge
  • Establishing VPN tunnels for control plane communication with F5 XC Global Controller
  • Management traffic and orchestration commands
  • Health monitoring and telemetry data transmission
• Requirements:
  • Must have Internet access to reach F5's global infrastructure
  • Should be considered as the "management interface" of the CE
  • Configured on the first interface (eth0/ens3)
  • Cannot be used for segment assignment

2. Site Local Inside (SLI) and Segment Interfaces

The remaining interfaces can be configured for data plane traffic:

• Site Local Inside (SLI):
  • Used for local network connectivity without segment assignment
• Segment Interfaces:
  • Dedicated to specific network segments (VRF-like isolation)
  • Each interface can belong to only one segment
  • Supports BGP peering within the segment context
  • Used for segmented connectivity

 

Interface Planning Considerations

When designing your CE deployment:

1. Two-Interface Minimum Deployment:
   • Interface 1: SLO for management and control plane
   • Interface 2: Segment or SLI for data plane traffic
2. Multi-Segment Deployments:
   • Require additional interfaces (one per segment plus SLO)
   • Example: 4 segments need 5 interfaces (1 SLO + 4 segment interfaces)
3. Cloud Deployments:
   • Ensure cloud instance types support the required number of network interfaces
   • Remember to disable source/destination checks on all interfaces
   • Consider network interfaces limits when planning for scale
4. Routing Considerations for Segments:
   • Plan for BGP peering if you need to advertise routes beyond connected subnets
   • Static routes cannot be advertised through segment interfaces yet
   • Each segment interface will only advertise:
     • It’s directly connected subnet
     • Routes learned via BGP on that interface
   • Design your IP addressing scheme accordingly

This interface architecture ensures proper separation between management/control plane traffic and data plane traffic, while providing the flexibility needed for complex network topologies.

 

Prerequisites

Before beginning the implementation, ensure you have:

• F5 Distributed Cloud account with appropriate permissions
• Three deployed Customer Edge nodes (SMSv2 sites)
• Basic understanding of BGP configuration (if implementing BGP peering)

 

Step-by-Step Configuration

Step 1: Create the Network Segment

Navigate to Multi-Cloud Network Connect → Manage → Networking → Segments

1. Click "Add Segment"
2. Configure your segment with appropriate naming and network policies
3. Define the segment scope based on your requirements
4. Save the configuration

Figure 2: Segment creation

The segment acts as a logical network overlay that spans across all participating sites, similar to extending a VRF across multiple locations in traditional MPLS networks.

 

Step 2: Assign Segments to CE Interfaces

Navigate to Multi-Cloud Network Connect → Manage → Site Management → Secure Mesh Sites v2

For each Customer Edge:

1. Select the CE and edit its configuration
2. Navigate to the node interface configuration
3. Modify the interface settings:
   • Select the appropriate interface (typically the second NIC, not the SLO interface)
   • Assign the created segment to this interface
   • Configure the interface mode as required
   • Ensure the SLO interface remains dedicated to management/control plane
4. Apply the changes

Figure 3: Node interface configuration showing segment assignment to the appropriate interface

Important: Remember that:

• The SLO interface (typically eth0/ens3) should not be used for segment assignment
• Each data plane interface can belong to only one segment
• Plan your interface allocation carefully based on your traffic segmentation requirements

Repeat this process for all participating CEs. Once complete, all sites will be connected through the assigned segment.

Figure 4: Overview of configured interfaces with segment assignments across all CE nodes

 

Step 3: Configure BGP Peering (Optional)

For sites requiring dynamic routing with local infrastructure:

1. Navigate to the CE's BGP configuration
2. Select the correct interface tied to the segment (e.g., "ens4")
3. Configure BGP parameters:
   • Local AS number
   • Peer AS number
   • Peer IP address
   • Network advertisements
4. Apply the configuration

Figure 5: BGP peering configuration showing interface selection tied to the segment

BGP peering enables automatic route exchange between your CE and local network infrastructure, with routes learned via BGP being contained within the assigned segment's routing domain.

 

Important Note on Route Advertisement:

• Segment interfaces only advertise connected routes (interface subnets) and BGP-learned routes
• Static routes are not currently supported for advertisement through segments
• If you need to advertise additional routes beyond the connected subnet, BGP peering is the only available method
• This makes BGP configuration essential for most production deployments where multiple subnets need to be accessible

 

Verifying Route Tables

To confirm proper route propagation:

1. Navigate to Multi-Cloud Network Connect → Overview → Infrastructure
2. Select your site name
3. Click on CE Routes
4. Apply filters as needed

Figure 6: CE Routes selection interface for viewing routing information

You should observe:

• Routes from remote sites appearing in the routing table
• Correct next-hop information pointing to remote CE IPs
• BGP-learned routes (if BGP is configured and Site Survivability is enabled)
• Routes properly isolated within their respective segments
• Only connected and BGP routes present (no static routes)

Figure 7: Route table showing routes received from other sites with next-hop information

 

Conclusion

F5 Distributed Cloud's Global Layer 3 Virtual Network with segments provides a robust solution for connecting distributed infrastructure across multiple environments. By leveraging segments as VRF-like constructs, organizations can achieve network isolation, multi-tenancy, and simplified management across their global infrastructure.

Key takeaways:

• Always use dual-NIC configurations for SMSv2 sites (minimum one SLO + one data plane interface)
• Understand the critical role of the SLO interface for management and control plane
• Plan interface allocation carefully - CEs support up to 8 physical interfaces plus VLAN sub-interfaces
• Understand segments as Layer 3 VRF equivalents for proper design
• Remember the one-to-one mapping between interfaces and segments
• Be aware that segments only advertise connected and BGP-learned routes (no static route support currently)
• Use BGP peering to advertise additional subnets beyond connected routes
• Disable source/destination checks for cloud-based CEs

As F5 Distributed Cloud continues to evolve, some of these considerations may change. Always refer to the latest documentation and test thoroughly in your environment.