F5 Distributed Cloud for Global Layer 3 Virtual Network Implementation
Introduction As organizations expand their infrastructure across multiple cloud providers and on-site locations, the need for seamless network connectivity becomes paramount. F5 Distributed Cloud provides a powerful solution for connecting distributed sites while maintaining network isolation and control. This article walks through implementing a global Layer 3 Virtual Network using segments with Secure Mesh Sites v2 (SMSv2) Customer Edges. It demonstrates connectivity between private data centers and AWS VPCs. We'll explore the configuration steps and BGP peering setup. The Challenge Organizations need to connect multiple isolated environments—private data centers and cloud VPCs, while maintaining: Network segmentation and isolation Dynamic routing capabilities Consistent connectivity across heterogeneous environments Simple management through a unified control plane Solution Architecture Our implementation consists of three distinct sites: Private Site: Running a Customer Edge (CE) in KVM with BGP peering to the local router for subnet exposure AWS VPC Site 1: Hosting a CE within the VPC AWS VPC Site 2: Another CE deployment with complete isolation (no VPC peering with Site 1) All sites utilize SMSv2 Customer Edges with dual-NIC configurations, connected through F5 Distributed Cloud's global network fabric. Figure 1: Global implementation diagram showing all IP subnets across the three sites with CE deployments and network segments Technical Deep Dive Before diving into the configuration, it's crucial to understand what segments are and how they function within F5 Distributed Cloud: Segments In F5 Distributed Cloud, segments can be considered the equivalent of Layer 3 VRFs in traditional networking. Just as VRFs create separate routing table instances in conventional routers, segments provide: Routing isolation: Each segment maintains its own routing table, ensuring traffic separation Multi-tenancy support: Different segments can overlap IP address spaces without conflict Security boundaries: Traffic between segments requires explicit policy configuration Simplified network management: Logical separation of different network domains or applications Key Segment Characteristics Interface Binding Requirements: Segments must be explicitly attached to CE interfaces Each interface can be part of only one segment This one-to-one mapping ensures clear traffic demarcation and prevents routing ambiguity Route Advertisement and Limitations: Supported Route Types: Connected Routes: Routes for subnets directly configured on the segment interface are automatically advertised BGP Learned Routes: Routes received via BGP peering on the segment interface are propagated to other sites in the same segment Current Limitations: No Static Route Support: Static routes cannot currently be advertised through segment interfaces This is an important consideration when planning your routing architecture Workaround: Use BGP to advertise routes that would traditionally be static Traffic Flow: Traffic entering a CE interface flows within the assigned segment Inter-segment communication requires a special configuration Routes learned on one segment remain isolated from other segments unless explicitly shared Only connected and BGP-learned routes are exchanged between sites within a segment Use Cases: Production/Development Separation: Different segments for prod and dev environments Multi-tenant Deployments: Isolated segments per customer or business unit Compliance Requirements: Segmented networks for PCI, HIPAA, or other regulated traffic This architectural approach provides the flexibility of traditional VRF implementations while leveraging F5 Distributed Cloud's global network capabilities. Customer Edge Interface Architecture Understanding CE Interface Requirements F5 Distributed Cloud Customer Edges require careful interface planning to function correctly, especially in SMSv2 deployments with segments. Understanding the interface architecture is crucial for successful implementations. Interface Capacity and Requirements Minimum Requirement: Each CE must be deployed with at least two physical interfaces Maximum Capacity: CEs support up to eight physical interfaces VLAN Support: Sub-interfaces can be created on top of physical interfaces Interface Types and Roles Customer Edge interfaces serve distinct purposes within the F5 Distributed Cloud architecture: 1. Site Local Outside (SLO) Interface The SLO interface is the "management" and control plane interface: Primary Functions: Zero-touch provisioning of the Customer Edge Establishing VPN tunnels for control plane communication with F5 XC Global Controller Management traffic and orchestration commands Health monitoring and telemetry data transmission Requirements: Must have Internet access to reach F5's global infrastructure Should be considered as the "management interface" of the CE Configured on the first interface (eth0/ens3) Cannot be used for segment assignment 2. Site Local Inside (SLI) and Segment Interfaces The remaining interfaces can be configured for data plane traffic: Site Local Inside (SLI): Used for local network connectivity without segment assignment Segment Interfaces: Dedicated to specific network segments (VRF-like isolation) Each interface can belong to only one segment Supports BGP peering within the segment context Used for segmented connectivity Interface Planning Considerations When designing your CE deployment: Two-Interface Minimum Deployment: Interface 1: SLO for management and control plane Interface 2: Segment or SLI for data plane traffic Multi-Segment Deployments: Require additional interfaces (one per segment plus SLO) Example: 4 segments need 5 interfaces (1 SLO + 4 segment interfaces) Cloud Deployments: Ensure cloud instance types support the required number of network interfaces Remember to disable source/destination checks on all interfaces Consider network interfaces limits when planning for scale Routing Considerations for Segments: Plan for BGP peering if you need to advertise routes beyond connected subnets Static routes cannot be advertised through segment interfaces yet Each segment interface will only advertise: It’s directly connected subnet Routes learned via BGP on that interface Design your IP addressing scheme accordingly This interface architecture ensures proper separation between management/control plane traffic and data plane traffic, while providing the flexibility needed for complex network topologies. Prerequisites Before beginning the implementation, ensure you have: F5 Distributed Cloud account with appropriate permissions Three deployed Customer Edge nodes (SMSv2 sites) Basic understanding of BGP configuration (if implementing BGP peering) Step-by-Step Configuration Step 1: Create the Network Segment Navigate to Multi-Cloud Network Connect → Manage → Networking → Segments Click "Add Segment" Configure your segment with appropriate naming and network policies Define the segment scope based on your requirements Save the configuration Figure 2: Segment creation The segment acts as a logical network overlay that spans across all participating sites, similar to extending a VRF across multiple locations in traditional MPLS networks. Step 2: Assign Segments to CE Interfaces Navigate to Multi-Cloud Network Connect → Manage → Site Management → Secure Mesh Sites v2 For each Customer Edge: Select the CE and edit its configuration Navigate to the node interface configuration Modify the interface settings: Select the appropriate interface (typically the second NIC, not the SLO interface) Assign the created segment to this interface Configure the interface mode as required Ensure the SLO interface remains dedicated to management/control plane Apply the changes Figure 3: Node interface configuration showing segment assignment to the appropriate interface Important: Remember that: The SLO interface (typically eth0/ens3) should not be used for segment assignment Each data plane interface can belong to only one segment Plan your interface allocation carefully based on your traffic segmentation requirements Repeat this process for all participating CEs. Once complete, all sites will be connected through the assigned segment. Figure 4: Overview of configured interfaces with segment assignments across all CE nodes Step 3: Configure BGP Peering (Optional) For sites requiring dynamic routing with local infrastructure: Navigate to the CE's BGP configuration Select the correct interface tied to the segment (e.g., "ens4") Configure BGP parameters: Local AS number Peer AS number Peer IP address Network advertisements Apply the configuration Figure 5: BGP peering configuration showing interface selection tied to the segment BGP peering enables automatic route exchange between your CE and local network infrastructure, with routes learned via BGP being contained within the assigned segment's routing domain. Important Note on Route Advertisement: Segment interfaces only advertise connected routes (interface subnets) and BGP-learned routes Static routes are not currently supported for advertisement through segments If you need to advertise additional routes beyond the connected subnet, BGP peering is the only available method This makes BGP configuration essential for most production deployments where multiple subnets need to be accessible Verifying Route Tables To confirm proper route propagation: Navigate to Multi-Cloud Network Connect → Overview → Infrastructure Select your site name Click on CE Routes Apply filters as needed Figure 6: CE Routes selection interface for viewing routing information You should observe: Routes from remote sites appearing in the routing table Correct next-hop information pointing to remote CE IPs BGP-learned routes (if BGP is configured and Site Survivability is enabled) Routes properly isolated within their respective segments Only connected and BGP routes present (no static routes) Figure 7: Route table showing routes received from other sites with next-hop information Conclusion F5 Distributed Cloud's Global Layer 3 Virtual Network with segments provides a robust solution for connecting distributed infrastructure across multiple environments. By leveraging segments as VRF-like constructs, organizations can achieve network isolation, multi-tenancy, and simplified management across their global infrastructure. Key takeaways: Always use dual-NIC configurations for SMSv2 sites (minimum one SLO + one data plane interface) Understand the critical role of the SLO interface for management and control plane Plan interface allocation carefully - CEs support up to 8 physical interfaces plus VLAN sub-interfaces Understand segments as Layer 3 VRF equivalents for proper design Remember the one-to-one mapping between interfaces and segments Be aware that segments only advertise connected and BGP-learned routes (no static route support currently) Use BGP peering to advertise additional subnets beyond connected routes Disable source/destination checks for cloud-based CEs As F5 Distributed Cloud continues to evolve, some of these considerations may change. Always refer to the latest documentation and test thoroughly in your environment.
