Leveraging BGP and ECMP for F5 Distributed Cloud Customer Edge, Part One

Introduction

Achieving high availability for application delivery while maintaining operational flexibility is a fundamental challenge for modern enterprises. When deploying F5 Distributed Cloud Customer Edge (CE) nodes in private data centers, on-premises environments or in some cases public cloud environments, the choice of how traffic reaches these nodes significantly impacts both service resilience and operational agility.

This article explores how Border Gateway Protocol (BGP) combined with Equal-Cost Multi-Path (ECMP) routing provides an elegant solution for two critical operational requirements:

• High availability of traffic for load balancers running on Customer Edge nodes
• Easier maintenance and upgrades of CE nodes without service disruption

By leveraging dynamic routing protocols instead of static configurations, you gain the ability to gracefully remove individual CE nodes from the traffic path, perform maintenance or upgrades, and seamlessly reintroduce them—all without impacting your application delivery services.

 

Understanding BGP and ECMP Benefits for Customer Edge Deployments

Why BGP and ECMP?

Traditional approaches to high availability often rely on protocols like VRRP, which create Active/Standby topologies. While functional, this model leaves standby nodes idle and creates potential bottlenecks on the active node. The BGP with ECMP fundamentally changes this paradigm.

The Power of ECMP

Equal-Cost Multi-Path routing allows your network infrastructure to distribute traffic across multiple CE nodes simultaneously. When each CE node advertises the same VIP prefix via BGP, your upstream router learns multiple equal-cost paths and distributes traffic across all available nodes:

This creates a true Active/Active topology where:

• All CE nodes actively process traffic
• Load is distributed across the entire set of CEs
• Failure of any single node automatically redistributes traffic to remaining nodes
• No manual intervention required for failover

Key Benefits

 

Understanding Customer Edge VIP Architecture

F5 Distributed Cloud Customer Edge nodes support a flexible VIP architecture that integrates seamlessly with BGP. Understanding how VIPs work is essential for proper BGP configuration.

The Global VIP

Each Customer Edge site can be configured with a Global VIP—a single IP address that serves as the default listener for all load balancers instantiated on that CE. Key characteristics:

• Configured at the CE site level in the F5 Distributed Cloud Console
• Acts as the default VIP for any load balancer that doesn’t have a dedicated VIP configured
• Advertised as a /32 prefix in the routing table
• To know: The Global VIP is NOT generated in the CE's routing table until at least one load balancer is configured on that CE

This last point is particularly important: if you configure a Global VIP but haven't deployed any load balancers, the VIP won't be advertised via BGP. This prevents advertising unreachable services.

For this article, we are going to use 192.168.100.0/24 as VIP subnet for all the examples.

Load Balancer Dedicated VIPs

Individual load balancers can be configured with their own Dedicated VIP, separate from the Global VIP. When a dedicated VIP is configured:

• The load balancer responds only to its dedicated VIP
• The load balancer does not respond to the Global VIP
• The dedicated VIP is also advertised as a /32 prefix
• Multiple load balancers can have different dedicated VIPs on the same CE

This flexibility allows you to:

• Separate different applications on different VIPs
• Implement different routing policies per application
• Maintain granular control over traffic distribution

VIP Summary

 

BGP Filtering Best Practices

Proper BGP filtering is essential for security and operational stability. This section covers the recommended filtering policies for both the upstream network device (firewall/router) and the Customer Edge nodes.

Design Principles

The filtering strategy follows the principle of explicit allow, implicit deny:

• Only advertise what is necessary
• Only accept what is expected
• Use prefix lists with appropriate matching for /32 routes

Upstream Device Configuration (Firewall/Router)

The device peering with your CE nodes should implement strict filtering:

Inbound policy on Firewall/Router

The firewall/router should accept only the CE VIP prefixes. In our example, all VIPs fall within 192.168.100.0/24:

Why "or longer" (le 32)?

Since VIPs are advertised as /32 prefixes, you need to match prefixes more specific than /24. The le 32 (less than or equal to 32) or "or longer" modifier ensures your filter matches the actual /32 routes while still using a manageable prefix range.

Outbound policy on Firewall/Router

By default, the firewall/router should not advertise any prefixes to the CE nodes.

Customer Edge Configuration

The CE nodes should implement complementary filtering:

Outbound Policy

CEs should advertise only their VIP prefixes.

Since all VIPs on Customer Edge nodes are /32 addresses, your prefix filters must also follow the "or longer" approach.

Inbound Policy

CEs should not accept any prefixes from the upstream firewall/router.

Filtering Summary Table

 

Graceful CE Isolation for Maintenance

One of the most powerful benefits of using BGP is the ability to gracefully remove a CE node from the traffic path for maintenance, upgrades, or troubleshooting. This section explains how to isolate a CE by manipulating its BGP route advertisements.

The Maintenance Challenge

When you need to perform maintenance on a CE node (OS upgrade, software update, reboot, troubleshooting), you want to:

1. Stop new traffic from reaching the node
2. Allow existing connections to complete gracefully
3. Perform your maintenance tasks
4. Reintroduce the node to the traffic pool

With VRRP, this can require manual failover procedures. With BGP, you simply stop advertising VIP routes.

Isolation Process Overview

Step 1: Configure BGP Route Filtering on the CE

To isolate a CE, you need to apply a BGP policy that prevents the VIP prefixes from being advertised or received.

Where to Apply the Policy?

There are two possible approaches to stop a CE from receiving traffic:

1. On the BGP peer (firewall/router): Configure an inbound filter on the upstream device to reject routes from the specific CE you want to isolate.
2. On the Customer Edge itself: Configure an outbound export policy on the CE to stop advertising its VIP prefixes.

We recommend the F5 Distributed Cloud approach (option 2) for several reasons:

In many organizations, the team responsible for managing the Customer Edge nodes is different from the team managing the network infrastructure (firewalls, routers). By implementing isolation policies on the CE side, you eliminate cross-team dependencies and enable self-service maintenance operations.

Applying the Filter

The filter is configured through the F5 Distributed Cloud Console on the specific CE site. The filter configuration will be:

• Match the VIP prefix range (192.168.100.0/24 or longer)
• Set the action to Deny
• Apply to the outbound direction (export policy)

Once applied, the CE stops advertising its VIP /32 routes to its BGP peers.

Step 2: Perform Maintenance

With the CE isolated from the traffic path, you can safely:

• Reboot the CE node
• Perform OS upgrades
• Apply software updates

Existing long-lived connections to the isolated CE will eventually timeout, while new connections are automatically directed to the remaining CEs.

Step 3: Reintroduce the CE in the data path

After maintenance is complete:

1. Remove or modify the BGP export filter to allow VIP advertisement
2. The CE will begin advertising its VIP /32 routes again
3. The upstream firewall/router will add the CE back to its ECMP paths
4. Traffic will automatically start flowing to the restored CE

Isolation Benefits Summary

Rolling Upgrade Strategy

Using this isolation technique, you can implement rolling upgrades across your CEs:

Rolling Upgrade Sequence:
                                                       
Step 1: Isolate CE1 → Upgrade CE1 → Put back CE1 in the data path
Step 2: Isolate CE2 → Upgrade CE2 → Put back CE2 in the data path  

Throughout this process:
  • At least 1 CE is always handling traffic
  • No service interruption occurs
  • Each CE is validated before moving to the next

 

Conclusion

BGP with ECMP provides a robust, flexible foundation for high-availability F5 Distributed Cloud Customer Edge deployments. By leveraging dynamic routing protocols:

• Traffic is distributed across all active CE nodes, maximizing resource utilization
• Failover is automatic when a CE becomes unavailable
• Maintenance is graceful through controlled route withdrawal
• Scaling is seamless as new CEs automatically join the traffic distribution once they are BGP-peered

The combination of proper BGP filtering (accepting only VIP prefixes, advertising only what’s necessary) and the ability to isolate individual CEs through route manipulation gives you complete operational control over your application delivery infrastructure.

Whether you’re performing routine maintenance, emergency troubleshooting, or rolling out upgrades, BGP-based CE deployments ensure your applications remain available and your operations remain smooth.