For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Leveraging BGP and ECMP for F5 Distributed Cloud Customer Edge, Part Two

Introduction

This is the second part of our series on leveraging BGP and ECMP for F5 Distributed Cloud Customer Edge deployments. In Part One, we explored the high-level concepts, architecture decisions, and design principles that make BGP and ECMP such a powerful combination for Customer Edge high availability and maintenance operations.

This article provides step-by-step implementation guidance, including:

  • High-level and low-level architecture diagrams
  • Complete BGP peering and routing policy configuration in F5 Distributed Cloud Console
  • Practical configuration examples for Fortinet FortiGate and Palo Alto Networks firewalls

By the end of this article, you'll have everything you need to implement BGP-based high availability for your Customer Edge deployment.

Architecture Overview

Before diving into configuration, let’s establish a clear picture of the architecture we’re implementing. We’ll examine this from two perspectives: a high-level logical view and a detailed low-level view showing specific IP addressing and AS numbers.

High-Level Architecture

The high-level architecture illustrates the fundamental traffic flow and BGP relationships in our deployment:

Key Components:

ComponentRole
InternetExternal connectivity to the network
Next-Generation FirewallActs as the BGP peer and performs ECMP distribution to Customer Edge nodes
Customer Edge Virtual Site Two or more CE nodes advertising identical VIP prefixes via BGP

The architecture follows a straightforward principle: the upstream firewall establishes BGP peering with each CE node. Each CE advertises its VIP addresses as /32 routes. The firewall, seeing multiple equal-cost paths to the same destination, distributes incoming traffic across all available CE nodes using ECMP.

Low-Level Architecture with IP Addressing

The low-level diagram provides the specific details needed for implementation, including IP addresses and AS numbers:

Network Details:

ComponentIP AddressRole
Firewall (Inside)10.154.4.119/24BGP Peer, ECMP Router
CE1 (Outside)10.154.4.160/24Customer Edge Node 1
CE2 (Outside)10.154.4.33/24Customer Edge Node 2
Global VIP192.168.100.10/32Load Balancer VIP

BGP Configuration:

ParameterFirewallCustomer Edge
AS Number6500165002
Router ID10.154.4.119Auto-assigned based on interface IP
Advertised PrefixNone192.168.100.0/24 le 32

This configuration uses eBGP (External BGP) between the firewall and CE nodes, with different AS numbers for each. The CE nodes share the same AS number (65002), which is the standard approach for multi-node CE deployments advertising the same VIP prefixes.

Configuring BGP in F5 Distributed Cloud Console

The F5 Distributed Cloud Console provides a centralized interface for configuring BGP peering and routing policies on your Customer Edge nodes. This section walks you through the complete configuration process.

Step 1: Configure the BGP peering

Go to: Multi-Cloud Network Connect --> Manage --> Networking --> External Connectivity --> BGP Peers & Policies

Click on Add BGP Peer

Then add the following information:

  • Object name
  • Site where to apply this BGP configuration
  • ASN
  • Router ID

Here is an example of the required parameters.

Then click on Peers --> Add Item

And filled the relevant fields like below by adapting the parameters for your requirements.

 

Step 2: Configure the BGP routing policies

Go to: Multi-Cloud Network Connect --> Manage --> Networking --> External Connectivity --> BGP Peers & Policies --> BGP Routing Policies

Click on Add BGP Routing Policy

Add a name for your BGP routing policy object and click on Configure to add the rules.

Click on Add Item to add a rule.

Here we are going to allow the /32 prefixes from our VIP subnet (192.168.100.0/24).

Save the BGP Routing Policy

Repeat the action to create another BGP routing policy with the exact same parameters except the Action Type, which should be of type Deny.

Now we have two BGP routing policies:

  • One to allow the VIP prefixes (for normal operations)
  • One to deny the VIP prefixes (for maintenance mode)

We still need to a a third and final BGP routing policy, in order to deny any prefixes on the CE.

For that, create a third BGP routing policy with this match.

Step 3: Apply the BGP routing policies

To apply the BGP routing policies in your BGP peer object, edit the Peer and:

  • Enable the BGP routing policy
  • Apply the BGP routing policy objects created before for Inbound and Outbound

 

Fortinet FortiGate Configuration

FortiGate firewalls are widely deployed as network security appliances and support robust BGP capabilities. This section provides the minimum configuration for establishing BGP peering with Customer Edge nodes and enabling ECMP load distribution.

Step 1: Configure the Router ID and AS Number

Configure the basic BGP settings:

config router bgp
    set as 65001
    set router-id 10.154.4.119
    set ebgp-multipath enable

Step 2: Configure BGP Neighbors

Add each CE node as a BGP neighbor:

    config neighbor
        edit "10.154.4.160"
            set remote-as 65002
            set route-map-in "ACCEPT-CE-VIPS"
            set route-map-out "DENY-ALL"
            set soft-reconfiguration enable
        next
        edit "10.154.4.63"
            set remote-as 65002
            set route-map-in "ACCEPT-CE-VIPS"
            set route-map-out "DENY-ALL"
            set soft-reconfiguration enable
        next
    end
end

Step 3: Create Prefix List for VIP Range

Define the prefix list that matches the CE VIP range:

config router prefix-list
    edit "CE-VIP-PREFIXES"
        config rule
            edit 1
                set prefix 192.168.100.0 255.255.255.0
                set ge 32
                set le 32
            next
        end
    next
end

Important: The ge 32 and le 32 parameters ensure we only match /32 prefixes within the 192.168.100.0/24 range, which is exactly what CE nodes advertise for their VIPs.

Step 4: Create Route Maps

Configure route maps to implement the filtering policies:

Inbound Route Map (Accept VIP prefixes):

config router route-map
    edit "ACCEPT-CE-VIPS"
        config rule
            edit 1
                set match-ip-address "CE-VIP-PREFIXES"
            next
        end
    next
end

Outbound Route Map (Deny all advertisements):

config router route-map
    edit "DENY-ALL"
        config rule
            edit 1
                set action deny
            next
        end
    next
end

Step 5: Verify BGP Configuration

After applying the configuration, verify the BGP sessions and routes:

Check BGP neighbor status:

get router info bgp summary

VRF 0 BGP router identifier 10.154.4.119, local AS number 65001
BGP table version is 4
1 BGP AS-PATH entries
0 BGP community entries

Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.154.4.33  4      65002    2092    2365        0    0    0 00:05:33        1
10.154.4.160 4      65002    2074    2346        0    0    0 00:14:14        1

Total number of neighbors 2

Verify ECMP routes:

get router info routing-table bgp
Routing table for VRF=0
B       192.168.100.10/32 [20/255] via 10.154.4.160 (recursive is directly connected, port2), 00:00:11, [1/0]
                          [20/255] via 10.154.4.33 (recursive is directly connected, port2), 00:00:11, [1/0]

 

Palo Alto Networks Configuration

Palo Alto Networks firewalls provide enterprise-grade security with comprehensive routing capabilities. This section covers the minimum BGP configuration for peering with Customer Edge nodes.

Note: This part is assuming that Palo Alto firewall is configured in the new "Advanced Routing Engine" mode. And we will use the logical-router named "default".

 

Step 1: Configure ECMP parameters

set network logical-router default vrf default ecmp enable yes
set network logical-router default vrf default ecmp max-path 4
set network logical-router default vrf default ecmp algorithm ip-hash

 

Step 2: Configure objects IPs and firewall rules for BGP peering

set address CE1 ip-netmask 10.154.4.160/32
set address CE2 ip-netmask 10.154.4.33/32
set address-group BGP_PEERS static [ CE1 CE2 ]
set address LOCAL_BGP_IP ip-netmask 10.154.4.119/32

set rulebase security rules ALLOW_BGP from service
set rulebase security rules ALLOW_BGP to service
set rulebase security rules ALLOW_BGP source LOCAL_BGP_IP
set rulebase security rules ALLOW_BGP destination BGP_PEERS
set rulebase security rules ALLOW_BGP application bgp
set rulebase security rules ALLOW_BGP service application-default
set rulebase security rules ALLOW_BGP action allow

 

Step 3: Palo Alto Configuration Summary (CLI Format)

set network routing-profile filters prefix-list ALLOWED_PREFIXES type ipv4 ipv4-entry 1 prefix entry network 192.168.100.0/24
set network routing-profile filters prefix-list ALLOWED_PREFIXES type ipv4 ipv4-entry 1 prefix entry greater-than-or-equal 32
set network routing-profile filters prefix-list ALLOWED_PREFIXES type ipv4 ipv4-entry 1 prefix entry less-than-or-equal 32
set network routing-profile filters prefix-list ALLOWED_PREFIXES type ipv4 ipv4-entry 1 action permit
set network routing-profile filters prefix-list ALLOWED_PREFIXES description "Allow only m32 inside 192.168.100.0m24"
set network routing-profile filters prefix-list DENY_ALL type ipv4 ipv4-entry 1 prefix entry network 0.0.0.0/0
set network routing-profile filters prefix-list DENY_ALL type ipv4 ipv4-entry 1 prefix entry greater-than-or-equal 0
set network routing-profile filters prefix-list DENY_ALL type ipv4 ipv4-entry 1 prefix entry less-than-or-equal 32
set network routing-profile filters prefix-list DENY_ALL type ipv4 ipv4-entry 1 action deny
set network routing-profile filters prefix-list DENY_ALL description "Deny all prefixes"


set network routing-profile bgp filtering-profile FILTER_INBOUND ipv4 unicast inbound-network-filters prefix-list ALLOWED_PREFIXES
set network routing-profile bgp filtering-profile FILTER_OUTBOUND ipv4 unicast inbound-network-filters prefix-list DENY_ALL


set network logical-router default vrf default bgp router-id 10.154.4.119
set network logical-router default vrf default bgp local-as 65001
set network logical-router default vrf default bgp install-route yes
set network logical-router default vrf default bgp enable yes

set network logical-router default vrf default bgp peer-group BGP_PEERS type ebgp
set network logical-router default vrf default bgp peer-group BGP_PEERS address-family ipv4 ipv4-unicast-default
set network logical-router default vrf default bgp peer-group BGP_PEERS filtering-profile ipv4 FILTER_INBOUND
set network logical-router default vrf default bgp peer-group BGP_PEERS filtering-profile ipv4 FILTER_OUTBOUND

set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE1 peer-as 65002
set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE1 local-address interface ethernet1/2
set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE1 local-address ip svc-intf-ip
set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE1 peer-address ip 10.154.4.160

set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE2 peer-as 65002
set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE2 local-address interface ethernet1/2
set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE2 local-address ip svc-intf-ip
set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE2 peer-address ip 10.154.4.33

 

Step 4: Verify BGP Configuration

After committing the configuration, verify the BGP sessions and routes:

Check BGP neighbor status:

run show advanced-routing bgp peer status logical-router default

Logical Router: default
==============
Peer Name:               CE2
BGP State:               Established, up for 00:01:55

Peer Name:               CE1
BGP State:               Established, up for 00:00:44

Verify ECMP routes:

run show advanced-routing route logical-router default

Logical Router: default
==========================
flags: A:active, E:ecmp, R:recursive, Oi:ospf intra-area, Oo:ospf inter-area, O1:ospf ext 1, O2:ospf ext 2

destination                             protocol       nexthop                                 distance  metric      flag      tag       age         interface                     
0.0.0.0/0                               static         10.154.1.1                              10        10          A                   01:47:33    ethernet1/1                   
10.154.1.0/24                           connected                                              0         0           A                   01:47:37    ethernet1/1                   
10.154.1.99/32                          local                                                  0         0           A                   01:47:37    ethernet1/1                   
10.154.4.0/24                           connected                                              0         0           A                   01:47:37    ethernet1/2                   
10.154.4.119/32                          local                                                  0         0           A                   01:47:37    ethernet1/2                   
192.168.100.10/32                       bgp            10.154.4.33                             20        255         A E                 00:01:03    ethernet1/2                   
192.168.100.10/32                       bgp            10.154.4.160                            20        255         A E                 00:01:03    ethernet1/2                   
total route shown: 7

Implementing CE Isolation for Maintenance

As discussed in Part One, one of the key advantages of BGP-based deployments is the ability to gracefully isolate CE nodes for maintenance. Here’s how to implement this in practice.

Isolation via F5 Distributed Cloud Console

To isolate a CE node from receiving traffic, in your BGP peer object, edit the Peer and:

  • Change the Outbound BGP routing policy from the one that is allowing the VIP prefixes to the one that is denying the VIP prefixes

The CE will stop advertising its VIP routes, and within seconds (based on BGP timers), the upstream firewall will remove this CE from its ECMP paths.

Verification During Maintenance

On your firewall, verify the route withdrawal (in this case we are using a Fortigate firewall):

get router info bgp summary

VRF 0 BGP router identifier 10.154.4.119, local AS number 65001
BGP table version is 4
1 BGP AS-PATH entries
0 BGP community entries

Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.154.4.33  4      65002    2070    2345        0    0    0 00:04:05        0
10.154.4.160 4      65002    2057    2326        0    0    0 00:12:46        1

Total number of neighbors 2

We are not receiving any prefixes anymore for the 10.154.4.33 peer.

get router info routing-table bgp 
Routing table for VRF=0
B       192.168.100.10/32 [20/255] via 10.154.4.160 (recursive is directly connected, port2), 00:06:34, [1/0]

End we have now only one path.

Restoring the CE in the data path

After maintenance is complete:

  1. Return to the BGP Peer configuration in the F5XC Console
  2. Restore the original export policy (permit VIP prefixes)
  3. Save the configuration
  4. On the upstream firewall, confirm that CE prefixes are received again and that ECMP paths are restored

 

Conclusion

This article has provided the complete implementation details for deploying BGP and ECMP with F5 Distributed Cloud Customer Edge nodes. You now have:

  • A clear understanding of the architecture at both high and low levels
  • Step-by-step instructions for configuring BGP in F5 Distributed Cloud Console
  • Ready-to-use configurations for both Fortinet FortiGate and Palo Alto Networks firewalls
  • Practical guidance for implementing graceful CE isolation for maintenance

By combining the concepts from the first article with the practical configurations in this article, you can build a robust, highly available application delivery infrastructure that maximizes resource utilization, provides automatic failover, and enables zero-downtime maintenance operations.

The BGP-based approach transforms your Customer Edge deployment from a traditional Active/Standby model into a full active topology where every node contributes to handling traffic, and any node can be gracefully removed for maintenance without impacting your users.

Published Dec 12, 2025
Version 1.0
automation
cloud
devops
Distributed Cloud
No CommentsBe the first to comment