F5 BIGIP & XC certbot plugin
Hi! As I maintain the certbot-f5bigip plugin to enable the certbot as ACME client to validate and install certificates, I now published the certbot-f5xc plugin! Currently only DNS validation with the DNS01 challenge is supported, so if you use F5 XC for DNS, you can use this plugin to verify domains for your certificates and use the certificate then for other infrastructure beside XC HTTP LBs (which can do auto-cert provisioning already). I will add an installer next, so certificates can be uploaded / installed to XC and re-used with HTTP LBs, so stay tuned! Here are the links to the repositories on gitlab.com: https://gitlab.com/emalzer/certbot-f5bigip https://gitlab.com/emalzer/certbot-f5xc https://gitlab.com/emalzer/certbot-ansibleFrom Chat to Config: Building an AI-Native MCP Server for F5 Distributed Cloud
The Problem: F5 Distributed Cloud is Powerful but Verbose Anyone who has worked with F5 Distributed Cloud (XC) knows the platform is incredibly capable. HTTP load balancers, WAF policies, API security, origin pools, namespaces, service policies—the feature set is deep. But with depth comes complexity. A single POST to create an HTTP load balancer with WAF, HTTPS auto-cert, and an origin pool involves carefully crafting nested JSON across three or four separate API calls, each with its own spec structure. For experienced engineers, this is manageable. But what if you could just say: "Create an HTTPS load balancer for test-namespace, attach a WAF policy in blocking mode, origin server at 10.10.10.10 port 80 with an HTTP health check, auto-cert on port 443 with HTTP redirect" …and have all of that happen automatically, correctly, with dry-run safety by default? That's exactly what I built. This article walks through the F5 XC MCP Server—an open-source Model Context Protocol server that translates natural language commands from Claude Code or GitHub Copilot directly into F5 XC API calls. What is MCP? Model Context Protocol (MCP) is an open standard introduced by Anthropic that lets AI assistants (like Claude) call external tools and services through a structured interface. Think of it as a plugin system for AI — instead of the AI just generating text, it can actually do things: query APIs, read files, run commands, interact with platforms. An MCP server exposes a set of tools—typed functions with names, descriptions, and input schemas. When you ask Claude Code something like "list all my namespaces in F5 XC," it finds the right tool (xc_list_namespaces), calls it with the right parameters, and shows you the result. No copy-pasting API tokens into curl commands. No hunting through docs for the right endpoint path. MCP clients could be the popular AI coding tool, or any custom build MCP client, such as: Claude Code (via VS Code extension—the one I primarily used) GitHub Copilot (via VS Code extension) Any MCP-compatible client MCP servers can run locally (via stdio) or remotely (via HTTP/HTTPS). Architecture The server is built in TypeScript using the @modelcontextprotocol/sdk, with axios for F5 XC API calls and zod for input validation. The structure is intentionally simple: Key design decisions: Dry-run by default. F5_XC_DRY_RUN=true is the default. Every mutating call returns a preview of what would be sent rather than actually calling the API. This makes it safe to explore and prototype without fear. Set F5_XC_DRY_RUN=false when you're ready to go live. Dual auth. Supports both API token (Authorization: APIToken …) and mTLS certificate auth (https.Agent with PEM cert + key). The certificate extracted from the F5 XC .p12 credential file works directly. Dual transport. stdio for local use with Claude Code/Copilot; streamable HTTP/HTTPS for team-shared remote deployment. Terraform as fallback. When the REST API doesn't support an operation (more on this below), tools automatically generate ready-to-apply Terraform HCL using the volterraedge/volterra provider. The Four Use Cases The server covers four areas matching common F5 XC workflows: UC Tools Example operations UC-1 Identity Namespace CRUD, API credentials Create namespace, list users/groups, audit credentials UC-2 Load Balancer Origin pool CRUD, HTTP LB CRUD Create HTTPS LB with auto-cert, add origin pool UC-3 Security App Firewall (WAF) CRUD, Service Policy CRUD Create WAF in blocking mode, attach to LB UC-4 API Security API definition CRUD, App API group CRUD, raw request Import OpenAPI spec, create API group In total the server exposes 39 tools—enough to cover the majority of day-to-day F5 XC operations from a conversation. A Live Demo Walkthrough Here's a real session—every one of these was a natural language instruction to Claude Code, which called the appropriate MCP tool automatically. Step 1: Create a namespace "Create a new namespace called test-namespace" Step 2: Create an HTTP load balancer — then upgrade it "Create an HTTP load balancer named test-http-lb in test-namespace" The tool calls xc_create_http_lb with a minimal spec. State returns as VIRTUAL_HOST_READY in seconds. "Change protocol from HTTP to HTTPS port 443, use automatic certificate management from XC, add origin server 10.10.10.10 port 80 with default HTTP health check" This triggers three tool calls automatically: xc_raw_request → creates an HTTP healthcheck object xc_create_origin_pool → creates origin pool with 10.10.10.10:80, references the healthcheck Delete old LB + xc_create_http_lb → recreates as HTTPS with https_auto_cert, HTTP→HTTPS redirect, pool attached Note: F5 XC does not allow changing the LB type (HTTP → HTTPS) via PUT. The MCP server detected this and handled the delete-recreate flow automatically. The result: Step 3: Create and attach a WAF policy "Create a WAF policy named test-waf-policy" xc_create_app_firewall — blocking mode, default OWASP detection, default bot settings. Done in one call. test-waf-policy created ✓ Field Value Name test-waf-policy Namespace test-namespace Mode Blocking (blocks detected attacks) Detection Default OWASP ruleset Bot protection Default bot settings UID 8d350b72-5c8c-4e7c-b827-248b03067ed3 The WAF policy is ready. Would you like to attach it to test-http-lb? "Enable this WAF policy on test-http-lb" xc_update_http_lb — removes disable_waf, adds app_firewall reference. Verified with a GET to confirm disable_waf is gone and app_firewall.name is set. WAF policy enabled on test-httlp-lb ✓ Field Value WAF policy test-waf-policy (blocking mode) disable_waf removed LB state VIRTUAL_HOST_PENDING_A_RECORD Total time from zero to a WAF-protected HTTPS load balancer: under 2 minutes, all from natural language. The API Limitation Discovery — and the Terraform Fallback One of the most interesting findings during development: F5 XC's public REST API does not expose user/group write operations. Every path I tried returned either 404 or 501 Not Implemented: This is intentional by design—F5 XC routes user management through its Console UI. The Terraform volterraedge/volterra provider also didn’t help for users, group management. Rather than leaving the user with a dead end, I built a Terraform fallback: when a user group write fails, the tool’s response automatically includes: The AI can then call xc_tf_apply directly to execute it—or the user can copy the HCL and apply it themselves. The Terraform runner operates in isolated temp directories, cleans up after itself, and respects the global dryRun flag (plan instead of apply when dry-run is active). This pattern—REST first, Terraform as fallback—turned out to be a very useful architectural choice. It gracefully handles the gap between what the API exposes and what the platform can actually do. Deploying to Production: HTTPS with Automatic Certificates For a shared team tool, local stdio mode isn't enough. The server needs to be always-on, accessible over HTTPS, and with a real TLS certificate. The deployment stack on an Azure Ubuntu VM: Node.js 20 (via nvm) running the MCP server on port 3000 as a systemd service Caddy as a TLS-terminating reverse proxy—one config file, automatic Let's Encrypt The entire Caddy config: Caddy handles the ACME HTTP-01 challenge automatically. The Let's Encrypt certificate was issued in under 10 seconds after DNS propagated. Auto-renewal is built in—no cron jobs, no certbot timers. One gotcha worth noting: the default Caddy proxy timeout (30s) is shorter than some F5 XC API calls (namespace creation can take ~45s). The response_header_timeout 90s setting above is necessary. With this setup, the MCP endpoint is https://your-domain/mcp — usable from any MCP client without VPN or local server setup. Connecting Claude Code to the Remote Server Add this to your Claude Code MCP configuration (~/.claude.json or .claude/settings.json in your project): That's it. After a /mcp reload in Claude Code, all 39 tools are available. You can verify with: "Show me the F5 XC server status" Which calls xc_server_status and returns tenant, auth method, dry-run state, and Terraform auth status. Lessons Learned The F5 XC REST API is comprehensive for data plane operations, limited for identity management. Load balancers, WAF policies, origin pools, API definitions — all fully CRUD-able via REST. User and group management is not. Plan accordingly if your use case involves IAM automation. Dry-run mode is not optional — it's essential. Without it, a misunderstood instruction could delete a production load balancer. Making dry-run the default (and requiring explicit override per-call or globally) is the right design for any AI-driven ops tool. Tool descriptions matter more than you think. The quality of an MCP tool's description directly affects how accurately the AI uses it. Spending time writing precise, example-rich descriptions — including what fields are required, what values are valid, and what the return looks like — significantly improves the AI's ability to compose multi-step operations correctly. Graceful degradation beats hard failures. The Terraform fallback pattern is a good example. Rather than returning a cryptic API error and stopping, surfacing the equivalent HCL and offering to apply it keeps the workflow moving. Users get an answer even when the API says no. LB type changes require delete+recreate. The F5 XC API rejects PUT requests that change the load balancer type (e.g., HTTP → HTTPS). The MCP server handles this automatically by detecting the error and orchestrating the delete-recreate sequence — a good example of where the AI layer can absorb platform-specific quirks. What's Next This is v1.0 — functional, deployed, and covering the core use cases. Areas I'm exploring for future versions: API security scanning integration: trigger XC's web application scanning from the MCP server and return findings Multi-tenant support: switch tenants within a session without restarting the server Policy-as-code export: serialize existing LBs and WAF configs to Terraform HCL for IaC migration Audit/diff mode: compare current live config against a desired state and report drift Try It Yourself The server is open source on GitHub: https://github.com/gavinw2006/F5_XC_MCP_Server Prerequisites: Node.js 18+, an F5 XC tenant with an API token, and Claude Code or any MCP-compatible client. The first thing to try once connected: "Show me the F5 XC server status, then list all namespaces" Happy to hear feedback, questions, and PRs from the DevCentral community. If you build something on top of this—a new tool module, a different transport, integration with another F5 product—I’d love to know about it.Bulk-Create Secondary DNS Zones in F5 Distributed Cloud (via API)
If you’ve ever had to onboard dozens (or hundreds) of domains as secondary DNS zones in F5 Distributed Cloud (XC), you know the drill: click through the console, fill in the domain, add the primary server IP, save, repeat. It works - but it doesn’t scale. I ran into exactly this situation recently and figured a quick shell script would save me (and maybe you) a lot of clicking. What it does The script reads a simple text file - one domain per line, comma-separated with its primary DNS server IPs - and creates the corresponding secondary DNS zones via the F5 XC API. The input file looks like this: example.com, 10.0.0.1, 10.0.0.2 example.org, 192.168.1.1 internal.example.net, 172.16.0.1, 172.16.0.2, 172.16.0.3 Run the script, and it works through the list one by one: $ ./create_secondary_dns.sh Starting secondary DNS zone creation... Tenant : acmecorp --- [line 1] Creating secondary zone: example.com (primaries: 10.0.0.1 10.0.0.2) [line 1] OK - example.com created (HTTP 200) [line 2] Creating secondary zone: example.org (primaries: 192.168.1.1) [line 2] OK - example.org created (HTTP 200) --- Done. Success: 2, Failed: 0, Total lines: 2 That’s it. No Python, no Terraform, no extra frameworks - just bash, curl, and jq. TSIG support If your primary DNS servers require TSIG authentication for zone transfers, the script handles that too. Set the TSIG key name, algorithm, and secret in the configuration block, and the script will encrypt the secret locally using vesctl (the F5 XC CLI) before sending it to the API. If vesctl isn’t installed yet, the script downloads it automatically. If you don’t need TSIG, just leave the TSIG variables empty and the script skips that part entirely. Getting started 1. Clone the repo and navigate to the tool: git clone https://github.com/de1chk1nd/resources-and-tools.git cd resources-and-tools/tools/dns-secondary 2. Create your domain list: cp domains.txt.example domains.txt 3. Open the script and fill in your tenant name and API token: vi create_secondary_dns.sh 4. Run it: chmod +x create_secondary_dns.sh ./create_secondary_dns.sh The full README in the repository covers all configuration options, dependencies, and common error messages. What happens when something goes wrong? The script validates every line before hitting the API - domain names are checked for valid DNS characters, IP addresses are checked for valid IPv4 format. If a line is malformed, it’s skipped with a clear message and the script moves on to the next one. If a zone already exists in XC, the API returns a 409 and the script logs it as a failure but keeps going. At the end, you get a summary: how many succeeded, how many failed. A note on the repository This is a personal project, not an official F5 tool. It’s not supported by F5 and comes with no warranty. That said, it works well for my use cases and I hope it’s useful for yours too. If you run into issues or have suggestions, please open an issue on GitHub: GIT Repo The script lives here: DNS ToolLeveraging BGP and ECMP for F5 Distributed Cloud Customer Edge, Part Two
Introduction This is the second part of our series on leveraging BGP and ECMP for F5 Distributed Cloud Customer Edge deployments. In Part One, we explored the high-level concepts, architecture decisions, and design principles that make BGP and ECMP such a powerful combination for Customer Edge high availability and maintenance operations. This article provides step-by-step implementation guidance, including: High-level and low-level architecture diagrams Complete BGP peering and routing policy configuration in F5 Distributed Cloud Console Practical configuration examples for Fortinet FortiGate and Palo Alto Networks firewalls By the end of this article, you'll have everything you need to implement BGP-based high availability for your Customer Edge deployment. Architecture Overview Before diving into configuration, let’s establish a clear picture of the architecture we’re implementing. We’ll examine this from two perspectives: a high-level logical view and a detailed low-level view showing specific IP addressing and AS numbers. High-Level Architecture The high-level architecture illustrates the fundamental traffic flow and BGP relationships in our deployment: Key Components: Component Role Internet External connectivity to the network Next-Generation Firewall Acts as the BGP peer and performs ECMP distribution to Customer Edge nodes Customer Edge Virtual Site Two or more CE nodes advertising identical VIP prefixes via BGP The architecture follows a straightforward principle: the upstream firewall establishes BGP peering with each CE node. Each CE advertises its VIP addresses as /32 routes. The firewall, seeing multiple equal-cost paths to the same destination, distributes incoming traffic across all available CE nodes using ECMP. Low-Level Architecture with IP Addressing The low-level diagram provides the specific details needed for implementation, including IP addresses and AS numbers: Network Details: Component IP Address Role Firewall (Inside) 10.154.4.119/24 BGP Peer, ECMP Router CE1 (Outside) 10.154.4.160/24 Customer Edge Node 1 CE2 (Outside) 10.154.4.33/24 Customer Edge Node 2 Global VIP 192.168.100.10/32 Load Balancer VIP BGP Configuration: Parameter Firewall Customer Edge AS Number 65001 65002 Router ID 10.154.4.119 Auto-assigned based on interface IP Advertised Prefix None 192.168.100.0/24 le 32 This configuration uses eBGP (External BGP) between the firewall and CE nodes, with different AS numbers for each. The CE nodes share the same AS number (65002), which is the standard approach for multi-node CE deployments advertising the same VIP prefixes. Configuring BGP in F5 Distributed Cloud Console The F5 Distributed Cloud Console provides a centralized interface for configuring BGP peering and routing policies on your Customer Edge nodes. This section walks you through the complete configuration process. Step 1: Configure the BGP peering Go to: Multi-Cloud Network Connect --> Manage --> Networking --> External Connectivity --> BGP Peers & Policies Click on Add BGP Peer Then add the following information: Object name Site where to apply this BGP configuration ASN Router ID Here is an example of the required parameters. Then click on Peers --> Add Item And filled the relevant fields like below by adapting the parameters for your requirements. Step 2: Configure the BGP routing policies Go to: Multi-Cloud Network Connect --> Manage --> Networking --> External Connectivity --> BGP Peers & Policies --> BGP Routing Policies Click on Add BGP Routing Policy Add a name for your BGP routing policy object and click on Configure to add the rules. Click on Add Item to add a rule. Here we are going to allow the /32 prefixes from our VIP subnet (192.168.100.0/24). Save the BGP Routing Policy Repeat the action to create another BGP routing policy with the exact same parameters except the Action Type, which should be of type Deny. Now we have two BGP routing policies: One to allow the VIP prefixes (for normal operations) One to deny the VIP prefixes (for maintenance mode) We still need to a a third and final BGP routing policy, in order to deny any prefixes on the CE. For that, create a third BGP routing policy with this match. Step 3: Apply the BGP routing policies To apply the BGP routing policies in your BGP peer object, edit the Peer and: Enable the BGP routing policy Apply the BGP routing policy objects created before for Inbound and Outbound Fortinet FortiGate Configuration FortiGate firewalls are widely deployed as network security appliances and support robust BGP capabilities. This section provides the minimum configuration for establishing BGP peering with Customer Edge nodes and enabling ECMP load distribution. Step 1: Configure the Router ID and AS Number Configure the basic BGP settings: config router bgp set as 65001 set router-id 10.154.4.119 set ebgp-multipath enable Step 2: Configure BGP Neighbors Add each CE node as a BGP neighbor: config neighbor edit "10.154.4.160" set remote-as 65002 set route-map-in "ACCEPT-CE-VIPS" set route-map-out "DENY-ALL" set soft-reconfiguration enable next edit "10.154.4.33" set remote-as 65002 set route-map-in "ACCEPT-CE-VIPS" set route-map-out "DENY-ALL" set soft-reconfiguration enable next end end Step 3: Create Prefix List for VIP Range Define the prefix list that matches the CE VIP range: config router prefix-list edit "CE-VIP-PREFIXES" config rule edit 1 set prefix 192.168.100.0 255.255.255.0 set ge 32 set le 32 next end next end Important: The ge 32 and le 32 parameters ensure we only match /32 prefixes within the 192.168.100.0/24 range, which is exactly what CE nodes advertise for their VIPs. Step 4: Create Route Maps Configure route maps to implement the filtering policies: Inbound Route Map (Accept VIP prefixes): config router route-map edit "ACCEPT-CE-VIPS" config rule edit 1 set match-ip-address "CE-VIP-PREFIXES" next end next end Outbound Route Map (Deny all advertisements): config router route-map edit "DENY-ALL" config rule edit 1 set action deny next end next end Step 5: Verify BGP Configuration After applying the configuration, verify the BGP sessions and routes: Check BGP neighbor status: get router info bgp summary VRF 0 BGP router identifier 10.154.4.119, local AS number 65001 BGP table version is 4 1 BGP AS-PATH entries 0 BGP community entries Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.154.4.33 4 65002 2092 2365 0 0 0 00:05:33 1 10.154.4.160 4 65002 2074 2346 0 0 0 00:14:14 1 Total number of neighbors 2 Verify ECMP routes: get router info routing-table bgp Routing table for VRF=0 B 192.168.100.10/32 [20/255] via 10.154.4.160 (recursive is directly connected, port2), 00:00:11, [1/0] [20/255] via 10.154.4.33 (recursive is directly connected, port2), 00:00:11, [1/0] Palo Alto Networks Configuration Palo Alto Networks firewalls provide enterprise-grade security with comprehensive routing capabilities. This section covers the minimum BGP configuration for peering with Customer Edge nodes. Note: This part is assuming that Palo Alto firewall is configured in the new "Advanced Routing Engine" mode. And we will use the logical-router named "default". Step 1: Configure ECMP parameters set network logical-router default vrf default ecmp enable yes set network logical-router default vrf default ecmp max-path 4 set network logical-router default vrf default ecmp algorithm ip-hash Step 2: Configure objects IPs and firewall rules for BGP peering set address CE1 ip-netmask 10.154.4.160/32 set address CE2 ip-netmask 10.154.4.33/32 set address-group BGP_PEERS static [ CE1 CE2 ] set address LOCAL_BGP_IP ip-netmask 10.154.4.119/32 set rulebase security rules ALLOW_BGP from service set rulebase security rules ALLOW_BGP to service set rulebase security rules ALLOW_BGP source LOCAL_BGP_IP set rulebase security rules ALLOW_BGP destination BGP_PEERS set rulebase security rules ALLOW_BGP application bgp set rulebase security rules ALLOW_BGP service application-default set rulebase security rules ALLOW_BGP action allow Step 3: Palo Alto Configuration Summary (CLI Format) set network routing-profile filters prefix-list ALLOWED_PREFIXES type ipv4 ipv4-entry 1 prefix entry network 192.168.100.0/24 set network routing-profile filters prefix-list ALLOWED_PREFIXES type ipv4 ipv4-entry 1 prefix entry greater-than-or-equal 32 set network routing-profile filters prefix-list ALLOWED_PREFIXES type ipv4 ipv4-entry 1 prefix entry less-than-or-equal 32 set network routing-profile filters prefix-list ALLOWED_PREFIXES type ipv4 ipv4-entry 1 action permit set network routing-profile filters prefix-list ALLOWED_PREFIXES description "Allow only m32 inside 192.168.100.0m24" set network routing-profile filters prefix-list DENY_ALL type ipv4 ipv4-entry 1 prefix entry network 0.0.0.0/0 set network routing-profile filters prefix-list DENY_ALL type ipv4 ipv4-entry 1 prefix entry greater-than-or-equal 0 set network routing-profile filters prefix-list DENY_ALL type ipv4 ipv4-entry 1 prefix entry less-than-or-equal 32 set network routing-profile filters prefix-list DENY_ALL type ipv4 ipv4-entry 1 action deny set network routing-profile filters prefix-list DENY_ALL description "Deny all prefixes" set network routing-profile bgp filtering-profile FILTER_INBOUND ipv4 unicast inbound-network-filters prefix-list ALLOWED_PREFIXES set network routing-profile bgp filtering-profile FILTER_OUTBOUND ipv4 unicast inbound-network-filters prefix-list DENY_ALL set network logical-router default vrf default bgp router-id 10.154.4.119 set network logical-router default vrf default bgp local-as 65001 set network logical-router default vrf default bgp install-route yes set network logical-router default vrf default bgp enable yes set network logical-router default vrf default bgp peer-group BGP_PEERS type ebgp set network logical-router default vrf default bgp peer-group BGP_PEERS address-family ipv4 ipv4-unicast-default set network logical-router default vrf default bgp peer-group BGP_PEERS filtering-profile ipv4 FILTER_INBOUND set network logical-router default vrf default bgp peer-group BGP_PEERS filtering-profile ipv4 FILTER_OUTBOUND set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE1 peer-as 65002 set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE1 local-address interface ethernet1/2 set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE1 local-address ip svc-intf-ip set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE1 peer-address ip 10.154.4.160 set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE2 peer-as 65002 set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE2 local-address interface ethernet1/2 set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE2 local-address ip svc-intf-ip set network logical-router default vrf default bgp peer-group BGP_PEERS peer CE2 peer-address ip 10.154.4.33 Step 4: Verify BGP Configuration After committing the configuration, verify the BGP sessions and routes: Check BGP neighbor status: run show advanced-routing bgp peer status logical-router default Logical Router: default ============== Peer Name: CE2 BGP State: Established, up for 00:01:55 Peer Name: CE1 BGP State: Established, up for 00:00:44 Verify ECMP routes: run show advanced-routing route logical-router default Logical Router: default ========================== flags: A:active, E:ecmp, R:recursive, Oi:ospf intra-area, Oo:ospf inter-area, O1:ospf ext 1, O2:ospf ext 2 destination protocol nexthop distance metric flag tag age interface 0.0.0.0/0 static 10.154.1.1 10 10 A 01:47:33 ethernet1/1 10.154.1.0/24 connected 0 0 A 01:47:37 ethernet1/1 10.154.1.99/32 local 0 0 A 01:47:37 ethernet1/1 10.154.4.0/24 connected 0 0 A 01:47:37 ethernet1/2 10.154.4.119/32 local 0 0 A 01:47:37 ethernet1/2 192.168.100.10/32 bgp 10.154.4.33 20 255 A E 00:01:03 ethernet1/2 192.168.100.10/32 bgp 10.154.4.160 20 255 A E 00:01:03 ethernet1/2 total route shown: 7 Implementing CE Isolation for Maintenance As discussed in Part One, one of the key advantages of BGP-based deployments is the ability to gracefully isolate CE nodes for maintenance. Here’s how to implement this in practice. Isolation via F5 Distributed Cloud Console To isolate a CE node from receiving traffic, in your BGP peer object, edit the Peer and: Change the Outbound BGP routing policy from the one that is allowing the VIP prefixes to the one that is denying the VIP prefixes The CE will stop advertising its VIP routes, and within seconds (based on BGP timers), the upstream firewall will remove this CE from its ECMP paths. Verification During Maintenance On your firewall, verify the route withdrawal (in this case we are using a Fortigate firewall): get router info bgp summary VRF 0 BGP router identifier 10.154.4.119, local AS number 65001 BGP table version is 4 1 BGP AS-PATH entries 0 BGP community entries Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.154.4.33 4 65002 2070 2345 0 0 0 00:04:05 0 10.154.4.160 4 65002 2057 2326 0 0 0 00:12:46 1 Total number of neighbors 2 We are not receiving any prefixes anymore for the 10.154.4.33 peer. get router info routing-table bgp Routing table for VRF=0 B 192.168.100.10/32 [20/255] via 10.154.4.160 (recursive is directly connected, port2), 00:06:34, [1/0] End we have now only one path. Restoring the CE in the data path After maintenance is complete: Return to the BGP Peer configuration in the F5XC Console Restore the original export policy (permit VIP prefixes) Save the configuration On the upstream firewall, confirm that CE prefixes are received again and that ECMP paths are restored Conclusion This article has provided the complete implementation details for deploying BGP and ECMP with F5 Distributed Cloud Customer Edge nodes. You now have: A clear understanding of the architecture at both high and low levels Step-by-step instructions for configuring BGP in F5 Distributed Cloud Console Ready-to-use configurations for both Fortinet FortiGate and Palo Alto Networks firewalls Practical guidance for implementing graceful CE isolation for maintenance By combining the concepts from the first article with the practical configurations in this article, you can build a robust, highly available application delivery infrastructure that maximizes resource utilization, provides automatic failover, and enables zero-downtime maintenance operations. The BGP-based approach transforms your Customer Edge deployment from a traditional Active/Standby model into a full active topology where every node contributes to handling traffic, and any node can be gracefully removed for maintenance without impacting your users.Moving HTTP Load Balancers Between F5 Distributed Cloud Namespaces — Why It's Harder Than You Think
The Problem If you have been working with F5 Distributed Cloud (XC) for a while, you have probably run into this: your namespace structure no longer reflects how your teams or applications are organized. Maybe the initial layout was a quick decision during onboarding. Maybe teams have merged, projects have grown, or your naming convention has evolved. Either way, you now want to move a handful of HTTP load balancers from one namespace to another. Simple enough, right? Just change the namespace field and save... Except you can't. There is no "move" operation on F5 XC - not in the UI, not in the API. Changing the namespace of a load balancer means deleting it in the source and re-creating it in the target. And that is where things get complicated. Why a Simple Delete-and-Recreate Is Not Enough On the surface, the API is straightforward: "GET" the config, "DELETE" the object, "POST" it into the new namespace. But a production HTTP load balancer on XC is rarely a standalone object. It sits at the top of a dependency tree that can include origin pools, health checks, TLS certificates, service policies, app firewalls, rate limiters, and more. Every one of those dependencies needs to be handled correctly - or the migration breaks. Here are the main challenges we might run into. Referential Integrity F5 XC enforces strict referential integrity. You cannot delete an origin pool that is still referenced by a load balancer. You cannot create a load balancer that references an origin pool that does not exist yet. This means the order of operations matters: delete top-down (LBs first, then dependencies), create bottom-up (dependencies first, then LBs). It also means that if two load balancers share an origin pool, you cannot move them independently. Delete the first LB, try to delete the shared pool, and the API returns a 409 Conflict because the second LB still references it. Both LBs - and all of their shared dependencies - have to be moved together as a single atomic unit. New CNAMEs After Every Move When you delete and re-create an HTTP load balancer, F5 XC assigns a new "host_name" (the CNAME target that your DNS records point to). If the LB uses Let's Encrypt auto-certificates, the ACME challenge CNAME changes too. That means after every move, someone needs to update external DNS records - and until that happens, the application is unreachable or the TLS certificate renewal fails. For tenants using XC-managed DNS zones with "Allow Application Loadbalancer Managed Records" enabled, this is handled automatically. But many customers manage their own DNS, and they need the old and new CNAME values for every moved LB. Certificates with Non-Portable Private Keys This one is subtle. When a load balancer uses a manually imported TLS certificate, the private key is stored in one of several formats: blindfolded (encrypted with the Volterra blindfold key) or clear secret. In both of these cases, the XC API never returns the private key material in its GET response. You get the certificate and metadata, but not the key. That means you cannot extract-and-recreate the certificate in a new namespace via the API. Cross-namespace certificate references (outside of "shared" namespace) are also not supported. So if an LB in namespace A uses a manually imported certificate stored in namespace A, and you want to move that LB to namespace B, you need to first manually upload the same certificate into namespace B (or into the "shared" namespace) before the migration can proceed. API Metadata The XC API returns a "referring_objects" field on every config GET response. In theory, this tells you what other objects reference a given resource - exactly what you need to know before deleting something. In practice, this field can be empty even when active references exist. The only reliable way to detect all external references is to actively scan: fetch the config of every load balancer in the namespace and check their specs for references to the objects you are about to move. Cross-Namespace References Are Not Allowed On F5 XC, an HTTP load balancer can only reference objects in its own namespace, in "system" or "shared" namespace. If your origin pool lives in namespace A and you move the LB to namespace B, the origin pool must either come along to namespace B or already exist there. There is no way to have the LB in namespace B point to a pool in namespace A. This means you need to discover the complete transitive dependency tree of every LB, determine which dependencies need to move, detect which are shared between multiple LBs, and batch everything accordingly. The Tool: XC Namespace migration To deal with all of this, (A)I built **xc-ns-mover** — a Python CLI tool that automates the entire process. It has two components: Scanner - scans all namespaces on your tenant, lists every HTTP load balancer, and writes a CSV report. This gives you the inventory to decide what to move. Mover - takes a CSV of load balancers, discovers all dependencies, groups LBs that share dependencies into atomic batches, runs a series of pre-flight checks, and then executes the migration - or generates a dry-run report so you can review everything first, or do the job manually (JSON Code blocks available in the report) What the Mover Does Before Touching Anything The mover runs six pre-flight phases before making any changes: Discovery and batching - fetches every LB config, walks the dependency tree, and uses a union-find algorithm to cluster LBs with shared dependencies into batches. External reference scan - for every dependency being moved, checks whether any LB outside the move list references it. If so, that dependency cannot be moved without breaking the external LB, and the batch is blocked. Conflict detection - lists all existing objects in the target namespace. If a name already exists, the user can skip the object or rename it with a configurable prefix (e.g., "migrated-my-pool"). All internal JSON references are updated automatically. Certificate pre-flight - identifies certificates with non-portable private keys, then searches the target and "shared" namespaces for a matching certificate by domain/SAN comparison (including wildcard matching per RFC 6125). If a match is found, the LB's certificate reference is automatically rewritten. If not, the batch is blocked until the certificate is manually created. DNS zone pre-flight - queries the tenant's DNS zones to detect which ones have managed LB records enabled. LBs under managed zones are flagged as "auto-managed" in the report — no manual DNS update needed. After all checks pass, the actual migration follows a strict order per batch: backup everything, delete top-down, create bottom-up, verify new CNAMEs. If anything fails, automatic rollback kicks in — objects created in the target are deleted, objects deleted from the source are restored from backups. The Reports Every run produces an HTML report. The dry-run report shows planned configurations, the full dependency graph , certificate issues, DNS changes required, and any blocking issues — all before a single API call mutates anything. The post-migration report includes old and new CNAME values, a DNS changes table with exactly which records need updating, and full configuration backups of everything that was touched. Things to Keep in Mind A few caveats that are worth highlighting: Brief interruption is unavoidable - The migration deletes and re-creates load balancers. During that window (typically seconds to a few minutes per batch), traffic to affected domains will be impacted. Plan a change window. Only HTTP load balancers are supported - TCP load balancers and other object types are not handled by this tool. DNS updates are your responsibility - The report gives you all the values - old CNAME, new CNAME, ACME challenge CNAME - but you need to update your DNS provider. Always run the dry-run first - The tool enforces this by default: it stores a fingerprint after a dry-run and verifies it before executing. If the config changes, a new dry-run is required. The project is open source and available on GitHub. This is privately maintained and not "officially supported": https://github.com/de1chk1nd/resources-and-tools/blob/main/tools/xc-ns-mover/README.md If you find bugs or have feature requests, please open a GitHub issue.Automating Certificate Management on F5 BIG-IP
The Certificate Lifespan Revolution Welcome to part one of our two-part series on certificate automation for the BIG-IP platform. Certificate lifecycle management is undergoing a seismic shift that will fundamentally change how organizations handle SSL/TLS certificates. The era of comfortable 12-13 month certificate lifespans is rapidly ending. Major Certificate Authorities and browser vendors are pushing aggressively toward 90-day or even 47-day certificates, and some are proposing even shorter durations. This transformation represents more than a simple policy change. It's a complete paradigm shift that renders traditional certificate management approaches obsolete. The familiar rhythm of annual renewals, managed through spreadsheets and calendar reminders, becomes not just inefficient but operationally impossible when certificates expire every three months or 47 days. The Challenge: Why Manual Processes Are Doomed Organizations worldwide are recognizing the writing on the wall. Large PKI environments that once managed hundreds or thousands of certificates annually now face the prospect of managing them on a quarterly basis. The mathematical reality is stark: a 75% reduction in certificate lifespan translates to a 400% increase in the frequency of renewals. At F5, we hear you, we understand the anxiety this creates. We hear from customers daily who are grappling with this operational challenge. The F5 Solution: ACME Protocol Implementation While third-party vendors like Venafi and DigiCert offer comprehensive certificate automation platforms, this guide focuses on F5 solutions that leverage your existing BIG-IP infrastructure. Our approach is based on the ACME (Automatic Certificate Management Environment) protocol. This is the same technology that powers Let’s Encrypt and other modern certificate authorities. The solution uses a specialized ACME implementation called "dehydrated," adapted for BIG-IP by F5's own Kevin Stewart, with Python code inspired by Jason Rahm. You can read Kevin's DevCentral article here. Getting Started: Prerequisites and Setup Before diving into the implementation, ensure you have: Active BIG-IP with the LTM module and basic networking configured Registered domain with DNS management access for A record modifications (if setting up a new domain for testing purposes) Internet connectivity for downloading the ACME script and communicating with certificate authorities This solution is straightforward to configure and deploy. Follow along below. Step 1: Download and Explore Connect to your BIG-IP shell via SSH and download the script: curl -s https://raw.githubusercontent.com/f5devcentral/kojot-acme/main/install.sh | bash Navigate to the installation directory and explore the config file. This is where you can customize key size, contact information, OCSP stapling and more. cd /shared/acme The script adds a few new shiny toys to your BIG-IP. You will find two new data groups and a new iRule. The data group dg_acme_config houses the subject and CA or your certificate. We will need to modify this data group before we run the script; more on this in a moment. The dg_acme_challenge holds the challenge tokens and is ephemeral, cleaning up when the process completes. The iRule handles the HTTP-01 challenge. You need this iRule applied to the port 80 VIP when you run the script. At a high level, the iRule intercepts the challenge request from the ACME server and responds with an appropriate challenge response. In scenarios without a proxy, such as the BIG-IP, the web server handles this. Alternatively, you can use a DNS challenge by adding a TXT record at your registrar. The HTTP method is much simpler, so that’s what we will use here. Step 2: Configure Your Virtual IP Create an HTTP Virtual IP (VIP) to handle the ACME challenges. Production environments usually need both HTTP (port 80) and HTTPS (port 443) VIPs. Our demonstration uses a simple HTTP-only setup. Critical requirement: The ACME iRule must be applied to your port 80 VIP to enable HTTP-01 challenge processing. Step 3: Configure the Data Group - dg_acme_config We need to add our domain and CA information at a minimum. Add your domain to the string field and your CA information to the value field. See the documentation for more examples. Step 4: Configure DNS If you already have a domain configured pointing to your HTTP VIP, then skip this step. However, if you are setting up a new domain, please make sure that DNS is pointing to your VIP IP address or A record. For this demo, I purchased a low-cost domain and configured the A record to point to my AWS Elastic IP, which is attached to my private IP in my VPC. Step 5: Run the script Ok, you have DNS configured, VIP configured, and the data group modified with your domain and CA information. You are ready to roll. Go to the directory and run the script. cd /shared/acme ./f5acmehandler.sh -verbose The verbose flag provides real-time feedback on the ACME communication process, invaluable for troubleshooting and understanding the workflow. Success? If you are successful, you should see output similar to the screenshot below. If not, then the details of the script workflow should clue you into the problem. The GitHub repo also offers some troubleshooting advice. Also, let's see if we got a brand new shiny certificate. Yep. What's Next? While this created a cert and key, it didn't place them into an SSL profile. This is possible by modifying the config file. Scheduling checks could also be on your list. Kevin's article highlights this and provides an example. This will run the script every Monday at 4 am. ./f5acmehandler.sh --schedule "00 04 * * 1" There are other advanced capabilities that I did not configure or explain in this article. I encourage you to visit the DevCentral repo for more information. A key consideration for Federal agencies is the ability to support air-gapped or closed networks. However, requirements and compliance hurdles will need to be examined further. In the following article, we will cover the Certificate Order capability built into the BIG-IP. Until next time.F5 Per-App AS3 Part 2 How to see if there are manual changes!
Code version: The code was tested on 17.1.5.3 AS3: 3.55 For more about AS3 and per-app AS3 see my previous code share Part 1 article: https://community.f5.com/kb/codeshare/f5-per-app-as3-part-1-how-share-tenant-specific-object/345072 First we will send Per-App AS3 declaration as shown below. { "id": "per-app-declarationn", "schemaVersion": "3.55.0", "controls": { "class": "Controls", "logLevel": "debug", "trace": true, "traceResponse": true }, "A2": { "class": "Application", "service": { "class": "Service_HTTP", "virtualAddresses": [ "10.0.3.10" ], "pool": "web2_pool" }, "web2_pool": { "class": "Pool", "monitors": [ "http" ], "members": [{ "servicePort": 80, "serverAddresses": [ "192.7.21.10", "192.7.21.11" ] }] } } } Then we will change for example virtual server ip from 10.0.3.10 to 10.0.3.11 and we will send the same declaration but with "dryRun" set to true as this will cause AS3 to validate the config but not to execute it and with trace and traceResponse we will get the difference 😎 { "id": "per-app-declarationn", "schemaVersion": "3.55.0", "controls": { "class": "Controls", "logLevel": "debug", "trace": true, "dryRun": true, "traceResponse": true }, "A2": { "class": "Application", "service": { "class": "Service_HTTP", "virtualAddresses": [ "10.0.3.10" ], "pool": "web2_pool" }, "web2_pool": { "class": "Pool", "monitors": [ "http" ], "members": [{ "servicePort": 80, "serverAddresses": [ "192.7.21.10", "192.7.21.11" ] }] } } } Now we see that the IP has been changed from 10.0.3.10 to 10.0.3.11 and here we go now we have the difference ! This can be added in CI/CD to always first do "dry-run" using the original declaration to see if there are changes before executing the new AS3 declaration that could be for example changing the IP address to 10.0.3.12 but using the official way. Look at the Json reply "diff" section that is seen thanks to trace and traceResponse options and an automation can just check this section and stop the new deployment if the manual changes need to be reviewed first. For AS3 basic declaration not Per-App actually the "dry-run" is a different. F5 likes changing the naming like Local Traffic policies to Endpoint Policies or naming of TLS profiles between GU/TMSH and AS3 😅 { "$schema": "https://raw.githubusercontent.com/F5Networks/f5-appsvcs-extension/refs/heads/main/schema/3.55.0/as3-schema.json", "class": "AS3", "action": "dry-run", "logLevel": "debug", "trace": true, "traceResponse": true, "persist": true, "declaration": { "class": "ADC", "schemaVersion": "3.55.0", "id": "BIG-IP-Example-Tenant", "Example": { "class": "Tenant", "Shared": { "class": "Application", "template": "shared", "Example_Response": { "remark": "Used for F5 response", "class": "iRule", "iRule": { "base64": "d2hlbiBIVFRQX1JFUVVFU1Qgew0KICAgSFRUUDo6cmVzcG9uZCAyMDAgY29udGVudCB7DQogICAgICA8aHRtbD4NCiAgICAgICAgIDxoZWFkPg0KICAgICAgICAgICAgPHRpdGxlPkFwb2xvZ3kgUGFnZTwvdGl0bGU+DQogICAgICAgICA8L2hlYWQ+DQogICAgICAgICA8Ym9keT4NCiAgICAgICAgICAgIFdlIGFyZSBzb3JyeSwgYnV0IHRoZSBzaXRlIHlvdSBhcmUgbG9va2luZyBmb3IgaXMgdGVtcG9yYXJpbHkgb3V0IG9mIHNlcnZpY2U8YnI+DQogICAgICAgICAgICBJZiB5b3UgZmVlbCB5b3UgaGF2ZSByZWFjaGVkIHRoaXMgcGFnZSBpbiBlcnJvciwgcGxlYXNlIHRyeSBhZ2Fpbi4NCiAgICAgICAgIDwvYm9keT4NCiAgICAgIDwvaHRtbD4NCiAgIH0NCn0=" } } } } } } This will not show if someone has manually added a vlan for example as only changes on the apps that were deployed with AS3 will be seen. For those you will get error like the one below when you try to delete the partition. "" 0107082a:3: All objects must be removed from a partition "" https://my.f5.com/manage/s/article/K02718312 https://my.f5.com/manage/s/article/K000138638 Github link: https://github.com/Nikoolayy1/AS3-Per-App-Manual-Changes/tree/mainF5 Per-App AS3 Part 1 How Share Tenant specific object
Code version: The code was tested on 17.1.5.3 AS3: 3.55 For testing F5 AS3 you can use combination of Postman as Visual Studio has option to post AS3 declarations but not Per-App specific as that will be needed for executing my code. Still I recommend using Visual Studio for general AS3 or Fast Templates and you can use the reference links for how to use Visual Studio F5 extensions and maybe F5 will update the extension in the future to work for Per-App AS3 with options for adding tenant and as variables 🤔 For FAST nice example is at https://github.com/Nikoolayy1/fast-template-examples The same way the /Shared objects can be configured under /Common can be done under a specific tenant and this is really useful if Per-App AS3 is enabled as each AS3 app will use separate declaration and all of them can use a customer iRule defined under /<Tenant>/Shared. References for AS3 Declarative API: https://www.youtube.com/watch?v=rrjrcnwAe1g&t=433s AS3 Foundations: Beyond Imperatives - What the Heck is AS3? | DevCentral https://www.youtube.com/watch?v=OHSBpOtQg_0&list=PLrn3yqXftAPUPdSP_kbCX6BDicVprsnfI Per-App AS3 is enabled by default for the AS3 3.50.x and above but if needed enable it. POST Endpoint: /mgmt/shared/appsvcs/settings { "perAppDeploymentAllowed": true } POST Endpoint for Creating the Tenant and the Shared irule that is base64 endpoint: /mgmt/shared/appsvcs/declare { "schemaVersion": "3.45.0", "Shared": { "class": "Application", "template": "shared", "test": { "class": "iRule", "remark": "yes", "iRule": { "base64": "d2hlbiBIVFRQX1JFUVVFU1Qgew0KICAgSFRUUDo6cmVzcG9uZCAyMDAgY29udGVudCB7DQogICAgICA8aHRtbD4NCiAgICAgICAgIDxoZWFkPg0KICAgICAgICAgICAgPHRpdGxlPkFwb2xvZ3kgUGFnZTwvdGl0bGU+DQogICAgICAgICA8L2hlYWQ+DQogICAgICAgICA8Ym9keT4NCiAgICAgICAgICAgIFdlIGFyZSBzb3JyeSwgYnV0IHRoZSBzaXRlIHlvdSBhcmUgbG9va2luZyBmb3IgaXMgdGVtcG9yYXJpbHkgb3V0IG9mIHNlcnZpY2U8YnI+DQogICAgICAgICAgICBJZiB5b3UgZmVlbCB5b3UgaGF2ZSByZWFjaGVkIHRoaXMgcGFnZSBpbiBlcnJvciwgcGxlYXNlIHRyeSBhZ2Fpbi4NCiAgICAgICAgIDwvYm9keT4NCiAgICAgIDwvaHRtbD4NCiAgIH0NCn0=" } } } } POST Endpoint for sending Per-App AP that references the Tenant shared iRule I: /mgmt/shared/appsvcs/declare/Example/applications { "schemaVersion": "3.45.0", "A1": { "class": "Application", "service": { "class": "Service_HTTP", "virtualAddresses": [ "10.0.1.10" ], "pool": "web_pool", "iRules": [ { "use": "/Test/Shared/test" }, "insert_xff" ] }, "web_pool": { "class": "Pool", "monitors": [ "http" ], "members": [ { "servicePort": 80, "serverAddresses": [ "192.0.1.10", "192.0.1.12" ] } ] }, "insert_xff": { "class": "iRule", "iRule": "when HTTP_REQUEST { HTTP::header insert X-Forwarded-For [IP::client_addr] }" } } } Github link: Editing AS3-Per-App-Shared-Tenant-Objects/README.md at main · Nikoolayy1/AS3-Per-App-Shared-Tenant-Objects Note! To see the Tenant config use GET Endpoint /mgmt/shared/appsvcs/declare/Example or for the app /mgmt/shared/appsvcs/declare/Example/A1 For only the Tenant shared config use GET /mgmt/shared/appsvcs/declare/Example/applications/Shared By placing the pools in the /Shared under the Tenant all Per-AS3 declarations for Tenant can use the pool like for the iRule example I showed and also the pools can share the node IP addresses like the "shareNodes" option but only for the tenant. https://my.f5.com/manage/s/article/K88250015
How can I get started with iCall
Hi all . Recently, I want to learn how to use iCall to do some automated operations work, but I haven't seen any comprehensive tutorials about iCall on askf5. Are there any good articles I can refer to for learning? Do I need to systematically learn Tcl first? I still have a question about iCall. What is the difference between using iCall and using shell scripts with scheduled tasks to achieve automated management and configuration of F5? Best Regards
