Deploying F5 BIG-IP HA into AWS GovCloud
This guide provides a walk-through for deploying an active/standby F5 BIG-IP High Availability (HA) cluster within AWS GovCloud (US) using the AWS Console. Code Repository: F5GovSolutions/f5-aws-cloudformation-v2-govcloud While the repository's examples/failover/GOVCLOUD-GUIDE.md handles the exact AWS CLI workflows, this document serves as the web console companion. The Partition Problem The upstream commercial F5 templates (F5Networks/f5-aws-cloudformation-v2) assume a standard AWS commercial partition. Pointing those unmodified templates at GovCloud breaks deployment for three distinct reasons: Partition Isolation (aws-us-gov): GovCloud uses unique ARNs, regional endpoints, and an entirely separate AMI catalog. Furthermore, CloudFormation mandates that a parent stack's nested templates reside in an S3 bucket within the exact same region and partition. CloudFormation cannot fetch objects across partitions, meaning all templates must be pre-staged in a local GovCloud bucket. Air-Gapped Realities: In isolated enclaves with no internet egress, a booting BIG-IP cannot pull the runtime-init installer or Automation Toolchain RPMs from public GitHub repositories or F5 CDNs. Everything must be hosted locally inside an S3 bucket, accessible via VPC Gateway and Interface endpoints. The 17.x Clustering Bug: On the BIG-IP 17.x code train, a startup-timing condition occasionally prevents the local device-trust domain (/Common/Root) from initializing properly on first boot. When this occurs, Declarative Onboarding (DO) cannot establish the trust domain or failover group, leaving both nodes deadlocked. This modified template embeds an automated, non-blocking self-heal script to orchestrate the recovery process without manual intervention. What it deploys The root template (failover.yaml) orchestrates a series of nested module stacks (network, access, ingress, application, and individual BIG-IP nodes) to build: An active/standby pair of BIG-IP Virtual Editions (VE) deployed across two Availability Zones using a 3-NIC topology (Management, External, Internal). Cloud Failover Extension (CFE) integration to re-map floating application IPs and AWS routes during a failover event. Automated onboarding via F5 BIG-IP Runtime Init, executing Declarative Onboarding (DO) for core network clustering, Application Services 3 (AS3) for virtual servers/WAF policies, and CFE for AWS API integration. The architecture uses Pay-As-You-Go (PAYG) marketplace licensing and is fully validated on the 3nic-payg...-with-app runtime-init configuration. Core Architecture Differences vs. Commercial Staged-Bucket Deployment Mode Because of partition constraints, your local GovCloud S3 bucket serves as the single source of truth. It must hold two types of data: the nested CloudFormation templates cloned from the repository, and the target installation binaries (the runtime-init .run installer and the accompanying DO/AS3/CFE RPM packages). The Unauthenticated 403 Pitfall CloudFormation reads your deployment templates using your active IAM user/role credentials. However, during the bootstrap phase, the individual BIG-IP instances download their configuration files and RPM packages over an unauthenticated HTTPS request. They do not sign these requests with AWS credentials. If your S3 bucket permissions block public read access completely, the instances receive an HTTP 403 Forbidden response, initialization fails, and CloudFormation rolls back the stack. Because GovCloud blocks public access by default, you must apply a scoped bucket policy allowing s3:GetObject to Principal: "*" paired with a DenyInsecureTransport block. Single-Toggle VPC Endpoints Isolated enclaves require four distinct VPC endpoints to function: S3 (Gateway) for fetching installation packages and AS3 WAF policies, and Interface endpoints for EC2, Secrets Manager, and CloudFormation APIs. Missing even one endpoint causes silent automation failures. To simplify this, the template consolidates these requirements into a single parameter: setting provisionS3Endpoint=true automatically provisions all four regional endpoints with private DNS enabled. Note that your bucket, stack, and endpoints must reside within the same AWS region. Automated Clustering Self-Heal To bypass the 17.x initialization bug without manual operator access, the template installs a localized orchestration loop during the pre_onboard hook. The mechanism drops three files onto the file system: cluster-heal.sh: The main orchestrator script. cluster-heal-trust.py: A native Python script that securely pulls the admin password from Secrets Manager via a SigV4 request using the instance's IAM role, then invokes the local device trust commands. /etc/cron.d/cluster-heal: A cron job that triggers the orchestrator every 3 minutes. This loop checks the cluster status. If it detects a missing root domain, it triggers a single controlled reboot, reads the peer node's address from the runtime logs, pulls the password, and establishes device trust. Once synchronized, it issues the CloudFormation success signal and disables its own cron job. This preserves DO as the declarative source of truth without creating configuration drift. During this process, the CloudFormation stack will remain in CREATE_IN_PROGRESS for roughly 25 to 30 minutes, which is normal behavior. Elastic IP Allocation Constraints The default AWS regional quota is 5 Elastic IPs (EIPs). Depending on your topology, this architecture can request up to 7. To prevent deployment failures due to exhausted quotas, use the template toggles to limit public allocations: Default Configuration (Public Mgmt, Public VIP, Public Self IPs): 7 EIPs Standard Air-Gap Profile (Public Mgmt, Private VIP, Private Self IPs): 4 EIPs Strict Isolated Profile (Private Mgmt via Bastion, Private VIP, Private Self IPs): 3 EIPs Verify your regional limit under Service Quotas → EC2-VPC Elastic IPs (L-0263D0A3) before executing the template. Deployment Steps (AWS Console) Before You Begin Select a single target GovCloud region (us-gov-east-1 or us-gov-west-1). Your staging bucket, CloudFormation stack, EC2 Key Pair, and Secrets Manager secret must share this region to prevent cross-region routing failures in isolated enclaves. 1. Provision the staging S3 bucket S3 → Create bucket, in your Region. Give it a globally unique name. [S3 Create bucket, Region selector highlighted.] 2. Stage the templates and artifacts While file uploads can be handled via the S3 web GUI, staging is more reliably managed via the AWS CLI from a local workstation. Installing the AWS CLI is outside the scope of this article but it's easy and instructions can be Googled. Execute the following commands from the root of your cloned repository to sync the template architecture and upload your pre-downloaded F5 binaries: Download commands: # F5 BIG-IP Runtime Init installer (note: this repo's tag has NO "v" prefix) curl -fL -o f5-bigip-runtime-init-2.0.3-1.gz.run \ https://github.com/F5Networks/f5-bigip-runtime-init/releases/download/2.0.3/f5-bigip-runtime-init-2.0.3-1.gz.run # Declarative Onboarding (DO) curl -fL -o f5-declarative-onboarding-1.47.0-14.noarch.rpm \ https://github.com/F5Networks/f5-declarative-onboarding/releases/download/v1.47.0/f5-declarative-onboarding-1.47.0-14.noarch.rpm # Application Services (AS3) curl -fL -o f5-appsvcs-3.56.0-10.noarch.rpm \ https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.56.0/f5-appsvcs-3.56.0-10.noarch.rpm # Cloud Failover Extension (CFE) curl -fL -o f5-cloud-failover-2.4.0-0.noarch.rpm \ https://github.com/F5Networks/f5-cloud-failover-extension/releases/download/v2.4.0/f5-cloud-failover-2.4.0-0.noarch.rpm Sync and upload commands (be sure to be at root of the cloned repo), set the variables accordingly: # Synchronize the nested CloudFormation infrastructure templates aws s3 sync ./examples/ "s3://$BUCKET/$PREFIX/" --region "$REGION" # Upload the core runtime engine and toolchain extensions aws s3 cp f5-bigip-runtime-init-2.0.3-1.gz.run "s3://$BUCKET/$PREFIX/" --region "$REGION" aws s3 cp f5-declarative-onboarding-1.47.0-14.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION" aws s3 cp f5-appsvcs-3.56.0-10.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION" aws s3 cp f5-cloud-failover-2.4.0-0.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION" 3. Grant anonymous read on the artifacts In the S3 Console, select your staging bucket and navigate to Permissions. Under Block public access (bucket settings), disable the option to Block public access to buckets and objects granted through new public bucket policies (leave ACL blocks enabled). Next, apply the following bucket policy to allow the instances to pull down the bootstrap packages securely over HTTPS: { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE/*" }, { "Sid": "DenyInsecureTransport", "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": [ "arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE", "arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE/*" ], "Condition": { "Bool": { "aws:SecureTransport": "false" } } } ] } Note: Ensure you replace YOUR-BUCKET-NAME-HERE with your actual bucket name. Do not include any trailing lines or comments in the JSON editor. [S3 Permissions — Block public access settings.] 4. Store the Admin Password Secret Navigate to AWS Secrets Manager → Store a new secret. Choose Other type of secret, select the Plaintext tab, clear the default JSON template, and type your raw password string. Name the secret (e.g., f5-bigip-admin) and save it using the default settings. Copy the resulting Secret ARN (arn:aws-us-gov:secretsmanager:...); both nodes query this specific object at startup. [Secrets Manager Store secret, plaintext tab.] [Secrets Manager: Name your secret.] 5. Create the SSH key pair EC2 → Key Pairs → Create key pair, download the .pem. The key pair is regional. This is optional but recommended you create one before template launch. [Screenshot: EC2 Create key pair.] (You can leave the parameter blank in the template and let the stack auto-create one, but then the private key only lives in SSM Parameter Store — pre-creating is simpler if you need to access the boxes vis SSH) 6. Confirm the BIG-IP AMI exists in your Region The deployment template discovers the target AMI using automated string matching. Ensure that your region has access to the target image by running the following lookup via your terminal, this can also be done via the GUI: aws ec2 describe-images --region "$REGION" --owners aws-marketplace \ --filters "Name=name,Values=*17.5*PAYG-Best Plus 25Mbps*" \ --query 'reverse(sort_by(Images,&CreationDate))[].[Name,ImageId,CreationDate]' --output table 7. Launch the stack (CloudFormation) Open CloudFormation, select your target region, and click Create stack (with new resources). Select Template is ready, choose Amazon S3 URL, and paste the direct object URL to your staged failover.yaml file. Configure the mandatory parameters: Stack Name: Define an environment-specific identifier. S3 Configurations: Populate s3BucketName and s3BucketRegion. Secrets Management: Paste your copied Secrets Manager ARN into bigIpSecretArn. Security Access: Set restrictedSrcAddressMgmt and restrictedSrcAddressApp to your administrative CIDR blocks to avoid exposing management interfaces. Air-Gap Toggles (For isolated VPCs): Change provisionS3Endpoint to true, and flip both provisionPublicIpExternalSelf and provisionPublicIpVip to false. In the options screen, consider setting Stack failure options to Preserve successfully provisioned resources during your initial test runs. This keeps instances alive for log inspection if a deployment fails. Acknowledge the CAPABILITY_NAMED_IAM prompt and click Submit. [CloudFormation Specify template — Amazon S3 URL field.] [Parameters page, air-gap toggles section.] [Review — Acknowledgement and Stack failure options.] 8. Wait it out Expect ~25–30 minutes in CREATE_IN_PROGRESS while the self-heal forms the cluster, then CREATE_COMPLETE. This is normal — it is not stuck. After 10 minutes, go ahead and admin SSH into the boxes, they should accept your admin secret by now, if not use your SSH key you created and referenced in the template. See the commands below or in the git repo guide. Validating the deployment From the stack Outputs, grab a management IP, then: Onboarding: grep -i 'All operations completed successfully' /var/log/cloud/bigIpRuntimeInit.log; the prompt should show failover01.local / failover02.local, not ip-x-x-x-x. Clustering: tmsh show cm sync-status → In Sync (green), Mode: high-availability, one node Active and one Standby. CFE: GET https://localhost/mgmt/shared/cloud-failover/inspect returns a populated object (instance, addresses, trafficGroup). The self-heal's own narration lives in /config/cluster-heal/log — the primary place to look if the stack runs long; you'll see the reboot → add-to-trust → creating failoverGroup → cluster In Sync → cfn-signal sent OK → disabling self-heal sequence. If clustering genuinely never forms (the rare case where the self-heal exhausts its attempts), the repo guide has a validated manual fallback that follows the same reboot → rebuild trust → reapply sequence. What's GovCloud-ready today — and what isn't Let's be clear about scope. The GovCloud template— staged-bucket defaults, the four VPC endpoints, the clustering self-heal, the bumped extensions — currently applies to one solution: the failover active/standby pair on the 3-NIC PAYG -with-app runtime-init config. The repo's autoscale, the quickstart standalone, and the other failover variants (2-NIC, BYOL, and the non--with-app configs) still carry upstream configuration and are not yet GovCloud-validated. Adapting them is planned work and would follow exactly the pattern documented here. And the self-heal is a workaround for an F5 platform bug, not a permanent fix. The right long-term resolution is the platform defect behind the KB; when a fixed build is in play, the self-heal simply stays inert. Where to go next The repo: F5GovSolutions/f5-aws-cloudformation-v2-govcloud Full CLI walkthrough (first-time operator): examples/failover/GOVCLOUD-GUIDE.md Parameter and architecture reference: examples/failover/README.md The self-heal sources: examples/failover/bigip-configurations/cluster-heal.sh and cluster-heal-trust.py If you're standing up BIG-IP HA in an enclave, start from the 4-EIP air-gap profile with provisionS3Endpoint=true, stage everything in one in-Region bucket, and let the self-heal do the clustering. File issues on the repo if you hit something the guide doesn't cover.The End of ClientAuth EKU…Oh Mercy…What to do?
If you’ve spent any time recently monitoring the cryptography and/or public key infrastructure (PKI) spaces…beyond that ever-present “post-quantum” thing, you may have read that starting in May of 2026, Google Chrome Root Program Policy will start requiring public certificate authorities (CAs) to stop issuing certificates with the Client Authentication Extended Key Usage (ClientAuth EKU) extension. While removing ClientAuth EKU from TLS server certificates correctly reduces the scope of these certificates, some internal client certificate authenticated TLS machine-to-machine and API workloads could fail when new/renewed certificates are received from a public CA. Read more here for details and options.Implementing Agentic Observability and Security
In April 2026, an AI coding agent named PocketOS wiped out a company's production database and its backups in nine seconds flat, executed through a single API call. Afterward, the agent described its behavior with unsettling bluntness: it ran a destructive action it was never explicitly asked to run, choosing to guess rather than verify. The detail that should keep every engineer awake at night isn't the deletion itself. It's that the team actually had a log of the action—the cloud provider recorded the destructive API call perfectly—but they had absolutely zero record of the reasoning that sparked it. By the time the deletion hit the logs, there was nothing left to salvage. That visibility gap is what we need to solve. Autonomous agents are hitting production environments faster than security teams can vet them, and they fail in ways traditional application security simply wasn't designed to intercept. Safely running them boils down to two critical capabilities: observability (seeing what an agent is deciding to do) and guardrails (restricting what it is allowed to do). You can't skip either one. If you want to simulate this exact failure mode on your own hardware, we built a hands-on lab called agent-security-lab. It spins up a five-agent SOC incident-response team, drives it rogue under realistic conditions, and hardens it step-by-step. I'll link to it at the end, but the takeaway here is that you don't have to take these risks on faith—you can test the defenses yourself. Agents fail differently Traditional software does exactly what the source code dictates. Its behavior is locked in at build time, meaning your gateways, logs, and audit trails are built around a static assumption: they record what happened because "what happened" belongs to a pre-defined, knowable set of outcomes. An agent behaves entirely differently. Its actions are decided at runtime by an LLM interpreting whatever text it was just handed. It maps out plans, chooses tools, spins up subordinate agents, and executes highly consequential actions without a human approving the intermediate steps. You cannot know the full scope of its actions in advance because its behavior depends entirely on inputs that an attacker might control. The most dangerous failure mode is the one that looks like a successful execution. In the PocketOS database incident, the agent wasn't acting maliciously—it was just trying to be helpful. It read a prompt stating "the table is corrupt, fix it," and it chose the most absolute resolution the prompt implied. If you are only looking at a log of side effects, a helpful-but-catastrophic action and a correct one look identical until you read the actual reasoning behind them. This is why model observability isn't just a compliance line-item; it is the only mechanism you have to differentiate between an intended fix and a disaster. Observability: Capturing decisions, not just the damage For autonomous systems, true observability requires deep visibility into the agent's chain of thought. You need to see the exact prompt it received, the conclusions it drew, the specific tools it selected, the exact arguments it generated, and how it interpreted the resulting data. This goes far beyond standard application logging, which merely timestamps the damage after the fact. The most practical architecture for capturing this data is routing all model traffic through an explicit security proxy, rather than letting agents call model endpoints directly. On the F5 AI Security platform, this is handled via our Bring Your Own Agent (BYOA) architecture and surfaced as Agentic Fingerprints. This gives you a per-agent, per-run audit trail of prompts, reasoning steps, and tool choices. Crucially, the proxy captures the model's decision independently of whether the downstream action succeeds. If an agent decides to drop a table, you see that intent and the verbatim arguments it formulated before the request ever reaches your database. Figure 1. An Agentic Fingerprints session: the agent's system prompt, the input it received, and its decision to call a destructive tool with verbatim arguments — captured before the action runs. However, there is a structural boundary here that teams often miss, leading to a false sense of security. In any tool-using agent, there is a specific line where a model's decision transitions into a physical action: the moment the runtime executes the tool the model selected. The proxy handles everything on the model-facing side of that line (the request, the reasoning, the tool request, and the model's reaction to the return payload). It cannot see the tool actually run. That execution happens on an entirely separate data path—from the agent to the tool server to your identity provider, governed by its own authorization checks and logs. Because of this, complete agent observability must span two separate layers: The Model-Facing Layer: This tracks what the agent concluded, which tool it wanted, and how it handled the return data. You monitor this via F5 AI Security (BYOA / Agentic Fingerprints views). The Tool-Facing Layer: This tracks which tool calls actually executed, which were blocked, and the specific permissions carried by that credential. You monitor this through your Tool/MCP server logs and identity-provider audit trails. Relying on either layer alone leaves a blind spot. The PocketOS incident occurred because the team only had tool-facing visibility—the cloud provider caught the destructive call—but they lacked the model-facing context to see the decision forming. You need to wire both layers together and correlate their logs before putting an autonomous agent anywhere near production data. Implementation is a configuration change The good news is that adding model-facing observability doesn't require rewriting your agentic framework; it’s a simple configuration adjustment. You just need to point your agent's model client to the proxy and tag the outbound traffic with a unique session identifier. Here is how you would configure a standard OpenAI-compatible client in Python: import datetime import uuid from openai import OpenAI # Generate a unique session ID for every single run. # Reusing IDs will merge separate runs into an unreadable trail. # Recommended format: {run_id}-{agent_name}-{timestamp}-{unique_tail} session_id = "-".join([ LAB_RUN_ID, # Shared across all agents in a single execution run AGENT_NAME, # Identifies the specific agent datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ"), uuid.uuid4().hex[:6] ]) # Initialize the client, routing traffic through the F5 AI Security proxy client = OpenAI( api_key=CALYPSOAI_TOKEN, # Your F5 platform token, not the raw LLM key base_url=CALYPSOAI_OPENAI_API_BASE, # Redirects requests through the proxy default_headers={ # Tags the traffic to this specific agent's audit trail "x-cai-metadata-session-id": session_id, } ) The first two parameters redirect the outbound calls through the security proxy, while the third converts raw LLM traffic into an organized audit trail. By assigning unique session IDs, the platform groups a single agent's calls into a coherent narrative rather than overlapping traffic from different processes. Without the uniqueness of each call/session, it would be impossible to isolate conversations. In a multi-agent system, this setup allows you to trace an entire workflow as it crosses boundaries: each agent generates its own session ID under a shared run prefix, letting you piece together the entire decision path from end to end. With just a few lines of configuration, you get full visibility across your environments, mapping directly to the telemetry shown in our platform dashboards. Work these standards into your governance frameworks, every agent deployed should have observability baked in. Designing multi-layered guardrails Observability gives you eyes, but it doesn't stop an action mid-flight. For enforcement, you need guardrails. Experience with production failures shows that a single defensive layer will eventually crack. You need independent, decoupled boundaries where each layer catches a different class of failure and fails closed by default. A robust agent defense architecture requires five distinct checkpoints: Content Scanning (Proxy Input/Output Scanners): Blocks poisoned or injected prompts before they hit the model. This is your primary defense against completely novel prompt injection techniques. Capability Scoping (Tool Server Manifests): Restricts the agent's environment so it can only see or select the specific tools required for its role. Authorization Scope (OAuth 2.1 / Per-Call Checks): Evaluates short-lived tokens to ensure that even if an agent requests a tool, its current credential actually possesses the authorization to run it. Agent-to-Agent Control (Signed Agent Cards): Governs multi-agent environments by validating whether one agent is permitted to invoke the skills of another, enforced at the receiving end. Underlying Privilege (Downstream Target Systems): The final backstop where the actual resource (database, identity provider, mail relay) enforces strict least-privilege access on the agent's service account. When designing these layers, focus on the difference between reactive and structural defenses. An authorization scope check is reactive: the agent attempts a destructive call, and the platform intercepts and denies it mid-flight. Capability scoping, on the other hand, is structural: the dangerous tool is completely omitted from the agent's available schema definition. Because the model doesn't even know the tool exists, it can't choose it in the first place, completely removing the attack vector. You should implement both, but structural defenses are far more resilient against novel bypasses because they actively shrink what the agent is capable of doing. It's also worth noting that content scanning is the only layer capable of identifying semantic attacks that authorization policies miss. A prompt injection payload hidden inside a customer support ticket or a scraped web page will easily bypass credential checks because, from an infrastructure perspective, reading that data is perfectly valid. Input scanners inspect the text itself before it reaches the LLM to verify what the agent is being asked to do. When a scanner intercepts an attack, the F5 Logs view isolates the specific rule that triggered the block and records the offending payload—giving you a clean, shareable artifact to explain the security decision to external stakeholders without forcing them to dig through raw system telemetry. Figure 2. The Logs view after a guardrail block: the scanner that fired, its verdict, and the content that triggered it — the explanation artifact for an automated decision. Visibility and enforcement must coexist It is common to view observability and guardrails as separate boxes to check on a security review. In practice, they are two sides of the exact same coin. A guardrail without observability requires blind faith; you can't validate that it's firing correctly, you can't tune it when it blocks legitimate business traffic, and you can't explain its decisions to your team. Conversely, observability without guardrails just gives you a pristine, high-resolution recording of a security breach you did nothing to stop. The deployment order matters here. Always deploy observability first. You cannot confidently evaluate or fine-tune a security policy you can't actively watch. Establish your visibility pipelines, verify that you can accurately map the agent's logic alongside its tool executions, and then begin layering your guardrails on top—using your new telemetry trails to validate each security layer as you turn it on. Next Steps If you are currently deploying or managing autonomous agents in production, you can lock down your environment with four immediate steps: Route all LLM traffic through a dedicated proxy to establish an immutable, server-side record of agent reasoning, rather than relying on standard application logs. Enforce unique session tagging across every agent run to keep your audit trails clean and searchable. Instrument both model-facing and tool-facing logs, ensuring you have a clear correlation key between the two. Layer your enforcement mechanisms sequentially: swap out static API keys for short-lived, scoped credentials; apply tool capability scoping; enforce agent-to-agent verification; and place content scanners in front of every ingress point. The most effective way to understand these interactions is to deliberately break them in a safe sandbox environment. Our agent-security-lab recreates the PocketOS database failure against a sandboxed environment, surfaces the agent's reasoning in real-time, and guides you through configuring OAuth 2.1 scoping, MCP capability scoping, agent cards, and F5 AI Guardrails to stop the exploit. The lab is built entirely for practitioners and provides all the required source code. Note that because the exercises interface with the F5 AI Security platform, you will need access to a platform tenant—if you need to set one up for your team, just reach out to your F5 account representatives to get a lab instance provisioned. Agents are the norm now, you are either planning for agentic AI or have already deployed them, its only a matter of time and observability is the blindspot. For autonomous agent architectures, deep observability forms your foundation, and multi-layered guardrails provide the structure to scale safely. References and further reading agent-security-lab (F5GovSolutions) — hands-on multi-agent security lab: github.com/F5GovSolutions/agent-security-lab F5 AI Security platform documentation: docs.aisecurity.f5.com F5 AI Guardrails: f5.com/company/blog/what-are-ai-guardrails F5 AI Explainability / Agentic Fingerprints: f5.com/company/blog/ai-explainability F5 Labs CASI and ARS leaderboards: f5.com/company/labs/casiProtecting Your MCP Server With F5 BIG-IP Advanced WAF
As AI assistants increasingly interface directly to production databases and APIs, building secure, structured interaction layers has become critical. The Model Context Protocol (MCP) standardizes these connections and eliminates the need for custom integrations (See APIs First: Why AI Systems Are Still API Systems). However, with this comes a new class of vulnerabilities that users must protect their service from if they are considering leveraging an MCP server to handle requests to their service. These vulnerabilities have begun to be identified and classified by the OWASP Foundation as OWASP MCP vulnerabilities and they have defined the Top 10 vulnerabilities. In the below demo, we will show the new MCP Policy Template for the BIG-IP Application Security Module and how it can block an attacker from using an AI Agent to execute Command injection against an MCP server. The vulnerabilities that will be demonstrated are as follows: MCP05:2025 - Command Injection & Execution MCP08:2025 - Lack of Audit and Telemetry This article will also serve as a container for more in-depth articles that will demonstrate how to block other OWASP MCP Top 10 vulnerabilities.
Securing AI with F5 AI Guardrails and Nutanix Enterprise AI
Enterprises are rapidly moving from AI experimentation to production-grade deployments, using Kubernetes to deliver and scale AI services consistently across environments. As AI workloads become embedded in business-critical workflows, new risks emerge—particularly at runtime. Generative AI systems can expose sensitive data, generate responses to restricted topics, or be manipulated through prompt injection and jailbreak techniques during user interactions. The challenge is no longer just how to deploy AI models, but how to control AI behavior in real-time—without adding operational complexity or slowing innovation. F5 and Nutanix address this challenge together by combining Nutanix Enterprise AI (NAI) with F5 AI Guardrails. Nutanix Enterprise AI provides a Kubernetes-based platform to deploy and operate AI models and inference services at scale, while F5 AI Guardrails enforces real-time security and policy controls on AI interactions. 🎬 See the Joint Solution in Action ✨ NEW✨🎬 Watch this demo video to see F5 AI Guardrails work seamlessly together with Nutanix Enterprise AI (NAI) Agent Gateway Unified Endpoint (New in NAI v2.7) to secure LLM inference at scale. 🎬 Watch this demo video to see how F5 and Nutanix work together to deliver secure, trustworthy, enterprise‑ready AI—running side by side on Nutanix Kubernetes Platform (NKP) deployed on Nutanix Infrastructure. F5 AI Guardrails Overview F5 AI Guardrails provides runtime security and governance for AI applications, models, and agents by enforcing policy‑based controls on how AI interacts with users and data. It includes a broad set of out‑of‑the‑box guardrails that enable organizations to immediately begin securing and governing AI interactions from day one. These built‑in guardrails address common AI risks such as prompt injection, jailbreaks, PII exposure, restricted topics, and more. In addition to built‑in guardrails, F5 AI Guardrails supports custom guardrails, allowing teams to tailor policy enforcement to specific use cases. One option for defining custom guardrails is through natural‑language policy creation, guided by an integrated F5 AI Assistant. The AI Assistant reviews and refines guardrail definitions using F5 best practices, ensuring that descriptions are clear, concise, and consistently structured for optimal performance. AI Guardrails simplifies AI observability by providing continuous visibility and traceability across AI interactions, with audit‑ready observability, scanning, and logging to support governance and compliance. As part of the F5 Application Delivery and Security Platform (ADSP), AI Guardrails delivers runtime protection for AI models, agents, and connected data, extending F5’s application delivery, security, and observability capabilities across applications and APIs. https://www.f5.com/products/ai-guardrails Nutanix Enterprise AI (NAI) Overview Nutanix Enterprise AI (NAI) is a cloud‑native, Kubernetes‑based AI platform designed to help organizations run AI inference at scale. NAI is supported on CNCF‑certified Kubernetes platforms, including Nutanix Kubernetes Platform (NKP). It enables enterprises to deploy, manage, and operate AI models, inference endpoints, and AI agents through a single, unified management experience, allowing organizations to run AI consistently wherever they operate Kubernetes while maintaining enterprise control, security, and visibility. https://www.nutanix.com/products/nutanix-enterprise-ai F5 and Nutanix: Better Together Nutanix Enterprise AI provides a Kubernetes‑based platform for deploying, managing, and operating AI models, inference endpoints, and AI agents, while F5 AI Guardrails provides runtime security and governance for AI interactions. F5 AI Guardrails can run side by side with Nutanix Enterprise AI, with both solutions deployed on the same Nutanix Kubernetes Platform (NKP)–managed Kubernetes cluster, enabling them to operate on a common Kubernetes foundation. Together, F5 and Nutanix deliver secure, trustworthy, enterprise‑ready AI deployments by combining an enterprise AI platform with runtime AI security—enabling organizations to move fast with AI without losing control over security, compliance, and governance.
HTTP Load Balancer Routes on F5 Distributed Cloud
Route misconfiguration is one of the most common configuration mistakes we see that can cause incidents on F5 Distributed Cloud (F5 XC). The four route types look deceptively simple in the console, but they have distinct behaviors, ordering rules, and gotchas. This article covers all four types with real field names, decision guidance, and the mistakes that actually happen in production. What Routes Do in F5 XC HTTP Load Balancer An HTTP Load Balancer in F5 XC is a full L7 proxy running at the Regional Edge (RE) and depending on the deployment model, Customer Edge (CE). We will use the Regional Edge as a deployment model for this article. When a request arrives, the RE evaluates the route list in order and applies the first matching route. That route determines what happens to the request: forward it to an origin, redirect the client, return a synthetic response, or apply advanced routing logic. Routes can be configured inside the HTTP Load Balancer configuration which opens a new Route Options window: Multi-Cloud App Connect > Load Balancers > HTTP Load Balancers > [your LB] > Routes > Route Options The four route types map to three underlying route actions: XC Route Type Route Action Traffic Goes To Simple Route route Origin Pool Redirect Route redirect Client (3xx response) Direct Response Route direct_response Client (fixed response body) Custom Route route / redirect / direct_response Depends on configuration Route Matching: How XC Evaluates Routes Route evaluation is sequential, stops on first match, and has no automatic specificity ranking. The order you set is the order XC uses. Evaluation Order The HTTP Load Balancer evaluates routes sequentially, top to bottom. The first route that matches the incoming request wins. No further routes are evaluated. This means: More specific routes must appear before broader ones. A catch-all route (prefix /) at the top will swallow everything. Nothing below it will ever match. Path Match Types Three path match types are available across all route types: Match Type Field Behavior Prefix path_prefix Path must begin with the specified string Exact exact Path must equal the value exactly (query string excluded) Regex regex Entire path (minus query string) must match the regex pattern Prefix matching pitfall: The prefix /api matches /api/v1/users but also /apikeys and /api-internal. If you want to match a path segment boundary, use /api/ (trailing slash) or switch to regex. Additional Matching Criteria Beyond path, routes can match on: HTTP methods: GET, POST, PUT, DELETE, etc. Request headers: presence, exact value, regex Query parameters: Retain, Remove, or Replace Combining criteria (e.g., path prefix + method + header) creates an AND condition: all specified criteria must match. Route Type 1: Simple Routes Simple routes are the workhorse of most HTTP Load Balancer configurations. They match a path (and optionally method/headers) and forward traffic to an Origin Pool. When to Use Standard application traffic forwarding Path-based routing to different backend services API versioning (/v1/ → pool A, /v2/ → pool B) Microservice fanout from a single domain Key Configuration Fields Field Description Path match type Prefix / Exact / Regex HTTP Method Any, GET, POST, PUT, etc... Origin Pool The backend pool receiving the request Host Rewrite Method Disable/ Hostname / Header value: rewrites the Host header sent to origin Query Parameters Retain, Remove, Replace Advanced Options Worth Knowing Path rewriting (under Advanced Options): Disabled: path sent to origin unchanged Prefix Replacement: replaces the matched prefix with a new string (e.g., strip /api/v1 prefix before sending to origin) Regex-based: full regex substitution on the path Retry policy: The default retry policy is 1 retry on 5xx responses. Set this explicitly for your application in every route: Disabled: no retries; required for write operations Default: 1 retry on 5xx Custom: specify retry conditions, count, and interval Per-route WAF override: Each simple route can attach its own WAF App Firewall policy. This completely replaces the load balancer-level WAF for matching requests; it is not additive. Use this to enforce stricter rules on sensitive paths (e.g., /admin/) or to relax inspection on certain paths. Example: API Path Routing Route 1: Prefix /api/v2/ → origin-pool-v2 (exact origin for v2) Route 2: Prefix /api/v1/ → origin-pool-v1 (legacy backend) Route 3: Prefix /api/ → origin-pool-api (catch-all for API paths) Route 4: Prefix / → origin-pool-web (catch-all for everything else) Order matters here. If route 3 or 4 appeared first, routes 1 and 2 would never fire. Route Type 2: Redirect Routes Redirect routes return an HTTP 3xx response directly to the client. No origin pool is involved: the RE handles the response entirely. When to Use HTTP → HTTPS redirect (though XC has a dedicated LB-level toggle for this) Domain canonicalization (www.example.com → example.com) Legacy URL migrations (/old-path/ → /new-path/) Temporary redirects during maintenance or A/B migrations Key Configuration Fields Field Description Protocol HTTP or HTTPS Host Target FQDN; supports non-standard ports Redirect Path / URI Target path; if left unset, original URI is preserved (including query string) Response Code 301 (Permanent), 302 (Temporary), 307, 308 Redirect Behavior: URI Preservation When you leave the redirect path unset, XC preserves the original URI path and query string in the Location header. This is useful for protocol/host-only redirects where you just want to change the scheme or domain without touching the path. Example: Redirect all HTTP traffic to HTTPS on the same host: Protocol: HTTPS Host: same-as-request (leave blank or match domain) Path: (unset — preserve original URI) Code: 301 Redirect Route Limitations Simple redirect routes (defined directly on the LB) do not support custom header manipulation on the redirect response. If you need to inject headers (e.g., Cache-Control: no-store on the redirect response), use a Custom Route object instead. Route Type 3: Direct Response Routes Direct response routes return a fully synthetic HTTP response to the client. The request never reaches an origin pool: the RE generates the response itself. When to Use Health check endpoints that should always return 200 (e.g., /healthz) without touching the app Maintenance mode pages: serve a 503 with a message body while origin is down Blocking specific paths with a meaningful error body (vs. a generic deny) Canary or feature-flag placeholders that return 404 before the feature ships Robots.txt or security.txt served from the edge without an origin Key Configuration Fields Field Description HTTP Status Code Any valid HTTP status code (200, 403, 503, etc.) Response Body Static text or HTML body returned to client Path match Same prefix/exact/regex options as other route types Example: Edge-Served Health Check Path: Exact /healthz Method: GET Action: Direct Response Status: 200 Body: OK This responds to health probes from AWS ALB, Kubernetes ingress controllers, or uptime monitors without any load on the backend. Particularly useful during blue/green deployments when the app might not yet be healthy. Route Type 4: Custom Routes Custom routes reference standalone Route objects created separately in XC and attached to one or more HTTP Load Balancers. Unlike the other three types, they follow a service-mesh model rather than a traditional LB model. When to Use Custom Routes Weighted traffic splitting between origin pools (canary releases, blue/green) Request/response header manipulation not available on simple routes Advanced retry policies with specific conditions and intervals Traffic mirroring (shadow traffic to a secondary backend for testing) Reusable route logic shared across multiple load balancers Architecture: Route Objects vs. Inline Routes Inline routes (simple, redirect, direct response) are defined directly on the HTTP Load Balancer. Custom route objects are: Created as standalone objects under Multi-Cloud App Connect Referenced by the HTTP Load Balancer Reusable: multiple LBs can reference the same route object Weighted Clusters Custom routes enable weighted traffic splitting across multiple upstream clusters, equivalent to BIG-IP pool ratio weights or AWS ALB weighted target groups. Route: Prefix /api/ Cluster A (origin-pool-v2): weight 90 Cluster B (origin-pool-v1): weight 10 This is the correct mechanism for canary deployments and gradual traffic shifts on F5 XC. The weights are percentage-based and must sum to 100. Header Manipulation Custom routes support header operations at the route level, applied before forwarding to origin: Operation Direction Example Add header Request X-Forwarded-For: {client-ip} Remove header Request Strip Authorization before certain paths Add header Response Strict-Transport-Security: max-age=31536000 Remove header Response Strip Server header from responses Header manipulation runs in order: route-level → virtual host-level → route configuration-level. Retry Policies Route retry policies take complete precedence over any virtual host-level retry policy. Configure explicitly: Field Description Retry on Conditions: 5xx, gateway-error, reset, connect-failure, retriable-4xx Number of retries Integer Per-try timeout Timeout applied to each individual retry attempt Retry interval Base interval between retries Traffic Mirroring Mirror policies shadow a copy of each request to a secondary cluster. The mirrored request is fire-and-forget. Example: Testing a new backend version against live traffic without affecting users Security analysis pipelines Route Ordering and Priority Route order is the most common source of routing bugs in XC deployments. There is no automatic specificity ranking; you own the order. The Rules Routes evaluate top to bottom. First match wins. Evaluation stops. Disabling a route (via the Route Activation Status toggle) does not remove it: traffic falls through to the next matching route. Recommended Ordering Pattern Order routes from most specific to least specific: Exact paths first Exact /api/v2/auth/token 2. Specific prefixes next Prefix /api/v2/ 3. Broader prefixes after Prefix /api/ 4. Path-specific exceptions Exact /healthz 5. Catch-all last Prefix / Common Ordering Mistakes Mistake Symptom Fix Catch-all prefix / first All traffic hits one origin; other routes never fire Move catch-all to last position /api prefix before /api/v2/ V2 traffic hits wrong origin Reverse the order Disabled/Unused route above active route Traffic silently hits next route with different behavior Remove disabled routes; don't rely on toggle for permanent changes Redirect route below a prefix match Redirect never fires Move redirect above the prefix that would match it first Common Mistakes and Gotchas Prefix /api matches /apikeys. The prefix match does not anchor to path segment boundaries. /api matches /api/, /api/v1/, and also /apikeys, /api-docs. Use /api/ (trailing slash) or regex if segment boundary matters. Per-route WAF is a full replacement, not additive. Attaching a WAF policy to a route does not stack with the LB-level WAF. It replaces it entirely for that route. If your LB WAF is in blocking mode and you attach a route-level WAF in monitoring mode, that route is now in monitoring mode only. Custom routes enforce TLS: test before production. If your origin uses a self-signed certificate and you switch from a simple route to a custom route without uploading the Root CA, connections will fail. Test in a staging environment first. Header manipulation on redirect requires a custom route object. Simple redirect routes in XC do not support response header injection. If you need Cache-Control or Vary headers on your redirects, you must use a standalone custom route object with redirect action. Regex route performance at scale. Regex routes require full path evaluation on every request. At high request volumes, a large number of regex routes adds measurable CPU overhead compared to prefix or exact routes. Use regex only where prefix or exact matching is insufficient. FAQ Q: What is the difference between a Simple Route and a Custom Route? A: Simple routes are inline on the HTTP Load Balancer and forward traffic directly to an Origin Pool. Custom routes are standalone objects that use an Endpoints → Clusters → Routes model, support weighted traffic splits, header manipulation, and mirroring, but cannot reference Origin Pools directly. Q: Why is my catch-all route matching everything instead of the specific routes below it? A: Route evaluation stops at the first match. If your catch-all prefix (/) is above more specific routes, it wins every time. Move the catch-all to the last position in the list. Q: Can I use a Custom Route to send traffic directly to an F5 XC Origin Pool? A: No. Custom routes do not support Origin Pools directly. They use an Endpoints → Clusters → Routes abstraction. If you need weighted splitting with Origin Pool support, custom routes are not the right fit; simple routes forward to Origin Pools but do not support weighted clusters. Q: My POST requests are creating duplicate records and I traced it to XC retries. What is happening? A: The default retry policy on simple routes is "1 retry on 5xx." A POST that hits a 500 gets retried once, potentially double-writing. Set the retry policy to Disabled on any route handling non-idempotent operations (POST, PUT, PATCH, DELETE). Q: Does attaching a WAF policy to a route add rules on top of my LB-level WAF? A: No. Per-route WAF replaces the LB-level WAF entirely for requests matching that route. If your LB WAF is in blocking mode and the route WAF is in monitoring mode, those requests are evaluated in monitoring mode only. Q: My Custom Route TLS connections to origin are failing but the same origin works fine on a Simple Route. Why? A: Custom routes enforce strict TLS with no skip-verify option. Simple routes do not have this requirement. For custom routes, upload the Root CA certificate for your origin, or use the use_volterra_trusted_ca_url flag via the API for public CAs. Self-signed certs without the Root CA uploaded will fail silently. Q: I disabled a route in the console but traffic behavior changed unexpectedly. What happened? A: Disabling a route via the Route Activation toggle does not remove it from evaluation. Traffic falls through to the next matching route in the list. If that route is a broad catch-all, the behavior shift may look correct until something that depends on specific routing breaks. Remove routes you no longer need rather than toggling them off.API Discovery and Enforcement with API Security Local Edition
API Security Local Edition is a self-hosted platform that discovers APIs from BIG-IP traffic insights, builds and maintains an inventory with risk scoring, and pushes enforcement back to BIG-IP. This article covers the architecture, the data flows between components, and the operator workflow from discovery to enforcement.
Automating ACMEv2 Certificate Management on BIG-IP
While we often associate and confuse Let's Encrypt with ACMEv2, the former is ultimately a consumer of the latter. The "Automated Certificate Management Environment" (ACME) protocol describes a system for automating the renewal of PKI certificates. The ACME protocol can be used with public services like Let's Encrypt, but also with internal certificate management services. In this article we explore the more generic support of ACME (version 2) on the F5 BIG-IP.Automatic Certificate Management with ACMEv2 in F5 BIG-IP
One of the most anticipated features of F5 BIG-IP is integration with ACMEv2. With the General Availability of BIG-IP 21.1.0 on May/26, this feature came into being. In this tutorial, we are going to configure it, using Let's Encrypt as the CA. The domain for which we are generating/renewing certificates is carlosf5lab.lat. The official docs for this feature are located in SSL Certificate Management | BIG-IP Documentation. Pre-requisite 1: DNS Resolver that can reach the internet (at least the CA endpoints). In this case, we are using the native DNS Resolver that comes with BIG-IP. Pre-requisite 2: The internal proxy that will make the connection with the CA. Pre-requisite 3: a self signed SSL certificate that the ACMEv2 protocol uses as the identifier for a device account. You don't have to fill the Subject Alternative Name. For the Common Name, an e-mail contact is advised. Now, we are going to create the ACME Provider object. Give it a name, and select the internal proxy previously created. For the CA Certificate to enable the secure connection with the Directory URL, you can use the default ca-bundle.crt. The Directory URL is the endpoint for the ACMEv2 protocol. In Let's Encrypt case, it is https://acme-v02.api.letsencrypt.org/directory For the Account Key, choose the previously created self-signed certificate. For the trickier part of all, the field "Contacts" is mandatory, and it must be an URL. That’s why you must use the format mailto:email_address. Check the Terms and Conditions, and the Create Account boxes. After a while, the Account Status must read as "Valid". To prove you own the domain whose certificate Let's Encrypt is going to create/renew, it must be pointing to an IP (A Record) where you must have your Virtual Server listening on Port 80 configured to respond to the ACMEv2 Challenge. (In this specific lab, the domain carlosf5lab.lat points to a Public IP mapped to an internal IP). Now you can order your first certificate via ACMEv2 on BIG-IP: After a while, the Key tab should read something like: Which means your certificate was generated: To track the ACME Provider, you can check its statistics: That's it, my friend! If it helped you, give a thumbs up to this post!NGINX Gateway Fabric - Data Plane Programmability with NGINX JavaScript
This post walks through a pattern for injecting NGINX JavaScript logic into NGINX Gateway Fabric using Kubernetes-native extension points to enable data plane programmability, with an F5 AI Guardrails integration as a worked example.
