F5 Distributed Cloud (XC) - Origins & Health Checks
This article provides a platform-level overview of how F5 Distributed Cloud and the Application Aware Fabric handle origin discovery and health checking. It explains how these platform mechanics impact resiliency, scale, and application uptime, while also highlighting why proper health-check and origin configuration from day one is important for long-term operational stability.Automating F5 ADSP — Part 1: F5 XC and BIG-IP for Delivery and Security
What this use case demonstrates This use case covers three of the four ADSP areas: Delivery, Security, and Deployment. Delivery — F5 Distributed Cloud (XC) load balancer at the edge, F5 BIG-IP LTM handling traffic management inside the VPC. Security — XC WAF at the edge, BIG-IP Advanced WAF (AWAF) applying in-path policy before traffic reaches the application servers. Deployment — XC consumed as SaaS, BIG-IP deployed as a Virtual Edition in GCP. Same article, two deployment models, both provisioned from code. You get two layers of delivery and two layers of WAF, across a SaaS edge and a self-managed VE. The whole stack, VPC through XC load balancer, comes up from a single git push. Architecture What gets deployed: A GCP VPC with management, external, internal, and application subnets BIG-IP with AWAF in a single-NIC configuration OWASP Juice Shop and crAPI as target applications F5 Distributed Cloud HTTP load balancer, origin pool, and WAF policy pointing at the BIG-IP The vulnerabilities in the apps are deliberate. They let you exercise the WAF stack against real attack signatures and API abuse patterns. Without them, you only know the controls deployed, not that they work. DevSecOps in practice The lead-in covers the approach. For UC1, that means: Terraform handles infrastructure, BIG-IP configuration, and F5 Distributed Cloud objects. No click-ops. State lives in a GCS bucket the workflow creates on the first run, with a separate state file per module. The same bucket carries the AS3 declaration BIG-IP pulls on boot, so the runner never needs network access to BIG-IP. GitHub Actions runs the pipeline. Branch names trigger deployments, so git history shows what was meant to happen. GCP Workload Identity Federation replaces static service account keys. The F5 XC API certificate lives in GitHub Actions secrets, not the repo. The pipeline Pushing to a branch runs the workflow. There is no manual terraform to apply. Action Branch Validate, plan, and apply deploy-adsp-uc1 Validate only (no apply) test-adsp-uc1 Destroy all resources destroy-adsp-uc1 This keeps intent visible in git, makes destroy as easy as deploying, and gives reviewers a real PR to look at when something changes. What's in the repo f5devcentral/F5-ADSP-Automation: Directory Purpose infra/gcp/ VPC, subnets, firewall rules compute/gcp/ Juice Shop and crAPI f5/ BIG-IP base config and AWAF policy config/uc1/gcp/env.json GCP project, region, prefix config/uc1/xc/env.json F5 Distributed Cloud config .github/workflows/ CI/CD workflows Demo Try it Prerequisites, secrets, and troubleshooting are in the Use Case Deployment Guide. Contribute Issues and PRs welcome at f5devcentral/F5-ADSP-Automation/Issues. Resources: F5 Application Delivery and Security Platform GitHub Repo and Automation Guide ADSP Architecture Article Series: Automating F5 ADSP Deployments (Intro) Automating F5 ADSP Deployments (Part 1 - F5 XC WAF and BIG-IP Adv. WAF) Automating F5 ADSP Deployments (Part 2 - F5 XC WAF and NGINX App Protect) Automating F5 ADSP Deployments (Part 3 - F5 XC API Protection and NGINX Ingress) Automating F5 ADSP Deployments (Part 4 - F5 XC BOT Defense and BIG-IP AdvWAF) Automating F5 ADSP Deployments (Part 5 - F5 XC, BIG-IP APM, CIS, and NGINX Ingress) Minimizing Security Complexity: Managing Distributed WAF Policies
Automating F5 Application Delivery and Security Platform Deployments
The F5 ADSP Architecture Automation Project The F5 Application Delivery and Security Platform (ADSP) reduces the complexity of modern applications by integrating operations, traffic management, performance optimization, and security controls into a single platform with multiple deployment options. This series outlines practical steps anyone can take to put these ideas into practice using the F5 ADSP Architectures GitHub repo and related projects. Each article in the series highlights a different deployment example. The examples can be run locally or integrated into CI/CD pipelines following DevSecOps practices. The repositories are community-supported and intended as reference code for demos, workshops, or as a stepping stone for your own F5 ADSP deployments. If you find any bugs or have any enhancement requests, open an issue, or better yet, contribute. The F5 Application Delivery and Security Platform (F5 ADSP) F5 ADSP addresses four core areas: how you operate day-to-day, how you deploy at scale, how you secure against evolving threats, and how you deliver reliably across environments. Each comes with its own challenges, but together they define the foundation for keeping systems fast, stable, and safe. xOps — day-to-day operations, observability, and lifecycle management Deployment — getting workloads where they need to go, at the scale they need Delivery — traffic management across hybrid and multi-cloud environments Security — protecting applications and APIs from current threats Each architecture deployment example in this series is designed to cover at least three of the four core areas. This ensures the examples demonstrate how multiple components of the platform work together in practice, rather than showcasing any single feature in isolation. DevSecOps: Integrating security into the software delivery lifecycle is a necessary part of building and maintaining secure applications. This project incorporates DevSecOps practices by using supported APIs and tooling, with each use case including a GitHub repository containing IaC code, CI/CD integration examples, and telemetry options. In practice across the series, that means Terraform for infrastructure and F5 configuration, GitHub Actions as the pipeline runner, federated cloud identity instead of long lived keys, secrets handled through the platform rather than committed, and vulnerable target applications so the security controls can be exercised end-to-end. Contribute The repos are community-supported. Open an issue, send a PR, or port a use case to another cloud. Resources: F5 Application Delivery and Security Platform GitHub Repo and Automation Guide ADSP Architecture Article Series: Automating F5 ADSP Deployments (Intro) Automating F5 ADSP Deployments (Part 1 - F5 XC WAF and BIG-IP Adv. WAF) Automating F5 ADSP Deployments (Part 2 - F5 XC WAF and NGINX App Protect) Automating F5 ADSP Deployments (Part 3 - F5 XC API Protection and NGINX Ingress) Automating F5 ADSP Deployments (Part 4 - F5 XC BOT Defense and BIG-IP AdvWAF) Automating F5 ADSP Deployments (Part 5 - F5 XC, BIG-IP APM, CIS, and NGINX Ingress) Minimizing Security Complexity: Managing Distributed WAF PoliciesAutomating F5 ADSP — Part 2: F5 XC and NGINX for Delivery and Security
What this use case demonstrates This use case deploys NGINX Ingress Controller (NIC) running NGINX Plus with NGINX App Protect V5 (NAP V5) as the in-cluster data plane on GKE. WAF runs on two layers: NAP V5 enforcing inside the cluster, F5 Distributed Cloud (XC) enforcing at the edge. XC also provides API protection driven by an OpenAPI spec. It covers all four ADSP areas: Delivery, Security, Deployment, and xOps. Delivery: F5 Distributed Cloud HTTPS load balancer at the edge, NGINX Ingress Controller handling in-cluster delivery through the NIC VirtualServer CRD. Security: Two layers of WAF. NAP V5 runs as NIC sidecars (waf-enforcer and waf-config-mgr) and enforces the WAF policy attached to the VirtualServer. XC WAF runs at the edge in blocking mode. XC API protection is driven by an OpenAPI spec. Deployment: XC consumed as SaaS, GKE Standard with private nodes, NIC and NAP installed via OCI Helm chart, the application installed via a separate OCI Helm chart. xOps: NAP policy lives in config/uc2/nap/policy.json. The workflow compiles it with the NAP waf-compiler container, uploads the compiled bundle to GCS, and NIC mounts the bundle read-only via the GCS Fuse CSI driver. The waf-config-mgr sidecar watches the mount and pushes updates to the waf-enforcer. Change the policy, push, and NAP follows. Architecture What gets deployed: A GCP VPC with a dedicated k8s subnet (with secondary ranges for pods and services), management subnet, and NAT for private nodes A GKE Standard zonal cluster with private nodes and a control plane locked down by authorized networks NGINX Ingress Controller running NGINX Plus, with NAP V5 enforcer and config-mgr sidecars Comfy Capybara deployed via an OCI Helm chart, exposed through a NIC VirtualServer that references the waf-policy CRD in the nginx-ingress namespace An F5 Distributed Cloud HTTP load balancer with WAF and API protection. The origin pool is resolved from the NIC LoadBalancer IP via Terraform remote state. The VirtualServer attaches waf-policy both server-wide and on the /api route by default, so the policy enforces everywhere as a baseline. DevSecOps in practice for UC2 The lead-in covers the approach. For UC2, that means: Terraform handles infrastructure, the GKE cluster, NIC and NAP, the application Helm release, and all F5 Distributed Cloud objects. No click-ops. State lives in a GCS bucket the workflow creates on the first run, with a separate state file per module. The same bucket carries the compiled NAP policy bundle that NIC mounts via the GCS Fuse CSI driver. The XC origin pool reads the NIC LoadBalancer IP from state/uc2/nic, so no IP is pasted between configs. GitHub Actions runs the pipeline. Branch names trigger deployments, so git history shows what was meant to happen. GCP Workload Identity Federation replaces static service account keys for the runner. NIC pods also use Workload Identity to impersonate the runtime service account when mounting the NAP bundle from GCS. The XC API certificate, NGINX Plus JWT, and NGINX registry credentials live in GitHub Actions secrets, not the repo. The OpenAPI spec at config/uc2/app/oas/openapi.json is base64-encoded by the workflow and referenced inline by the XC API definition. Change the spec, push, and API protection follows. The pipeline Pushing to a branch runs the workflow. There is no manual terraform apply or helm install. Action Branch Validate, plan, and apply deploy-adsp-uc2 Validate only (no apply) test-adsp-uc2 Destroy all resources destroy-adsp-uc2 Modules deploy sequentially: state bucket - infra - GKE - compile NAP policy - NIC and NAP - app - XC. Destroy runs in reverse. What's in the repo f5devcentral/F5-ADSP-Automation: Directory Purpose infra/gcp/ VPC, subnets with pod and service secondary ranges, NAT, firewall k8s/gcp/ GKE Standard cluster and node pool f5/nic/gcp/ NGINX Ingress Controller and NAP V5 Helm release f5/xc/ F5 Distributed Cloud HTTP LB, WAF, API definition (shared with other XC use cases) app/gcp/ Comfy Capybara Helm release and VirtualServer config/uc2/gcp/env.json GCP, GKE, and NIC config config/uc2/nap/policy.json NAP policy source, compiled in the workflow config/uc2/app/env.json Application chart and VirtualServer config config/uc2/app/oas/openapi.json OpenAPI spec the XC API definition is built from config/uc2/xc/env.json XC tenant, LoadBalancer, WAF and API feature flags .github/workflows/ CI/CD workflows Prerequisites, secrets, and troubleshooting are in the UC2 deployment guide. Demo Try it Fork f5devcentral/F5-ADSP-Automation, set the secrets and tfvars from the deployment guide, and push to deploy-adsp-uc2. Push to destroy-adsp-uc2 to tear it down. Contribute Issues and PRs welcome at f5devcentral/F5-ADSP-Automation.F5 WAF on NGINX Gateway Fabric: Container-Native WAF for the Kubernetes Gateway API
The Kubernetes Gateway API is quickly becoming the standard for managing ingress and traffic routing — replacing the older Ingress resource with a more expressive, role-oriented model. But while the Gateway API provides platform teams with powerful traffic management, it comes with limited built-in application-layer security. It handles routing. Protection is not included. F5 WAF for NGINX Gateway Fabric closes that gap. It brings enterprise-grade WAF protection directly into the Kubernetes data plane — container-native, declaratively managed, and integrated with NGINX One Console for centralized policy management. In this article, I'll walk through the end-to-end workflow: creating a WAF policy in NGINX One Console, applying it to F5 NGINX Gateway Fabric, blocking a live XSS attack, and masking sensitive data with DataGuard — all without changing a single line of application code. Why WAF at the Gateway? Traditional WAF deployments were designed for monolithic architectures — sitting at the network perimeter, inspecting traffic before it reached a handful of servers. In Kubernetes, applications are distributed across dozens or hundreds of microservices, each with its own API surface. The perimeter model doesn't scale. F5 WAF for NGINX Gateway Fabric embeds security directly into the data plane. Every HTTPRoute behind the gateway gets WAF protection — declaratively, through the same Kubernetes-native workflows your platform team already uses. Key capabilities: Container-native — runs natively with NGINX Gateway Fabric, no sidecar required OWASP Top 10 protection with advanced attack signatures REST, GraphQL, and gRPC API protection DataGuard — automatic PII masking in API responses Centralized management through NGINX One Console NGINX One Console Integration The integration between NGINX Gateway Fabric and NGINX One Console is what makes this solution practical for real-world operations. It creates a clean separation between security policy ownership and application deployment. Policy Management SecOps teams create and manage WAF policies entirely within NGINX One Console — using the visual form editor or the JSON tab for advanced configurations. Policies are defined once and consumed by any number of gateways. There's no need for SecOps to touch Kubernetes resources or learn YAML. The console supports policy presets (like NGINX Strict for aggressive blocking), custom violation rules, attack signature set configuration, and application-specific exceptions. Once a policy is saved, it compiles instantly and generates an Object ID that platform teams reference from Kubernetes. Security Dashboard Every WAF event — blocked attacks, DataGuard alerts, policy violations — flows back to the Security Dashboard in real time. SecOps gets full visibility without leaving the console. The dashboard surfaces: Attack types, blocked requests, and violation counts per policy Top attack signatures — including XSS, SQLi, and bot patterns Attacker IP addresses and targeted endpoints Event drilldown with raw request payloads and signature match details For incident response, the Event Logs tab provides full forensic detail — the raw request, violation rating, matched signatures with accuracy and risk classification, and the support ID returned to the attacker. Log Profiles Security logging is configured as part of the WAF policy configuration. Logs are sent back to NGINX One Console using the secops_dashboard log profile, which feeds the Security Dashboard. This means every gateway running the policy contributes to the same centralized view — no separate log aggregation needed. Automatic Policy Updates When SecOps updates a policy — adding a new signature set, tightening a violation rule, or enabling DataGuard — NGINX Gateway Fabric detects the change automatically. The updated policy propagates to every gateway that references it. No pod restarts, no redeployments, no manual sync. This is the operating model: SecOps owns the security posture in the console. Platform teams consume it declaratively in Kubernetes. Both teams work independently without stepping on each other. How It Works — The WAFPolicy Resource The connection between NGINX One Console and NGINX Gateway Fabric is established via a WAFPolicy custom resource. This is the only Kubernetes resource the platform team needs to apply. The WAFPolicy does three things: Points the gateway to the specific policy created in NGINX One Console (by Object ID) Provides secure credentials to pull the policy (via a Kubernetes secret) Routes security events back to the console for the Security Dashboard apiVersion: gateway.nginx.org/v1alpha1 kind: WAFPolicy metadata: name: gateway-base-protection spec: targetRefs: - group: gateway.networking.k8s.io kind: Gateway name: gateway type: N1C policySource: n1cSource: url: https://your-console.example.com namespace: your-namespace policyObjectID: pol_xxxxxxxxxxxxx auth: secretRef: name: n1c-credentials securityLogs: - destination: type: syslog syslog: server: localhost:1514 logSource: n1cSource: url: https://your-console.example.com namespace: your-namespace profileName: "secops_dashboard" auth: secretRef: name: n1c-credentials Once applied, the gateway status confirms the policy is active: $ kubectl describe wafpolicy gateway-base-protection ... Status: Conditions: Message: The Policy is accepted Reason: Accepted Status: True Message: Policy is programmed in the data plane Reason: Programmed Status: True All traffic through the gateway is now inspected by F5 WAF. Demo Walkthrough The video below demonstrates the complete workflow — from policy creation to attack blocking to DataGuard masking. Here's a summary of what you'll see. Resources NGINX Gateway Fabric documentation — https://docs.nginx.com/nginx-gateway-fabric/ F5 WAF for NGINX Gateway Fabric Blog: https://blog.nginx.org/blog/nginx-gateway-fabric-2-6-f5-waf-for-nginx-comes-to-the-gateway-api NGINX One Console — https://docs.nginx.com/nginx-one/ GitHub — NGINX Gateway Fabric — https://github.com/nginx/nginx-gateway-fabricDeploying F5 BIG-IP HA into AWS GovCloud
This guide provides a walk-through for deploying an active/standby F5 BIG-IP High Availability (HA) cluster within AWS GovCloud (US) using the AWS Console. Code Repository: F5GovSolutions/f5-aws-cloudformation-v2-govcloud While the repository's examples/failover/GOVCLOUD-GUIDE.md handles the exact AWS CLI workflows, this document serves as the web console companion. The Partition Problem The upstream commercial F5 templates (F5Networks/f5-aws-cloudformation-v2) assume a standard AWS commercial partition. Pointing those unmodified templates at GovCloud breaks deployment for three distinct reasons: Partition Isolation (aws-us-gov): GovCloud uses unique ARNs, regional endpoints, and an entirely separate AMI catalog. Furthermore, CloudFormation mandates that a parent stack's nested templates reside in an S3 bucket within the exact same region and partition. CloudFormation cannot fetch objects across partitions, meaning all templates must be pre-staged in a local GovCloud bucket. Air-Gapped Realities: In isolated enclaves with no internet egress, a booting BIG-IP cannot pull the runtime-init installer or Automation Toolchain RPMs from public GitHub repositories or F5 CDNs. Everything must be hosted locally inside an S3 bucket, accessible via VPC Gateway and Interface endpoints. The 17.x Clustering Bug: On the BIG-IP 17.x code train, a startup-timing condition occasionally prevents the local device-trust domain (/Common/Root) from initializing properly on first boot. When this occurs, Declarative Onboarding (DO) cannot establish the trust domain or failover group, leaving both nodes deadlocked. This modified template embeds an automated, non-blocking self-heal script to orchestrate the recovery process without manual intervention. What it deploys The root template (failover.yaml) orchestrates a series of nested module stacks (network, access, ingress, application, and individual BIG-IP nodes) to build: An active/standby pair of BIG-IP Virtual Editions (VE) deployed across two Availability Zones using a 3-NIC topology (Management, External, Internal). Cloud Failover Extension (CFE) integration to re-map floating application IPs and AWS routes during a failover event. Automated onboarding via F5 BIG-IP Runtime Init, executing Declarative Onboarding (DO) for core network clustering, Application Services 3 (AS3) for virtual servers/WAF policies, and CFE for AWS API integration. The architecture uses Pay-As-You-Go (PAYG) marketplace licensing and is fully validated on the 3nic-payg...-with-app runtime-init configuration. Core Architecture Differences vs. Commercial Staged-Bucket Deployment Mode Because of partition constraints, your local GovCloud S3 bucket serves as the single source of truth. It must hold two types of data: the nested CloudFormation templates cloned from the repository, and the target installation binaries (the runtime-init .run installer and the accompanying DO/AS3/CFE RPM packages). The Unauthenticated 403 Pitfall CloudFormation reads your deployment templates using your active IAM user/role credentials. However, during the bootstrap phase, the individual BIG-IP instances download their configuration files and RPM packages over an unauthenticated HTTPS request. They do not sign these requests with AWS credentials. If your S3 bucket permissions block public read access completely, the instances receive an HTTP 403 Forbidden response, initialization fails, and CloudFormation rolls back the stack. Because GovCloud blocks public access by default, you must apply a scoped bucket policy allowing s3:GetObject to Principal: "*" paired with a DenyInsecureTransport block. Single-Toggle VPC Endpoints Isolated enclaves require four distinct VPC endpoints to function: S3 (Gateway) for fetching installation packages and AS3 WAF policies, and Interface endpoints for EC2, Secrets Manager, and CloudFormation APIs. Missing even one endpoint causes silent automation failures. To simplify this, the template consolidates these requirements into a single parameter: setting provisionS3Endpoint=true automatically provisions all four regional endpoints with private DNS enabled. Note that your bucket, stack, and endpoints must reside within the same AWS region. Automated Clustering Self-Heal To bypass the 17.x initialization bug without manual operator access, the template installs a localized orchestration loop during the pre_onboard hook. The mechanism drops three files onto the file system: cluster-heal.sh: The main orchestrator script. cluster-heal-trust.py: A native Python script that securely pulls the admin password from Secrets Manager via a SigV4 request using the instance's IAM role, then invokes the local device trust commands. /etc/cron.d/cluster-heal: A cron job that triggers the orchestrator every 3 minutes. This loop checks the cluster status. If it detects a missing root domain, it triggers a single controlled reboot, reads the peer node's address from the runtime logs, pulls the password, and establishes device trust. Once synchronized, it issues the CloudFormation success signal and disables its own cron job. This preserves DO as the declarative source of truth without creating configuration drift. During this process, the CloudFormation stack will remain in CREATE_IN_PROGRESS for roughly 25 to 30 minutes, which is normal behavior. Elastic IP Allocation Constraints The default AWS regional quota is 5 Elastic IPs (EIPs). Depending on your topology, this architecture can request up to 7. To prevent deployment failures due to exhausted quotas, use the template toggles to limit public allocations: Default Configuration (Public Mgmt, Public VIP, Public Self IPs): 7 EIPs Standard Air-Gap Profile (Public Mgmt, Private VIP, Private Self IPs): 4 EIPs Strict Isolated Profile (Private Mgmt via Bastion, Private VIP, Private Self IPs): 3 EIPs Verify your regional limit under Service Quotas → EC2-VPC Elastic IPs (L-0263D0A3) before executing the template. Deployment Steps (AWS Console) Before You Begin Select a single target GovCloud region (us-gov-east-1 or us-gov-west-1). Your staging bucket, CloudFormation stack, EC2 Key Pair, and Secrets Manager secret must share this region to prevent cross-region routing failures in isolated enclaves. 1. Provision the staging S3 bucket S3 → Create bucket, in your Region. Give it a globally unique name. [S3 Create bucket, Region selector highlighted.] 2. Stage the templates and artifacts While file uploads can be handled via the S3 web GUI, staging is more reliably managed via the AWS CLI from a local workstation. Installing the AWS CLI is outside the scope of this article but it's easy and instructions can be Googled. Execute the following commands from the root of your cloned repository to sync the template architecture and upload your pre-downloaded F5 binaries: Download commands: # F5 BIG-IP Runtime Init installer (note: this repo's tag has NO "v" prefix) curl -fL -o f5-bigip-runtime-init-2.0.3-1.gz.run \ https://github.com/F5Networks/f5-bigip-runtime-init/releases/download/2.0.3/f5-bigip-runtime-init-2.0.3-1.gz.run # Declarative Onboarding (DO) curl -fL -o f5-declarative-onboarding-1.47.0-14.noarch.rpm \ https://github.com/F5Networks/f5-declarative-onboarding/releases/download/v1.47.0/f5-declarative-onboarding-1.47.0-14.noarch.rpm # Application Services (AS3) curl -fL -o f5-appsvcs-3.56.0-10.noarch.rpm \ https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.56.0/f5-appsvcs-3.56.0-10.noarch.rpm # Cloud Failover Extension (CFE) curl -fL -o f5-cloud-failover-2.4.0-0.noarch.rpm \ https://github.com/F5Networks/f5-cloud-failover-extension/releases/download/v2.4.0/f5-cloud-failover-2.4.0-0.noarch.rpm Sync and upload commands (be sure to be at root of the cloned repo), set the variables accordingly: # Synchronize the nested CloudFormation infrastructure templates aws s3 sync ./examples/ "s3://$BUCKET/$PREFIX/" --region "$REGION" # Upload the core runtime engine and toolchain extensions aws s3 cp f5-bigip-runtime-init-2.0.3-1.gz.run "s3://$BUCKET/$PREFIX/" --region "$REGION" aws s3 cp f5-declarative-onboarding-1.47.0-14.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION" aws s3 cp f5-appsvcs-3.56.0-10.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION" aws s3 cp f5-cloud-failover-2.4.0-0.noarch.rpm "s3://$BUCKET/$PREFIX/bigip-extensions/" --region "$REGION" 3. Grant anonymous read on the artifacts In the S3 Console, select your staging bucket and navigate to Permissions. Under Block public access (bucket settings), disable the option to Block public access to buckets and objects granted through new public bucket policies (leave ACL blocks enabled). Next, apply the following bucket policy to allow the instances to pull down the bootstrap packages securely over HTTPS: { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE/*" }, { "Sid": "DenyInsecureTransport", "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": [ "arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE", "arn:aws-us-gov:s3:::YOUR-BUCKET-NAME-HERE/*" ], "Condition": { "Bool": { "aws:SecureTransport": "false" } } } ] } Note: Ensure you replace YOUR-BUCKET-NAME-HERE with your actual bucket name. Do not include any trailing lines or comments in the JSON editor. [S3 Permissions — Block public access settings.] 4. Store the Admin Password Secret Navigate to AWS Secrets Manager → Store a new secret. Choose Other type of secret, select the Plaintext tab, clear the default JSON template, and type your raw password string. Name the secret (e.g., f5-bigip-admin) and save it using the default settings. Copy the resulting Secret ARN (arn:aws-us-gov:secretsmanager:...); both nodes query this specific object at startup. [Secrets Manager Store secret, plaintext tab.] [Secrets Manager: Name your secret.] 5. Create the SSH key pair EC2 → Key Pairs → Create key pair, download the .pem. The key pair is regional. This is optional but recommended you create one before template launch. [Screenshot: EC2 Create key pair.] (You can leave the parameter blank in the template and let the stack auto-create one, but then the private key only lives in SSM Parameter Store — pre-creating is simpler if you need to access the boxes vis SSH) 6. Confirm the BIG-IP AMI exists in your Region The deployment template discovers the target AMI using automated string matching. Ensure that your region has access to the target image by running the following lookup via your terminal, this can also be done via the GUI: aws ec2 describe-images --region "$REGION" --owners aws-marketplace \ --filters "Name=name,Values=*17.5*PAYG-Best Plus 25Mbps*" \ --query 'reverse(sort_by(Images,&CreationDate))[].[Name,ImageId,CreationDate]' --output table 7. Launch the stack (CloudFormation) Open CloudFormation, select your target region, and click Create stack (with new resources). Select Template is ready, choose Amazon S3 URL, and paste the direct object URL to your staged failover.yaml file. Configure the mandatory parameters: Stack Name: Define an environment-specific identifier. S3 Configurations: Populate s3BucketName and s3BucketRegion. Secrets Management: Paste your copied Secrets Manager ARN into bigIpSecretArn. Security Access: Set restrictedSrcAddressMgmt and restrictedSrcAddressApp to your administrative CIDR blocks to avoid exposing management interfaces. Air-Gap Toggles (For isolated VPCs): Change provisionS3Endpoint to true, and flip both provisionPublicIpExternalSelf and provisionPublicIpVip to false. In the options screen, consider setting Stack failure options to Preserve successfully provisioned resources during your initial test runs. This keeps instances alive for log inspection if a deployment fails. Acknowledge the CAPABILITY_NAMED_IAM prompt and click Submit. [CloudFormation Specify template — Amazon S3 URL field.] [Parameters page, air-gap toggles section.] [Review — Acknowledgement and Stack failure options.] 8. Wait it out Expect ~25–30 minutes in CREATE_IN_PROGRESS while the self-heal forms the cluster, then CREATE_COMPLETE. This is normal — it is not stuck. After 10 minutes, go ahead and admin SSH into the boxes, they should accept your admin secret by now, if not use your SSH key you created and referenced in the template. See the commands below or in the git repo guide. Validating the deployment From the stack Outputs, grab a management IP, then: Onboarding: grep -i 'All operations completed successfully' /var/log/cloud/bigIpRuntimeInit.log; the prompt should show failover01.local / failover02.local, not ip-x-x-x-x. Clustering: tmsh show cm sync-status → In Sync (green), Mode: high-availability, one node Active and one Standby. CFE: GET https://localhost/mgmt/shared/cloud-failover/inspect returns a populated object (instance, addresses, trafficGroup). The self-heal's own narration lives in /config/cluster-heal/log — the primary place to look if the stack runs long; you'll see the reboot → add-to-trust → creating failoverGroup → cluster In Sync → cfn-signal sent OK → disabling self-heal sequence. If clustering genuinely never forms (the rare case where the self-heal exhausts its attempts), the repo guide has a validated manual fallback that follows the same reboot → rebuild trust → reapply sequence. What's GovCloud-ready today — and what isn't Let's be clear about scope. The GovCloud template— staged-bucket defaults, the four VPC endpoints, the clustering self-heal, the bumped extensions — currently applies to one solution: the failover active/standby pair on the 3-NIC PAYG -with-app runtime-init config. The repo's autoscale, the quickstart standalone, and the other failover variants (2-NIC, BYOL, and the non--with-app configs) still carry upstream configuration and are not yet GovCloud-validated. Adapting them is planned work and would follow exactly the pattern documented here. And the self-heal is a workaround for an F5 platform bug, not a permanent fix. The right long-term resolution is the platform defect behind the KB; when a fixed build is in play, the self-heal simply stays inert. Where to go next The repo: F5GovSolutions/f5-aws-cloudformation-v2-govcloud Full CLI walkthrough (first-time operator): examples/failover/GOVCLOUD-GUIDE.md Parameter and architecture reference: examples/failover/README.md The self-heal sources: examples/failover/bigip-configurations/cluster-heal.sh and cluster-heal-trust.py If you're standing up BIG-IP HA in an enclave, start from the 4-EIP air-gap profile with provisionS3Endpoint=true, stage everything in one in-Region bucket, and let the self-heal do the clustering. File issues on the repo if you hit something the guide doesn't cover.log local0. is not a debugging strategy!
But let's be honest...with iRules, it's pretty much all we've had. If you have ever debugged an iRule, you know the ritual. Former F5er Jibin_Han in an article once called the log command is "the crudest of debug facilities." He was not wrong. It is the printf of application delivery, and at high traffic the logging pipeline will happily truncate your output just to keep things spicy. So back in TMOS 13.1, he shipped something much better: ltm rule-profiler. It is a passive tracer baked right into TMM. (Did you know this? Most don't, including most F5ers I talk to!) You do not touch your script. You tell it which virtual server, events, and occurrences to watch, you flip it on, and it emits a timestamped execution trace of everything your iRule did. Events firing, rules matching, the Tcl VM grinding through bytecode, native commands getting dispatched, variables changing. The whole shebang. There is just one catch. Ok actually two. It is tmsh-only. No GUI, no REST endpoint, no nothing. You configure it by hand and you start and stop it by hand. The not so fun part, the output looks like this: 1780079189187194,RP_EVENT_ENTRY,/Common/testvip-http,CLIENT_ACCEPTED,22623,0x70373707000576,10.1.10.6,36086,0,10.1.10.50,80,0 1780079189187210,RP_RULE_ENTRY,/Common/testvip-http,/Common/myrule,22623,0x70373707000576,... 1780079189187225,RP_RULE_VM_ENTRY,... 1780079189187240,RP_CMD_BYTECODE,/Common/testvip-http,push1,... Now multiply that by a few hundred lines, sprinkle in microsecond timestamps you are expected to subtract in your head, and remember that every single custom iRules command is actually a round trip out of the Tcl VM and back into TMM. A command, inside a VM, inside the microkernel. We need to go deeper. Reading it raw is less "performance analysis" and more "staring at the green rain in The Matrix and pretending you can see the woman in the red dress." The data is genuinely great. It's just wearing a CSV trench coat and refusing to make eye contact. But that ends now. Let's talk about Rültracer. What Rültracer is Rültracer is an iApps LX extension that gives ltm rule-profiler the face it always deserved. When Jibin_Han released his 3-part article series (linked at the bottom of this article) introducing the rule-profiler, we had a couple interns who built an analysis engine called Campfire that used a perl-based flamegraph package to display the trace in context of the "lift" of each occurence within a trace. The challenge was you had to manually configure everything on BIG-IP, send the logs somewhere, collect those logs, then import them into where you built campfire to run. A lot of manual work to get to the good. But Rültracer? It runs on the BIG-IP. An on-box Node worker handles the unglamorous parts: it configures the profiler, sets up (and tears down) the log publisher, captures the trace stream into a per-session file, and serves that file to a browser app. The browser does all the parsing and visualization client side, so the box just ships raw CSV and gets out of the way. What you get: A sequence diagram and step-through debugger. The trace becomes a UML-style sequence diagram across six lifelines (Users, Event, Rule, Rule VM, Command VM, Command), with the TMM and Tcl VM sides color coded so you can actually see every handoff between the microkernel and the interpreter. Which matters, because that back-and-forth is exactly where iRule inefficiency likes to hide. Next to it sits a linked step-through with a timeline scrubber, variable state, and command replay. Click the diagram, the table follows. Click the table, the diagram follows. There's a Tclsh disassembler you can enable as well, and as long as you're willing to modify the parts of the iRules code under test. I wrote an article years ago on this disassembly functionality, it's worth the read to see what this functionality affords you. iRule source mapping. It pulls your actual iRule source and lights it up: which commands fired (with microseconds and counts), which branches never ran, and which lines were ambiguous multi-matches. Your code, annotated by what the trace really did. (this part is early stages, it needs work.) Flamegraphs, with diff. An interactive flamegraph where width equals inclusive time, so the widest frame is your prime suspect. Find the slow command without playing Where's Waldo. Captured a "before" and an "after"? The diff view paints frames red and blue by how their self-time shifted, so you can prove your optimization actually optimized. Cycles versus CPU. Rültracer takes the box's own ltm rule stats hardware cycle counters and turns them into honest performance tables: cycles to microseconds, percent of a CPU per request, and max requests per second before your iRule becomes the bottleneck. It even reconciles the authoritative cycle counts against the trace-derived numbers, and the gap between them is the profiler's own overhead. These are numbers you can bring to a capacity-planning meeting without getting laughed out of the room. This is based on Deb Allen's yesteryear work in excel for computing capacity with iRules and my update in doing this with python much later. Reports and exports. Export a capture as self-contained HTML, JSON, a Mermaid sequence diagram, or Brendan Gregg folded stacks. Share it, attach it to a ticket, or feed it to your own tooling. Multi-TMM aware. Captures that span multiple TMMs get partitioned by context id with a scope selector, because of course your traffic did not politely land on a single TMM. No (post RPM install) build steps, no cloud, no telemetry, no agent. Vanilla JS in the browser, a small ES5 worker on the box, fully self-contained. And it is a lab tool on purpose: tracing adds significant TMM overhead, so this is not something you run in production. Rültracer tears the profiler and publisher down for you on teardown, so you never accidentally leave the tap open. Installing it Rültracer installs over SSH. You build the RPM on your workstation, copy it to the BIG-IP, and run the installer on the box as root. The installer provisions the persistent data directory, installs the package through the iApps LX framework, runs the post-install step, and confirms the workers came up. Replace and with your BIG-IP's SSH host and port. If it answers on plain old port 22, drop the -P / -p flags. First time on a fresh box The installer script lives outside the RPM, so it rides along once: Copy the installer on the box (one time only) scp -O -P <port> build/install-onbox.sh root@<host>:/shared/images/ Build, ship, and install ./build/build-rpm.sh 0.7.1 0001 scp -O -P <port> build/dist/rultracer-0.7.1-0001.noarch.rpm root@<host>:/shared/images/ ssh -p <port> root@<host> /shared/images/install-onbox.sh 0.7.1-0001 Because the installer runs as root, it creates the session data directory owned by the restnoded worker user before the workers start (the worker is uid 198 and cannot create directories under /shared/ on its own). When it finishes, it prints your UI URL: https://BIG-IP-host/mgmt/shared/rultracer/ui/ Open that, and you are in. Follow-on updates The installer is already on the box, so the next rounds are just bump, build, ship, install like above. That is an in-place upgrade and it keeps your saved sessions. If you ever want a clean slate, pass --reinstall, but note it wipes session data, so hit the Sessions tab's "Download backup" button first if you care about what is in there. See it in action (The walkthrough covers a live capture end to end: pointing the profiler at a virtual server, driving a little traffic, then digging through the sequence diagram, flamegraph, and cycle stats on a real trace.) Conclusion This was a fun project to bring together something I've played with a lot since the v13.1 release but could never quite figure out a packaging solution to make it functional enough to rely on. You can find the code in the Rültracer repo on Github. Let me know in the comments if you take a look at this and submit any bugs ore feature requests as an issue out on Github.
Implementing Agentic Observability and Security
In April 2026, an AI coding agent named PocketOS wiped out a company's production database and its backups in nine seconds flat, executed through a single API call. Afterward, the agent described its behavior with unsettling bluntness: it ran a destructive action it was never explicitly asked to run, choosing to guess rather than verify. The detail that should keep every engineer awake at night isn't the deletion itself. It's that the team actually had a log of the action—the cloud provider recorded the destructive API call perfectly—but they had absolutely zero record of the reasoning that sparked it. By the time the deletion hit the logs, there was nothing left to salvage. That visibility gap is what we need to solve. Autonomous agents are hitting production environments faster than security teams can vet them, and they fail in ways traditional application security simply wasn't designed to intercept. Safely running them boils down to two critical capabilities: observability (seeing what an agent is deciding to do) and guardrails (restricting what it is allowed to do). You can't skip either one. If you want to simulate this exact failure mode on your own hardware, we built a hands-on lab called agent-security-lab. It spins up a five-agent SOC incident-response team, drives it rogue under realistic conditions, and hardens it step-by-step. I'll link to it at the end, but the takeaway here is that you don't have to take these risks on faith—you can test the defenses yourself. Agents fail differently Traditional software does exactly what the source code dictates. Its behavior is locked in at build time, meaning your gateways, logs, and audit trails are built around a static assumption: they record what happened because "what happened" belongs to a pre-defined, knowable set of outcomes. An agent behaves entirely differently. Its actions are decided at runtime by an LLM interpreting whatever text it was just handed. It maps out plans, chooses tools, spins up subordinate agents, and executes highly consequential actions without a human approving the intermediate steps. You cannot know the full scope of its actions in advance because its behavior depends entirely on inputs that an attacker might control. The most dangerous failure mode is the one that looks like a successful execution. In the PocketOS database incident, the agent wasn't acting maliciously—it was just trying to be helpful. It read a prompt stating "the table is corrupt, fix it," and it chose the most absolute resolution the prompt implied. If you are only looking at a log of side effects, a helpful-but-catastrophic action and a correct one look identical until you read the actual reasoning behind them. This is why model observability isn't just a compliance line-item; it is the only mechanism you have to differentiate between an intended fix and a disaster. Observability: Capturing decisions, not just the damage For autonomous systems, true observability requires deep visibility into the agent's chain of thought. You need to see the exact prompt it received, the conclusions it drew, the specific tools it selected, the exact arguments it generated, and how it interpreted the resulting data. This goes far beyond standard application logging, which merely timestamps the damage after the fact. The most practical architecture for capturing this data is routing all model traffic through an explicit security proxy, rather than letting agents call model endpoints directly. On the F5 AI Security platform, this is handled via our Bring Your Own Agent (BYOA) architecture and surfaced as Agentic Fingerprints. This gives you a per-agent, per-run audit trail of prompts, reasoning steps, and tool choices. Crucially, the proxy captures the model's decision independently of whether the downstream action succeeds. If an agent decides to drop a table, you see that intent and the verbatim arguments it formulated before the request ever reaches your database. Figure 1. An Agentic Fingerprints session: the agent's system prompt, the input it received, and its decision to call a destructive tool with verbatim arguments — captured before the action runs. However, there is a structural boundary here that teams often miss, leading to a false sense of security. In any tool-using agent, there is a specific line where a model's decision transitions into a physical action: the moment the runtime executes the tool the model selected. The proxy handles everything on the model-facing side of that line (the request, the reasoning, the tool request, and the model's reaction to the return payload). It cannot see the tool actually run. That execution happens on an entirely separate data path—from the agent to the tool server to your identity provider, governed by its own authorization checks and logs. Because of this, complete agent observability must span two separate layers: The Model-Facing Layer: This tracks what the agent concluded, which tool it wanted, and how it handled the return data. You monitor this via F5 AI Security (BYOA / Agentic Fingerprints views). The Tool-Facing Layer: This tracks which tool calls actually executed, which were blocked, and the specific permissions carried by that credential. You monitor this through your Tool/MCP server logs and identity-provider audit trails. Relying on either layer alone leaves a blind spot. The PocketOS incident occurred because the team only had tool-facing visibility—the cloud provider caught the destructive call—but they lacked the model-facing context to see the decision forming. You need to wire both layers together and correlate their logs before putting an autonomous agent anywhere near production data. Implementation is a configuration change The good news is that adding model-facing observability doesn't require rewriting your agentic framework; it’s a simple configuration adjustment. You just need to point your agent's model client to the proxy and tag the outbound traffic with a unique session identifier. Here is how you would configure a standard OpenAI-compatible client in Python: import datetime import uuid from openai import OpenAI # Generate a unique session ID for every single run. # Reusing IDs will merge separate runs into an unreadable trail. # Recommended format: {run_id}-{agent_name}-{timestamp}-{unique_tail} session_id = "-".join([ LAB_RUN_ID, # Shared across all agents in a single execution run AGENT_NAME, # Identifies the specific agent datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ"), uuid.uuid4().hex[:6] ]) # Initialize the client, routing traffic through the F5 AI Security proxy client = OpenAI( api_key=CALYPSOAI_TOKEN, # Your F5 platform token, not the raw LLM key base_url=CALYPSOAI_OPENAI_API_BASE, # Redirects requests through the proxy default_headers={ # Tags the traffic to this specific agent's audit trail "x-cai-metadata-session-id": session_id, } ) The first two parameters redirect the outbound calls through the security proxy, while the third converts raw LLM traffic into an organized audit trail. By assigning unique session IDs, the platform groups a single agent's calls into a coherent narrative rather than overlapping traffic from different processes. Without the uniqueness of each call/session, it would be impossible to isolate conversations. In a multi-agent system, this setup allows you to trace an entire workflow as it crosses boundaries: each agent generates its own session ID under a shared run prefix, letting you piece together the entire decision path from end to end. With just a few lines of configuration, you get full visibility across your environments, mapping directly to the telemetry shown in our platform dashboards. Work these standards into your governance frameworks, every agent deployed should have observability baked in. Designing multi-layered guardrails Observability gives you eyes, but it doesn't stop an action mid-flight. For enforcement, you need guardrails. Experience with production failures shows that a single defensive layer will eventually crack. You need independent, decoupled boundaries where each layer catches a different class of failure and fails closed by default. A robust agent defense architecture requires five distinct checkpoints: Content Scanning (Proxy Input/Output Scanners): Blocks poisoned or injected prompts before they hit the model. This is your primary defense against completely novel prompt injection techniques. Capability Scoping (Tool Server Manifests): Restricts the agent's environment so it can only see or select the specific tools required for its role. Authorization Scope (OAuth 2.1 / Per-Call Checks): Evaluates short-lived tokens to ensure that even if an agent requests a tool, its current credential actually possesses the authorization to run it. Agent-to-Agent Control (Signed Agent Cards): Governs multi-agent environments by validating whether one agent is permitted to invoke the skills of another, enforced at the receiving end. Underlying Privilege (Downstream Target Systems): The final backstop where the actual resource (database, identity provider, mail relay) enforces strict least-privilege access on the agent's service account. When designing these layers, focus on the difference between reactive and structural defenses. An authorization scope check is reactive: the agent attempts a destructive call, and the platform intercepts and denies it mid-flight. Capability scoping, on the other hand, is structural: the dangerous tool is completely omitted from the agent's available schema definition. Because the model doesn't even know the tool exists, it can't choose it in the first place, completely removing the attack vector. You should implement both, but structural defenses are far more resilient against novel bypasses because they actively shrink what the agent is capable of doing. It's also worth noting that content scanning is the only layer capable of identifying semantic attacks that authorization policies miss. A prompt injection payload hidden inside a customer support ticket or a scraped web page will easily bypass credential checks because, from an infrastructure perspective, reading that data is perfectly valid. Input scanners inspect the text itself before it reaches the LLM to verify what the agent is being asked to do. When a scanner intercepts an attack, the F5 Logs view isolates the specific rule that triggered the block and records the offending payload—giving you a clean, shareable artifact to explain the security decision to external stakeholders without forcing them to dig through raw system telemetry. Figure 2. The Logs view after a guardrail block: the scanner that fired, its verdict, and the content that triggered it — the explanation artifact for an automated decision. Visibility and enforcement must coexist It is common to view observability and guardrails as separate boxes to check on a security review. In practice, they are two sides of the exact same coin. A guardrail without observability requires blind faith; you can't validate that it's firing correctly, you can't tune it when it blocks legitimate business traffic, and you can't explain its decisions to your team. Conversely, observability without guardrails just gives you a pristine, high-resolution recording of a security breach you did nothing to stop. The deployment order matters here. Always deploy observability first. You cannot confidently evaluate or fine-tune a security policy you can't actively watch. Establish your visibility pipelines, verify that you can accurately map the agent's logic alongside its tool executions, and then begin layering your guardrails on top—using your new telemetry trails to validate each security layer as you turn it on. Next Steps If you are currently deploying or managing autonomous agents in production, you can lock down your environment with four immediate steps: Route all LLM traffic through a dedicated proxy to establish an immutable, server-side record of agent reasoning, rather than relying on standard application logs. Enforce unique session tagging across every agent run to keep your audit trails clean and searchable. Instrument both model-facing and tool-facing logs, ensuring you have a clear correlation key between the two. Layer your enforcement mechanisms sequentially: swap out static API keys for short-lived, scoped credentials; apply tool capability scoping; enforce agent-to-agent verification; and place content scanners in front of every ingress point. The most effective way to understand these interactions is to deliberately break them in a safe sandbox environment. Our agent-security-lab recreates the PocketOS database failure against a sandboxed environment, surfaces the agent's reasoning in real-time, and guides you through configuring OAuth 2.1 scoping, MCP capability scoping, agent cards, and F5 AI Guardrails to stop the exploit. The lab is built entirely for practitioners and provides all the required source code. Note that because the exercises interface with the F5 AI Security platform, you will need access to a platform tenant—if you need to set one up for your team, just reach out to your F5 account representatives to get a lab instance provisioned. Agents are the norm now, you are either planning for agentic AI or have already deployed them, its only a matter of time and observability is the blindspot. For autonomous agent architectures, deep observability forms your foundation, and multi-layered guardrails provide the structure to scale safely. References and further reading agent-security-lab (F5GovSolutions) — hands-on multi-agent security lab: github.com/F5GovSolutions/agent-security-lab F5 AI Security platform documentation: docs.aisecurity.f5.com F5 AI Guardrails: f5.com/company/blog/what-are-ai-guardrails F5 AI Explainability / Agentic Fingerprints: f5.com/company/blog/ai-explainability F5 Labs CASI and ARS leaderboards: f5.com/company/labs/casiHTTP Load Balancer Routes on F5 Distributed Cloud
Route misconfiguration is one of the most common configuration mistakes we see that can cause incidents on F5 Distributed Cloud (F5 XC). The four route types look deceptively simple in the console, but they have distinct behaviors, ordering rules, and gotchas. This article covers all four types with real field names, decision guidance, and the mistakes that actually happen in production. What Routes Do in F5 XC HTTP Load Balancer An HTTP Load Balancer in F5 XC is a full L7 proxy running at the Regional Edge (RE) and depending on the deployment model, Customer Edge (CE). We will use the Regional Edge as a deployment model for this article. When a request arrives, the RE evaluates the route list in order and applies the first matching route. That route determines what happens to the request: forward it to an origin, redirect the client, return a synthetic response, or apply advanced routing logic. Routes can be configured inside the HTTP Load Balancer configuration which opens a new Route Options window: Multi-Cloud App Connect > Load Balancers > HTTP Load Balancers > [your LB] > Routes > Route Options The four route types map to three underlying route actions: XC Route Type Route Action Traffic Goes To Simple Route route Origin Pool Redirect Route redirect Client (3xx response) Direct Response Route direct_response Client (fixed response body) Custom Route route / redirect / direct_response Depends on configuration Route Matching: How XC Evaluates Routes Route evaluation is sequential, stops on first match, and has no automatic specificity ranking. The order you set is the order XC uses. Evaluation Order The HTTP Load Balancer evaluates routes sequentially, top to bottom. The first route that matches the incoming request wins. No further routes are evaluated. This means: More specific routes must appear before broader ones. A catch-all route (prefix /) at the top will swallow everything. Nothing below it will ever match. Path Match Types Three path match types are available across all route types: Match Type Field Behavior Prefix path_prefix Path must begin with the specified string Exact exact Path must equal the value exactly (query string excluded) Regex regex Entire path (minus query string) must match the regex pattern Prefix matching pitfall: The prefix /api matches /api/v1/users but also /apikeys and /api-internal. If you want to match a path segment boundary, use /api/ (trailing slash) or switch to regex. Additional Matching Criteria Beyond path, routes can match on: HTTP methods: GET, POST, PUT, DELETE, etc. Request headers: presence, exact value, regex Query parameters: Retain, Remove, or Replace Combining criteria (e.g., path prefix + method + header) creates an AND condition: all specified criteria must match. Route Type 1: Simple Routes Simple routes are the workhorse of most HTTP Load Balancer configurations. They match a path (and optionally method/headers) and forward traffic to an Origin Pool. When to Use Standard application traffic forwarding Path-based routing to different backend services API versioning (/v1/ → pool A, /v2/ → pool B) Microservice fanout from a single domain Key Configuration Fields Field Description Path match type Prefix / Exact / Regex HTTP Method Any, GET, POST, PUT, etc... Origin Pool The backend pool receiving the request Host Rewrite Method Disable/ Hostname / Header value: rewrites the Host header sent to origin Query Parameters Retain, Remove, Replace Advanced Options Worth Knowing Path rewriting (under Advanced Options): Disabled: path sent to origin unchanged Prefix Replacement: replaces the matched prefix with a new string (e.g., strip /api/v1 prefix before sending to origin) Regex-based: full regex substitution on the path Retry policy: The default retry policy is 1 retry on 5xx responses. Set this explicitly for your application in every route: Disabled: no retries; required for write operations Default: 1 retry on 5xx Custom: specify retry conditions, count, and interval Per-route WAF override: Each simple route can attach its own WAF App Firewall policy. This completely replaces the load balancer-level WAF for matching requests; it is not additive. Use this to enforce stricter rules on sensitive paths (e.g., /admin/) or to relax inspection on certain paths. Example: API Path Routing Route 1: Prefix /api/v2/ → origin-pool-v2 (exact origin for v2) Route 2: Prefix /api/v1/ → origin-pool-v1 (legacy backend) Route 3: Prefix /api/ → origin-pool-api (catch-all for API paths) Route 4: Prefix / → origin-pool-web (catch-all for everything else) Order matters here. If route 3 or 4 appeared first, routes 1 and 2 would never fire. Route Type 2: Redirect Routes Redirect routes return an HTTP 3xx response directly to the client. No origin pool is involved: the RE handles the response entirely. When to Use HTTP → HTTPS redirect (though XC has a dedicated LB-level toggle for this) Domain canonicalization (www.example.com → example.com) Legacy URL migrations (/old-path/ → /new-path/) Temporary redirects during maintenance or A/B migrations Key Configuration Fields Field Description Protocol HTTP or HTTPS Host Target FQDN; supports non-standard ports Redirect Path / URI Target path; if left unset, original URI is preserved (including query string) Response Code 301 (Permanent), 302 (Temporary), 307, 308 Redirect Behavior: URI Preservation When you leave the redirect path unset, XC preserves the original URI path and query string in the Location header. This is useful for protocol/host-only redirects where you just want to change the scheme or domain without touching the path. Example: Redirect all HTTP traffic to HTTPS on the same host: Protocol: HTTPS Host: same-as-request (leave blank or match domain) Path: (unset — preserve original URI) Code: 301 Redirect Route Limitations Simple redirect routes (defined directly on the LB) do not support custom header manipulation on the redirect response. If you need to inject headers (e.g., Cache-Control: no-store on the redirect response), use a Custom Route object instead. Route Type 3: Direct Response Routes Direct response routes return a fully synthetic HTTP response to the client. The request never reaches an origin pool: the RE generates the response itself. When to Use Health check endpoints that should always return 200 (e.g., /healthz) without touching the app Maintenance mode pages: serve a 503 with a message body while origin is down Blocking specific paths with a meaningful error body (vs. a generic deny) Canary or feature-flag placeholders that return 404 before the feature ships Robots.txt or security.txt served from the edge without an origin Key Configuration Fields Field Description HTTP Status Code Any valid HTTP status code (200, 403, 503, etc.) Response Body Static text or HTML body returned to client Path match Same prefix/exact/regex options as other route types Example: Edge-Served Health Check Path: Exact /healthz Method: GET Action: Direct Response Status: 200 Body: OK This responds to health probes from AWS ALB, Kubernetes ingress controllers, or uptime monitors without any load on the backend. Particularly useful during blue/green deployments when the app might not yet be healthy. Route Type 4: Custom Routes Custom routes reference standalone Route objects created separately in XC and attached to one or more HTTP Load Balancers. Unlike the other three types, they follow a service-mesh model rather than a traditional LB model. When to Use Custom Routes Weighted traffic splitting between origin pools (canary releases, blue/green) Request/response header manipulation not available on simple routes Advanced retry policies with specific conditions and intervals Traffic mirroring (shadow traffic to a secondary backend for testing) Reusable route logic shared across multiple load balancers Architecture: Route Objects vs. Inline Routes Inline routes (simple, redirect, direct response) are defined directly on the HTTP Load Balancer. Custom route objects are: Created as standalone objects under Multi-Cloud App Connect Referenced by the HTTP Load Balancer Reusable: multiple LBs can reference the same route object Weighted Clusters Custom routes enable weighted traffic splitting across multiple upstream clusters, equivalent to BIG-IP pool ratio weights or AWS ALB weighted target groups. Route: Prefix /api/ Cluster A (origin-pool-v2): weight 90 Cluster B (origin-pool-v1): weight 10 This is the correct mechanism for canary deployments and gradual traffic shifts on F5 XC. The weights are percentage-based and must sum to 100. Header Manipulation Custom routes support header operations at the route level, applied before forwarding to origin: Operation Direction Example Add header Request X-Forwarded-For: {client-ip} Remove header Request Strip Authorization before certain paths Add header Response Strict-Transport-Security: max-age=31536000 Remove header Response Strip Server header from responses Header manipulation runs in order: route-level → virtual host-level → route configuration-level. Retry Policies Route retry policies take complete precedence over any virtual host-level retry policy. Configure explicitly: Field Description Retry on Conditions: 5xx, gateway-error, reset, connect-failure, retriable-4xx Number of retries Integer Per-try timeout Timeout applied to each individual retry attempt Retry interval Base interval between retries Traffic Mirroring Mirror policies shadow a copy of each request to a secondary cluster. The mirrored request is fire-and-forget. Example: Testing a new backend version against live traffic without affecting users Security analysis pipelines Route Ordering and Priority Route order is the most common source of routing bugs in XC deployments. There is no automatic specificity ranking; you own the order. The Rules Routes evaluate top to bottom. First match wins. Evaluation stops. Disabling a route (via the Route Activation Status toggle) does not remove it: traffic falls through to the next matching route. Recommended Ordering Pattern Order routes from most specific to least specific: Exact paths first Exact /api/v2/auth/token 2. Specific prefixes next Prefix /api/v2/ 3. Broader prefixes after Prefix /api/ 4. Path-specific exceptions Exact /healthz 5. Catch-all last Prefix / Common Ordering Mistakes Mistake Symptom Fix Catch-all prefix / first All traffic hits one origin; other routes never fire Move catch-all to last position /api prefix before /api/v2/ V2 traffic hits wrong origin Reverse the order Disabled/Unused route above active route Traffic silently hits next route with different behavior Remove disabled routes; don't rely on toggle for permanent changes Redirect route below a prefix match Redirect never fires Move redirect above the prefix that would match it first Common Mistakes and Gotchas Prefix /api matches /apikeys. The prefix match does not anchor to path segment boundaries. /api matches /api/, /api/v1/, and also /apikeys, /api-docs. Use /api/ (trailing slash) or regex if segment boundary matters. Per-route WAF is a full replacement, not additive. Attaching a WAF policy to a route does not stack with the LB-level WAF. It replaces it entirely for that route. If your LB WAF is in blocking mode and you attach a route-level WAF in monitoring mode, that route is now in monitoring mode only. Custom routes enforce TLS: test before production. If your origin uses a self-signed certificate and you switch from a simple route to a custom route without uploading the Root CA, connections will fail. Test in a staging environment first. Header manipulation on redirect requires a custom route object. Simple redirect routes in XC do not support response header injection. If you need Cache-Control or Vary headers on your redirects, you must use a standalone custom route object with redirect action. Regex route performance at scale. Regex routes require full path evaluation on every request. At high request volumes, a large number of regex routes adds measurable CPU overhead compared to prefix or exact routes. Use regex only where prefix or exact matching is insufficient. FAQ Q: What is the difference between a Simple Route and a Custom Route? A: Simple routes are inline on the HTTP Load Balancer and forward traffic directly to an Origin Pool. Custom routes are standalone objects that use an Endpoints → Clusters → Routes model, support weighted traffic splits, header manipulation, and mirroring, but cannot reference Origin Pools directly. Q: Why is my catch-all route matching everything instead of the specific routes below it? A: Route evaluation stops at the first match. If your catch-all prefix (/) is above more specific routes, it wins every time. Move the catch-all to the last position in the list. Q: Can I use a Custom Route to send traffic directly to an F5 XC Origin Pool? A: No. Custom routes do not support Origin Pools directly. They use an Endpoints → Clusters → Routes abstraction. If you need weighted splitting with Origin Pool support, custom routes are not the right fit; simple routes forward to Origin Pools but do not support weighted clusters. Q: My POST requests are creating duplicate records and I traced it to XC retries. What is happening? A: The default retry policy on simple routes is "1 retry on 5xx." A POST that hits a 500 gets retried once, potentially double-writing. Set the retry policy to Disabled on any route handling non-idempotent operations (POST, PUT, PATCH, DELETE). Q: Does attaching a WAF policy to a route add rules on top of my LB-level WAF? A: No. Per-route WAF replaces the LB-level WAF entirely for requests matching that route. If your LB WAF is in blocking mode and the route WAF is in monitoring mode, those requests are evaluated in monitoring mode only. Q: My Custom Route TLS connections to origin are failing but the same origin works fine on a Simple Route. Why? A: Custom routes enforce strict TLS with no skip-verify option. Simple routes do not have this requirement. For custom routes, upload the Root CA certificate for your origin, or use the use_volterra_trusted_ca_url flag via the API for public CAs. Self-signed certs without the Root CA uploaded will fail silently. Q: I disabled a route in the console but traffic behavior changed unexpectedly. What happened? A: Disabling a route via the Route Activation toggle does not remove it from evaluation. Traffic falls through to the next matching route in the list. If that route is a broad catch-all, the behavior shift may look correct until something that depends on specific routing breaks. Remove routes you no longer need rather than toggling them off.Use MinIO AIStor in Kubernetes with BIG-IP Container Ingress Services for S3 data delivery
AIStor is a massively scalable S3 object storage solution, one which can exist outside or inside of Kubernetes (k8s) clusters. Modern workflows, including AI workflows, often gravitate to orchestrated container schemes, something to alleviate hands-on keyboard tasks. One example would be a “vanilla” k8s solution, the so-called upstream Kubernetes which reflects the original, unmodified, and open-source version of the Kubernetes codebase hosted by the Cloud Native Computing Foundation (CNCF). Popular as well are “opinionated” Kubernetes, solutions like RedHat OpenShift Container Platform (OCP) which can integrate enterprise-grade tools and can often install with thought-out default settings. Let’s consider a couple of examples of k8s workloads that might leverage highly performant in-the-cluster S3 object storage; the first could be an AI data delivery task. Data scientists can initiate machine learning jobs within k8s that access large, centralized datasets—often hundreds of gigabytes of image or text data—stored through S3. During AI training, compute pods read and process this data, then write the resulting model artifacts, such as checkpoints and weights, back via S3. Another example would be in the world of “Big Data” and analytics processing, chief among these examples would be Apache Spark jobs. ETL (Extract, Transform, Load) workloads running on Kubernetes consume raw log or database files from S3 storage. Processing frameworks, such as Spark clusters, transform the data and persist the output back to S3 in optimized formats like Parquet. With the need for scalable and performant S3 storage being made, the question is how to seamlessly access this “inside” Kubernetes S3 to all of the S3 application users that exist outside of Kubernetes. This is where a modern k8s ingress controller offering, such as BIG-IP Container Ingress Services (CIS), will shine. This article aims to set up a simple working Kubernetes lab environment, equipped with both AIStor inside k8s and a BIG-IP appliance, to demonstrate the ease of setup and streamlined S3 data delivery that can be offered. Build a Kubernetes lab with MinIO AIStor installed To keep aligned with a simple, quick-win philosophy, the simplest Kubernetes new deployment was conducted. This entailed installing on just a couple of hosts a modern Ubuntu operating system and then downloading the constituent components of a k8s control-plane node and worker node from the kubernetes.io website site. The CNCF fully documents this “build your own cluster” approach on their site. In production cases, a large node set, made up of multiple control-plane nodes and even more worker nodes is expected; within a learning-first oriented lab environment just two nodes were started with. In order to deploy the MinIO AIStor, a simple lab setup as depicted below was laid out. Note worker nodes in the Kubernetes cluster should have large, performant disks locally installed. This is important, when the time comes to use persistent volumes and the required persistent volume claims, to be described later, one does not want to cede performance by using an external storage solution, like a remote and shared legacy SAN as an example. Create MinIO DirectPV and assign your drives Simply put, Kubernetes is a container orchestration scheme, think of the difficulty of manually launching containers to run the entirety of a modern application. There could be many processes, leading to an eclectic set of different containers. The intelligence to launch the appropriate number of each container type, at the exact right time, and provide critical auto recovery around failed containers, forget about it, this job is best left for Kubernetes and not your hands and keyboard. In a nutshell, a container storage interface (CSI) driver is the “storage adapter” for Kubernetes. Kubernetes itself doesn’t know how to talk to storage systems, think AWS EBS, Azure Disk or NFS-based NAS appliances, as examples. The CSI driver is the plugin that translates Kubernetes requests into storage actions. Just like Windows needs a driver for a USB disk, Kubernetes needs a CSI driver to use its storage. Enough said. What will happen with a Kubernetes app, when it asks for storage for its own purposes, it will make a persistent volume claim (PVC), simple enough. It is the CSI driver which handles creating volumes and attaching volumes to nodes. DirectPV, created by MinIO, is a specialized CSI designed specifically for Direct Attached Storage (DAS). Unlike traditional block or file-oriented CSIs that, for instance, can rely on a central Storage Area Network (SAN) or Network Attached Storage (NAS), DirectPV provisions volumes directly from the physical drives attached to your Kubernetes node. This lets one ride the micro-latency, price-to-performance improvement curve over time. Locally installed solid state drives (SSD) keep getting larger, exceeding 100TB regularly, cheaper per TB, and more performant than ever. DirectPV will therefore infuse these benefits of local storage distributed directly on your nodes into the Kubernetes S3 object environment. For those looking to double click on storage basics or just want a quick refresher on three terms pertaining to allocating storage in Kubernetes, the rest of this section is for you. These are terms I had to get re-acquainted with in setting up the lab. The following infographic may be a useful summary. Think of MinIO AIStor as an application, a set of running k8s pods, that turns a set of mounted disks into a distributed S3 object store. Keep in mind that k8s never gives an app like AIStor raw disks directly. Instead, it flows like this: Physical disks (SSD/NVMe on nodes) ↓ Storage backend / CSI (e.g., DirectPV) ↓ Persistent Volumes (PV) ↓ Persistent Volume Claims (PVC) ↓ AIStor Pods (mounted as /data) ↓ AIStor object storage service (users engage the S3 API to BIG-IP virtual servers, proxied onwards towards pods) AIStor is Kubernetes-native and thus logically uses PVCs to store data. The PVC is indirectly seen in the configuration through the use of “volumes”; each MinIO pod mounts one or more PVC-backed volumes. As depicted above, PVs are foundational for those PVCs, think of PVCs “binding” to PVs which represent actual portions of storage capacity in the cluster StorageClass defines how PVs are provisioned, using local and highspeed disks in our case. Although it is unnecessary to know day to day, the specific StorageClass recommended and implemented for use with AIStor is labelled as follows: storageClassName: directpv-min-io In summary, together the StorageClass specification and the CSI driver flavor, in our case MinIO’s DirectPV, together provisions PVs. Assign disks to a solution using DirectPV and Helm DirectPV installation is a breeze as it leverages the simplicity of a modern Helm chart-based approach. A Helm chart is a packaged set of Kubernetes configuration files (YAML) that describe how to deploy an application. It includes templates and default values so you can deploy the same app consistently across environments with different settings. Think of it like an installable “app package” for Kubernetes that you can deploy, upgrade, or roll back applications with a single command. Some people like to think of Helm as similar to Yum for Linux. The DirectPV Helm install command for upstream Kubernetes can be found in the MinIO docs here. To assign local disks, as recall we want storage to be local to worker nodes, the steps are listed in the docs. Largely the process is quick, one command will discover eligible disks on each worker node: $ kubectl directpv discover Now one can initialize the disks wanted as part of the AIStor solution, it doesn’t have to be all your disk but let’s be generous as this will scale out the S3 object storage. The “init” command referenced in the docs, as with many things in Kubernetes, will generate a YAML file which is what will place our drives under the purview of directpv. MinIO AIStor will make persistent volume claims through mounting volumes, as desired the PVCs are pointing at our disks in the end. You will now see the drives listed: $ kubectl directpv list drives In our lab worker node, a couple of starter 50 GB drives have been added to begin with (double-click to enlarge): Install AIStor within your Kubernetes cluster A nice aspect of AIStor in k8s is the reliance on Operators, a time saving and complexity-reducing feature heavily employed in modern containerized applications. A Kubernetes Operator is software that extends k8s to automatically manage a very specific application and its lifecycle. Orchestration takes things even further. It uses custom resources and a controller to continuously watch the application and make sure it stays in the desired state. Specifically, for AIStor, the net result is CustomResourceDefinitions, StatefulSets, and Secrets applied into a new or existing k8s Namespace. Think of an Operator as an automated expert—essentially a “robo-admin” for your application. It manages deployment, scaling, upgrades, and recovery, handling the routine operational tasks you’d otherwise need to do manually. From the installation guide, you need to add the AIStor Helm repository before installing: $ helm repo add minio https://helm.min.io/ Now, the helm installation looks like this, the license value can be downloaded from a customer’s support (“MinIO Subnet”) landing page, or an eval license can be created in a self-server format. $ helm install aistor minio/aistor-operator \ -n aistor --create-namespace \ --set license="eyJhbGciOiJFUzM4NCIsInR..." We can now see the result through: $ kubectl get all -n aistor This shows the AIStor installation is now running, albeit we have not installed the actual S3 object storage yet, which is what applications will use. This is the administrative portion of the installation. Minio AIStor separates its Kubernetes management into three distinct deployments to follow microservice and security best practices. This is seen in the middle “deployment” list of the previous kubectl get all command output. The three components are: object-store-operator: The core control plane. It watches for declarative Kubernetes Custom Resources (like ObjectStores) and automates the creation, scaling, and lifecycle management of the actual MinIO AIStor storage clusters. object-store-webhook: A validating webhook. It validates cluster configuration during deployment and dynamically injects/manages TLS certificates for in-pod version updates and secure, node-to-node communication. adminjob-operator: Handles administrative and management tasks outside of the core data path. It processes declarative kubernetes jobs, such as running the mcli admin client. Create your first AIStor S3 object storage service With all the foundational tasks behind us, we are now ready to again use Helm to install the first AIStor S3 object service. A nice feature is that multi-tenancy is baked into the solution. Although this quick lab setup will use just one, in production many communities of interest, say individual departments within an enterprise, will receive a unique tenant and isolated S3 services. Just to recap the big picture, the AIStor object storage will be instantiated as k8s pods. As one example, Pod 1 will mount a volume /data (PVC → PV → SSD on Node A). The more nodes and disks you have the larger the solution will grow. Once mounted, MinIO pods see each volume as a “disk”, together the pods enact a resilient and distributed solution. All pods are peers; there is no chief among them. The solution will present single S3 endpoints for configured buckets, clients never see anything pertaining to PVs and PVCs, it's simplicity first. One major requirement in production environments is to add data robustness through erasure coding (EC). EC will break larger objects into data chunks, sometimes called shards, along with parity chunks, and distribute these across nodes and disks. As such, any production deployment, as an absolute minimum, will have at least four nodes or disks to automatically "turn on" MinIO erasure coding. If you have ever used Helm before, a common and useful aspect is that a “values” file can be fed in when applying the Helm chart, key variables can all be set in place at once. A default AIStor values file is provided, an administrator just removes all values except those requiring customization. It’s easy and you can exercise your delete key in this exercise. In our lab setup, the values file was whittled down to the following small file, with three key elements highlighted, small storage was specified as this is a lab investigation and not a production grade offering: The first highlighted item, the number of servers specifies how many AIStor pods will be run by our worker nodes in the cluster; we have requested three. The k8s namespace is to be “primary-object-storage”. In a production cluster, think three to twenty nodes as reasonable examples, our specifying of three would mean the AIStor pods would likely run upon three different nodes. Our lab, although only consisting of one worker node, can be easily scaled out in the future. We see the three pods running in the following kubectl command directed at our namespace (double-click to enlarge). As was seen in our minimal Helm values file, we have requested the MinIO web console (“myminio-console”) to be exposed using NodePort on TCP port 31000 and the actual AIStor S3 service itself (“myminio”) using TPC port 31001. The next section will dive into leveraging F5 BIG-IP and the Container Ingress Services (CIS) solution in order to provide S3 applications and users outside the cluster with the high-performance S3 service that AIStor is running inside the cluster. BIG-IP Container Ingress Services deployed to allow easy S3 data delivery Using the BIG-IP to expose the AIStor S3 object storage service to external users is an obvious win for IT teams. It can support exposing S3, as well as the MinIO console, through common ports like TCP 443 which are very firewall friendly. It also allows a Kubernetes service to be configured in either ClusterIP or NodePort modes. The following diagram hits on principal differences between each mode. Using ClusterIP with BIG-IP has some major advantages. The origin pool consists of the actual AIStor pods themselves, not the K8S nodes, and is automatically built without administrative tasks. This automatic pool creation is through the use of the Ingress controller pod that BIG-IP CIS introduces in the cluster. ClusterIP also offers more fidelity than NodePort, each pod can be load balanced with consideration of pod-level criteria like, perhaps, current least connections to pods as an algorithm. NodePort on the other hand also offers automatic origin pool creation, albeit differently. The nodes in the cluster, and the TCP port values that services are being exposed on, are set up automatically in the BIG-IP origin pools. The S3 consumers again benefit from reaching S3 on low port values like 443, so trying to expose high ports which run up against common firewall rules, like TCP 31000 and 31001 in our lab setup, becomes a moot point. A drawback to NodePort is that the nodes, not the actual MinIO pods, are what is load balanced. Each node receiving S3 traffic itself will load balance to the full set of AIStor pods running anywhere in the cluster. As such, an S3 transaction first arrives at a node IP address and is then proxied again to an individual pod, since internal nodes are aware of pod addresses. This additional proxy step may or may not be significant to users. As well the fact that NodePort forgoes applying a load balancing algorithm directly against pods, and instead balances against nodes, might be taken into consideration. The reason this article will focus upon demonstrating NodePort is due to simplicity, it is quicker to set up and requires less ruminations on the network aspects of k8s networking. A follow up article will walk the reader through ClusterIP mode, however a CNI plug-in will need to be decided upon. A CNI (Container Networking Interface) is the “network plumbing” of Kubernetes — the CNI makes sure pods can talk to each other. By infusing the BIG-IP with a CNI, and enabling ClusterIP mode, we can directly participate in pod networking and bonus, load balance S3 directly to MinIO AIStor pods. BIG-IP Container Ingress Services – NodePort S3 delivery The full setup sequence for BIG-IP as a load balancer into Kubernetes is documented here, including upstream Kubernetes, RedHat OpenShift and VMware Tanzu. In our lab, the step will start with cloning the k8s BIG-IP Ingress controller GitHub: $ git clone https://github.com/F5Networks/k8s-bigip-ctlr.git The last step, in a similar vein to our MinIO AIStor installation, is to use a Helm chart and a corresponding edited values file: $ helm install -f values.yaml <new-chart-name> f5-stable/f5-bigip-ctlr There are a number of approaches to harnessing F5 CIS to expose Kubernetes for a NodePort solution. Top of mind are using the k8s Ingress object, or the ConfigMap object or custom resource definitions (CRDs), all in conjunction with the F5 Ingress Controller pod now running in the cluster. For those curious about using Ingress or ConfigMap, an intuitive step-by-step lab guide is available here which may be informative to step through. One major advantage to CRDs is that they allow independent DevOps teams to focus on their own applications, there is a separate YAML file per service/application. Early adopters of CIS often first used BIG-IP AS3 configured config map objects, however that is essentially one large JSON declaration shared across many applications. The idea of edits affecting all namespaces in an enterprise’s k8s applications is less attractive. A one-page overview of migrating AS3 config map setups to an equivalent CRD style is found here. CIS does not support combination of CRDs with any of Ingress/Routes and ConfigMaps. To follow the CRD approach, the only requirement is to ensure the Helm values file has the following attribute active, not prefaced with a #. Here is an example of the Helm values file used in this lab exercise, with portions removed for brevity. The notable settings are highlighted in yellow. Custom resources are extensions of the Kubernetes API. A resource is an endpoint in the Kubernetes API that stores a collection of API objects. For example, the k8s built-in Pods resource contains a collection of Pod objects, add a custom resource and it too will have a collection of objects. The custom resources offered by the BIG-IP CIS solution are documented here, we are most interested in using (a) the virtual server and (b) TLS profile CRDs. Our objective is to offer TLS-encrypted S3 service on both sides of the BIG-IP load balancer, essentially a security first stance. The lab uses an internal DNS, as such the external clients need just know FQDNs, whether that be clients in the form of application servers with fully automated S3 requirements or perhaps human clients interactively using storage. The names used for the S3 service and AIStor GUI in this private DNS arrangement were, respectively: aistors3.lab.com aistorgui.lab.com The configuration is perhaps analogous to putting a full self-driving (FSD) car into action, presumably someday simply putting in the end state, a destination, and all steps to reaching that target would be automated. With BIG-IP CIS, we need only give two pieces of data, a TLS profile in a small YAML file and a virtual server profile, again in YAML format. The Ingress Controller pod will then connect and configure a full BIG-IP origin pool pointing at the k8s service. Beyond this, virtual servers for our two services made available to our external users will also be created. The BIG-IP GUI will automatically update itself with these live published applications, in a BIG-IP partition by default we have named “kubernetes”, all within seconds. Samples for each type of setup file can be found in this maintained Github location. Here are the two TLSProfiles used for each application: The virtual server definitions for each are similarly intuitive: Note, there is no need for the administrator to even know which NodePort values were assigned in Kubernetes, you may recall mention of ports 31000 and 31001 earlier, we simply just need to provide the k8s service names and the service (native) port of each, that’s it. Validation of S3 services through BIG-IP to Kubernetes-based AIStor The deployment of the 4 small YAML files is done with 4 consecutive kubectl commands, starting with the TLS profiles and then the virtual servers: $ kubectl apply -f file_name.yaml We can then open the BIG-IP GUI and, voila, notice that two virtual services are waiting for us, already processing S3 traffic, in the kubernetes partitions. Although we have no tasks within the BIG-IP GUI, a quick check on the S3 service virtual server is apropos, to ensure all is as expected. The key takeaway is the address is correct; this corresponds to our private DNS entry for s3.lab.local. Another point to note is this is a layer7 http profile being used. Advanced features like iRules, URL awareness and modern load balancing persistence schemes like cookie persistence are all now in play. It’s worth noting for experienced BIG-IP users, the origin pool that maps to our NodePort entries is not called out in the Resource tab as a pool. Rather, the mapping is done with iRules and local policies, all created automatically. With HTTPS/TLS underpinning S3, it’s important to use fully qualified domain names as opposed to just IP addresses that map to BIG-IP virtual servers. One can see the local policy supporting our S3 application (10.150.92.68) explicitly looks for names in its logic (double-click to enlarge). Let’s put everything to the test, first a simple login attempt from an external Windows client to the AIStor web console, at https://aistorgui.lab.com. The following demonstrates all is well, successful login and perusing the one bucket that has been configured and populated with six files. Note, to avoid the certificate trust issue I need simply to load the default TLS profile client certificate being presented into the Windows machine’s trusted certificates store, Microsoft Management Console or Active Directory could be leaned upon. One could quickly create a new, unique certificate authority (CA) using a tool like OpenSSL, a common approach for lab work, and install trust of all certificates issued by that CA on the client. To validate that the S3 object service, not just the AIStor GUI, is working, a simple graphical utility, S3 browser, is leveraged along with AIStor access credentials. Both username and password or S3 access key and access secrets could be used. Using the “Upload” button we immediately uploaded a seventh file, noted with highlights in the event log as having been successfully completed. We are now offering external users BIG-IP facilitated secure, load balanced S3 into AIStor in a Kubernetes cluster. This is what we came to do. A couple of features of S3Browser that are useful to note for lab work, the advanced options allow S3 certificate validation to be bypassed, as such we receive no complaints when using self-signed certificates in the lab. The other key feature is an industry absolute must, S3 multi-part uploads. A common setting is for all objects greater than, say, 100 megabytes to be broken into multiple chunks, for instance a 3-gigabyte file upload can be handled by 30 separate uploads for reassembly at the object store, where in turn the object is likely to be erasure coded. The value of multi-part uploads includes alleviating transient network conditions that might cause extremely large objects to fail after the majority of data has been delivered. Also, the individual chunks can enable parallelism, chunks hit BIG-IP and could be load balanced across multiple S3 nodes. The last validation is to examine the BIG-IP origin pool in use for the S3 application. As you will recall, this is a small lab setup using NodePort and two k8s nodes. As such, our expectation would be for two auto-discovered node entries, and the solution should have automatically determined the TCP high value port to use in the back end. The following demonstrates the AIStor GUI traffic origin pool. We observed that indeed the nodes and the ports assigned to the two services, TCP port 31000/31001 for Console/S3 delivery, respectively, were automatically discovered and configured. Traffic counters indicate communications to both. One last note, Minio has S3 services exposed by AIStor in encrypted format on 443, which we have selected a NodePort of TCP 31001 for. By default, an unencrypted port, using HTTP as transport, is also available in the backend. This normally is on service (native) port 9000. NodePort will automatically expose this as well, on a randomized high port, in the screenshot below we see 30547 has been assigned. As such, the BIG-IP will discover this as a second set of NodePort values in the S3 origin pool. Simply delete the pool entries for port 30547 as we wish to only support encrypted S3 in this lab. S3 traffic, once distributed to cluster nodes on port 31001, is then proxied to any of the individual AIStor pods running in the cluster. As you may recall, our Helm Chart requested three instances of AIStor servers (pods) to always be running and healthy in the cluster. As with normal deployments in k8s, this translates into a replica set. Should any pod or entire node fail, automatically Kubernetes orchestration will kick in and provide the specified number of AIStor pods. Surely even self-driving cards will be challenged to reach this level of hands-free automation. Summary and next steps A simple, small scale upstream Kubernetes cluster was deployed in a lab environment on Ubuntu hosts. Using the MinIO documentation, a DirectPV storage solution was quickly created. MinIO DirectPV is a Kubernetes add-on that lets you use local, performant disks on your servers as the underlying storage for a modern, scalable S3 solution—automatically and efficiently. Using Helm charts, a Kubernetes-native AIStor Operator was installed, followed by one instance of the AIStor S3 object storage service itself. Multiple tenants can easily be added. To enable a single endpoint for external S3 users to interact with the k8s-based solution, BIG-IP Container Ingress Services was deployed. This, also using Helm charts for a quick install, resulted in an ingress controller pod running in the cluster and fully automatic setup of an adjacent BIG-IP load balancer. The setup used CRDs and four small YAML files to described the desired S3 and AIStor GUI virtual servers. S3 traffic was observed to be load balanced as expected from external sources to the AIStor service within Kubernetes. As a next step, an investigation will be conducted into ClusterIP operations, where BIG-IP can directly load balance to the AIStor pods themselves. This normally involves selecting a container networking interface, like Flannel, Calico or Celium, all effectively allow pod networking such that pods and the BIG-IP can directly communicate. This completed lab exercise used NodePort rather than ClusterIP, and load balances S3 traffic to the individual cluster nodes for successful forwarding to pods.
