Deploying the F5 AI Security Certified OpenShift Operator: A Validated Playbook
Introduction As enterprises race to deploy Large Language Models (LLMs) in production, securing AI workloads has become as critical as securing traditional applications. The F5 AI Security Operator installs two products on your cluster — F5 AI Guardrails and F5 AI Red Team — both powered by CalypsoAI. Together they provide inline prompt/response scanning, policy enforcement, and adversarial red-team testing, all running natively on your own OpenShift cluster. This article is a validated deployment runbook for F5 AI Security on OpenShift (version 4.20.14) with NVIDIA GPU nodes. It is based on the official Red Hat Operator installation baseline, in a real lab deployment on a 3×A40 GPU cluster. If you follow these steps in order, you will end up with a fully functional AI Security stack, avoiding the most common pitfalls along the way. What Gets Deployed F5 AI Security consists of four main components, each running in its own OpenShift namespace: Component Namespace Role Moderator + PostgreSQL cai-moderator Web UI, API gateway, policy management, and backing database Prefect Server + Worker prefect Workflow orchestration for scans and red-team runs AI Guardrails Scanner cai-scanner Inline scanning against your OpenAI-compatible LLM endpoint AI Red Team Worker cai-redteam GPU-backed adversarial testing; reports results to Moderator via Prefect The Moderator is CPU-only. The Scanner and Red Team Worker can leverage GPUs depending on the policies and models you configure. Infrastructure Requirements Before you begin, verify your cluster meets these minimums: CPU / Control Node 16 vCPUs, 32 GiB RAM, x86_64, 100 GiB persistent storage Worker Nodes (per GPU-enabled component) 4 vCPUs, 16 GiB RAM (32 GiB recommended for Red Team), 100 GiB storage GPU Nodes AI Guardrails: CUDA-compatible GPU, minimum 24 GB VRAM, 100 GiB storage AI Red Team: CUDA-compatible GPU, minimum 48 GB VRAM, 200 GiB storage GPU must NOT be shared with other workloads Verify your cluster: # Check nodes oc get nodes -o wide # Check GPU allocatable resources oc get node -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}' # Check available storage classes oc get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE lvms-vg1 (default) topolvm.io Delete WaitForFirstConsumer true 15d Step 1 — Install Prerequisites 1.1 Node Feature Discovery (NFD) Operator NFD labels your nodes with hardware capabilities, which NVIDIA GPU Operator relies on to target the right nodes. OpenShift Console → Ecosystem → Software Catalog → Search Node Feature Discovery Operator → Install After installation: Installed Operators → Node Feature Discovery → Create NodeFeatureDiscovery → Accept defaults Verify: oc get pods -n openshift-nfd oc get node --show-labels | grep feature.node.kubernetes.io || true 1.2 NVIDIA GPU Operator OpenShift Console → Ecosystem → Software Catalog → Search GPU Operator → Install After installation: Installed Operators → NVIDIA GPU Operator → Create ClusterPolicy → Accept defaults Verify: oc get pods -n nvidia-gpu-operator oc describe node <gpu-node> | grep -i nvidia nvidia-smi</gpu-node> Step 2 — Install F5 AI Security Operator Prerequisites: You will need registry credentials and a valid license from the F5 AI Security team before proceeding. Contact F5 Sales: https://www.f5.com/products/get-f5 2.1 Create the Namespace and Pull Secret export DOCKER_USERNAME='<registry-username>' export DOCKER_PASSWORD='<registry-password>' export DOCKER_EMAIL='<your-email>' oc new-project f5-ai-sec oc create secret docker-registry regcred \ -n f5-ai-sec \ --docker-username=$DOCKER_USERNAME \ --docker-password=$DOCKER_PASSWORD \ --docker-email=$DOCKER_EMAIL</your-email></registry-password></registry-username> 2.2 Install from OperatorHub OpenShift Console → Ecosystem → Software Catalog → Search F5 AI Security Operator → Install into namespace f5-ai-sec Verify your F5 AI Security Operator: # Verify the controller-manager pod is Running oc -n f5-ai-sec get pods # NAME READY STATUS RESTARTS AGE # controller-manager-6f784bd96d-z6sbh 1/1 Running 1 43s # Verify the CSV reached Succeeded phase oc -n f5-ai-sec get csv # NAME DISPLAY VERSION PHASE # f5-ai-security-operator.v0.4.3 F5 Ai Security Operator 0.4.3 Succeeded # Verify the CRD is registered oc -n f5-ai-sec get crd | grep ai.security.f5.com # securityoperators.ai.security.f5.com 2.3 Deploy the SecurityOperator Custom Resource After installation: Installed Operators → F5 AI Security Operator → Create SecurityOperator Choose YAML and copy the below Custom Resource Template in there, changing select values to match your installation. apiVersion: ai.security.f5.com/v1alpha1 kind: SecurityOperator metadata: name: security-operator-demo namespace: f5-ai-sec spec: registryAuth: existingSecret: "regcred" # Internal PostgreSQL — convenient for labs, not recommended for production postgresql: enabled: true values: postgresql: auth: password: "pass" jobManager: enabled: true moderator: enabled: true values: env: CAI_MODERATOR_BASE_URL: https://<your-hostname> secrets: CAI_MODERATOR_DB_ADMIN_PASSWORD: "pass" CAI_MODERATOR_DEFAULT_LICENSE: "<valid_license_from_f5>" scanner: enabled: true redTeam: enabled: true</valid_license_from_f5></your-hostname> Key values to customize: Field What to set CAI_MODERATOR_BASE_URL Your cluster's public hostname for the UI (e.g., https://aisec.apps.mycluster.example.com ) CAI_MODERATOR_DEFAULT_LICENSE License string provided by F5 CAI_MODERATOR_DB_ADMIN_PASSWORD DB password — must match the value set in the PostgreSQL block For external PostgreSQL (recommended for production), replace the postgresql block with: moderator: values: env: CAI_MODERATOR_DB_HOST: <my-external-db-hostname> secrets: CAI_MODERATOR_DB_ADMIN_PASSWORD: <my-external-db-password></my-external-db-password></my-external-db-hostname> Verify your F5 AI Security Operator: oc -n f5-ai-sec get securityoperator oc -n f5-ai-sec get securityoperator security-operator-demo -o yaml | sed -n '/status:/,$p' Step 3 — Required OpenShift Configuration This is where most deployments hit problems. OpenShift's default restricted Security Context Constraint (SCC) blocks these containers from running. You must explicitly grant anyuid to each service account. 3.1 Apply SCC Policies oc adm policy add-scc-to-user anyuid -z cai-moderator-sa -n cai-moderator oc adm policy add-scc-to-user anyuid -z default -n cai-moderator oc adm policy add-scc-to-user anyuid -z default -n prefect oc adm policy add-scc-to-user anyuid -z prefect-server -n prefect oc adm policy add-scc-to-user anyuid -z prefect-worker -n prefect oc adm policy add-scc-to-user anyuid -z cai-scanner -n cai-scanner oc adm policy add-scc-to-user anyuid -z cai-redteam-worker -n cai-redteam 3.2 Force PostgreSQL to Restart (if Stuck at 0/1) If PostgreSQL was stuck before the SCC was applied, bounce it manually: oc -n cai-moderator scale sts/cai-moderator-postgres-cai-postgresql --replicas=0 oc -n cai-moderator scale sts/cai-moderator-postgres-cai-postgresql --replicas=1 3.3 Restart All Components oc -n cai-moderator rollout restart deploy oc -n prefect rollout restart deploy oc -n cai-scanner rollout restart deploy oc -n cai-redteam rollout restart deploy 3.4 Verify ➜ oc -n cai-moderator get statefulset NAME READY AGE cai-moderator-postgres-cai-postgresql 1/1 3d4h ➜ oc -n cai-moderator get pods | grep postgres cai-moderator-postgres-cai-postgresql-0 1/1 Running 0 3d4h ➜ oc -n cai-moderator get pods | grep cai-moderator cai-moderator-75c47fc9db-sl8t2 1/1 Running 0 3d4h cai-moderator-postgres-cai-postgresql-0 1/1 Running 0 3d4h ➜ oc -n cai-moderator get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cai-moderator ClusterIP 172.30.123.197 <none> 5500/TCP,8080/TCP 3d4h cai-moderator-headless ClusterIP None <none> 8080/TCP 3d4h cai-moderator-postgres-postgresql ClusterIP None <none> 5432/TCP 3d4h ➜ oc -n cai-moderator get endpoints Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice NAME ENDPOINTS AGE cai-moderator 10.130.0.139:8080,10.130.0.139:5500 3d4h cai-moderator-headless 10.130.0.139:8080 3d4h cai-moderator-postgres-postgresql 10.128.0.177:5432 3d4h</none></none></none> Step 4 — Create OpenShift Routes (Required for UI Access) The Moderator exposes two ports that must be routed separately: port 5500 for the UI and port 8080 for the /auth path. Skipping the auth route is the most common cause of the blank/black page issue. # UI route oc -n cai-moderator create route edge cai-moderator-ui \ --service=cai-moderator \ --port=5500 \ --hostname=<your-hostname> \ --path=/ # Auth route — required, or the UI will render blank oc -n cai-moderator create route edge cai-moderator-auth \ --service=cai-moderator \ --port=8080 \ --hostname=<your-hostname> \ --path=/auth</your-hostname></your-hostname> Verify all pods are running: oc get pods -n cai-moderator oc get pods -n cai-scanner oc get pods -n cai-redteam oc get pods -n prefect Access the UI Open https:// in a browser. Log in with the default credentials: admin / pass Log in and update the admin email address immediately. You should be able to log in successfully and see the Guardrails dashboard. Step 5 — Grant Prefect Worker Cluster-scope RBAC The Prefect worker watches Kubernetes Pods and Jobs at cluster scope to monitor scan and red-team workflow execution. Without this RBAC, prefect-worker fills its logs with 403 Forbidden errors. The Guardrails UI still loads, but scheduled workflows and Red Team runs will fail silently. # ClusterRole: allow prefect-worker to list/watch pods, jobs, and events cluster-wide oc apply -f - <<'YAML' apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prefect-worker-watch-cluster rules: - apiGroups: ["batch"] resources: ["jobs"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["pods","pods/log","events"] verbs: ["get","list","watch"] YAML # ClusterRoleBinding: bind to the prefect-worker ServiceAccount oc apply -f - <<'YAML' apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prefect-worker-watch-cluster subjects: - kind: ServiceAccount name: prefect-worker namespace: prefect roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prefect-worker-watch-cluster YAML # Restart to pick up the new permissions oc -n prefect rollout restart deploy/prefect-worker Verify RBAC errors are gone: oc -n prefect logs deploy/prefect-worker --tail=200 \ | egrep -i 'forbidden|rbac|permission|denied' \ || echo "OK: no RBAC errors detected" oc get clusterrolebinding prefect-worker-watch-cluster LlamaStack Integration F5 AI Security works alongside any OpenAI-compatible LLM inference endpoint. In our lab we pair it with LlamaStack running a quantized Llama 3.2 model on the same OpenShift cluster — F5 AI Guardrails then scans every prompt and response inline before it reaches your application. A dedicated follow-up post will walk through the full LlamaStack deployment and end-to-end integration in detail. Stay tuned. Summary Deploying F5 AI Security on OpenShift is straightforward once you know the OpenShift-specific friction points: SCC policies, the dual-route requirement, and the Prefect cluster-scope RBAC. Following this runbook in sequence — prerequisites, operator install, SCC grants, routes, Prefect RBAC — gets you to a fully operational AI guardrailing stack in a single pass. If you run into anything not covered here, drop a comment below. Tested on: OpenShift 4.20.14 · F5 AI Security Operator v0.4.3 · NVIDIA A40 GPUs · LlamaStack with Llama-3.2-1B-Instruct-quantized.w8a8 Additional Resources F5 AI Security Operator — Red Hat CatalogF5 Container Ingress Services (CIS) deployment using Cilium CNI and static routes
F5 Container Ingress Services (CIS) supports static route configuration to enable direct routing from F5 BIG-IP to Kubernetes/OpenShift Pods as an alternative to VXLAN tunnels. Static routes are enabled in the F5 CIS CLI/Helm yaml manifest using the argument --static-routing-mode=true. In this article, we will use Cilium as the Container Network Interface (CNI) and configure static routes for an NGINX deployment For initial configuration of the BIG-IP, including AS3 installation, please see https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/installation.html and https://clouddocs.f5.com/containers/latest/userguide/kubernetes/#cis-installation The first step is to install Cilium CNI using the steps below on Linux host: CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} cilium install --version 1.18.5 cilium status cilium status --wait root@ciliumk8s-ubuntu-server:~# cilium status --wait /¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: OK \__/¯¯\__/ Hubble Relay: disabled \__/ ClusterMesh: disabled DaemonSet cilium Desired: 1, Ready: 1/1, Available: 1/1 DaemonSet cilium-envoy Desired: 1, Ready: 1/1, Available: 1/1 Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 Containers: cilium Running: 1 cilium-envoy Running: 1 cilium-operator Running: 1 clustermesh-apiserver hubble-relay Cluster Pods: 6/6 managed by Cilium Helm chart version: 1.18.3 Image versions cilium quay.io/cilium/cilium:v1.18.3@sha256:5649db451c88d928ea585514746d50d91e6210801b300c897283ea319d68de15: 1 cilium-envoy quay.io/cilium/cilium-envoy:v1.34.10-1761014632-c360e8557eb41011dfb5210f8fb53fed6c0b3222@sha256:ca76eb4e9812d114c7f43215a742c00b8bf41200992af0d21b5561d46156fd15: 1 cilium-operator quay.io/cilium/operator-generic:v1.18.3@sha256:b5a0138e1a38e4437c5215257ff4e35373619501f4877dbaf92c89ecfad81797: 1 cilium connectivity test root@ciliumk8s-ubuntu-server:~# cilium connectivity test ℹ️ Monitor aggregation detected, will skip some flow validation steps ✨ [default] Creating namespace cilium-test-1 for connectivity check... ✨ [default] Deploying echo-same-node service... ✨ [default] Deploying DNS test server configmap... ✨ [default] Deploying same-node deployment... ✨ [default] Deploying client deployment... ✨ [default] Deploying client2 deployment... ✨ [default] Deploying ccnp deployment... ⌛ [default] Waiting for deployment cilium-test-1/client to become ready... ⌛ [default] Waiting for deployment cilium-test-1/client2 to become ready... ⌛ [default] Waiting for deployment cilium-test-1/echo-same-node to become ready... ⌛ [default] Waiting for deployment cilium-test-ccnp1/client-ccnp to become ready... ⌛ [default] Waiting for deployment cilium-test-ccnp2/client-ccnp to become ready... ⌛ [default] Waiting for pod cilium-test-1/client-645b68dcf7-s5mdb to reach DNS server on cilium-test-1/echo-same-node-f5b8d454c-qkgq9 pod... ⌛ [default] Waiting for pod cilium-test-1/client2-66475877c6-cw7f5 to reach DNS server on cilium-test-1/echo-same-node-f5b8d454c-qkgq9 pod... ⌛ [default] Waiting for pod cilium-test-1/client-645b68dcf7-s5mdb to reach default/kubernetes service... ⌛ [default] Waiting for pod cilium-test-1/client2-66475877c6-cw7f5 to reach default/kubernetes service... ⌛ [default] Waiting for Service cilium-test-1/echo-same-node to become ready... ⌛ [default] Waiting for Service cilium-test-1/echo-same-node to be synchronized by Cilium pod kube-system/cilium-lxjxf ⌛ [default] Waiting for NodePort 10.69.12.2:32046 (cilium-test-1/echo-same-node) to become ready... 🔭 Enabling Hubble telescope... ⚠️ Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:4245: connect: connection refused" ℹ️ Expose Relay locally with: cilium hubble enable cilium hubble port-forward& ℹ️ Cilium version: 1.18.3 🏃[cilium-test-1] Running 126 tests ... [=] [cilium-test-1] Test [no-policies] [1/126] .................... [=] [cilium-test-1] Skipping test [no-policies-from-outside] [2/126] (skipped by condition) [=] [cilium-test-1] Test [no-policies-extra] [3/126] <- snip -> For this article, we will install k3s with Cilium CNI root@ciliumk8s-ubuntu-server:~# curl -sfL https://get.k3s.io | sh -s - --flannel-backend=none --disable-kube-proxy --disable servicelb --disable-network-policy --disable traefik --cluster-init --node-ip=10.69.12.2 --cluster-cidr=10.42.0.0/16 root@ciliumk8s-ubuntu-server:~# mkdir -p $HOME/.kube root@ciliumk8s-ubuntu-server:~# sudo cp -i /etc/rancher/k3s/k3s.yaml $HOME/.kube/config root@ciliumk8s-ubuntu-server:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config root@ciliumk8s-ubuntu-server:~# echo "export KUBECONFIG=$HOME/.kube/config" >> $HOME/.bashrc root@ciliumk8s-ubuntu-server:~# source $HOME/.bashrc API_SERVER_IP=10.69.12.2 API_SERVER_PORT=6443 CLUSTER_ID=1 CLUSTER_NAME=`hostname` POD_CIDR="10.42.0.0/16" root@ciliumk8s-ubuntu-server:~# cilium install --set cluster.id=${CLUSTER_ID} --set cluster.name=${CLUSTER_NAME} --set k8sServiceHost=${API_SERVER_IP} --set k8sServicePort=${API_SERVER_PORT} --set ipam.operator.clusterPoolIPv4PodCIDRList=$POD_CIDR --set kubeProxyReplacement=true --helm-set=operator.replicas=1 root@ciliumk8s-ubuntu-server:~# cilium config view | grep cluster bpf-lb-external-clusterip false cluster-id 1 cluster-name ciliumk8s-ubuntu-server cluster-pool-ipv4-cidr 10.42.0.0/16 cluster-pool-ipv4-mask-size 24 clustermesh-enable-endpoint-sync false clustermesh-enable-mcs-api false ipam cluster-pool max-connected-clusters 255 policy-default-local-cluster false root@ciliumk8s-ubuntu-server:~# cilium status --wait The F5 CIS yaml manifest for deployment using Helm Note that these arguments are required for CIS to leverage static routes static-routing-mode: true orchestration-cni: cilium-k8s We will also be installing custom resources, so this argument is also required 3. custom-resource-mode: true Values yaml manifest for Helm deployment bigip_login_secret: f5-bigip-ctlr-login bigip_secret: create: false username: password: rbac: create: true serviceAccount: # Specifies whether a service account should be created create: true # The name of the service account to use. # If not set and create is true, a name is generated using the fullname template name: k8s-bigip-ctlr # This namespace is where the Controller lives; namespace: kube-system ingressClass: create: true ingressClassName: f5 isDefaultIngressController: true args: # See https://clouddocs.f5.com/containers/latest/userguide/config-parameters.html # NOTE: helm has difficulty with values using `-`; `_` are used for naming # and are replaced with `-` during rendering. # REQUIRED Params bigip_url: X.X.X.S bigip_partition: <BIG-IP_PARTITION> # OPTIONAL PARAMS -- uncomment and provide values for those you wish to use. static-routing-mode: true orchestration-cni: cilium-k8s # verify_interval: # node-poll_interval: # log_level: DEBUG # python_basedir: ~ # VXLAN # openshift_sdn_name: # flannel_name: cilium-vxlan # KUBERNETES # default_ingress_ip: # kubeconfig: # namespaces: ["foo", "bar"] # namespace_label: # node_label_selector: pool_member_type: cluster # resolve_ingress_names: # running_in_cluster: # use_node_internal: # use_secrets: insecure: true custom-resource-mode: true log-as3-response: true as3-validation: true # gtm-bigip-password # gtm-bigip-url # gtm-bigip-username # ipam : true image: # Use the tag to target a specific version of the Controller user: f5networks repo: k8s-bigip-ctlr pullPolicy: Always version: latest # affinity: # nodeAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # nodeSelectorTerms: # - matchExpressions: # - key: kubernetes.io/arch # operator: Exists # securityContext: # runAsUser: 1000 # runAsGroup: 3000 # fsGroup: 2000 # If you want to specify resources, uncomment the following # limits_cpu: 100m # limits_memory: 512Mi # requests_cpu: 100m # requests_memory: 512Mi # Set podSecurityContext for Pod Security Admission and Pod Security Standards # podSecurityContext: # runAsUser: 1000 # runAsGroup: 1000 # privileged: true Installation steps for deploying F5 CIS using helm can be found in this link https://clouddocs.f5.com/containers/latest/userguide/kubernetes/ Once F5 CIS is validated to be up and running, we can now deploy the following application example root@ciliumk8s-ubuntu-server:~# cat application.yaml apiVersion: cis.f5.com/v1 kind: VirtualServer metadata: labels: f5cr: "true" name: goblin-virtual-server namespace: nsgoblin spec: host: goblin.com pools: - path: /green service: svc-nodeport servicePort: 80 - path: /harry service: svc-nodeport servicePort: 80 virtualServerAddress: X.X.X.X --- apiVersion: apps/v1 kind: Deployment metadata: name: goblin-backend namespace: nsgoblin spec: replicas: 2 selector: matchLabels: app: goblin-backend template: metadata: labels: app: goblin-backend spec: containers: - name: goblin-backend image: nginx:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: svc-nodeport namespace: nsgoblin spec: selector: app: goblin-backend ports: - port: 80 targetPort: 80 type: ClusterIP k apply -f application.yaml We can now verify the k8s pods are created. Then we will create a sample html page to test access to the backend NGINX pod root@ciliumk8s-ubuntu-server:~# k -n nsgoblin get po -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES goblin-backend-7485b6dcdf-d5t48 1/1 Running 0 6d2h 10.42.0.70 ciliumk8s-ubuntu-server <none> <none> goblin-backend-7485b6dcdf-pt7hx 1/1 Running 0 6d2h 10.42.0.97 ciliumk8s-ubuntu-server <none> <none> root@ciliumk8s-ubuntu-server:~# k -n nsgoblin exec -it po/goblin-backend-7485b6dcdf-pt7hx -- /bin/sh # cat > green <<'EOF' <!DOCTYPE html> > > <html> > <head> <title>Green Goblin</title> <style> body { background-color: #4CAF50; color: white; text-align: center; padding: 50px; } h1 { font-size: 3em; } > > > > > </style> </head> <body> <h1>I am the green goblin!</h1> <p>Access me at /green</p> </body> </html> > > > > > > > EOF root@ciliumk8s-ubuntu-server:~# k -n nsgoblin exec -it goblin-backend-7485b6dcdf-d5t48 -- /bin/sh # cat > green <<'EOF' > <!DOCTYPE html> <html> <head> <title>Green Goblin</title> <style> body { background-color: #4CAF50; color: white; text-align: center; padding: 50px; } h1 { font-size: 3em; } </style> > </head> <body> <h1>I am the green goblin!</h1> <p>Access me at /green</p> </body> </html> EOF> > > > > > > > > > > > > We can now validate the pools are created on the F5 BIG-IP root@(ciliumk8s-bigip)(cfg-sync Standalone)(Active)(/kubernetes/Shared)(tmos)# list ltm pool all ltm pool svc_nodeport_80_nsgoblin_goblin_com_green { description "crd_10_69_12_40_80 loadbalances this pool" members { /kubernetes/10.42.0.70:http { address 10.42.0.70 } /kubernetes/10.42.0.97:http { address 10.42.0.97 } } min-active-members 1 partition kubernetes } ltm pool svc_nodeport_80_nsgoblin_goblin_com_harry { description "crd_10_69_12_40_80 loadbalances this pool" members { /kubernetes/10.42.0.70:http { address 10.42.0.70 } /kubernetes/10.42.0.97:http { address 10.42.0.97 } } min-active-members 1 partition kubernetes } root@(ciliumk8s-bigip)(cfg-sync Standalone)(Active)(/kubernetes/Shared)(tmos)# list ltm virtual crd_10_69_12_40_80 ltm virtual crd_10_69_12_40_80 { creation-time 2025-12-22:10:10:37 description Shared destination /kubernetes/10.69.12.40:http ip-protocol tcp last-modified-time 2025-12-22:10:10:37 mask 255.255.255.255 partition kubernetes persist { /Common/cookie { default yes } } policies { crd_10_69_12_40_80_goblin_com_policy { } } profiles { /Common/f5-tcp-progressive { } /Common/http { } } serverssl-use-sni disabled source 0.0.0.0/0 source-address-translation { type automap } translate-address enabled translate-port enabled vs-index 2 } CIS log output 2025/12/22 18:10:25 [INFO] [Request: 1] cluster local requested CREATE in VIRTUALSERVER nsgoblin/goblin-virtual-server 2025/12/22 18:10:25 [INFO] [Request: 1][AS3] creating a new AS3 manifest 2025/12/22 18:10:25 [INFO] [Request: 1][AS3][BigIP] posting request to https://10.69.12.1 for tenants 2025/12/22 18:10:26 [INFO] [Request: 2] cluster local requested UPDATE in ENDPOINTS nsgoblin/svc-nodeport 2025/12/22 18:10:26 [INFO] [Request: 3] cluster local requested UPDATE in ENDPOINTS nsgoblin/svc-nodeport 2025/12/22 18:10:43 [INFO] [Request: 1][AS3][BigIP] post resulted in SUCCESS 2025/12/22 18:10:43 [INFO] [AS3][POST] SUCCESS: code: 200 --- tenant:kubernetes --- message: success 2025/12/22 18:10:43 [INFO] [Request: 3][AS3] Processing request 2025/12/22 18:10:43 [INFO] [Request: 3][AS3] creating a new AS3 manifest 2025/12/22 18:10:43 [INFO] [Request: 3][AS3][BigIP] posting request to https://10.69.12.1 for tenants 2025/12/22 18:10:43 [INFO] Successfully updated status of VirtualServer:nsgoblin/goblin-virtual-server in Cluster W1222 18:10:49.238444 1 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 2025/12/22 18:10:52 [INFO] [Request: 3][AS3][BigIP] post resulted in SUCCESS 2025/12/22 18:10:52 [INFO] [AS3][POST] SUCCESS: code: 200 --- tenant:kubernetes --- message: success 2025/12/22 18:10:52 [INFO] Successfully updated status of VirtualServer:nsgoblin/goblin-virtual-server in Cluster Troubleshooting: 1. If static routes are not added, the first step is to inspect CIS logs for entries similar to these: Cilium annotation warning logs 2025/12/22 17:44:45 [WARNING] Cilium node podCIDR annotation not found on node ciliumk8s-ubuntu-server, node has spec.podCIDR ? 2025/12/22 17:46:41 [WARNING] Cilium node podCIDR annotation not found on node ciliumk8s-ubuntu-server, node has spec.podCIDR ? 2025/12/22 17:46:42 [WARNING] Cilium node podCIDR annotation not found on node ciliumk8s-ubuntu-server, node has spec.podCIDR ? 2025/12/22 17:46:43 [WARNING] Cilium node podCIDR annotation not found on node ciliumk8s-ubuntu-server, node has spec.podCIDR ? 2. These are resolved by adding annotations to the node using the reference: https://clouddocs.f5.com/containers/latest/userguide/static-route-support.html Cilium annotation for node root@ciliumk8s-ubuntu-server:~# k annotate node ciliumk8s-ubuntu-server io.cilium.network.ipv4-pod-cidr=10.42.0.0/16 root@ciliumk8s-ubuntu-server:~# k describe node | grep -E "Annotations:|PodCIDR:|^\s+.*pod-cidr" Annotations: alpha.kubernetes.io/provided-node-ip: 10.69.12.2 io.cilium.network.ipv4-pod-cidr: 10.42.0.0/16 PodCIDR: 10.42.0.0/24 3. Verify a static route has been created and test connectivity to k8s pods root@(ciliumk8s-bigip)(cfg-sync Standalone)(Active)(/kubernetes)(tmos)# list net route net route k8s-ciliumk8s-ubuntu-server-10.69.12.2 { description 10.69.12.1 gw 10.69.12.2 network 10.42.0.0/16 partition kubernetes } Using pup (command line HTML parser) -> https://commandmasters.com/commands/pup-common/ root@ciliumk8s-ubuntu-server:~# curl -s http://goblin.com/green | pup 'body text{}' I am the green goblin! Access me at /green 1 0.000000 10.69.12.34 ? 10.69.12.40 TCP 78 34294 ? 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=2984295232 TSecr=0 WS=128 2 0.000045 10.69.12.40 ? 10.69.12.34 TCP 78 80 ? 34294 [SYN, ACK] Seq=0 Ack=1 Win=23360 Len=0 MSS=1460 WS=512 SACK_PERM TSval=1809316303 TSecr=2984295232 3 0.001134 10.69.12.34 ? 10.69.12.40 TCP 70 34294 ? 80 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2984295234 TSecr=1809316303 4 0.001151 10.69.12.34 ? 10.69.12.40 HTTP 149 GET /green HTTP/1.1 5 0.001343 10.69.12.40 ? 10.69.12.34 TCP 70 80 ? 34294 [ACK] Seq=1 Ack=80 Win=23040 Len=0 TSval=1809316304 TSecr=2984295234 6 0.002497 10.69.12.1 ? 10.42.0.97 TCP 78 33707 ? 80 [SYN] Seq=0 Win=23360 Len=0 MSS=1460 WS=512 SACK_PERM TSval=1809316304 TSecr=0 7 0.003614 10.42.0.97 ? 10.69.12.1 TCP 78 80 ? 33707 [SYN, ACK] Seq=0 Ack=1 Win=64308 Len=0 MSS=1410 SACK_PERM TSval=1012609408 TSecr=1809316304 WS=128 8 0.003636 10.69.12.1 ? 10.42.0.97 TCP 70 33707 ? 80 [ACK] Seq=1 Ack=1 Win=23040 Len=0 TSval=1809316307 TSecr=1012609408 9 0.003680 10.69.12.1 ? 10.42.0.97 HTTP 149 GET /green HTTP/1.1 10 0.004774 10.42.0.97 ? 10.69.12.1 TCP 70 80 ? 33707 [ACK] Seq=1 Ack=80 Win=64256 Len=0 TSval=1012609409 TSecr=1809316307 11 0.004790 10.42.0.97 ? 10.69.12.1 TCP 323 HTTP/1.1 200 OK [TCP segment of a reassembled PDU] 12 0.004796 10.42.0.97 ? 10.69.12.1 HTTP 384 HTTP/1.1 200 OK 13 0.004820 10.69.12.40 ? 10.69.12.34 TCP 448 HTTP/1.1 200 OK [TCP segment of a reassembled PDU] 14 0.004838 10.69.12.1 ? 10.42.0.97 TCP 70 33707 ? 80 [ACK] Seq=80 Ack=254 Win=23552 Len=0 TSval=1809316308 TSecr=1012609410 15 0.004854 10.69.12.40 ? 10.69.12.34 HTTP 384 HTTP/1.1 200 OK Summary: There we have it, we have successfully deployed an NGINX application on a Kubernetes cluster managed by F5 CIS using static routes to forward traffic to the kubernetes podsF5 BIG-IP deployment with Red Hat OpenShift - keeping client IP addresses and egress flows
Controlling the egress traffic in OpenShift allows to use the BIG-IP for several use cases: Keeping the source IP of the ingress clients Providing highly scalable SNAT for egress flows Providing security functionalities for egress flowsDeploying F5 Distributed Cloud Customer Edge in Red Hat OpenShift Virtualization
Introduction Red Hat OpenShift Virtualization is a feature that brings virtual machine (VM) workloads into the Kubernetes platform, allowing them to run alongside containerized applications in a seamless, unified environment. Built on the open-source KubeVirt project, OpenShift Virtualization enables organizations to manage VMs using the same tools and workflows they use for containers. Why OpenShift Virtualization? Organizations today face critical needs such as: Rapid Migration: "I want to migrate ASAP" from traditional virtualization platforms to more modern solutions. Infrastructure Modernization: Transitioning legacy VM environments to leverage the benefits of hybrid and cloud-native architectures. Unified Management: Running VMs alongside containerized applications to simplify operations and enhance resource utilization. OpenShift Virtualization addresses these challenges by consolidating legacy and cloud-native workloads onto a single platform. This consolidation simplifies management, enhances operational efficiency, and facilitates infrastructure modernization without disrupting existing services. Integrating F5 Distributed Cloud Customer Edge (XC CE) into OpenShift Virtualization further enhances this environment by providing advanced networking and security capabilities. This combination offers several benefits: Multi-Tenancy: Deploy multiple CE VMs, each dedicated to a specific tenant, enabling isolation and customization for different teams or departments within a secure, multi-tenant environment. Load Balancing: Efficiently manage and distribute application traffic to optimize performance and resource utilization. Enhanced Security: Implement advanced threat protection at the edge to strengthen your security posture against emerging threats. Microservices Management: Seamlessly integrate and manage microservices, enhancing agility and scalability. This guide provides a step-by-step approach to deploying XC CE within OpenShift Virtualization, detailing the technical considerations and configurations required. Technical Overview Deploying XC CE within OpenShift Virtualization involves several key technical steps: Preparation Cluster Setup: Ensure an operational OpenShift cluster with OpenShift Virtualization installed. Access Rights: Confirm administrative permissions to configure compute and network settings. F5 XC Account: Obtain access to generate node tokens and download the XC CE images. Resource Optimization: Enable CPU Manager: Configure the CPU Manager to allocate CPU resources effectively. Configure Topology Manager: Set the policy to single-numa-node for optimal NUMA performance. Network Configuration: Open vSwitch (OVS) Bridges: Set up OVS bridges on worker nodes to handle networking for the virtual machines. NetworkAttachmentDefinitions (NADs): Use Multus CNI to define how virtual machines attach to multiple networks, supporting both external and internal connectivity. Image Preparation: Obtain XC CE Image: Download the XC CE image in qcow2 format suitable for KubeVirt. Generate Node Token: Create a one-time node token from the F5 Distributed Cloud Console for node registration. User Data Configuration: Prepare cloud-init user data with the node token and network settings to automate the VM initialization process. Deployment: Create DataVolumes: Import the XC CE image into the cluster using the Containerized Data Importer (CDI). Deploy VirtualMachine Resources: Apply manifests to deploy XC CE instances in OpenShift. Network Configuration Setting up the network involves creating Open vSwitch (OVS) bridges and defining NetworkAttachmentDefinitions (NADs) to enable multiple network interfaces for the virtual machines. Open vSwitch (OVS) Bridges Create a NodeNetworkConfigurationPolicy to define OVS bridges on all worker nodes: apiVersion: nmstate.io/v1 kind: NodeNetworkConfigurationPolicy metadata: name: ovs-vms spec: nodeSelector: node-role.kubernetes.io/worker: '' desiredState: interfaces: - name: ovs-vms type: ovs-bridge state: up bridge: allow-extra-patch-ports: true options: stp: true port: - name: eno1 ovn: bridge-mappings: - localnet: ce2-slo bridge: ovs-vms state: present Replace eno1 with the appropriate physical network interface on your nodes. This policy sets up an OVS bridge named ovs-vms connected to the physical interface. NetworkAttachmentDefinitions (NADs) Define NADs using Multus CNI to attach networks to the virtual machines. External Network (ce2-slo): External Network (ce2-slo): Connects VMs to the physical network with a specific VLAN ID. This setup allows the VMs to communicate with external systems, services, or networks, which is essential for applications that require access to resources outside the cluster or need to expose services to external users. apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: ce2-slo namespace: f5-ce spec: config: | { "cniVersion": "0.4.0", "name": "ce2-slo", "type": "ovn-k8s-cni-overlay", "topology": "localnet", "netAttachDefName": "f5-ce/ce2-slo", "mtu": 1500, "vlanID": 3052, "ipam": {} } Internal Network (ce2-sli): Internal Network (ce2-sli): Provides an isolated Layer 2 network for internal communication. By setting the topology to "layer2", this network operates as an internal overlay network that is not directly connected to the physical network infrastructure. The mtu is set to 1400 bytes to accommodate any overhead introduced by encapsulation protocols used in the internal network overlay. apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: ce2-sli namespace: f5-ce spec: config: | { "cniVersion": "0.4.0", "name": "ce2-sli", "type": "ovn-k8s-cni-overlay", "topology": "layer2", "netAttachDefName": "f5-ce/ce2-sli", "mtu": 1400, "ipam": {} } VirtualMachine Configuration Configuring the virtual machine involves preparing the image, creating cloud-init user data, and defining the VirtualMachine resource. Image Preparation Obtain XC CE Image: Download the qcow2 image from the F5 Distributed Cloud Console. Generate Node Token: Acquire a one-time node token for node registration. Cloud-Init User Data Create a user-data configuration containing the node token and network settings: #cloud-config write_files: - path: /etc/vpm/user_data content: | token: <your-node-token> slo_ip: <IP>/<prefix> slo_gateway: <Gateway IP> slo_dns: <DNS IP> owner: root permissions: '0644' Replace placeholders with actual network configurations. This file automates the VM's initial setup and registration. VirtualMachine Resource Definition Define the VirtualMachine resource, specifying CPU, memory, disks, network interfaces, and cloud-init configurations. Resources: Allocate sufficient CPU and memory. Disks: Reference the DataVolume containing the XC CE image. Interfaces: Attach NADs for network connectivity. Cloud-Init: Embed the user data for automatic configuration. Conclusion Deploying F5 Distributed Cloud CE in OpenShift Virtualization enables organizations to leverage advanced networking and security features within their existing Kubernetes infrastructure. This integration facilitates a more secure, efficient, and scalable environment for modern applications. For detailed deployment instructions and configuration examples, please refer to the attached PDF guide. Related Articles: BIG-IP VE in Red Hat OpenShift Virtualization VMware to Red Hat OpenShift Virtualization Migration OpenShift Virtualization
feature request: container egress service
After installing cis in a test environment and getting ready to install in a new production environment I wonder if there also will be a container egress service (CES)? It is very easy to set a gateway for selected namespaces with AdminPolicyBasedExternalRoute in Openshift. See, F5 BIG-IP deployment with Red Hat OpenShift - keeping client IP addresses and egress flows | DevCentral The solution above does not scale well if multiple namespace-egress IP address mappings are desired. A nice solution would be a CES that watches the creating and deletion of pods in selected namespaces. Then it can manage address lists with the pods ip addresses in the F5 ltm. Forwarding ip virtual services will use these address lists to match pod ip addresses to an egress ip defined in a snat pool. Also the creation and deletion of forwarding ip virtual servers and address lists could be managed with a "CES". A possible issue is that a container in a pod can start network connections before the forwarding IP virtual server accepts the new pod IP address. But this can easily be solved with adding an initcontainer in the pod that tests the network connectivity. This would be a good alternative for Openshift egress IPs or Istio gateways. Reason to want this, is to offer applications on Openshift an own egress IP address and stop using the node IP address for external network connections of the pods.
Health Monitor unable to connect to OpenShift Router
Hi, We have F5 VS routing traffic to a service behind OpenShift Router ( We are not using F55 CIS ). The OpenShift Route is configured as TLS Passthrough. I want to re-encrypt TLS at F5. In case of TLS Passthrough configuration OpenShift Router determines the route based on TLS Client Hello hostname. So I have OPS Route with host name “my-tls-passthrough-service.com” and I have F5 VS with hostname “my-f5vs.com” and a pool with single member pointing to OPS Router IP and port 443 . I have configured Client and Server SSL profiles. Also, in server SSL profile I have set “Server Name” attribute to “my-tls-passthrough-service.com” . Everything works as expected – the request reaches the service through F5 . The problem I have is when I configure Health Monitor. The generic HTTPS monitor doesn’t help as it checks the status of OPS Router , not the service behind it. But when I add ServerSSL profile to Health check monitor I get pool member marked down and message in local traffic log “Unable to connect “ Can you please help - without health monitor the set up is uselessGlobal Live Webinar (02/05): Securing Model Serving in Red Hat OpenShift AI with F5API Security
Securing Model Serving in Red Hat OpenShift AI with F5 Distributed Cloud API Security This webinar event is open to all regardless of geographic location. Date: Wednesday, February 5, 2025 Time: 10:00am PT | 1:00pm ET F5 Speaker: Eric Ji. Sr. Solutions Architect, F5 Guest Speaker: E.G. Nadhan Field CTO Ambassador Red Hat What's the webinar about? Join industry experts from Red Hat and F5 in an insightful webinar focused on securely scaling generative AI workloads through the integration of Red Hat OpenShift AI and F5 Distributed Cloud API Security across the open hybrid cloud. Discover how OpenShift AI offers a powerful, scalable MLOps platform for developing, training, and deploying AI models, while F5 Distributed Cloud provides advanced API security to protect your inference endpoints. In this session, we will also delve into how F5's AI Factory approach facilitates the deployment of AI applications across hybrid and multicloud environments, ensuring they are secure, scalable, and reliable. Learn more, register today https://www.f5.com/company/events/webinars/securing-model-serving-in-red-hat-openshift-ai-with-f5-distributed-cloud-api-security
