Hey did you know?
Most folks know F5 for application delivery and security. You may not know that F5 is also out in front on AI application security. Gartner recently positioned F5 as a Market Shaper in its Emerging Market Quadrant for AI application security.
As part of that work, F5 developed the F5 AI Reference Architecture. It lays out the building blocks for developing, orchestrating and delivering AI workloads and services. It also spells out the challenges and risks an enterprise takes on once it starts delivering those workloads.
A reference architecture looks great on a whiteboard, but I wanted to see what it looks like when you actually build it. Over the next few installments of “How I did it”, we’ll do that by following an example enterprise, RedSeaLab Bank, as it starts its AI journey.
Meet RedSeaLab Bank
RedSeaLab’s executive team sees the value and business impact AI can bring. They’ve given management a new initiative with two goals:
- Integrate AI into the bank’s products and services.
- Find ways to put AI tools in the hands of employees.
It’s a bank, so there are conditions. Every implementation has to account for:
- Accessibility and visibility
- Scalability and availability
- Security of applications, services and models, (both publicly hosted and privately hosted)
Let’s Chat!
For the first project, management decided to add a virtual assistant to the bank’s public-facing online banking application. Customers can use it for general chat. More importantly, it has to answer banking-specific questions about the bank’s products and services, rates and fees, and so on. A general-purpose model doesn’t know RedSeaLab’s current CD rates or overdraft fees. That information lives in the bank’s own documents, so the assistant needs Retrieval-Augmented Generation (RAG).
The online banking app has been deployed on [Red Hat OpenShift] (Red Hat OpenShift) and irrespective of its new AI initiatives the current infrastructure includes multiple F5 products and services to support availability, scalability, and security. To support the new virtual assistant, the bank has integrated several services.
-
NVIDIA GPU Operator - The operator automates the provisioning and management of NVIDIA GPUs running on Red Hat OpenShift, including GPU drivers, device plugins, monitoring, and container runtime integration. It enables workloads such as the models used in this demonstration to request and consume GPU resources through standard Kubernetes resource definitions.
For this infrastructure, the bank has deployed a total of three (3) GPUs in the cluster and configured time-slicing. GPU time-slicing allows multiple workloads to share a single physical GPU by rapidly scheduling each workload for a slice of GPU execution time. Currently the cluster hosts models to support f5 AI Guardrails/Red Team and embedding.
-
Hugging Face - During initial startup, the vLLM runtime retrieves the model artifacts used for embedding from the Hugging Face model repository and stores in local OpenShift persistent volume. vLLM then loads the model onto the assigned NVIDIA GPU.
-
Flowise - Visual AI application/orchestration layer that loads documents and coordinates document parsing and chunking. Flowise queries Qdrant to retrieve the most relevant chunks for the request.
-
QDrant - Vector database designed to store and search embeddings using vector similarity. It enables Flowise to quickly retrieve the most relevant document chunks and provide them as context.
-
MinIO AIStor - Provides S3-compatible storage of bank knowledge base documents used in Retrieval-Augmented Generation (RAG). The documents, (PDFs, Word files, Excel spreadsheets, etc.) are subsequently loaded into Flowise for chunking and embedding generation.
-
Inference and Embedding models - The architecture utilizes a locally hosted model for embedding that converts each document chunk into a numerical vector that is stored in Qdrant along with metadata about the source chunk. The bank currently utilizes a publicly available model for inference.
Securing inference with F5 ADSP
Exposing and securing the front door
To ensure availability, scalability and security of the bank’s online presence, RedSeaLab utilizes a number of F5 ADSP platform product and services. These services work together to enable secure access to the bank’s online banking platform as well as control plane acess to servcces used by the bank to deliver contextual inference.
-
NGINX Gateway Link - To expose its container-based banking platform running in OpenShfit, I used GatewayLink to connect CIS and BIG-IP to NGINX Gateway Fabric, (NGF). This gave BIG-IP a stable NGINX data plane endpoint rather than exposing it to the constant changes happening at the pod level while allowing NetOps and DevOps to maintain their respective areas of control. NGF provides the layer-7 load balancing, TLS termination and WAF security services. BIG-IP CIS provides external layer-4 load balancing of NGINX gateway services.
-
F5 BIG-IP Container Ingress Services (CIS) - In addition to the aforementioned integration via Gateway Link, the bank uses CIS to provide direct layer-7 access to OpenShift hosted Flowise and Qdrant services. Access to these services is more limited and less dynamic nature. Subsequently, related cluster changes are minimized and direct CIS integration provides for more efficient delivery without the concern for excessive control plane traffic on the BIG-IP.
-
F5 WAF for BIG-IP - In conjunction with CIS, the BIG-IP provides Web Application and API Protection, (WAAP) security for Flowise and Qdrant management endpoints.
Securing inference
-
F5 AI Guardrails - AI Guardrails sits inline between the chat application and the OpenAI endpoint. It inspects every prompt before it goes out and every response before it reaches your users. Because it doesn’t touch the model, it works with a hosted service like OpenAI. The flow works like this:
- The user sends a prompt to the app or orchestrator.
- The orchestrator calls AI Guardrails. Guardrails scans the prompt and forwards it to the model endpoint only if it’s clean.
- The model’s response comes back through Guardrails, which scans it again before it goes to the user.
What does it protect against? On the way in, it catches prompt injection attempts meant to make the assistant ignore its instructions. On the way out, it stops sensitive data from leaking into a response. For a customer-facing bank chat, the outbound check matters as much as the inbound one. Red Hat provides a quick start for easily deploying F5 AI Guardrails and F5 Red Team on OpenShift.
-
F5 AI Red Team - AI Red Team tests the chat app with adversarial attacks. Its findings feed back into the Guardrails policies, so a weakness found in testing gets covered by a policy in production.
-
F5 BIG-IP Next for Kubernetes (BNK) - BNK runs in host mode and provides load balancing and secure access to the locally hosted models. Right now that covers the embedding model and the models that support F5 AI Guardrails. The bank uses a publicly hosted model for inference today. In a future installment we’ll add a locally hosted inference model and configure an AI gateway that provides, among other features, failover to the public model.
Video walkthrough - Inference with RAG
RAG corpus management
If someone slips a bad document into the knowledge base, the assistant will repeat it to customers without hesitation. That makes the corpus as important to protect as the model.
RedSeaLab Bank stores every knowledge base document it uses for RAG in MinIO AIStor, which is currently hosted on OpenShift. Administrators can reach AIStor in several ways, (CLI, UI, 3rd party clients, SDK, etc.).
External S3 access
The bank exposes the AIStor endpoint externally using BIG-IP CIS. It applies the S3-specific tuning based on the F5 BIG-IP and MinIO AIStor Deployment Guide for S3 Data Delivery and puts BIG-IP WAAP in front of the endpoint.
Guarded document uploads
The bank also built a custom feature into the online app that lets administrators upload documents to AIStor. This endpoint connects to AIStor directly inside the OpenShift cluster and calls AI Guardrails out-of-band through its API.
Using a mix of canned and custom guardrails, each document is scanned, then either sanitized or blocked depending on which guardrails it triggers. In effect it works as a customizable DLP solution that keeps the corpus from being poisoned. The RedSeaLab Bank repo on GitHub walks through how I built the three-step guardrails scan pipeline.
Video Walkthrough - RAG Corpus Management
What’s next
So far RedSeaLab has a working RAG chat assistant built on the reference architecture, with F5 placed on each connection that needs it. In an upcoming installment, the bank will bring inference in-house with a locally hosted model. It will also add an AI gateway that can fail over to the public model when needed. See you then.
Getting by with a little help from my AI friend
Yes, even an article about architecting secure AI pipelines got a little help from generative AI with grammar checking, formatting polish, and a dash of wordsmithing. Rest assured, strict human guardrails were enforced from draft to final publish. The AI just helped me sound smarter than I do before my first coffee.






