If you’re managing Kubernetes clusters right now, you already know the drill: standard Layer 7 load balancing works flawlessly for web APIs where requests resolve in milliseconds. But the moment you start hosting Large Language Models (LLMs), that traditional routing logic falls apart.
AI inference workloads are a completely different beast. You have to account for GPU memory, active inference queues, and KV-caches. If you rely on basic proxying, you usually end up with incredibly expensive GPUs tied up handling lightweight tasks, while your developers are forced to write custom middleware just to orchestrate traffic.
We needed a way to bring intent-driven networking directly to the AI edge. That’s exactly what the Gateway API Inference Extension does. By pairing this with F5 NGINX Gateway FabricWe can transform a standard Kubernetes Gateway into a dedicated Inference Gateway.
Let’s look at how this changes the game for platform teams.
The Two-Stage Architecture
To make intelligent routing decisions, we use a two-stage architecture that separates high-level routing intent from real-time endpoint selection.
- Stage 1 (Intent): We use standard Kubernetes HTTPRoutes to define exactly where traffic should go based on paths, headers, or weights.
- Stage 2 (Real-Time Selection): Instead of routing blindly to backend pods, we target an InferencePool CRD. This pool uses an Endpoint Picker to evaluate real-time node telemetry (like queue depth) and picks the absolute best pod for the job.
To prove this is running under the hood, we can describe our GPU and CPU InferencePools. Notice how each pool has a dedicated Endpoint Picker attached and ready to route traffic based on real-time node health.
GPU Pool Endpoint Picker:
kubectl describe inferencepool ollama-inferencepool -n ollama | grep -A 10 "Endpoint Picker"
Endpoint Picker Ref:
Failure Mode: FailClose
Group:
Kind: Service
Name: ollama-inferencepool-epp
Port:
Number: 9002
Selector:
Match Labels:
App: ollama --> GPU
Target Ports:
CPU Pool Endpoint Picker:
kubectl describe inferencepool ollama-cpu-pool -n ollama | grep -A 10 "Endpoint Picker"
Endpoint Picker Ref:
Failure Mode: FailOpen
Group:
Kind: Service
Name: ollama-cpu-pool-epp
Port:
Number: 9002
Selector:
Match Labels:
App: ollama-cpu --> CPU
Target Ports:
This separation of routing intent from real-time endpoint selection allows platform engineers to solve three critical AI infrastructure challenges without requiring developers to write custom middleware.
A quick note on resilience: Notice the Failure Mode in those outputs? This defines what happens if the Endpoint Picker itself goes offline. For our expensive GPU pool, we set it to FailClose (rejecting traffic so we don’t overwhelm premium hardware blindly). For our CPU efficiency tier, we set it to FailOpen (falling back to standard round-robin load balancing to keep the application alive).
Let’s look at three practical ways to use this setup to optimize your hardware.
1. Model-Aware Routing: Stop Using GPUs for Everything
Diagram 1: Model Aware Routing
Treating all AI traffic equally is the fastest way to exhaust a hardware budget. High-performance GPUs should be strictly reserved for complex generation tasks, while efficiency tiers (such as CPUs) should handle lightweight tasks, such as audio transcription or basic summarization.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: model-aware-httproute
namespace: ollama
spec:
parentRefs:
- name: inference-gateway
namespace: nginx-gateway
sectionName: http
rules:
# --- UI Preservation Rules ---
- matches:
- path: { type: PathPrefix, value: /ui }
filters:
- type: URLRewrite
urlRewrite:
path: { type: ReplacePrefixMatch, replacePrefixMatch: / }
backendRefs:
- name: chatbot
port: 8501
- matches:
- path: { type: PathPrefix, value: /static }
- path: { type: Exact, value: /favicon.png }
backendRefs:
- name: chatbot
port: 8501
# --- AI Routing Rules ---
- matches:
- path: { type: PathPrefix, value: /v1/audio }
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: ollama-cpu-pool. # CPU Pool Fallback
port: 11434
- matches:
- path: { type: PathPrefix, value: /v1/chat }
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: ollama-inferencepool. # GPU Pool Fallback
port: 11434
Using path-based routing, the Gateway acts as an intelligent traffic cop. In the configuration below, we map the /v1/audio path directly to a CPU InferencePool. If a request hits this endpoint, the Gateway seamlessly offloads it to the efficiency tier, protecting our premium GPUs from trivial workloads.
2. Canary Deployments
In MLOps, rolling out a new LLM is inherently risky. Performance regressions, latency spikes, and hallucinations are real threats to the user experience. You cannot simply cut over all production traffic to a new model overnight.
Diagram 2: Canary Testing
Native traffic splitting provides a safety net for model validation. By configuring a deterministic weight at the Gateway level, 90% of users continue to be served by the stable production GPU pool, while 10% of traffic is routed to the efficiency tier for real-time validation.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: canary-httproute
namespace: ollama
spec:
parentRefs:
- name: inference-gateway
namespace: nginx-gateway
sectionName: http
rules:
# --- UI Preservation Rules ---
- matches:
- path: { type: PathPrefix, value: /ui }
filters:
- type: URLRewrite
urlRewrite:
path: { type: ReplacePrefixMatch, replacePrefixMatch: / }
backendRefs:
- name: chatbot
port: 8501
- matches:
- path: { type: PathPrefix, value: /static }
- path: { type: Exact, value: /favicon.png }
backendRefs:
- name: chatbot
port: 8501
# --- Use Case 2: 90/10 Canary Split ---
- matches:
- path: { type: PathPrefix, value: /v1/chat }
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: ollama-inferencepool. # GPU Pool Fallback
weight: 90
port: 11434
- group: inference.networking.k8s.io
kind: InferencePool
name: ollama-cpu-pool. # CPU Pool Fallback
weight: 10
port: 11434
Because this is handled purely via declarative YAML, platform teams can execute risk-free canary tests without altering any application code.
3. Cost Optimization (Header-Based):
When an AI cluster is under maximum pressure, standard load balancers process requests on a first-in, first-out basis. In an enterprise environment, this is unacceptable. Mission-critical workflows and premium users must have guaranteed access to the best compute resources.
Diagram 3: Cost Optimization
By utilizing custom HTTP headers, client applications can signal their importance to the Gateway. The NGINX semantic engine reads these headers in real-time. If the x-query-complexity: high header is present, the request is immediately fast-tracked to the premium GPU pool. Every other request falls back to the CPU tier.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: header-based-httproute
namespace: ollama
spec:
parentRefs:
- name: inference-gateway
namespace: nginx-gateway
sectionName: http
rules:
# --- UI Preservation Rules ---
- matches:
- path: { type: PathPrefix, value: /ui }
filters:
- type: URLRewrite
urlRewrite:
path: { type: ReplacePrefixMatch, replacePrefixMatch: / }
backendRefs:
- name: chatbot
port: 8501
- matches:
- path: { type: PathPrefix, value: /static }
- path: { type: Exact, value: /favicon.png }
backendRefs:
- name: chatbot
port: 8501
# --- Use Case 3: Priority Steering ---
- matches:
- headers:
- type: Exact
name: x-query-complexity
value: high
path: { type: PathPrefix, value: /v1/chat }
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: ollama-inferencepool # GPU Pool
port: 11434
- matches:
- path: { type: PathPrefix, value: /v1/chat }
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: ollama-cpu-pool # CPU Pool Fallback
port: 11434
This enforces strict SLAs regardless of overall cluster load, so your most valuable transactions stay running.
See the Architecture in Action
Managing complex AI traffic across heterogeneous hardware doesn’t have to be a headache. By utilizing F5 NGINX Gateway Fabric, you can optimize compute, validate models safely, and prioritize critical traffic—all through declarative, intent-driven configurations.
To see exactly how these routing rules are executed, check out my full technical walk-through below:
Resources :
[NGINX Community Blog] Read the official announcement and dive deeper into how NGF supports this new extension.
[NGINX Gateway Fabric Documentation] The official documentation for deploying and configuring NGINX Gateway Fabric in your Kubernetes environment.
[NGINX Gateway Fabric (GitHub)] Explore the upstream development, CRD definitions, and architecture of the Inference Extension project.