Manage Thousands of F5 NGINX Ingress VirtualServerRoutes with Kustomize
Introduction
F5 NGINX Ingress VirtualServerRoutes require maintaining route files and manually updating VirtualServer references — a dual-maintenance problem that doesn't scale. This tutorial automates the integration so you only manage route files, and everything else is generated automatically.
The Challenge
A recent customer engagement highlighted a common scalability problem with F5 NGINX Ingress Controller: managing thousands of routes across multiple domains within monolithic VirtualServer manifests.
The Scale Problem:
- Few domains (2-5 host configurations)
- Thousands of routes per domain (1000+ first-level paths like /api, /admin, /user, and more)
- Multiple teams contributing routes independently
- Frequent updates requiring full manifest redeployment
Why Single Manifests Don't Scale:
A traditional VirtualServer with 1000+ routes becomes:
- Unmaintainable: 5000+ line YAML files that are difficult to read and edit
- Error-prone: Small syntax errors break the entire routing configuration
- Collaboration nightmare: Multiple teams editing the same massive file creates merge conflicts
- Deployment bottleneck: Any route change requires redeploying the entire configuration
- Rollback complexity: Failed deployments affect all routes, not just the problematic ones
Example of the problem:
# A single VirtualServer with hundreds of routes becomes unwieldy
spec:
routes:
- path: /api/v1/users
action: { pass: users-service }
- path: /api/v1/orders
action: { pass: orders-service }
- path: /api/v1/payments
action: { pass: payments-service }
# ... 997 more routes
- path: /admin/dashboard
action: { pass: admin-service }The Standard VirtualServerRoute Problem
Even when breaking routes into individual VirtualServerRoute manifests, NGINX Ingress Controller requires dual maintenance:
1. Create the VirtualServerRoute file:
# routes/payment-route.yaml
apiVersion: k8s.nginx.org/v1
kind: VirtualServerRoute
metadata:
name: payment-route
spec:
# Route configuration here2. Manually add the route reference to the VirtualServer:
# virtualserver.yaml - Must be edited separately
spec:
routes:
- path: /payment
route: nginx-ingress/payment-route # Manual reference requiredThis creates ongoing maintenance overhead:
- Two-place editing: Every route addition/removal requires changes in two files
- Synchronization errors: Easy to forget updating the VirtualServer references
- Manual bookkeeping: Developers must remember to add route references
- Deployment complexity: Both files must be updated and deployed together
The Solution
This tutorial eliminates the dual-maintenance problem through complete automation:
Your workflow becomes:
- ✅ Add route: Create new-route.yaml in the routes/ directory
- ✅ Remove route: Delete the file from routes/ directory
- ✅ Deploy: Run ./build-kustomize.sh && kustomize build . | kubectl apply -f -
Everything else is automatically generated:
- Route references in the VirtualServer
- Kustomize configuration files
- Proper resource inclusion and patching
The result: Transform thousands of routes requiring dual maintenance into a streamlined workflow where you only manage route files - all integration is automated.
Critical Requirements for VirtualServerRoute
Before creating any route files, ensure each VirtualServerRoute includes:
- Required host field: Must exactly match the host in your main VirtualServer
- Correct path prefixes: Subroute paths must begin with the route path defined in the VirtualServer
Example showing the relationship:
# In VirtualServer (auto-generated by script):
- path: /login
route: nginx-ingress/login-route
# In VirtualServerRoute file:
spec:
host: myapp.example.com # REQUIRED: Must match VirtualServer host
subroutes:
- path: /login/details # REQUIRED: Must start with /login
- path: /login/status # REQUIRED: Must start with /loginDirectory Structure
Your project should follow this structure. The kustomization.yaml and routes-patch.yaml files are auto-generated.
.
├── base/
│ ├── kustomization.yaml
│ └── virtualserver.yaml # Base VS definition
├── routes/
│ ├── login-route.yaml # Example VSR
│ └── profile-route.yaml # Example VSR
├── build-kustomize.sh # The single script to generate Kustomize configs
├── .gitignore
├── kustomization.yaml # <-- Generated by script
└── routes-patch.yaml # <-- Generated by scriptSetup Instructions
1. Create the Base VirtualServer
Define your main VirtualServer without any of the dynamic route references. This file should contain defaults, TLS configuration, and upstreams that might be shared or used by the base path.
# base/virtualserver.yaml
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: main-application
namespace: nginx-ingress # Ensure this matches your environment
spec:
host: myapp.example.com
tls:
secret: myapp-tls
redirect:
enable: true
upstreams:
- name: main-app
service: nginx-test-svc
port: 80
# Define a base/default route here if needed.
routes:
- path: /
action:
pass: main-app
# Dynamic routes for VirtualServerRoutes will be added via patch.2. Create Base Kustomization File
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- virtualserver.yaml3. Create the Kustomize Generator Script
This is the core of the automated solution. This single script finds all *-route.yaml files in the routes/ directory and generates both the kustomization.yaml and routes-patch.yaml files.
#!/bin/bash
# build-kustomize.sh
#
# This script automatically generates BOTH the kustomization.yaml AND the
# routes-patch.yaml files based on the contents of the 'routes/' directory.
#
set -e
# --- Configuration ---
ROUTES_DIR="routes"
KUSTOMIZATION_FILE="kustomization.yaml"
PATCH_FILE="routes-patch.yaml"
NAMESPACE="nginx-ingress" # Ensure this matches your VS/VSR namespace
# --- Part 1: Generate kustomization.yaml ---
echo "Generating ${KUSTOMIZATION_FILE}..."
# Start with the static parts of the kustomization file
cat > $KUSTOMIZATION_FILE << EOL
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# Include the base VirtualServer definition
- base
EOL
# Find all route files and append them to the resources list
# The shell's glob expansion (*) is reliable and used here to build the file.
for route_file in ${ROUTES_DIR}/*-route.yaml; do
if [[ -f "$route_file" ]]; then
# Add the file path to the kustomization resources list
echo " - ${route_file}" >> $KUSTOMIZATION_FILE
fi
done
# Add the final static patch section
cat >> $KUSTOMIZATION_FILE << EOL
patches:
- path: ${PATCH_FILE}
EOL
echo "Successfully generated ${KUSTOMIZATION_FILE}."
# --- Part 2: Generate routes-patch.yaml ---
echo -e "\nGenerating ${PATCH_FILE}..."
# Start with the patch header
cat > $PATCH_FILE << EOL
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: main-application
namespace: ${NAMESPACE}
spec:
routes:
# Include the base route with its action
- path: /
action:
pass: main-app
EOL
# Find all route files and append their references to the patch
find "${ROUTES_DIR}" -maxdepth 1 -name '*-route.yaml' -print0 | while IFS= read -r -d $'\0' route_file; do
if [[ -f "$route_file" ]]; then
resource_name=$(basename "$route_file" .yaml)
path_prefix=$(echo "$resource_name" | sed 's/-route$//')
cat >> $PATCH_FILE << EOL
# Route added from ${route_file}
- path: /${path_prefix}
route: ${NAMESPACE}/${resource_name}
EOL
fi
done
echo "Successfully generated ${PATCH_FILE}."Make the script executable:
chmod +x build-kustomize.sh4. Create an Initial Route File
Create at least one VirtualServerRoute file in the routes/ directory so the script has something to find.
# routes/login-route.yaml
apiVersion: k8s.nginx.org/v1
kind: VirtualServerRoute
metadata:
name: login-route
namespace: nginx-ingress # Should match NAMESPACE in script and VS
spec:
host: myapp.example.com # REQUIRED: Must match VirtualServer host
upstreams:
- name: login-service
service: login-svc
port: 80
subroutes:
- path: /login/details # REQUIRED: Must start with /login prefix
action:
pass: login-service
- path: /login/status # REQUIRED: Must start with /login prefix
action:
pass: login-service5. Git Ignore Generated Files
Add the generated files to your .gitignore to avoid committing them to version control.
# .gitignore
kustomization.yaml
routes-patch.yaml
Workflow
Your daily workflow is now streamlined. You only need to manage files in the routes/ directory and run the helper script.
Initial Generation and Deployment
1. Run the generator script to create your Kustomize configuration for the first time
./build-kustomize.sh2. (Optional) Verify the changes with a dry-run build.
kustomize build .This shows you the final, combined manifest that will be sent to the cluster. Inspect it to ensure your base VirtualServer has the new route reference and that the VirtualServerRoute resource is included.
3. Apply the changes to your cluster.
kustomize build . | kubectl apply -f -Adding a New Route
1. Create a new VirtualServerRoute file in the routes/ directory (e.g., account-route.yaml). Ensure the filename follows the name-route.yaml convention and includes the required host field
# routes/account-route.yaml
apiVersion: k8s.nginx.org/v1
kind: VirtualServerRoute
metadata:
name: account-route
namespace: nginx-ingress
spec:
host: myapp.example.com # REQUIRED: Must match VirtualServer host
upstreams:
- name: account-service
service: account-svc
port: 80
subroutes:
- path: /account/profile # REQUIRED: Must start with /account
action:
pass: account-service
- path: /account/settings # REQUIRED: Must start with /account
action:
pass: account-service2. Run the generator script.
./build-kustomize.sh3. Apply the changes.
kustomize build . | kubectl apply -f -Removing an Existing Route
1. Delete the file from the routes/ directory (e.g., rm routes/login-route.yaml).
2. Run the generator script. The script will see the file is gone and will automatically regenerate kustomization.yaml and routes-patch.yaml without the corresponding entries.
./build-kustomize.sh3. Apply the changes. kubectl apply will update the VirtualServer to remove the route reference and will remove the VirtualServerRoute object from the cluster (if using --prune or a GitOps tool).
kustomize build . | kubectl apply -f -
Troubleshooting
Validating Your Configuration
After applying changes, always check the status of your resources to ensure they were accepted:
# Check VirtualServer status
kubectl describe virtualserver main-application -n nginx-ingress
# Check VirtualServerRoute status
kubectl describe virtualserverroutes -n nginx-ingressUnderstanding Events and Warnings
When troubleshooting, pay attention to the most recent events in the output. Early warning events are often remnants from the deployment sequence and can be safely ignored if recent events show success.
Example of a normal deployment sequence:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning AddedOrUpdatedWithWarning 55s nginx-ingress-controller Configuration for nginx-ingress/main-application was added or updated with warning(s): VirtualServerRoute nginx-ingress/login-route doesn't exist or invalid
Warning AddedOrUpdatedWithWarning 55s nginx-ingress-controller Configuration for nginx-ingress/main-application was added or updated with warning(s): VirtualServerRoute nginx-ingress/logout-route doesn't exist or invalid
Normal AddedOrUpdated 54s nginx-ingress-controller Configuration for nginx-ingress/main-application was added or updated
Normal AddedOrUpdated 54s nginx-ingress-controller Configuration for nginx-ingress/main-application was added or updatedInterpretation:
- 55s ago: VirtualServer applied/updated before VirtualServerRoutes existed - NGINX warns about missing dependencies
- 54s ago: VirtualServerRoutes created, VirtualServer automatically becomes valid
- Current state: All dependencies satisfied, configuration valid (focus on the most recent AddedOrUpdated events)
Common Issues
1. Missing host field: VirtualServerRoute resources must include spec.host that matches the VirtualServer host
Error: "spec.host: Required value"
Solution: Add host: myapp.example.com to each VirtualServerRoute2. Invalid path prefixes: Subroute paths must start with the route path from the VirtualServer
Error: Path mismatch
Solution: Ensure /login/details starts with /login prefix3. Namespace mismatch: All resources must be in the same namespace
Error: VirtualServerRoute doesn't exist
Solution: Verify all resources use the same namespace4. Upstream not found: VirtualServer references an upstream that doesn't exist
Error: "spec.routes[0].action.pass: Not found: 'upstream-name'"
Solution: Ensure upstream name in routes matches upstream name in upstreams sectionChecking Resource Status
Look for these indicators of success:
VirtualServer:
- State: Valid
- Reason: AddedOrUpdated
- Recent Normal AddedOrUpdated events
VirtualServerRoute:
- State: Valid
- Referenced By: nginx-ingress/main-application
- Recent Normal AddedOrUpdated events
If you see State: Invalid or Reason: Rejected, check the error message and events for specific guidance on what needs to be fixed.
Eliminating Dependency Warnings
The warnings shown above are normal during deployment but can be eliminated by applying resources in the correct order:
# Apply VirtualServerRoutes first
kustomize build . | yq 'select(.kind == "VirtualServerRoute")' | kubectl apply -f -
# Then apply VirtualServer
kustomize build . | yq 'select(.kind == "VirtualServer")' | kubectl apply -f -This ensures VirtualServerRoutes exist before the VirtualServer references them, preventing dependency warnings.
Employee