Deploying an NGINX App across Kubernetes Multi-clusters with F5 BIG-IP Container Ingress Services

Articles F5 Technical Articles
, , , , , , ,
1

Overview

Modern enterprise applications demand high availability, geographic distribution, and resilient infrastructure that can withstand regional failures. As Kubernetes adoption accelerates, organizations are increasingly deploying applications across multiple clusters to achieve these goals. However, managing traffic distribution, service discovery, and load balancing across disparate Kubernetes environments introduces significant complexity.

F5 Container Ingress Services (CIS) addresses these challenges by providing intelligent, centralized traffic management for multi-cluster Kubernetes deployments.

In this article we will walk through deploying an NGINX application across two Kubernetes clusters kube-east and kube-west. In the example configuration, a virtual server custom resource (CR) will be used to expose this application. Please see  https://clouddocs.f5.com/containers/latest/userguide/multicluster/ for additional details regarding multi-cluster support and deployment.

Architecture Overview

The deployment architecture simulates a multi-cluster environment using the following components:

  • Infrastructure: Single control plane (k8s-master) managing two worker nodes representing geographically distributed clusters
  • Network: Flannel CNI with NodePort services exposing applications on ports 30080 (HTTP) and 30443 (HTTPS)
  • Traffic Management: F5 BIG-IP (10.124.207.1) with AS3 extension routing traffic to distributed application instances. A partition called “multicluster” for AS3 configuration objects
  • Application: Nginx 1.25-alpine containers serving custom HTML content over HTTP/HTTPS with TLS certificates
  • Load Distribution: 50/50 traffic split between kube-east (10.124.207.3) and kube-west (10.124.207.4) clusters

image_346063.png
image_346063.png1454Ă—838 61.7 KB

Kubeadm Multi-Cluster Deployment with BIG-IP

The multi-cluster architecture will be simulated by:

  • Creating separate kubeconfig contexts (kube-east, kube-west)
  • Both contexts point to the same API server (10.124.207.2:6443)
  • F5 CIS sees each context as “external clusters”
  • An extended configmap will be deployed to enable CIS run in multi-cluster mode

By the end of this guide, we will have a fully functional multi-cluster Kubernetes environment managed by F5 CIS, capable of intelligently routing HTTPS traffic to a distributed Nginx application with health monitoring, TLS encryption, and  load balancing.

Kubeadm is selected as the Kubernetes distribution to bootstrap the cluster using the networking reference details below

image_346063.png
image_346063.png1476Ă—506 51.1 KB

Kubeadm Network Configuration

Kubeadm deployment steps have been condensed into this Bash script which is designed to be run on each node in the cluster

#!/bin/bash set -e NODE_TYPE=${1} NODE_HOSTNAME=${2} NODE_IP=${3} if [ -z "$NODE_TYPE" ] || [ -z "$NODE_HOSTNAME" ] || [ -z "$NODE_IP" ]; then echo "Usage: $0 <master|worker> <hostname> <ip-address>" exit 1 fi echo "==========================================" echo "Preparing Node: $NODE_HOSTNAME ($NODE_IP)" echo "==========================================" # Set hostname echo " Setting hostname..." sudo hostnamectl set-hostname $NODE_HOSTNAME # Update /etc/hosts (UPDATED with BIG-IP) echo " Updating /etc/hosts..." sudo tee -a /etc/hosts <<EOF 10.124.207.1 k8s-multicluster-bigip bigip 10.124.207.2 k8s-master 10.124.207.3 kube-east 10.124.207.4 kube-west 10.155.90.184 bigip-external 10.155.90.185 k8s-master-external 10.155.90.186 kube-east-external 10.155.90.187 kube-west-external EOF # Disable swap echo " Disabling swap..." sudo swapoff -a sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab # Load kernel modules echo "Loading kernel modules..." cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF sudo modprobe overlay sudo modprobe br_netfilter # Configure sysctl echo "Configuring sysctl..." cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF sudo sysctl --system # Install containerd echo "Installing containerd..." sudo apt-get update sudo apt-get install -y ca-certificates curl gnupg lsb-release sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg sudo chmod a+r /etc/apt/keyrings/docker.gpg echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install -y containerd.io # Configure containerd echo "Configuring containerd..." sudo mkdir -p /etc/containerd sudo containerd config default | sudo tee /etc/containerd/config.toml sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml sudo systemctl restart containerd sudo systemctl enable containerd # Install Kubernetes echo "Installing Kubernetes components..." curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl sudo systemctl enable kubelet echo " Kubeadm Node preparation complete!" echo "=========================================="

Once kubeadm is installed on all nodes, next step is to initialize the control plane on node k8s-master - 10.124.207.2

Initialize Control Plane Node

# Initialize cluster root@k8s-control:~# sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/16 --apiserver-advertise-address=10.124.207.2 --control-plane-endpoint=k8s-master --node-name=k8s-master --upload-certs # Configure kubectl root@k8s-control:~# mkdir -p $HOME/.kube root@k8s-control:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config root@k8s-control:~# chown $(id -u):$(id -g) $HOME/.kube/config # Install Flannel CNI root@k8s-control:~# kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml # Wait for node to be Ready root@k8s-control:~# kubectl wait --for=condition=ready node k8s-master --timeout=300s # Save join command root@k8s-control:~# kubeadm token create --print-join-command > /tmp/join-command.sh root@k8s-control:~# chmod +x /tmp/join-command.sh root@k8s-control:~# cat /tmp/join-command.sh kubeadm join k8s-control:6443 --token a3wqwg.qhjrja8w0t2322f5 --discovery-token-ca-cert-hash sha256:cb52788c410fc115e622af816bead975b3076175acea7335dff24561238ee2a8

After initializing the control plane node, join the worker nodes kube-east - 10.124.207.3 and kube-west - 10.124.207.4 to the k8s cluster

kube-east - 10.124.207.3 # Join cluster root@kube-east:~# kubeadm join k8s-master:6443 --token a3wqwg.qhjrja8w0t2322f5 --discovery-token-ca-cert-hash sha256:cb52788c410fc115e622af816bead975b3076175acea7335dff24 --node-name=kube-east kube-west - 10.124.207.4 root@kube-west:~# kubeadm join k8s-master:6443 --token a3wqwg.qhjrja8w0t2322f5 --discovery-token-ca-cert-hash sha256:cb52788c410fc115e622af816bead975b3076175acea7335dff24 --node-name=kube-west

Now we can verify the cluster state once the worker nodes have joined

Verify k8s cluster root@k8s-control:~# alias k=kubectl root@k8s-control:~# k get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-control Ready control-plane 21h v1.28.15 10.124.207.2 <none> Ubuntu 22.04.5 LTS 5.15.0-168-generic containerd://2.2.1 kube-east Ready <none> 21h v1.28.15 10.124.207.3 <none> Ubuntu 22.04.5 LTS 5.15.0-168-generic containerd://2.2.1 kube-west Ready <none> 21h v1.28.15 10.124.207.4 <none> Ubuntu 22.04.5 LTS 5.15.0-168-generic containerd://2.2.1

Now that we have the Kubernetes cluster set up we can now move on to configure the BIG-IP using the following resources

  • Install the AS3 extension detailed in Downloading and installing the BIG-IP AS3 package
  • Creating a BIG-IP partition to manage Kubernetes objects. This partition can be created either via the GUI (System > Users > Partition List) or via our TMOS CLI
  • Installing F5 CIS which is documented in installation guide: Kubernetes

F5 CIS multi-cluster deployments require these parameters in the deployment manifest

  • multi_cluster_mode: standalone
  • local_cluster_name: k8s-control
  • extended_spec_configmap: kube-system/extended-cm

The example values yaml file below includes these configuration parameters

root@k8s-control:~# cat cis-values.yaml bigip_login_secret: f5-bigip-ctlr-login bigip_secret: create: false username: password: rbac: create: true serviceAccount: # Specifies whether a service account should be created create: true # The name of the service account to use. # If not set and create is true, a name is generated using the fullname template name: k8s-bigip-ctlr # This namespace is where the Controller lives; namespace: kube-system ingressClass: create: false ingressClassName: f5 isDefaultIngressController: true args: # See https://clouddocs.f5.com/containers/latest/userguide/config-parameters.html # NOTE: helm has difficulty with values using `-`; `_` are used for naming # and are replaced with `-` during rendering. # REQUIRED Params bigip_url: 10.124.207.1 bigip_partition: multicluster as3_validation: true ## OPTIONAL PARAMS -- uncomment and provide values for those you wish to use. ## all config options ## https://clouddocs.f5.com/containers/latest/userguide/config-parameters.html ## here are some deployment parameters for your considerations # verify_interval: # node_poll_interval: log_level: DEBUG # python_basedir: ~ # VXLAN # openshift_sdn_name: # flannel_name: # KUBERNETES # default_ingress_ip: # kubeconfig: # namespaces: ["foo", "bar"] # namespace_label: # node_label_selector: pool_member_type: nodeport ## See the page for pool member type support, https://clouddocs.f5.com/containers/latest/userguide/config-options.html ## if pool-member-type is set to cluster, ## see static routes support, https://clouddocs.f5.com/containers/latest/userguide/static-route-support.html ## for Calico CNI see https://clouddocs.f5.com/containers/latest/userguide/calico-config.html ## for Clilium CNI see https://clouddocs.f5.com/containers/latest/userguide/cilium-config.html ## for vxlan tunnel configuration see https://clouddocs.f5.com/containers/latest/userguide/cis-installation.html#creating-vxlan-tunnels ## for vxlan tunnel parameters see https://clouddocs.f5.com/containers/latest/userguide/config-parameters.html#vxlan # resolve_ingress_names: # running_in_cluster: # use_node_internal: # use_secrets: ## below parameter is a recommended parameter to verify the bigip certificate # trusted_certs_cfgmap: kube-system/trusted-certs ## Use below parameter only for non-production environments insecure: true ## for using custom resources, see more on https://clouddocs.f5.com/containers/latest/userguide/crd/ custom_resource_mode: true log_as3_response: true # gtm_bigip_password # gtm_bigip_url # gtm_bigip_username ## for using the F5 IPAM Controller, see more on https://clouddocs.f5.com/containers/latest/userguide/ipam/ # ipam : true controller_mode: customresource ## for configuring nextgen openshift routes, see more on https://clouddocs.f5.com/containers/latest/userguide/next-gen-routes/ # extended_spec_configmap: default/extended-cm ## for configuring the multi-cluster mode, see more on https://clouddocs.f5.com/containers/latest/userguide/multicluster/ multi_cluster_mode: standalone local_cluster_name: k8s-control extended_spec_configmap: kube-system/extended-cm image: # Use the tag to target a specific version of the Controller user: f5networks repo: k8s-bigip-ctlr pullPolicy: Always version: 2.20.3 # affinity: # nodeAffinity: # requiredDuringSchedulingIgnoredDuringExecution: # nodeSelectorTerms: # - matchExpressions: # - key: kubernetes.io/arch # operator: Exists securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 # If you want to specify resources, uncomment the following limits_cpu: 100m limits_memory: 512Mi requests_cpu: 100m requests_memory: 512Mi # Set podSecurityContext for Pod Security Admission and Pod Security Standards podSecurityContext: runAsNonRoot: true allowPrivilegeEscalation: false capabilities: drop: [ "ALL" ] seccompProfile: type: RuntimeDefault

F5 CIS can be deployed either manually or using Helm. Please see

F5 CIS will be deployed on the control plane node. Once deployed we can verify F5 CIS is in the running state:

root@k8s-control:~# k -n kube-system get deploy/k8s-multicluster-f5-bigip-ctlr NAME READY UP-TO-DATE AVAILABLE AGE k8s-multicluster-f5-bigip-ctlr 1/1 1 1 46d

The next step is to create the kubeconfig secrets needed by the k8s control plane node to manage the worker nodes

The kubeconfig files for kube-east and kube-west clusters will be created on the control plane node and secrets created for control plane access to the worker nodes

Kubeconfig for east/west clusters

# Create kubeconfig for kube-east root@k8s-control:~# cat > /tmp/kubeconfig-east.yaml <<EOF apiVersion: v1 kind: Config clusters: - cluster: certificate-authority-data: $(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}') server: https://10.124.207.2:6443 name: kube-east contexts: - context: cluster: kube-east user: kube-east name: kube-east current-context: kube-east users: - name: kube-east user: client-certificate-data: $(kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}') client-key-data: $(kubectl config view --raw -o jsonpath='{.users[0].user.client-key-data}') EOF # Create kubeconfig for kube-west root@k8s-control:~# cat > /tmp/kubeconfig-west.yaml <<EOF apiVersion: v1 kind: Config clusters: - cluster: certificate-authority-data: $(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}') server: https://10.124.207.2:6443 name: kube-west contexts: - context: cluster: kube-west user: kube-west name: kube-west current-context: kube-west users: - name: kube-west user: client-certificate-data: $(kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}') client-key-data: $(kubectl config view --raw -o jsonpath='{.users[0].user.client-key-data}') EOF # Create secrets root@k8s-control:~# kubectl create secret generic kubeconfig-east --from-file=kubeconfig=/tmp/kubeconfig-east.yaml -n kube-system root@k8s-control:~# kubectl create secret generic kubeconfig-west --from-file=kubeconfig=/tmp/kubeconfig-west.yaml -n kube-system

These secrets will then be referenced in the extended configmap yaml manifest

Nginx TLS Configuration

Extended configmap root@k8s-control:~# cat extended-cm.yaml apiVersion: v1 kind: ConfigMap metadata: name: extended-cm namespace: kube-system labels: f5type: virtual-server as3: "true" data: extendedSpec: | mode: default externalClustersConfig: - clusterName: kube-east secret: kube-system/kubeconfig-east - clusterName: kube-west secret: kube-system/kubeconfig-west

Our use-case for our multi-cluster deployment is a secure Nginx application. So we will create TLS certificates using these steps and then store in the k8s secrets:

Create TLS certificates, store in secret and verify secret # Generate self-signed certificate root@k8s-control:~# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=goblin.multicluster.local/O=Goblin Multi-Cluster" # Create Kubernetes TLS secret root@k8s-control:~# kubectl create secret tls goblin-tls --cert=/tmp/tls.crt --key=/tmp/tls.key -n kube-system # Verify secret root@k8s-control:~# k get secret goblin-tls -n kube-system NAME TYPE DATA AGE goblin-tls kubernetes.io/tls 2 17h # Clean up temporary files root@k8s-control:~# rm /tmp/tls.key /tmp/tls.crt

Now we can create a configmap to inject the TLS configuration (secrets) in the eventual NGINX application containers

root@k8s-control:~# kubectl apply -f - <<'EOF' apiVersion: v1 kind: ConfigMap metadata: name: nginx-ssl-config namespace: kube-system data: default.conf: | server { listen 80; listen [::]:80; server_name _; # Root location location / { root /usr/share/nginx/html; index index.html; } # /green endpoint location /green { root /usr/share/nginx/html; try_files /green /green.html =404; } } server { listen 443 ssl http2; listen [::]:443 ssl http2; server_name _; # SSL certificates ssl_certificate /etc/nginx/ssl/tls.crt; ssl_certificate_key /etc/nginx/ssl/tls.key; # SSL configuration ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384; ssl_prefer_server_ciphers on; ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; # Security headers add_header Strict-Transport-Security "max-age=31536000" always; add_header X-Frame-Options DENY; add_header X-Content-Type-Options nosniff; # Root location location / { root /usr/share/nginx/html; index index.html; } # /green endpoint location /green { root /usr/share/nginx/html; try_files /green /green.html =404; } } EOF

Our NGINX application consisting of the configmap, deployment and service objects will be scaffolded in the yaml manifest below

Application configmap, deployment and service objects for application

--- apiVersion: v1 kind: ConfigMap metadata: name: goblin-html namespace: kube-system data: index.html: | <!DOCTYPE html> <html> <head> <title>Green Goblin Multi-Cluster</title> <style> body { background-color: #4CAF50; color: white; text-align: center; padding: 50px; font-family: Arial, sans-serif; } h1 { font-size: 3em; } .info { margin: 30px; font-size: 1.2em; } .badge { background: #27ae60; padding: 10px 20px; border-radius: 5px; display: inline-block; margin: 10px; } </style> </head> <body> <h1> Green Goblin Multi-Cluster!</h1> <div class="info"> <div class="badge">Protocol: <span id="protocol"></span></div> <div class="badge">Port: <span id="port"></span></div> <div class="badge">BIG-IP: 10.124.207.1</div> </div> <p>Served via F5 CIS Multi-Cluster</p> <script> document.getElementById('protocol').textContent = window.location.protocol; document.getElementById('port').textContent = window.location.port || (window.location.protocol === 'https:' ? '443' : '80'); </script> </body> </html> green: | <!DOCTYPE html> <html> <head> <title>Green Goblin - /green</title> <style> body { background-color: #2E7D32; color: white; text-align: center; padding: 50px; font-family: Arial, sans-serif; } h1 { font-size: 3em; } .secure { background: #1b5e20; padding: 15px; border-radius: 5px; margin: 20px auto; max-width: 400px; } </style> </head> <body> <h1> Green Goblin at /green!</h1> <div class="secure"> <p> SSL/TLS Enabled</p> <p>Multi-Cluster via BIG-IP</p> </div> </body> </html> --- apiVersion: apps/v1 kind: Deployment metadata: name: goblin-deployment namespace: kube-system labels: app: goblin spec: replicas: 2 selector: matchLabels: app: goblin-app template: metadata: labels: app: goblin-app spec: containers: - name: nginx image: nginx:1.25-alpine ports: - name: http containerPort: 80 - name: https containerPort: 443 volumeMounts: # HTML content - name: html-content mountPath: /usr/share/nginx/html # SSL certificates - name: ssl-certs mountPath: /etc/nginx/ssl readOnly: true # Nginx SSL configuration - name: nginx-config mountPath: /etc/nginx/conf.d resources: requests: memory: "128Mi" cpu: "100m" limits: memory: "256Mi" cpu: "200m" readinessProbe: httpGet: path: /green port: 443 scheme: HTTPS initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /green port: 443 scheme: HTTPS initialDelaySeconds: 15 periodSeconds: 20 volumes: - name: html-content configMap: name: goblin-html - name: ssl-certs secret: secretName: goblin-tls - name: nginx-config configMap: name: nginx-ssl-config --- apiVersion: v1 kind: Service metadata: name: multi-cluster namespace: kube-system labels: app: goblin f5cr: "true" spec: type: NodePort ports: - name: http port: 80 targetPort: 80 nodePort: 30080 protocol: TCP - name: https port: 443 targetPort: 443 nodePort: 30443 protocol: TCP selector: app: goblin-app

Next we follow the steps below to deploy the NGINX application

Application deployment

# Apply updated configuration root@k8s-control:~# kubectl apply -f multicluster-apps-https.yaml # Wait for rollout root@k8s-control:~# kubectl rollout status deployment/goblin-deployment -n kube-system deployment "goblin-deployment" successfully rolled out # Verify pods root@k8s-control:~# kubectl get pods -n kube-system -l app=goblin-app -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES goblin-deployment-7c558fbd4-6f9dj 1/1 Running 0 17h 10.244.1.2 kube-east <none> <none> goblin-deployment-7c558fbd4-db5td 1/1 Running 0 17h 10.244.2.3 kube-west <none> <none> # Verify service root@k8s-control:~# kubectl get svc multi-cluster -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE multi-cluster NodePort 10.96.44.56 <none> 80:30080/TCP,443:30443/TCP 17h

Verifying the application post-deployment

# Saving pod name in a variable root@k8s-control:~# POD=$(kubectl get pods -n kube-system -l app=goblin-app -o jsonpath='{.items[0].metadata.name}') root@k8s-control:~# echo $POD goblin-deployment-7c558fbd4-6f9dj # Testing HTTP inside pod root@k8s-control:~# kubectl exec -n kube-system $POD -- curl -s http://localhost/green | grep -o "Green Goblin" Green Goblin Green Goblin # Testing HTTPS inside pod root@k8s-control:~# kubectl exec -n kube-system $POD -- curl -sk https://localhost/green | grep -o "Green Goblin" Green Goblin Green Goblin # Install pup command line HTML parser (https://github.com/ericchiang/pup) root@k8s-control:~# wget https://github.com/ericchiang/pup/releases/download/v0.4.0/pup_v0.4.0_linux_amd64.zip | unzip pup_v0.4.0_linux_amd64.zip -d /usr/local/bin # Testing HTTP NodePort on kube-east root@k8s-control:~# curl -s http://10.124.207.3:30080/green | pup 'body text{}' Green Goblin at /green! SSL/TLS Enabled Multi-Cluster via BIG-IP

We will now deploy virtual server CR and verify our backend pods for the kube-east and kube-west clusters are deployed

Virtual Server HTTPs CR

root@k8s-control:~# cat virtual-https.yaml apiVersion: cis.f5.com/v1 kind: VirtualServer metadata: labels: f5cr: "true" name: goblin-https-virtual-server namespace: kube-system spec: host: goblin.multicluster.local policyName: goblin-secure-policy tlsProfileName: reencrypt-tls pools: - path: /green monitor: interval: 10 recv: "200 OK" send: "GET /green HTTP/1.1\r\nHost: goblin.multicluster.local\r\nConnection: Close\r\n\r\n" timeout: 31 type: https #nodeMemberLabel: node-role.kubernetes.io/control-plane=true multiClusterServices: - clusterName: kube-east namespace: kube-system service: multi-cluster servicePort: https weight: 50 - clusterName: kube-west namespace: kube-system service: multi-cluster servicePort: https weight: 50 virtualServerAddress: 10.124.207.40 virtualServerHTTPSPort: 443 root@k8s-control:~# k apply -f virtual-https.yaml root@k8s-control:~# kn get virtualserver NAME HOST TLSPROFILENAME HTTPTRAFFIC IPADDRESS IPAMLABEL IPAMVSADDRESS STATUS AGE goblin-https-virtual-server goblin.multicluster.local reencrypt-tls 10.124.207.40 10.124.207.40 OK 3h30m

Let’s head over the BIG-IP config utility and verify our pods are up and running

image_346063.jpg
image_346063.jpg3430Ă—594 273 KB

We can also verify using tmsh utility

BIG-IP pool members

root@(k8s-multcluster-bigip)(cfg-sync Standalone)(Active)(/multicluster/Shared)(tmos)# list ltm pool all ltm pool multi_cluster_https_kube_system_goblin_multicluster_local_kube_east_green { description "crd_10_124_207_40_443 loadbalances this pool" members { /Common/10.124.207.2:30443 { address 10.124.207.2 session monitor-enabled state up } /Common/10.124.207.3:30443 { address 10.124.207.3 session monitor-enabled state up } /Common/10.124.207.4:30443 { address 10.124.207.4 session monitor-enabled state up } } min-active-members 1 monitor min 1 of { kube_system_10_124_207_40_goblin_multicluster_local_green } partition multicluster slow-ramp-time 60 } ltm pool multi_cluster_https_kube_system_goblin_multicluster_local_kube_west_green { description "crd_10_124_207_40_443 loadbalances this pool" members { /Common/10.124.207.2:30443 { address 10.124.207.2 session monitor-enabled state up } /Common/10.124.207.3:30443 { address 10.124.207.3 session monitor-enabled state up } /Common/10.124.207.4:30443 { address 10.124.207.4 session monitor-enabled state up } } min-active-members 1 monitor min 1 of { kube_system_10_124_207_40_goblin_multicluster_local_green } partition multicluster slow-ramp-time 60 }

Notice the F5 CIS also creates pool member for the control plane node even though the intent was only for the kube-east/kube-west clusters

This can be rectified by applying a taint to the control plane node to repel any scheduling of pods

taints: - effect: NoSchedule key: node-role.kubernetes.io/control-plane

Conclusion

The goal of this tutorial was to demonstrate how F5 Container Ingress Services (CIS) enables seamless multi-cluster application deployment across geographically distributed Kubernetes environments.

By simulating a multi-cluster architecture using separate kubeconfig contexts within a single Kubeadm cluster, we’ve shown how CIS can intelligently manage traffic distribution across multiple worker nodes acting as distinct clusters. In practical terms, the key benefits of our deployed multi-cluster environment are summarized below:

  • High Availability: Applications deployed across kube-east and kube-west clusters provide redundancy and fault tolerance
  • Geographic Distribution: Simulated cluster separation demonstrates how workloads can span multiple data centers or regions
  • Centralized Management: Single F5 CIS instance on the control plane manages all cluster configurations through extended ConfigMaps
  • Load Distribution: Traffic weight distribution (50/50 in this example) ensures balanced load across clusters

Technical Implementation Highlights:

  • Kubeconfig Secrets: Separate kubeconfig contexts enable CIS to treat worker nodes as external clusters while maintaining centralized control
  • Extended ConfigMap: The extended-cm ConfigMap serves as the bridge connecting CIS to multiple cluster contexts via kubeconfig secrets
  • NodePort Service Type: Using NodePort (30080/30443) allows BIG-IP to reach application pods across all nodes in the cluster
  • TLS Re-encryption: The reencrypt-tls profile ensures end-to-end encryption from client to BIG-IP to backend pods
1 Like