VMware VKS integration with F5 BIG-IP and CIS

Introduction

vSphere Kubernetes Service (VKS) is the Kubernetes runtime built directly into VMware Cloud Foundation (VCF). With CNCF certified Kubernetes, VKS enables platform engineers to deploy and manage Kubernetes clusters while leveraging a comprehensive set of cloud services in VCF. Cloud admins benefit from the support for N-2 Kubernetes versions, enterprise-grade security, and simplified lifecycle management for modern apps adoption.

Alike with other Kubernetes platforms, the integration with BIG-IP is done through the use of the Container Ingress Services (CIS) component, which is hosted in the Kubernetes platform and allows to configure the BIG-IP using the Kubernetes API. Under the hood, it uses the F5 AS3 declarative API.

Overall Integration

Note from the picture that BIG-IP integration with VKS is not limited to BIG-IP´s load balancing capabilities and that most BIG-IP features can be configured using this integration. These features include:

• Advanced TLS encryption, including safe key storage with Hardware Security Module (HSM) or Network & Cloud HSM support.
  
  
• Advanced WAF, L7 bot and API protection.
  
  
• L3-L4 High-performance firewall with IPS for protocol conformance.
  
  
• Behavioral DDoS protection with cloud scrubbing support.
  
  
• Visibility into TLS traffic for inspection with 3^rd party solutions.
  
  
• Identity-aware ingress with Federated SSO and integration with leading MFAs.
  
  
• AI inference and agentic support thanks to JSON and MCP protocol support.

Planning the deployment of CIS for VMware VKS

The installation of CIS in VMware VKS is performed through the standard Helm charts facility. The platform owner needs to determine beforehand:

• Whether the deployment is hosted on a vSphere (VDS) network or an NSX network.
  
  It has to be taken into account that on an NSX network, VKS doesn´t currently allow to place the load balancers in the same segment as the VKS cluster. 
  
  No special considerations have to be taken when hosting BIG-IP in a vSphere (VDS) network.
  
  
• Whether this is a single-cluster or a multi-cluster deployment.
  
  When using the multi-cluster option and clusterIP mode (only possible with Calico in VKS), it has to be taken into account that the POD networks of the clusters cannot have overlapping prefixes.
  
  
• What Kubernetes networking (CNI) is desired to be used.
  
  CIS supports both VKS supported CNIs: Antrea (default) and Calico. From the CIS point of view, the CNI is only relevant when sending traffic directly to the PODs. See next.
  
  
• What integration with the CNI is desired between the BIG-IP and VKS
  
  
  • NodePort mode
    
    This is done by making applications discoverable using Services of type NodePort.
    
    From the BIG-IP, the traffic is sent to the Node´s IPs where it is redistributed to the POD depending on the TrafficPolicies of the Service. This is CNI agnostic. Any CNI can be used. 
     
  • Direct-to-POD mode
    
    This is done by making applications discoverable using the Services of type ClusterIP. 
    
    Note that the CIS integration with Antrea uses Antrea´s nodePortLocal mechanism, which requires an additional annotation in the Service declaration. See the CIS VKS page in F5 CloudDocs for details.
    
    This Antrea nodePortLocal mechanism allows to send the traffic directly to the POD without actually using the POD IP address. This is especially relevant for NSX because it allows to access the PODs without actually re-distributing the PODs IPs across the NSX network, which is not allowed. 
    
    When using vSphere (VDS) networking, either Antrea’s nodePortLocal or clusterIP with Calico can be used.
    
    Another way (but not frequent) is the use of hostNetwork POD networking because it requires privileges for the application PODs or ingress controllers. Network-wise, this would have a similar behavior to nodePortLocal, but without the automatic allocation of ports.
    
    
• Whether the deployment is a single-tier or a two-tier deployment.
  
  
  • A single-tier deployment is a deployment where the BIG-IP sends the traffic directly to the application PODs. This has a simpler traffic flow and easier persistence and end-to-end monitoring.
    
    
  • A two-tier deployment sends the traffic to an ingress controller POD instead of the application PODs. This ingress controller could be Contour, NGINX Gateway Fabric, Istio or an API gateway. This type of deployment offers the ultimate scalability and provides additional segregation between the BIG-IPs (typically owned by NetOps) and the Kubernetes cluster (typically owned by DevOps). 

Once CIS is deployed, applications can be published either using the Kubernetes standard Ingress resource or F5’s Custom Resources. This latter is the recommended way because it allows to expose most of the BIG-IPs capabilities.

Details on the Ingress resource and F5 custom annotations can be found here.

Details on the F5 CRDs can be found here. Please note that at time of this writing Antrea nodePortLocal doesn´t support the TransportServer CRD. Please consult your F5 representative for its availability. 

DevOps flows

Detailed instructions on how to deploy CIS for VKS can be found on this CIS VKS page in F5 CloudDocs. 

 

Application-aware MultiCluster support

 

MultiCluster allows to expose applications that are hosted in multiple VKS clusters and publish them in a single VIP.  BIG-IP & CIS are in charge of:

• Discover where the PODs of the applications are hosted. Note that a given application doesn´t need to be available in all clusters.
  
  
• Upon receiving the request for a given application, decide to which cluster and Node/Pod the request has to be sent. This decision is based on the weight of each cluster, the application availability and the load balancing algorithm being applied.

Single-tier or Two-tier architectures are possible. NodePort and ClusterIP modes are possible as well. Note that at the time of this writing, Antrea in ClusterIP mode (nodePortLocal) is not supported currently. Please consult your F5 representative for availability of this feature.

Considerations for NSX

Load Balancers cannot be placed in the same VPC segment where the VMware VKS cluster is. These can be placed in a separate VPC segment of the same VPC gateway as shown in the next diagram.

In this arrangement the BIG-IP can be configured as either 1NIC mode or as a regular deployment, in which case the MGMT interface is typically configured through an infrastructure VLAN instead of an NSX segment. 

The data segment is only required to have enough prefixes to host the self-IPs of the BIG-IP units. The prefixes of the VIPs might not belong to the Data Segment´s subnet. These additional prefixes have to be configured as static routes in the VPC Gateway and Route Redistribution for these must be enabled.

Given that the Load Balancers are not in line with the traffic flow towards the VKS Cluster, it is required to use SNAT. When using SNAT pools, the prefixes of these can optionally be configured as additional prefixes of the Data Segment, like the VIPs.

Specifically for Calico, clusterIP mode cannot be used in NSX because this would require the BIG-IP to be in the same VPC segment as VMware VKS. Note also that BGP multi-hop is not feasible either because it would require the POD cluster network prefixes to be redistributed through NSX, which is not possible either.

Conclusion and final remarks

F5 BIG-IPs provides unmatched deployment options and features for VMware VKS; these include:

• Support for all VKS CNIs, which allows sending the traffic directly instead of using hostNetwork (which implies a security risk) or using the common NodePort, which can incur an additional kube-proxy indirection.
  
  
• Both 1-tier or 2-tier arrangements (or both types simultaneously) are possible.
  
  
• F5´s Container Ingress Services provides the ability to handle multiple VMware VKS clusters with application-aware VIPs. This is a unique feature in the industry.
  
  
• Securing applications with the wide range of L3 to L7 security features provided by BIG-IP, including Advanced WAF and Application Access.
• To complete the circle, this integration also provides IP address management (IPAM) which provides great flexibility to DevOps teams.

All these are available regardless of the form factor of  the  BIG-IP: Virtual Edition, appliance or chassis, allowing great scalability and multi-tenancy options. In NSX deployments, the recommended form-factor is Virtual Edition in order to connect to the NSX segments.

We look forward to hearing your experience and feedback on this article.