BIG-IP integration with Antrea and NodePortLocal on VMware Tanzu
Summary
The NodePortLocal (NPL) feature of the Antrea CNI is now supported by CIS for integrating BIG-IP with Kubernetes (K8S) workloads.
See this YouTube clip for a quick overview and demo by Mark Dittmer and Lavanya Sirigudi.
What is NodePortLocal?
Service types in Kubernetes
Services in Kubernetes can be of type ClusterIP, NodePort, or Loadbalancer. When you want to get traffic into your cluster from the network that is outside of the cluster, often NodePort is the chosen service type. This is because ClusterIP only exposes your pods internally to the cluster (and requires extra work to expose externally), and Loadbalancer is aimed at public cloud automation (and requires extra support for non-public-cloud deployments).
For a review of what a NodePort service does, review this nice post or k8s docs. Here's a diagram that shows BIG-IP integration with a NodePort service:
NodePort limitations
Given that we've landed on NodePort as our service type, let's review some drawbacks of NodePort. A NodePort service will:
- be available on all nodes, even those not running an exposed pod
- potentially add an additional hop between nodes
- probably load-balance imperfectly to backend pods
- traverse kube-proxy, adding another layer of loadbalancing and potentially minor latency
NodePort is great for some use cases, but improvements can be made.
Enter NodePortLocal
I first heard about NPL when a customer using VMware Tanzu asked me about it. My Googling led me to this discussion and I eventually to this overview.
For clusters using Antrea as CNI, the NodePortLocal feature option can be used; a Pod can be directly reached from an external network through a port in the Node. In this mode, instead of relying on NodePort Services implemented by kube-proxy, CIS consumes NPL port mappings published by the Antrea Agent (as K8s Pod annotations) to read NodePort and NodeIP information to load balance traffic to backend pods.
The Antrea CNI has now released NodePortLocal as a GA feature. It's clever because it overcomes the drawbacks of NodePort that we listed above:
- Only nodes that are running an exposed pod will receive traffic from an integrated load-balancer
- Nodes with more pods than other nodes should receive more traffic, since pods are exposed individually
- Performance improvements may be seen because kube-proxy is bypassed and additional network hops should never occur
How is VMware Tanzu related?
VMware Tanzu is a suite of products that helps users run and manage multiple Kubernetes clusters across public and private “clouds”. Tanzu Kubernetes Grid (TKG) is a multi-cloud Kubernetes footprint that you can run both on-premises in vSphere and in the public cloud on Amazon EC2 and Microsoft Azure. TKG by default uses Antrea CNI and, according to release notes, supports NPL from version 1.5.
What about other CNI's?
To my knowledge to date, only Antrea CNI has implemented a feature like NodePortLocal. I think it's very clever. We'll see how the adoption of Antrea continues with this feature. Today I believe Calico is still the most popular CNI, but I also see customers running other CNI's, like Canal (especially Rancher users) or Flannel, or OpenShift's CNI.
Where can I learn more?
Great question! Watch this YouTube clip, or read the official Antrea docs. For F5 BIG-IP integration, our CIS documentation has an overview and code example on github.
Lastly, a big thanks to Mark Dittmer, Lavanya Sirigudi, and Ulises Alonso Camaro at F5 for delivering this integration! Thanks for reading and leave a comment if you'd like to ask a question or learn more.