Using Direct Server Return (DSR) in Kubernetes can have benefits when you have workloads that require low latency, high throughput, and/or you want to preserve the source IP address of the connection. The following will guide you through how to configure Kubernetes and BIG-IP to use DSR for traffic to a Kubernetes Pod.
I’m not a huge fan of DSR. It’s a weird way of having a client send traffic to a Load Balancer (LB), the LB forwards to a backend server WITHOUT rewriting the destination address, and the backend server responds directly back to the client.
It looks WEIRD! But there are some benefits, the backend server sees the original client IP address without the need for the LB to be in the return path of traffic and the LB only has to handle one side of the connection. This is also the downside because it’s not straightforward to do any type of intelligent LB if you only see half the conversation. It also involves doing weird things on your backend servers to configure loopback devices so that it will answer for the traffic when it is received, but not create an IP conflict on the network.
DSR in Kubernetes
The following uses IP Virtual Server (IPVS) to setup DSR in Kubernetes. IPVS has been supported in Kubernetes since 1.11. When using IPVS it replaces IP Tables for the kube-proxy (internal LB). When you provision a LoadBalancer or NodePort service (method to expose traffic outside the cluster) you can add “externalTrafficPolicy: Local” to enable DSR. This is mentioned in the Kubernetes documentation for GCP and Azure environments.
DSR in BIG-IP
On the BIG-IP DSR is referred to as “nPath”. K11116 discusses the steps involved in getting it setup. The steps create a profile that will disable destination address translation and allow the BIG-IP to not maintain the state of TCP connections (since it will only see half the conversation).
Putting the Pieces Together
To enable DSR from Kubernetes the first step is to create a LoadBalancer service where you define the external LB IP address.
You can verify that DSR is working by connecting to the external IP address and observing that the MAC address that the traffic is sent to is different than the MAC address that the reply is sent from.
Using Container Ingress Services we can automate this setup with the following AS3 declaration (note the formatting is off and this will not copy-and-paste cleanly, only provided for illustrative purposes).
DSR is a weird way to load balance traffic, but it can have some benefits. For a more exhaustive list of the reasons not to do DSR; we can reach back to 2008 for the following gem from Lori MacVittie. What is old is new again!