Double Trouble: Multiple Controllers Handling the Same Kubernetes LoadBalancer Service

This is a summary of a confusing problem I saw recently. It also reminds us of two best practices: use the LoadBalancerClass field always and deploy with automation.

 

The problem statement

A customer called with a challenge:

"I'm using F5 CIS to create a VIP on my BIG-IP. But my VIP isn't reachable, not even via PING. However, if I use CIS to delete this VIP, and then create the exact same VIP manually, it's reachable. What is CIS doing that results in it creating a VIP that is not reachable?"

 

The Setup

Let's walk through a deeper dive:

F5 CIS and service of type LoadBalancer

This customer was using F5 Container Ingress Services (CIS) to watch for Service objects in their Kubernetes cluster. Specifically, services of type LoadBalancer. 

They created a Service object that looks something like this: 

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: default
  annotations:
    cis.f5.com/ip: 192.0.2.1
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
status:
  loadBalancer:
    ingress:
    - ip: 192.0.2.1  # Assigned by one controller, but read by both!

When CIS is configured correctly, it will create a VIP on BIG-IP with an IP address of 192.0.2.1. Then CIS reads the annotation above and then updates the status section with the IP address. 

 

LoadBalancerClass

The spec.loadBalancerClass field (introduced in K8s 1.24) allows an admin to use multiple controllers to watch services of type LB, which will not step on each other's toes. 

It is key to know that CIS supports use of the LoadBalancerClass field. The important thing to take away is that by default, CIS will process all services of type LB if they do not have this field. If they do have this field, it will not process them unless the field's value matches what we can configure with CIS's arguments. 

 

Where Things Go Sideways

Kubernetes doesn’t prevent multiple controllers from handling the same Service. In fact, from Kubernetes’ perspective, it’s just a chunk of YAML describing an abstract networking resource. 

This is just before things go sideways.

Multiple Controllers

Maybe you’re also trying out another controller. You could try a cloud load balancer controller like GCP’s L4 controller, or another on-prem load balancer integration like MetalLB or Avi. These controllers may all watch for Service objects of type LoadBalancer, and when they see one, they respond by provisioning a VIP somewhere.

If two controllers are watching services and they both think they should provision a VIP, then they will each do so.

 

Question: What happens if they both respond to the same Service?

Answer: They might both use the same IP address.

 

How?

One controller (say CIS) might assign an IP address and update the status.loadBalancer.ingress.ip field of the Service object. But that update is visible to all controllers. Unless the other controller is smart enough to back off, it might assume that IP was assigned for its own use.

The result: two different systems might attempt to own the same IP address on the same network.

 

IP Conflict in Action

When two network systems try to assign the same IP address to interfaces on the same VLAN, we get the kind of fun that keeps network engineers up at night:

• ARP table confusion
• Flapping routes
• Intermittent connection drops
• VIPs that work intermittently or "sometimes”
• App owners screaming into the void

In the case of my customer, their Rancher cluster had a controller that created a VIP in their Harvester environment. They also had a CIS controller that created a VIP on the F5 BIG-IP. Upon inspecting the local router's ARP table, we learned that the MAC address belonging to the Harvester environment was the first to respond to the ARP request, not the CIS. So in this case, Harvester won the race to reply and created the ARP entry for the wrong system. This meant that the F5 BIG-IP had a VIP that was not reachable from the network because the VS IP address resolved to the wrong MAC address.

 

When 2x controllers attempt to create VIPs with the same IP address.

 

 

Best Practices to Avoid the Clash

 

One Controller per Service

The most obvious solution: don’t have multiple controllers managing the same Service.

Here's our YAML again, this time with the spec.LoadBalancerClass defined:

apiVersion: v1
kind: Service
...
spec:
  type: LoadBalancer
  loadBalancerClass: f5 # let's add this line to our original Service object
  ...

Now, ensure your controllers manage this field as expected. This could mean:

• ensuring every service of type LB has spec.loadBalancerClass defined, and/or
• only allow a single controller to process any service that does not have spec.loadBalancerClass defined

For CIS, we would add these arguments to the CIS deployment:

--load-balancer-class=f5   #this will tell CIS to only consider services where spec.loadBalancerClass has a value of "f5"
--manage-load-balancer-class-only=true   #this will stop CIS from processing services without this field, since it is not a mandatory field (default is false)

Or

• configuring controllers to watch only services with a given loadBalancerClass (i.e., no "default" LB controller)

 

Configure controllers to watch different namespaces

For CIS, we can limit the namespaces watched like this:

--namespace=nginx-ingress   #only watch this namespace. For multiple namespaces, we can define this field multiple times
--namespace-label=cis=true   #another way to watch multiple namespaces with CIS

 

Isolate IP Pools

If you must run multiple controllers, consider assigning each one a dedicated subnet or IP range. That way, even if both provision VIPs, they can’t step on each other.

 

 

Final Thoughts

Kubernetes offers flexibility—but with great power comes great responsibility. Multiple controllers watching the same object can be a recipe for disaster unless they’re well-coordinated.

When using F5 CIS in tandem with another controller, remember: it’s not just about deploying the VIP. It’s about making sure only one system thinks it owns it.

Got questions or want to share your experience with multi-controller setups? Drop a comment below or reach out to me directly.