We're having issues with network traffic in our tanzu kubernetes cluster during upgrades. The upgrade process deletes nodes 1 by 1, and creates a new node with the upgraded k8s version.
We have tried modifying the node-poll-interval and verify-interval, but there seems to be an issue with this approach. Our traffic is directed to a random node, and from that node, is forwarded to the correct node/pod inside the cluster. When a node is deleted, it takes up to 30 seconds for that node to be removed from bigip.
This results in substantial packet loss during every upgrade of our cluster. Is there a way to perhaps remove/disable nodes based on labels/taints? unschedulable taint would be perfect for this. Any other suggestions?