Skip to content

Workers, CNI, and VIP Failover

First PublishedByAtif Alam

Third page of the HA Kubernetes on Two Proxmox Hosts series. Both control planes are up behind the VIP from the previous page; this page adds the workers and the pod network, then validates VIP failover.

How the VIP Actually Fails Over

Section titled “How the VIP Actually Fails Over”

kube-vip was installed pre-init on both control planes; this section explains what it is doing so the validation below makes sense.

  • Both kube-vip static pods run leader election through the Kubernetes API. The current leader binds the VIP to its network interface and answers ARP for it (gratuitous ARP on takeover, so the switch updates its table immediately).
  • When the leader’s node dies or its kubelet stops, the lease expires and the other control plane claims the VIP — typically within a few seconds.
  • ARP mode requires the VIP and both control planes on the same L2 segment, which is why the hub’s network plan uses one flat subnet. Keep the VIP outside the DHCP pool — a DHCP lease colliding with the VIP produces intermittent, maddening API failures.

Step 1: Join the Four Workers

Section titled “Step 1: Join the Four Workers”

Run the worker join (saved from kubeadm init) on each worker — and attach a zone label marking which physical host the VM lives on. This label is what topology spread and Longhorn use later to keep replicas on different physical machines.

On worker-1 and worker-2 (Host A):

Terminal window
sudo kubeadm join <VIP-IP>:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Then from a control plane (or any admin kubeconfig):

Terminal window
kubectl label node worker-1 topology.kubernetes.io/zone=host-a
kubectl label node worker-2 topology.kubernetes.io/zone=host-a
kubectl label node worker-3 topology.kubernetes.io/zone=host-b
kubectl label node worker-4 topology.kubernetes.io/zone=host-b

Without these labels the scheduler has no idea two “nodes” share one physical machine — host-loss HA for workloads silently fails. The labels are load-bearing, not cosmetic.

Verify:

Terminal window
kubectl get nodes -L topology.kubernetes.io/zone

Expected: six nodes (still NotReady — no CNI yet), workers showing host-a/host-b zones.

Step 2: Install Cilium

Section titled “Step 2: Install Cilium”

Pinned to the 1.17 minor from the hub’s version matrix, installed via Helm, with kube-proxy left in place (simplest working configuration for this cluster):

Terminal window
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.17.6 \
  --namespace kube-system \
  --set k8sServiceHost=<VIP-IP> \
  --set k8sServicePort=6443

Pointing Cilium at the VIP (rather than one control plane’s IP) keeps the CNI’s own API access highly available.

Verify:

Terminal window
kubectl -n kube-system get pods -l k8s-app=cilium
kubectl get nodes

Expected: one cilium-... pod Running per node, and all six nodes flip to Ready within a minute or two. If you have the Cilium CLI installed, cilium status --wait and cilium connectivity test give a deeper check.

Step 3: Confirm Control Planes Stay Dedicated

Section titled “Step 3: Confirm Control Planes Stay Dedicated”

kubeadm already tainted both control planes:

Terminal window
kubectl describe node cp1 | grep -A1 Taints

Expected: node-role.kubernetes.io/control-plane:NoSchedule. Leave it — dedicated control planes keep failover behavior predictable and protect etcd’s latency from noisy app neighbors.

Capacity tradeoff: on hardware this small, some homelabs untaint the control planes to schedule regular workloads (kubectl taint nodes cp1 cp2 node-role.kubernetes.io/control-plane:NoSchedule-). This lab keeps them dedicated; if you untaint, expect slower and less predictable recovery during host-loss drills.

Step 4: Validate VIP Failover

Section titled “Step 4: Validate VIP Failover”
  1. Find the current VIP holder:
Terminal window
kubectl -n kube-system logs -l name=kube-vip-ds --tail=5 2>/dev/null || \
  ip -br addr show | grep <VIP-IP>   # run on each CP; the leader has the VIP bound
  1. Stop the kubelet on the leader (say cp1) and watch the API from elsewhere:
Terminal window
# On cp1:
sudo systemctl stop kubelet
# From your workstation:
while true; do kubectl --server=https://<VIP-IP>:6443 get --raw /healthz; sleep 1; done

Expected: at most a few seconds of failed requests, then ok resumes — cp2 has claimed the VIP. Confirm with ip -br addr show | grep <VIP-IP> on cp2.

  1. Restart kubelet on cp1 (sudo systemctl start kubelet). The VIP stays on cp2 until the next election — that is fine; there is no “home” node.

Verify Before Moving On

Section titled “Verify Before Moving On”
  • kubectl get nodes — six nodes Ready, workers labeled with zones.
  • kubectl -n kube-system get pods — Cilium, CoreDNS, kube-proxy all Running.
  • VIP failover demonstrated above.

Next: Longhorn and Failure Drills.