HA Kubernetes on Two Proxmox Hosts
This lab series builds a kubeadm-based Kubernetes cluster spanning two physical Proxmox hosts that keeps serving workloads when either physical machine fails or is powered off — with no manual intervention. When the failed host returns, it rejoins and resyncs automatically: etcd catches up, the kubelet reconnects, and Longhorn rebuilds storage replicas.
This is a full production-topology build (not k3s/k0s): external etcd with a third tiebreaker member on an always-on Linux box, a kube-vip floating VIP for the API server, and Longhorn replicated block storage. The final page proves the design with mandatory failure drills.
The HA claim only holds end to end: control-plane HA comes from etcd quorum and the VIP, but workload HA additionally requires the node labels, topology spread, and Longhorn anti-affinity configured in the final page. Skipping those leaves both replicas of an app on one physical host.
Architecture
Section titled “Architecture”flowchart TB
subgraph hostA [Physical Host A]
cp1["cp1 VM: control plane + etcd1"]
w1[worker-1 VM]
w2[worker-2 VM]
end
subgraph hostB [Physical Host B]
cp2["cp2 VM: control plane + etcd2"]
w3[worker-3 VM]
w4[worker-4 VM]
end
subgraph tiebreaker ["Always-On Linux Box / NAS"]
etcd3["etcd3 (Docker container)"]
qnetd[corosync-qnetd QDevice]
nfs["NFS share: backups"]
end
vip["Floating VIP :6443 (kube-vip)"]
vip --> cp1
vip --> cp2
cp1 <-->|"etcd peer 2380"| cp2
cp1 <-->|"etcd peer 2380"| etcd3
cp2 <-->|"etcd peer 2380"| etcd3
hostA <-->|corosync| hostB
hostA <-->|quorum vote| qnetd
hostB <-->|quorum vote| qnetd
w1 -.->|"Longhorn backup (daily)"| nfs
w3 -.->|"Longhorn backup (daily)"| nfs
Failure Model
Section titled “Failure Model”- Either physical host fails → etcd retains quorum (2 of 3 members: surviving host + tiebreaker), the surviving control-plane VM keeps serving the API through the kube-vip VIP, workloads reschedule onto the surviving host’s workers, and Longhorn’s surviving replica keeps volumes available.
- Failed host returns → its etcd member rejoins and catches up, the kubelet reconnects, and Longhorn resyncs replicas — with zero manual steps (verified in the drills page).
Residual risk: the tiebreaker box is a single point of failure for quorum tie-breaking and the backup target. If it is down while one host is also down, etcd loses quorum and the API goes read-only. Accepted for this lab; mitigating it needs a second always-on device.
Design Decisions
Section titled “Design Decisions”| Decision | Choice | Why |
|---|---|---|
| Topology | External etcd (3 members) | A third quorum member can run on a lightweight box without a third hypervisor |
| API HA | kube-vip floating VIP (ARP mode) | No external load balancer VM to own |
| Live storage | Longhorn, replica count 2 | Replicated block storage across workers; survives a host loss |
| Backups | NFS share on the tiebreaker box, backup target only | Never NFS for live PVs — it reintroduces a runtime SPOF |
| Control planes | Dedicated (tainted) | Predictable failover behavior; untainting is a documented tradeoff |
| VM placement | Pinned to hosts, no live migration for CP/etcd VMs | The quorum math depends on etcd1/etcd2 staying on separate physical hosts |
| Proxmox | No-subscription repos | Homelab deployment |
External vs stacked, one nuance: etcd1 and etcd2 run on the same VMs as the control planes to save resources, but the topology is still external — etcd runs as a systemd service that kubeadm does not manage, rather than as kubeadm’s stacked static pods. kubeadm is configured with an etcd.external stanza. See etcd and Control Plane Health for the stacked-vs-external tradeoffs.
Tiebreaker host requirements: any always-on Debian-based Linux box works (a Debian-capable NAS, a mini PC, a Pi). It needs apt for corosync-qnetd and Docker for etcd3. Appliance NAS operating systems without apt need qnetd run another way or a different tiebreaker host — and etcd3’s data directory must sit on SSD/NVMe, never spinning disk (etcd is fsync-latency sensitive).
Hardware and VM Layout
Section titled “Hardware and VM Layout”Reference hardware: two hosts with ~16 GB RAM, 6-core CPU, and a 500 GB NVMe each.
| VM | vCPU | RAM | Disk | Host |
|---|---|---|---|---|
| cp1 (control plane + etcd1) | 2 | 4 GB | 60 GB | A |
| worker-1 | 2 | 4 GB | 150 GB | A |
| worker-2 | 2 | 4 GB | 150 GB | A |
| cp2 (control plane + etcd2) | 2 | 4 GB | 60 GB | B |
| worker-3 | 2 | 4 GB | 150 GB | B |
| worker-4 | 2 | 4 GB | 150 GB | B |
RAM headroom policy: with three 4 GB VMs per 16 GB host, only ~4 GB remains for the Proxmox host itself. Do not enable memory ballooning or overcommit; keep ~4 GB reserved for the host. If a host shows memory pressure, trim the worker VMs to 3.5 GB rather than squeezing the hypervisor.
Disk budget per host: ~32 GB for Proxmox OS/local storage, 360 GB allocated to VMs, ~108 GB headroom (target ~20% free on the NVMe).
Network Plan
Section titled “Network Plan”Everything lives on one flat L2 subnet — required for kube-vip ARP mode and simplest for etcd peers. Reserve static IPs before starting; the VIP must be outside your DHCP pool.
| Role | Example IP (adapt to your subnet) |
|---|---|
| Proxmox Host A | 10.0.10.11 |
| Proxmox Host B | 10.0.10.12 |
| Tiebreaker box (etcd3 + QDevice + NFS) | 10.0.10.13 |
| cp1 | 10.0.10.21 |
| cp2 | 10.0.10.22 |
| worker-1 … worker-4 | 10.0.10.31–10.0.10.34 |
| Kubernetes API VIP | 10.0.10.100 |
Port Matrix
Section titled “Port Matrix”| Port | Protocol | Between | Purpose |
|---|---|---|---|
| 6443 | TCP | everyone → VIP/CPs | Kubernetes API |
| 2379 | TCP | CPs → all etcd members | etcd client |
| 2380 | TCP | etcd members ↔ etcd members | etcd peer |
| 10250 | TCP | CPs ↔ all nodes | kubelet API |
| 5403 | TCP | Proxmox hosts → tiebreaker | corosync-qnetd (QDevice) |
| 8472 | UDP | all nodes ↔ all nodes | Cilium VXLAN overlay |
| 4240 | TCP | all nodes ↔ all nodes | Cilium health checks |
| 3260, 9500–9504 | TCP | workers ↔ workers | Longhorn iSCSI + manager/engine |
| 2049 | TCP | workers → tiebreaker | NFS backup target |
Version Matrix (Tested Combination)
Section titled “Version Matrix (Tested Combination)”Every page in this series uses versions from this table.
| Component | Version |
|---|---|
| Proxmox VE | 8.x |
| Guest OS | Debian 12 (cloud image) |
| Kubernetes (kubeadm/kubelet/kubectl) | v1.33.x |
| etcd | v3.5.x |
| containerd | 1.7.x |
| kube-vip | v0.8.x |
| Cilium | 1.17.x |
| Longhorn | 1.8.x |
Build Phases
Section titled “Build Phases”Work through the pages in order — each ends with verification steps the next page depends on.
- Proxmox Cluster and VMs — Install Proxmox on both hosts (no-subscription), form the 2-node cluster with a QDevice tiebreaker, provision the six VMs from a cloud-init template, and prep every node OS for kubeadm.
- External etcd and kubeadm Bootstrap — Generate etcd TLS certs, bring up the 3-member etcd cluster (including the Docker member on the tiebreaker), install kube-vip pre-init, and bootstrap both control planes against the VIP.
- Workers, CNI, and VIP Failover — Join the four workers with host-zone labels, install Cilium, and validate VIP failover behavior.
- Longhorn and Failure Drills — Install Longhorn with cross-host replica anti-affinity, configure daily NFS backups, and run the mandatory power-off drills for both hosts.
Next Steps After the Lab
Section titled “Next Steps After the Lab”- App traffic: the VIP only covers the API server. For
LoadBalancerServices and north-south app traffic, add kube-vip Services mode (or MetalLB) plus an ingress controller — see Ingress Controllers. - Automate provisioning: replace the manual
qmcommands with Terraform’sbpg/proxmoxprovider once the layout is stable. - Observability: Installing Prometheus and Grafana adapts directly to this cluster.
- Concepts behind this design: Kubernetes Architecture, etcd and Control Plane Health, Storage, and Production Patterns.