Longhorn and Failure Drills

First PublishedAug 6, 2026ByAtif Alam

Final page of the HA Kubernetes on Two Proxmox Hosts series. Storage goes in, then the design gets proven: the drills are the deliverable. Do not skip them.

Step 1: Install Longhorn

Workers already have open-iscsi and nfs-common from node prep. Install Longhorn at the pinned 1.8 minor:

helm repo add longhorn https://charts.longhorn.io
helm install longhorn longhorn/longhorn \
  --namespace longhorn-system --create-namespace \
  --version 1.8.2 \
  --set defaultSettings.defaultReplicaCount=2 \
  --set defaultSettings.replicaZoneSoftAntiAffinity=true \
  --set defaultSettings.storageReservedPercentageForDefaultDisk=25 \
  --set persistence.defaultClassReplicaCount=2

Why these values:

• defaultReplicaCount=2 — one replica per physical host; a third replica has nowhere independent to live.
• replicaZoneSoftAntiAffinity=true — Longhorn spreads the two replicas across the topology.kubernetes.io/zone labels (host-a/host-b) applied at worker join. This is what guarantees a volume survives a host loss — without it both replicas can land on one machine.
• storageReservedPercentageForDefaultDisk=25 — the workers’ 150 GB disks also hold the OS and images; reserving 25% (~37 GB) leaves ~110 GB schedulable per worker and keeps the filesystem out of the danger zone.

Verify:

kubectl -n longhorn-system get pods
kubectl get storageclass

Expected: manager/driver/instance-manager pods Running on all four workers; longhorn StorageClass present (default).

Never use the NFS share as live PV storage. NFS on the tiebreaker box is a backup target only — putting live volumes on it makes the tiebreaker a runtime SPOF and defeats the whole design. See Storage for the replicated-CSI vs NFS tradeoff.

Step 2: Daily Backups to the NFS Target

In the Longhorn UI (kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80), or via settings, point the backup target at the tiebreaker’s share:

nfs://<TIEBREAKER-IP>:/volume1/longhorn-backup

Then create a daily backup job for all volumes with the default group:

apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: daily-backup
  namespace: longhorn-system
spec:
  cron: "0 2 * * *"
  task: backup
  groups: ["default"]
  retain: 7
  concurrency: 1

kubectl apply -f recurring-backup.yaml

This covers application data. Cluster state is covered separately by the etcd snapshot timers on both control planes. Backup drills and restore-testing discipline: Stateful Backup and Restore.

Step 3: Deploy the Drill Workloads

Two workloads, each proving a different half of the HA claim:

• drill-web — a stateless Deployment with 2 replicas and topologySpreadConstraints, one per physical host. Proves scheduling spread and rescheduling.
• drill-data — a single-replica Deployment with a Longhorn PVC. A ReadWriteOnce volume attaches to one node at a time, so the real storage test is the pod rescheduling to the other host and reattaching the volume from the surviving replica.

The spread constraint is the app-level counterpart of Longhorn’s replica anti-affinity. Pattern background: Production Patterns.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: drill-web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: drill-web
  template:
    metadata:
      labels:
        app: drill-web
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: drill-web
      containers:
        - name: web
          image: mccutchen/go-httpbin:v2.15.0
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: drill-data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drill-data
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: drill-data
  template:
    metadata:
      labels:
        app: drill-data
    spec:
      containers:
        - name: data
          image: mccutchen/go-httpbin:v2.15.0
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: drill-data

whenUnsatisfiable: ScheduleAnyway (soft) is deliberate: with a hard constraint (DoNotSchedule), losing a host would leave rescheduled pods Pending instead of running degraded on the surviving host. Likewise strategy: Recreate on the data app — a rolling update cannot work when the volume only attaches to one node.

kubectl apply -f drill-workloads.yaml
kubectl get pods -l 'app in (drill-web,drill-data)' -o wide

Expected: the two drill-web pods Running on different zones (host-a and host-b workers), drill-data Running on either. Check the drill-data volume in the Longhorn UI: one replica in each zone.

Step 4: Drill 1 — Power Off Host A

Physically power off (or poweroff from the Proxmox shell on) Host A, then observe:

# API stays up via the VIP (cp2 holds or claims it):
kubectl --server=https://<VIP-IP>:6443 get nodes
# etcd keeps quorum (run from cp2):
sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  endpoint status -w table
# Workloads reschedule (default tolerance ~5 minutes after node goes NotReady):
kubectl get pods -l 'app in (drill-web,drill-data)' -o wide -w

Success Criteria Checklist

• API answers through the VIP the whole time (blips of a few seconds are acceptable).
• etcd reports 2 healthy members; writes still succeed (kubectl create configmap drill-write-test --from-literal=a=b).
• Host A’s nodes go NotReady; the Host A drill-web pod and (if it was there) the drill-data pod reschedule onto Host B workers within ~5–7 minutes.
• The drill-data volume degrades (1 of 2 replicas) but reattaches on Host B from the surviving replica, and /data is still writable.

Record actual timings — they are your cluster’s real RTO.

Step 5: Drill 2 — Host A Returns

Power Host A back on and watch — touch nothing:

kubectl get nodes -w

Success Criteria Checklist

• etcd1 rejoins and catches up automatically (endpoint status shows 3 healthy members again).
• cp1 and both Host A workers return to Ready with no manual steps.
• Longhorn rebuilds the second replica automatically (watch the volume in the UI go from Degraded to Healthy).

If any step needed manual intervention, the design goal is not met — find and fix the cause (usual suspects: etcd data dir permissions, static IP not restored, iscsid not enabled) and re-run the drill.

Step 6: Repeat for Host B, Then Check Backups

Run both drills again powering off Host B — this also proves the VIP fails back and that cp1’s etcd member tolerates being on the surviving side.

Finally, verify the backup machinery actually ran:

ls -lh /mnt/backup/etcd/          # on either CP: snapshots from both hosts, dated today/yesterday
kubectl -n longhorn-system get backups.longhorn.io

If the Host Never Comes Back

Permanent loss (dead board, dead NVMe) needs manual day-2 steps — this is the one scenario auto-recovery cannot cover:

1. Remove the dead nodes from Kubernetes: kubectl delete node cp1 worker-1 worker-2.
2. Remove the dead etcd member so quorum math stays correct: etcdctl member list then etcdctl member remove <ID>.
3. Rebuild the replacement VMs from the Proxmox template (page 1), re-run node prep.
4. Re-add the etcd member (etcdctl member add etcd1 --peer-urls=https://<NEW-CP1-IP>:2380, start etcd with --initial-cluster-state existing), then kubeadm join --control-plane.
5. Join the replacement workers, re-apply zone labels, and let Longhorn rebuild replicas onto them.

Done

The cluster now survives the loss of either physical machine and recovers without hands-on-keyboard. Where to go next is on the hub page — app ingress, Terraform provisioning, and observability.