External etcd and kubeadm Bootstrap

First PublishedAug 6, 2026ByAtif Alam

Second page of the HA Kubernetes on Two Proxmox Hosts series. It assumes the six VMs from Proxmox Cluster and VMs are running and node-prepped.

The etcd Topology

Three members form the quorum: etcd1 on cp1 (Host A), etcd2 on cp2 (Host B), and etcd3 as a Docker container on the tiebreaker box. Losing any one member — including a whole physical host — leaves 2 of 3 and keeps the cluster writable.

etcd1 and etcd2 run as systemd services on the control-plane VMs, not as kubeadm static pods — that is what makes this an external etcd topology even though they share VMs with the control planes. Background on why external topology decouples quorum from control-plane lifecycle: etcd and Control Plane Health.

On the tiebreaker box, etcd3’s data directory must live on SSD/NVMe storage. etcd fsyncs on every write, and raft only commits once a quorum of members has persisted the entry — a spinning-disk etcd3 slows every write in the cluster.

Step 1: Generate etcd TLS Certificates

Generate everything on cp1 with kubeadm’s cert phases, then distribute. (cfssl works too if you prefer a standalone PKI tool; this lab uses kubeadm to avoid a second toolchain.)

Create the etcd CA

sudo kubeadm init phase certs etcd-ca

This writes /etc/kubernetes/pki/etcd/ca.crt and ca.key.

Generate Per-Member Certificates

Create one small config per member so each cert carries the right SANs. Example etcd1.yaml (repeat with the right name/IP for etcd2.yaml and etcd3.yaml):

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
etcd:
  local:
    serverCertSANs: ["<ETCD1-IP>", "127.0.0.1"]
    peerCertSANs: ["<ETCD1-IP>"]

For each member, generate server and peer certs (run these three times on cp1, moving the outputs aside between runs so they are not overwritten):

sudo kubeadm init phase certs etcd-server --config=etcd1.yaml
sudo kubeadm init phase certs etcd-peer --config=etcd1.yaml
sudo mkdir -p /root/etcd-certs/etcd1
sudo mv /etc/kubernetes/pki/etcd/{server,peer}.* /root/etcd-certs/etcd1/

Then generate the two client certs the API servers and health checks use (once):

sudo kubeadm init phase certs etcd-healthcheck-client
sudo kubeadm init phase certs apiserver-etcd-client

Distribute the Files

File(s)	cp1	cp2	Tiebreaker box
etcd/ca.crt	yes	yes	yes (cert only, not ca.key)
etcd1 server + peer certs	yes	—	—
etcd2 server + peer certs	—	yes	—
etcd3 server + peer certs	—	—	yes (into the Docker cert mount)
apiserver-etcd-client.*	yes	yes	—

On cp1/cp2 the files live under /etc/kubernetes/pki/etcd/ (named server.* and peer.*) — so after generating all three sets, copy etcd1’s server and peer certs back from /root/etcd-certs/etcd1/ into /etc/kubernetes/pki/etcd/ on cp1, and scp etcd2’s set to the same path on cp2. On the tiebreaker box, place etcd3’s set in a directory you will mount into the container, for example /opt/etcd/certs/.

Step 2: Start the Three etcd Members

etcd1 and etcd2 (systemd)

On both control-plane VMs, install the etcd binary at the pinned v3.5.x release:

ETCD_VER=v3.5.21
curl -L https://github.com/etcd-io/etcd/releases/download/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz | tar xz --strip-components=1 -C /usr/local/bin etcd-${ETCD_VER}-linux-amd64/etcd etcd-${ETCD_VER}-linux-amd64/etcdctl

Create /etc/systemd/system/etcd.service on cp1 (swap name/IPs for cp2):

[Unit]
Description=etcd (external member)
After=network-online.target

[Service]
ExecStart=/usr/local/bin/etcd \
  --name etcd1 \
  --data-dir /var/lib/etcd \
  --listen-peer-urls https://<CP1-IP>:2380 \
  --listen-client-urls https://<CP1-IP>:2379,https://127.0.0.1:2379 \
  --initial-advertise-peer-urls https://<CP1-IP>:2380 \
  --advertise-client-urls https://<CP1-IP>:2379 \
  --initial-cluster etcd1=https://<CP1-IP>:2380,etcd2=https://<CP2-IP>:2380,etcd3=https://<TIEBREAKER-IP>:2380 \
  --initial-cluster-state new \
  --cert-file=/etc/kubernetes/pki/etcd/server.crt \
  --key-file=/etc/kubernetes/pki/etcd/server.key \
  --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt \
  --peer-key-file=/etc/kubernetes/pki/etcd/peer.key \
  --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \
  --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt \
  --client-cert-auth --peer-client-cert-auth
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload && sudo systemctl enable --now etcd

etcd3 (Docker on the Tiebreaker Box)

docker run -d --name etcd3 --restart unless-stopped \
  -v /opt/etcd/data:/etcd-data \
  -v /opt/etcd/certs:/etcd-certs:ro \
  -p 2379:2379 -p 2380:2380 \
  quay.io/coreos/etcd:v3.5.21 \
  etcd --name etcd3 \
  --data-dir /etcd-data \
  --listen-peer-urls https://0.0.0.0:2380 \
  --listen-client-urls https://0.0.0.0:2379 \
  --initial-advertise-peer-urls https://<TIEBREAKER-IP>:2380 \
  --advertise-client-urls https://<TIEBREAKER-IP>:2379 \
  --initial-cluster etcd1=https://<CP1-IP>:2380,etcd2=https://<CP2-IP>:2380,etcd3=https://<TIEBREAKER-IP>:2380 \
  --initial-cluster-state new \
  --cert-file=/etcd-certs/server.crt \
  --key-file=/etcd-certs/server.key \
  --peer-cert-file=/etcd-certs/peer.crt \
  --peer-key-file=/etcd-certs/peer.key \
  --trusted-ca-file=/etcd-certs/ca.crt \
  --peer-trusted-ca-file=/etcd-certs/ca.crt \
  --client-cert-auth --peer-client-cert-auth

Start all three within a few minutes of each other — with --initial-cluster-state new they bootstrap as one cluster.

Verify Quorum

From cp1:

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://<CP1-IP>:2379,https://<CP2-IP>:2379,https://<TIEBREAKER-IP>:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  endpoint health

Expected: all three endpoints is healthy. Also check endpoint status -w table — exactly one member shows IS LEADER: true.

Step 3: kube-vip Before Init

kubeadm will be configured with controlPlaneEndpoint: <VIP>:6443, so the VIP must answer before kubeadm init runs. kube-vip runs as a static pod that kubelet starts as soon as the manifest exists — install it now, on cp1.

export VIP=<VIP-IP>
export INTERFACE=eth0
KVVERSION=v0.8.10
sudo ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION
sudo mkdir -p /etc/kubernetes/manifests
sudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip \
  /kube-vip manifest pod \
  --interface $INTERFACE --address $VIP \
  --controlplane --arp --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

Kubernetes 1.29+ gotcha: the generated manifest mounts /etc/kubernetes/admin.conf, but since 1.29 that kubeconfig is not usable until init completes (kubeadm bootstraps with super-admin.conf). Point kube-vip at super-admin.conf for first boot, and switch back after init.

sudo sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml

Step 4: kubeadm Init on cp1

Create kubeadm-config.yaml:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.33.4
controlPlaneEndpoint: "<VIP-IP>:6443"
etcd:
  external:
    endpoints:
      - https://<CP1-IP>:2379
      - https://<CP2-IP>:2379
      - https://<TIEBREAKER-IP>:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

sudo kubeadm init --config kubeadm-config.yaml --upload-certs

After init succeeds, revert the kube-vip workaround:

sudo sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml

Save the kubeadm join ... --control-plane and worker join commands the output prints.

Step 5: Join cp2 as a Second Control Plane

On cp2, drop the same kube-vip static pod manifest first (repeat Step 3 on cp2 — including the super-admin.conf note; on join it can reference admin.conf directly since the cluster CA already exists, but keeping the same sequence is harmless). Then:

sudo kubeadm join <VIP-IP>:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key>

With external etcd, --upload-certs/--certificate-key distributes the cluster CA bundle; the etcd client certs from Step 1 must already be in place on cp2.

Step 6: Nightly etcd Snapshots on Both Control Planes

Longhorn (next pages) backs up application volumes — not cluster state. Snapshot etcd separately, on both control planes so snapshots continue when either physical host is down. Mount the tiebreaker’s NFS share at /mnt/backup, then create /etc/systemd/system/etcd-snapshot.service:

[Unit]
Description=etcd snapshot to NFS

[Service]
Type=oneshot
ExecStart=/bin/bash -c '/usr/local/bin/etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  snapshot save /mnt/backup/etcd/$(hostname)-$(date +%%F).db'

And /etc/systemd/system/etcd-snapshot.timer — schedule 1:00 AM on cp1, 2:00 AM on cp2 (staggered):

[Unit]
Description=Nightly etcd snapshot

[Timer]
OnCalendar=*-*-* 01:00:00
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload && sudo systemctl enable --now etcd-snapshot.timer

Restore procedure and snapshot theory: etcd and Control Plane Health.

Verify Before Moving On

kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes

Expected: both control planes listed (NotReady is normal — no CNI yet).

Now prove the VIP survives a control-plane outage:

# On cp1:
sudo systemctl stop kubelet
# From cp2 (or any machine with the admin kubeconfig):
kubectl --server=https://<VIP-IP>:6443 get nodes

Expected: the API still answers within a few seconds (kube-vip on cp2 claims the VIP). Restart kubelet on cp1 afterwards.

Next: Workers, CNI, and VIP Failover.