<p>I set up a Kubernetes cluster with a single master node and two worker nodes using <code>kubeadm</code>, and I am trying to figure out how to recover from node failure.</p>

<p>When a worker node fails, recovery is straightforward: I create a new worker node from scratch, run <code>kubeadm join</code>, and everything's fine.</p>

<p>However, I cannot figure out how to recover from master node failure (without interrupting the deployments running on the worker nodes). Do I need to backup and restore the original certificates or can I just run <code>kubeadm init</code> to create a new master from scratch? How do I join the existing worker nodes?</p>

<p>I ended up writing a Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/" rel="nofollow noreferrer">CronJob</a> backing up the etcd data. If you are interested: I wrote a blog post about it: <a href="https://labs.consol.de/kubernetes/2018/05/25/kubeadm-backup.html" rel="nofollow noreferrer">https://labs.consol.de/kubernetes/2018/05/25/kubeadm-backup.html</a></p>

<p>In addition to that you may want to backup all of <code>/etc/kubernetes/pki</code> to avoid issues with secrets (tokens) having to be renewed.</p>

<p>For example, kube-proxy uses a secret to store a token and this token becomes invalid if only the etcd certificate is backed up.</p>
