<p>I've started learning kubernetes with docker and I've been thinking, what happens if master node dies/fails. I've already read the answers <a href="https://stackoverflow.com/q/39172131/3906965">here</a>. But it doesn't answer the remedy for it.</p>
<p>Who is responsible to bring it back? And how to bring it back? Can there be a backup master node to avoid this? If yes, how?</p>
<p>Basically, I'm asking a recommended way to handle master failure in kubernetes setup.</p>

<p>You should have multiple VMs serving as master node to avoid single point of failure.An odd number of 3 or 5 master nodes are recommended for quorum. Have a load balancer in-front of all the VMs serving as master node which can do load balancing and in case one master node dies loadbalancer should remove the VMs IP and make it as unhealthy and not send traffic to it.</p>
<p>Also ETCD cluster is the brain of a kubernetes cluster. So you should have multiple VMs serving as ETCD nodes. Those VMs can be same VMs as of master node or for reduced blast radius you can have separate VMs for ETCD. Again the odd number of VMs should should be 3 or 5. Make sure to take periodic backup of ETCD nodes data so that you can restore the cluster state to pervious state in case of a disaster.</p>
<p>Check the official <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/" rel="nofollow noreferrer">doc</a> on how to install a HA kubernetes cluster using Kubeadm.</p>
<p><a href="https://i.stack.imgur.com/T3suN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/T3suN.png" alt="HA with stacked ETCD" /></a>
<a href="https://i.stack.imgur.com/yiqgC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/yiqgC.png" alt="enter image description here" /></a></p>
