<p>I have a kubernetes cluster with one master and two nodes.
For some reason, a node became unreachable for the cluster so all pods were moved to the other node. The problem is that the broken node keep in the cluster, but i think the master should remove the node automatically and create another one.</p>

<p>Can anyone help me?</p>

<p><strong>I option:</strong></p>
<p>If you work on GKE and have HA cluster, node with <strong>NotReady</strong> state should have been automatically deleted after couple of minutes if you have autoscaling mode on. After a while new node will be added.</p>
<p><strong>II option:</strong>
If you use kubeadm:</p>
<p>Nodes with state <strong>NotReady</strong> aren't automatically deleted if you don't have autoscaling mode on and HA cluster. Node will be continuously check and restart.</p>
<p>If you have Prometheus check metrics what happened on your node which has NotReady state or from unreachable node execute command:</p>
<p><code> $ sudo journalctl -u kubelet</code></p>
<p>If you want node with <strong>NotReady</strong> state to be deleted you should do it manually:</p>
<p>You should first drain the node and make sure that the node is empty before shutting it down.</p>
<p><code> $ kubectl drain &lt;node name&gt; --delete-local-data --force --ignore-daemonsets</code></p>
<p><code> $ kubectl delete node &lt;node name&gt;</code></p>
<p>Then, on the node being removed, reset all kubeadm installed state:</p>
<p><code>$ kubeadm reset</code></p>
<p>The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually:</p>
<p><code>$ iptables -F &amp;&amp; iptables -t nat -F &amp;&amp; iptables -t mangle -F &amp;&amp; iptables -X</code></p>
<p>If you want to reset the IPVS tables, you must run the following command:</p>
<p><code>$ ipvsadm -C</code></p>
<p>You can also simply shutdown desire node:</p>
<p><code>$ shutdown -h</code></p>
<p>The <strong>-h</strong> means halt while now clearly means that the instruction should be carried out immediately. Different delays can be used. For example, you might use +6 instead, which will tell the computer to run the shutdown procedure in six minutes.</p>
<p>In this case new node will <strong>not</strong> be added automatically.</p>
<p>I hope this helps.</p>
