<p>I have a K8s cluster that was working properly but because of power failure, all the nodes got rebooted.</p>

<p>At the moment I have some problem recovering the master (and other nodes):</p>

<ol>
<li><code>sudo systemctl kubelet status</code> is returning <code>Unknown operation kubelet.</code> but when I run <code>kubeadm init ...</code> (the command that I set up the cluster with) it returns:</li>
</ol>

<pre><code>error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR Port-6443]: Port 6443 is in use
    [ERROR Port-10251]: Port 10251 is in use
    [ERROR Port-10252]: Port 10252 is in use
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
    [ERROR Port-10250]: Port 10250 is in use
    [ERROR Port-2379]: Port 2379 is in use
    [ERROR Port-2380]: Port 2380 is in use
    [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
</code></pre>

<p>and when I checked those ports I can see that kubelet and other K8s components are using them:</p>

<pre><code>~/k8s-multi-node$ sudo lsof -i :10251
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
kube-sche 26292 root    3u  IPv6 104933      0t0  TCP *:10251 (LISTEN)

~/k8s-multi-node$ sudo lsof -i :10252
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
kube-cont 26256 root    3u  IPv6 115541      0t0  TCP *:10252 (LISTEN)

~/k8s-multi-node$ sudo lsof -i :10250
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
kubelet 24781 root   27u  IPv6 106821      0t0  TCP *:10250 (LISTEN)
</code></pre>

<p>I tried to kill them but they start to use those ports again.</p>

<ol start="2">
<li>My second problem is because of the power failure my machines don't have access to internet at the moment.</li>
</ol>

<p>So what is the proper way to recover such a cluster? Do I need to remove kubelet and all the otehr components and install them again?</p>

AVarf
<p>You need to first stop kubelet using <code>sudo systemctl stop kubelet.service</code></p>

<p>After that run <code>kubeadm reset</code> and then <code>kubeadm init</code>. Note that this will clean up existing cluster and create a new cluster altogether.</p>

<p>Regarding proper way to recover check this <a href="https://stackoverflow.com/questions/49480707/how-to-recover-from-master-failure-with-kubeadm">question</a></p>
