<h3>Context</h3>

<p>We are currently using a few clusters with v1.8.7 (which was created by currently unavailable developers, months ago) and are trying to upgrade to a higher version.
However, we wanted to try the same on an cluster we use for experimental &amp; POCs.</p>

<h3>What we tried</h3>

<p>In doing the same, we tried to run a few kubeadm commands on one of the master nodes, but kubeadm was not found.</p>

<p>So, we tried installing the same with commands -</p>

<pre><code>apt-get update &amp;&amp; apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat &lt;&lt;EOF &gt;/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
</code></pre>

<h3>What happened</h3>

<p>However, now that node has status Not Ready and kubelet service is failing</p>

<p>Any pointers on how to fix this and what we should've done ?</p>

<pre><code>root@k8s-master-dev-0:/home/azureuser# kubectl get nodes
NAME                     STATUS     ROLES     AGE       VERSION
k8s-master-dev-0         NotReady   master    118d      v1.8.7
k8s-master-dev-1         Ready      master    118d      v1.8.7
k8s-master-dev-2         Ready      master    163d      v1.8.7
k8s-agents-dev-0         Ready      agent     163d      v1.8.7
k8s-agents-dev-1         Ready      agent     163d      v1.8.7
k8s-agents-dev-2         Ready      agent     163d      v1.8.7

root@k8s-master-dev-0:/home/azureuser# systemctl status kubelet.service
● kubelet.service - Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: failed (Result: start-limit-hit) since Thu 2018-12-13 14:33:25 UTC; 18h ago

Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Control process exited, code=exited status=2
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: Failed to start Kubelet.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Unit entered failed state.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: Stopped Kubelet.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Start request repeated too quickly.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: Failed to start Kubelet.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Unit entered failed state.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Failed with result 'start-limit-hit'.
</code></pre>

<p>The reason your kubelet went into bad state is that you upgraded kubelet package and service file for kubelet must be renewed and If you earlier did some changes must be lost.</p>

<p>Following things you can try:</p>

<ol>
<li>Disabling you swap memory: <code>swapoff -a</code></li>
<li>Check your kubelet service file, for kubeadm it is located at <code>/etc/systemd/system/kubelet.service.d/10-kubeadm.conf</code> and check the value <code>--cgroup-driver</code> and if it is <code>systemd</code> make it <code>cgroupfs</code> and then:</li>
</ol>

<p>Reload the daemon and restart kubelet:</p>

<pre><code>systemctl daemon-reload
systemctl restart kubelet
</code></pre>

<p>Now check if your kubelet started or not. </p>

<p>PS: Live upgrade of kubeadm control plane should be done carefully, check my answer on how to upgrade kubeadm</p>

<p><a href="https://stackoverflow.com/questions/53771883/how-to-upgrade-kubernetes-from-v1-10-0-to-v1-10-11/53773310#53773310">how to upgrade kubernetes from v1.10.0 to v1.10.11</a></p>
