<p>After I update the backend code (pushing update to gcr.io), I delete the pod. Usually a new pod spins up.</p>
<p>But after today the whole cluster just breaks down. I really cannot comprehend what is happening here (I did not touch any of the other items).</p>
<p>I am really looking in the dark here. Where do I start looking?</p>
<p>I see that the logs show:</p>
<pre><code>0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
</code></pre>
<p>when I look this up:</p>
<pre><code>kubectl describe node | grep -i taint
Taints:             node.kubernetes.io/unreachable:NoSchedule
Taints:             node.kubernetes.io/unreachable:NoSchedule
</code></pre>
<p>But I have no clue what this is or how they even get there.</p>
<p>EDIT:</p>
<p>It looks like I need to remove the taints, but I am not able to (taint not found?)</p>
<pre><code>kubectl taint nodes --all node-role.kubernetes.io/unreachable-
taint &quot;node-role.kubernetes.io/unreachable&quot; not found
taint &quot;node-role.kubernetes.io/unreachable&quot; not found
</code></pre>
<p><a href="https://i.stack.imgur.com/mSx6Q.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mSx6Q.png" alt="enter image description here" /></a></p>

<p>Likely problem with the nodes. Debug with some of these (sample):</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   Ready      master    1d    v1.14.2
k8s-node1    NotReady   &lt;none&gt;    1d    v1.14.2
k8s-node2    NotReady   &lt;none&gt;    1d    v1.14.2 &lt;-- Does it say NotReady?
</code></pre>
<pre class="lang-sh prettyprint-override"><code>$ kubectl describe node k8s-node1
...
 # Do you see something like this? What's the event message?
 MemoryPressure...
 DiskPressure...
 PIDPressure...
</code></pre>
<p>Check if the kubelet is running on every node (it might be crashing and restarting)</p>
<pre class="lang-sh prettyprint-override"><code>ssh k8s-node1
# ps -Af | grep kubelet
# systemctl status kubelet
# journalctl -xeu kubelet
</code></pre>
<p>Nuclear option:</p>
<p>If you are using a <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/node-pools" rel="nofollow noreferrer">node pool</a>, delete your nodes and let the autoscaler restart brand new nodes.</p>
<p><a href="https://stackoverflow.com/questions/56614136/how-to-remove-kube-taints-from-worker-nodes-taints-node-kubernetes-io-unreachab">Related question/answer</a>.</p>
<p>✌️</p>
