<p>I am having some issues with a fairly new cluster where a couple of nodes (always seems to happen in pairs but potentially just a coincidence) will become NotReady and a <code>kubectl describe</code> will say that the Kubelet stopped posting node status for memory, disk, PID and ready.</p>
<p>All of the running pods are stuck in Terminating (can use k9s to connect to the cluster and see this) and the only solution I have found is to cordon and drain the nodes. After a few hours they seem to be being deleted and new ones created. Alternatively I can delete them using kubectl.</p>
<p>They are completely inaccessible via ssh (timeout) but AWS reports the EC2 instances as having no issues.</p>
<p>This has now happened three times in the past week. Everything does recover fine but there is clearly some issue and I would like to get to the bottom of it.</p>
<p>How would I go about finding out what has gone on if I cannot get onto the boxes at all? (Actually just occurred to me to maybe take a snapshot of the volume and mount it so will try that if it happens again, but any other suggestions welcome)</p>
<p>Running kubernetes v1.18.8</p>

<p>There are two most common possibilities here, both most likely caused by a large load:</p>
<ul>
<li><p><code>Out of Memory</code> error on the kubelet host. Can be solved by adding proper <code>--kubelet-extra-args</code> to <code>BootstrapArguments</code>. For example: <code>--kubelet-extra-args &quot;--kube-reserved memory=0.3Gi,ephemeral-storage=1Gi --system-reserved memory=0.2Gi,ephemeral-storage=1Gi --eviction-hard memory.available&lt;200Mi,nodefs.available&lt;10%&quot; </code></p>
</li>
<li><p>An issue explained <a href="https://github.com/kubernetes/kubernetes/issues/74302#issuecomment-467359176" rel="nofollow noreferrer">here</a>:</p>
</li>
</ul>
<blockquote>
<p>kubelet cannot patch its node status sometimes, ’cos more than 250
resources stay on the node, kubelet cannot watch more than 250 streams
with kube-apiserver at the same time. So, I just adjust kube-apiserver
--http2-max-streams-per-connection to 1000 to relieve the pain.</p>
</blockquote>
<p>You can either adjust the values provided above or try to find the cause of high load/iops and try to tune it down.</p>
