<p>I am having some issues with a fairly new cluster where a couple of nodes (always seems to happen in pairs but potentially just a coincidence) will become NotReady and a <code>kubectl describe</code> will say that the Kubelet stopped posting node status for memory, disk, PID and ready.</p>
<p>All of the running pods are stuck in Terminating (can use k9s to connect to the cluster and see this) and the only solution I have found is to cordon and drain the nodes. After a few hours they seem to be being deleted and new ones created. Alternatively I can delete them using kubectl.</p>
<p>They are completely inaccessible via ssh (timeout) but AWS reports the EC2 instances as having no issues.</p>
<p>This has now happened three times in the past week. Everything does recover fine but there is clearly some issue and I would like to get to the bottom of it.</p>
<p>How would I go about finding out what has gone on if I cannot get onto the boxes at all? (Actually just occurred to me to maybe take a snapshot of the volume and mount it so will try that if it happens again, but any other suggestions welcome)</p>
<p>Running kubernetes v1.18.8</p>

<p>I had the same issue, after 20-30 min my nodes became in <code>NotRready</code> status, and all pods linked to these nodes became stuck in <code>Terminating</code> status.<br/>I tried to connect to my nodes via SSH, sometimes I faced a timeout, sometimes I could (hardly) connect, and I executed the <code>top</code> command to check the running processes.<br/>The most consuming process was <code>kswapd0</code>.<br/>My instance memory and CPU were both full (!), because it tried to swap a lot (due to a lack of memory), causing the <code>kswapd0</code> process to consume more than 50% of the CPU!<p><strong>Root cause</strong>:<br/>Some pods consumed 400% of their memory request (defined in Kubernetes deployment), because they were initially under-provisioned. As a consequence, when my nodes started, Kubernetes placed them on nodes with only 32Mb of memory request per pod (the value I had defined), but that was insufficient.<p><strong>Solution</strong>:<br/>The solution was to increase containers requests:</p>
<pre><code>requests:
  memory: &quot;32Mi&quot;
    cpu: &quot;20m&quot;
    limits:
       memory: &quot;256Mi&quot;
       cpu: &quot;100m&quot;
</code></pre>
<p>With these values (in my case):</p>
<pre><code>requests:
  memory: &quot;256Mi&quot;
    cpu: &quot;20m&quot;
    limits:
       memory: &quot;512Mi&quot;
       cpu: &quot;200m&quot;
</code></pre>
<p><strong>Important</strong>:<br/>
After that I processed a rolling update (<em>cordon</em> &gt; <em>drain</em> &gt; <em>delete</em>) of my nodes in order to ensure that Kubernetes reserve directly enough memory for my freshly started pods.<p>
<strong>Conclusion</strong>:<br/>
Regularly check your pods' memory consumption, and adjust your resources requests over time.<br/>
The goal is to never leave your nodes be surprised by a memory saturation, because the swap can be fatal for your nodes.</p>
