<p>I have a cluster where the free memory on the nodes recently dipped to %5. When this happens, the nodes CPU (load) spikes while it tries to free up some memory, from cache/buffer. One consequence of the high load, low memory is that I sometimes end up with Pods that get into an Error state or get stuck in Terminating. These Pods sit around until I manually intervene, which can further exacerbate the low memory issue that caused it.</p>

<p>My question is why Kubernetes leaves these Pods stuck in this state? My hunch is that kubernetes didn’t get the right feedback from the Docker daemon and never tries again. I need to know how to have Kubernetes cleanup or repair Error and Terminating Pods. Any ideas?</p>

<p>I'm currently on:</p>

<pre><code>~ # kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p><strong>UPDATE:</strong>
Here are some of the Events listed in pods. You can see that some of them sit around for days. You will also see that one shows a Warning, but the others show Normal.</p>

<pre><code>Events:
  Type     Reason         Age                  From                 Message
  ----     ------         ----                 ----                 -------
  Warning  FailedKillPod  25m                  kubelet, k8s-node-0  error killing pod: failed to "KillContainer" for "kubectl" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
  Normal   Killing        20m (x2482 over 3d)  kubelet, k8s-node-0  Killing container with id docker://docker:Need to kill Pod
  Normal   Killing        15m (x2484 over 3d)  kubelet, k8s-node-0  Killing container with id docker://maven:Need to kill Pod
  Normal   Killing        8m (x2487 over 3d)   kubelet, k8s-node-0  Killing container with id docker://node:Need to kill Pod
  Normal   Killing        4m (x2489 over 3d)   kubelet, k8s-node-0  Killing container with id docker://jnlp:Need to kill Pod

Events:
  Type    Reason   Age                 From                 Message
  ----    ------   ----                ----                 -------
  Normal  Killing  56m (x125 over 5h)  kubelet, k8s-node-2  Killing container with id docker://owasp-zap:Need to kill Pod
  Normal  Killing  47m (x129 over 5h)  kubelet, k8s-node-2  Killing container with id docker://jnlp:Need to kill Pod
  Normal  Killing  38m (x133 over 5h)  kubelet, k8s-node-2  Killing container with id docker://dind:Need to kill Pod
  Normal  Killing  13m (x144 over 5h)  kubelet, k8s-node-2  Killing container with id docker://maven:Need to kill Pod
  Normal  Killing  8m (x146 over 5h)   kubelet, k8s-node-2  Killing container with id docker://docker-cmds:Need to kill Pod
  Normal  Killing  1m (x149 over 5h)   kubelet, k8s-node-2  Killing container with id docker://pmd:Need to kill Pod

Events:
  Type    Reason   Age                  From                 Message
  ----    ------   ----                 ----                 -------
  Normal  Killing  56m (x2644 over 4d)  kubelet, k8s-node-0  Killing container with id docker://openssl:Need to kill Pod
  Normal  Killing  40m (x2651 over 4d)  kubelet, k8s-node-0  Killing container with id docker://owasp-zap:Need to kill Pod
  Normal  Killing  31m (x2655 over 4d)  kubelet, k8s-node-0  Killing container with id docker://pmd:Need to kill Pod
  Normal  Killing  26m (x2657 over 4d)  kubelet, k8s-node-0  Killing container with id docker://kubectl:Need to kill Pod
  Normal  Killing  22m (x2659 over 4d)  kubelet, k8s-node-0  Killing container with id docker://dind:Need to kill Pod
  Normal  Killing  11m (x2664 over 4d)  kubelet, k8s-node-0  Killing container with id docker://docker-cmds:Need to kill Pod
  Normal  Killing  6m (x2666 over 4d)   kubelet, k8s-node-0  Killing container with id docker://maven:Need to kill Pod
  Normal  Killing  1m (x2668 over 4d)   kubelet, k8s-node-0  Killing container with id docker://jnlp:Need to kill Pod
</code></pre>

<p>This is typically related to the <a href="https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#advanced-topics" rel="nofollow noreferrer">metadata.finalizers</a> on your objects (pod, deployment, etc)</p>

<p>You can also read more about <a href="https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/#foreground-cascading-deletion" rel="nofollow noreferrer">Foreground Cascading Deleting</a> and how it uses metadata.finalizers.</p>

<p>If not it could be a networking issue, you could check the kubelet logs, typically:</p>

<pre><code>journalctl -xeu kubelet 
</code></pre>

<p>You can also check the docker daemon logs, typically:</p>

<pre><code>cat /var/log/syslog | grep dockerd
</code></pre>
