title: How can kubernetes recover Pods stuck in Error or Terminating
body: <p>I have a cluster where the free memory on the nodes recently dipped to %5. When this happens, the nodes CPU (load) spikes while it tries to free up some memory, from cache/buffer. One consequence of the high load, low memory is that I sometimes end up with Pods that get into an Error state or get stuck in Terminating. These Pods sit around until I manually intervene, which can further exacerbate the low memory issue that caused it.</p>
  
  <p>My question is why Kubernetes leaves these Pods stuck in this state? My hunch is that kubernetes didn’t get the right feedback from the Docker daemon and never tries again. I need to know how to have Kubernetes cleanup or repair Error and Terminating Pods. Any ideas?</p>
  
  <p>I'm currently on:</p>
  
  <pre><code>~ # kubectl version
  Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"bb9ffb1654d4a729bb4cec18ff088eacc153c239", GitTreeState:"clean", BuildDate:"2018-08-07T23:17:28Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
  </code></pre>
  
  <p><strong>UPDATE:</strong>
  Here are some of the Events listed in pods. You can see that some of them sit around for days. You will also see that one shows a Warning, but the others show Normal.</p>
  
  <pre><code>Events:
    Type     Reason         Age                  From                 Message
    ----     ------         ----                 ----                 -------
    Warning  FailedKillPod  25m                  kubelet, k8s-node-0  error killing pod: failed to "KillContainer" for "kubectl" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
    Normal   Killing        20m (x2482 over 3d)  kubelet, k8s-node-0  Killing container with id docker://docker:Need to kill Pod
    Normal   Killing        15m (x2484 over 3d)  kubelet, k8s-node-0  Killing container with id docker://maven:Need to kill Pod
    Normal   Killing        8m (x2487 over 3d)   kubelet, k8s-node-0  Killing container with id docker://node:Need to kill Pod
    Normal   Killing        4m (x2489 over 3d)   kubelet, k8s-node-0  Killing container with id docker://jnlp:Need to kill Pod
  
  Events:
    Type    Reason   Age                 From                 Message
    ----    ------   ----                ----                 -------
    Normal  Killing  56m (x125 over 5h)  kubelet, k8s-node-2  Killing container with id docker://owasp-zap:Need to kill Pod
    Normal  Killing  47m (x129 over 5h)  kubelet, k8s-node-2  Killing container with id docker://jnlp:Need to kill Pod
    Normal  Killing  38m (x133 over 5h)  kubelet, k8s-node-2  Killing container with id docker://dind:Need to kill Pod
    Normal  Killing  13m (x144 over 5h)  kubelet, k8s-node-2  Killing container with id docker://maven:Need to kill Pod
    Normal  Killing  8m (x146 over 5h)   kubelet, k8s-node-2  Killing container with id docker://docker-cmds:Need to kill Pod
    Normal  Killing  1m (x149 over 5h)   kubelet, k8s-node-2  Killing container with id docker://pmd:Need to kill Pod
  
  Events:
    Type    Reason   Age                  From                 Message
    ----    ------   ----                 ----                 -------
    Normal  Killing  56m (x2644 over 4d)  kubelet, k8s-node-0  Killing container with id docker://openssl:Need to kill Pod
    Normal  Killing  40m (x2651 over 4d)  kubelet, k8s-node-0  Killing container with id docker://owasp-zap:Need to kill Pod
    Normal  Killing  31m (x2655 over 4d)  kubelet, k8s-node-0  Killing container with id docker://pmd:Need to kill Pod
    Normal  Killing  26m (x2657 over 4d)  kubelet, k8s-node-0  Killing container with id docker://kubectl:Need to kill Pod
    Normal  Killing  22m (x2659 over 4d)  kubelet, k8s-node-0  Killing container with id docker://dind:Need to kill Pod
    Normal  Killing  11m (x2664 over 4d)  kubelet, k8s-node-0  Killing container with id docker://docker-cmds:Need to kill Pod
    Normal  Killing  6m (x2666 over 4d)   kubelet, k8s-node-0  Killing container with id docker://maven:Need to kill Pod
    Normal  Killing  1m (x2668 over 4d)   kubelet, k8s-node-0  Killing container with id docker://jnlp:Need to kill Pod
  </code></pre>