<p>I have 3 node in k8s cluster, where all are masters, i.e. I have removed the <code>node-role.kubernetes.io/master</code> taint.</p>

<p>I physically removed the network cable on <code>foo2</code>, so I have </p>

<pre><code>kubectl get nodes
NAME   STATUS     ROLES    AGE     VERSION
foo1   Ready      master   3d22h   v1.13.5
foo2   NotReady   master   3d22h   v1.13.5
foo3   Ready      master   3d22h   v1.13.5
</code></pre>

<p>After several hours some of the pods are still in <code>STATUS = Terminating</code> though I think they should be in <code>Terminated</code> ?</p>

<p>I read at <a href="https://www.bluematador.com/docs/troubleshooting/kubernetes-pod" rel="nofollow noreferrer">https://www.bluematador.com/docs/troubleshooting/kubernetes-pod</a></p>

<blockquote>
  <p>In rare cases, it is possible for a pod to get stuck in the terminating state. This is detected by finding any pods where every container has been terminated, but the pod is still running. Usually, this is caused when a node in the cluster gets taken out of service abruptly, and the cluster scheduler and controller-manager do not clean up all of the pods on that node.     </p>
  
  <p>Solving this issue is as simple as manually deleting the pod using kubectl delete pod .</p>
</blockquote>

<p>The pod describe says if unreachable for 5 minutes will be tolerated ...</p>

<pre><code>Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  etcd-data:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
    SizeLimit:   &lt;unset&gt;
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          &lt;none&gt;
</code></pre>

<p>I have tried <code>kubectl delete pod etcd-lns4g5xkcw</code> which just hung, though forcing it <strong>does</strong> work as per this <a href="https://stackoverflow.com/questions/55935173/kubernetes-pods-stuck-with-in-terminating-state">answer</a> ...</p>

<pre><code>kubectl delete pod etcd-lns4g5xkcw  --force=true --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "etcd-lns4g5xkcw" force deleted
</code></pre>

<p>(1) Why is this happening ? Shouln't it change to terminated? </p>

<p>(2) Where even is <code>STATUS = Terminating</code> coming from? At <a href="https://v1-13.docs.kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/" rel="nofollow noreferrer">https://v1-13.docs.kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/</a> I See only Waiting/Running/Terminated as the options ?</p>

k1eran
<p>Pods volume and network cleanup can consume more time while in <code>termination</code> status.
Proper way to do it is to drain node in order to get pods terminated successfully in grace period.
Because you plugged out the network cable the node has changed its status to <code>not ready</code> with pods already running on it.
Due to this pod could not terminate.</p>

<p>You may find this information from k8s documentation about <code>terminating</code> status useful:</p>

<blockquote>
  <p>Kubernetes (versions 1.5 or newer) will not delete Pods just because a
  Node is unreachable. The Pods running on an unreachable Node enter the
  ‘Terminating’ or ‘Unknown’ state after a timeout. Pods may also enter
  these states when the user attempts graceful deletion of a Pod on an
  unreachable Node:</p>
  
  <p>There are 3 suggested ways to remove it from apiserver:</p>
  
  <p>The Node object is deleted (either by you, or by the Node Controller).
  The kubelet on the unresponsive Node starts responding, kills the Pod
  and removes the entry from the apiserver. Force deletion of the Pod by
  the user.</p>
</blockquote>

<p>Here you can find more information about background deletion from k8s <a href="https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/" rel="nofollow noreferrer">offical documentation</a></p>
