<p>I am running a self managed Kubernetes cluster 1.18.1. I have deployed some pods with persistence volumes (based on the longhorn project).
Now after doing some testing I observe the following behavior:</p>
<p>If I simulate a hard shutdown of one node, after a while (5 minutes) Kubernetes is recognizing the loss and starts rescheduling PODs form the dead node to another.</p>
<p>Because of the fact that my nodes had persistence volumes the new POD will never start. The reason is that the old pod (on the dead node) is now durable in the the status terminating.</p>
<p>The fact that pods that reside on a crashed node did not terminate seems to be an well known Kubernetes limitation. See also the problem description <a href="https://www.ibm.com/support/knowledgecenter/SSRQ8T_1.0.0/csi_ug_node_crash.html" rel="nofollow noreferrer">here</a>.</p>
<p>My question is: Why does Kubernetes not provide a function to automatically terminate old PODs and resources like persistence volumes. Why do I have to intervene manually as an administrator? To me, this behavior seems not logical regarding to the promises that Kubernetes makes.</p>
<p>This is for example how my yaml file looks like:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
  labels: 
    app: db
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: db
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - env:
        - name: POSTGRES_DB
          value: office
        image: postgres:9.6.1
        name: db

        livenessProbe:
          tcpSocket:
            port: 5432
          initialDelaySeconds: 30
          periodSeconds: 10

        ports:
          - containerPort: 5432        
        resources: {}
        volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: dbdata
          subPath: postgres
      restartPolicy: Always
      volumes:
      - name: dbdata
        persistentVolumeClaim:
          claimName: office-demo-dbdata-pvc


# Storage
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: office-demo-dbdata-pv
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  claimRef:
    namespace: default
    name: office-demo-dbdata-pvc
  csi:
    driver: io.rancher.longhorn 
    fsType: ext4
    volumeHandle: office-demo-dbdata
  storageClassName: longhorn-durable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: office-demo-dbdata-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-durable
  resources:
    requests:
      storage: 2Gi
  volumeName: &quot;office-demo-dbdata-pv&quot;
</code></pre>
<p>As explained the volume is created on Longhorn. But the attachement is not released even after kubernetes starts to reschedule the pod to another node.</p>
<p>The pod hanging in the termination status can be released if I manually delete the 'volumeattachment'</p>
<pre><code>$ kubectl delete volumeattachment csi-08d9842e.......
</code></pre>
<p>But in any case this is a manual action.</p>

<p>According to <a href="https://longhorn.io/docs/0.8.0/users-guide/node-failure/" rel="nofollow noreferrer">Longhorn Documentation on Node Failure</a>:</p>

<blockquote>
  <p>When a Kubernetes node fails with CSI driver installed (all the following are based on Kubernetes v1.12 with default setup):</p>
  
  <ol>
  <li>After  <strong>one minute</strong>,  <code>kubectl get nodes</code>  will report  <code>NotReady</code>  for the failure node.</li>
  <li>After about  <strong>five minutes</strong>, the states of all the pods on the  <code>NotReady</code>  node will change to either  <code>Unknown</code>  or  <code>NodeLost</code>.</li>
  <li>If you’re deploying using StatefulSet or Deployment, you need to decide is if it’s safe to force deletion the pod of the workload running on the lost node. See  <a href="https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/" rel="nofollow noreferrer">here</a>.
  
  <ol>
  <li>StatefulSet has stable identity, so Kubernetes won’t force delete the Pod for the user.</li>
  <li>Deployment doesn’t have stable identity, but Longhorn is a Read-Write-Once type of storage, which means it can only attached to one Pod. So the new Pod created by Kubernetes won’t be able to start due to the Longhorn volume still attached to the old Pod, on the lost Node.</li>
  <li>In both cases, Kubernetes will automatically evict the pod (set deletion timestamp for the pod) on the lost node, then try to  <strong>recreate a new one with old volumes</strong>. Because the evicted pod gets stuck in  <code>Terminating</code>  state and the attached Longhorn volumes cannot be released/reused, the new pod will get stuck in  <code>ContainerCreating</code>  state. That’s why users need to decide is if it’s safe to force deleting the pod.</li>
  </ol></li>
  </ol>
</blockquote>

<p>This is the current state of LongHorn (which is still in beta).</p>

<p>There is an open Issue on Github: <a href="https://github.com/longhorn/longhorn/issues/1105" rel="nofollow noreferrer">Improve node failure handling #1105</a> to address exacly that, but for now as stated in the documentation the admin has to intervene manually.</p>

<p>There is more Issues in Kubernetes Github like <a href="https://github.com/kubernetes/kubernetes/issues/69697#issuecomment-429112187" rel="nofollow noreferrer">this</a> one, which is a issue that I believe lies in the edge between kubernetes and the CSI, it's a mutual validation: The CSI is read-write-once and sets the storage as stable and locks the storage. In kubernetes part, it sees the pods has finalizers (as in the issue above) and won't delete them until the task is finished.</p>

<p>Unfortunately either way, as of today, requires manual interference.</p>

<ul>
<li>From longhorn perspective you should run the <code>delete volumeattachment</code>.</li>
<li>From kubernetes side you can remove the <a href="https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#finalizers" rel="nofollow noreferrer">pod finalizers</a> allowing it to remove the pods, <a href="https://github.com/kubernetes/kubernetes/issues/77258#issuecomment-540534981" rel="nofollow noreferrer">here</a> is an example of how to change the finalizers.</li>
</ul>

<p><strong>Edit:</strong></p>

<p><a href="https://github.com/kubernetes/kubernetes/issues/69697#issuecomment-489876217" rel="nofollow noreferrer">Github Kubernetes Issue #69697</a> Example for removing finalizers:</p>

<pre><code>kubectl patch pvc &lt;PVC_NAME&gt; -p '{"metadata":{"finalizers":null}}'
kubectl patch pod &lt;POD_NAME&gt; -p '{"metadata":{"finalizers":null}}'
</code></pre>

<p>You can create a script to remove your finalizers so you don't have to make it manually, like suggested in another <a href="https://github.com/kubernetes/kubernetes/issues/77258#issuecomment-514543465" rel="nofollow noreferrer">Kubernetes Open Issue #77258</a>:</p>

<blockquote>
  <p>Here's a one-liner to remove finalizers from all pv in the system:</p>
</blockquote>

<pre><code>kubectl get pv | tail -n+2 | awk '{print $1}' | xargs -I{} kubectl patch pv {} -p '{"metadata":{"finalizers": null}}'
</code></pre>

<p>The great issue here, is that the finalizers are added by the LongHorn, so in my understanding you can't create the pods without it because it is added afterwards by the LongHorn.</p>

<p>I added the documentation and the Open Issues from Github to show you that this is a current issue and is yet to be resolved by the developers both from Longhorn and Kubernetes. </p>
