<p>I am running a self managed Kubernetes cluster 1.18.1. I have deployed some pods with persistence volumes (based on the longhorn project).
Now after doing some testing I observe the following behavior:</p>
<p>If I simulate a hard shutdown of one node, after a while (5 minutes) Kubernetes is recognizing the loss and starts rescheduling PODs form the dead node to another.</p>
<p>Because of the fact that my nodes had persistence volumes the new POD will never start. The reason is that the old pod (on the dead node) is now durable in the the status terminating.</p>
<p>The fact that pods that reside on a crashed node did not terminate seems to be an well known Kubernetes limitation. See also the problem description <a href="https://www.ibm.com/support/knowledgecenter/SSRQ8T_1.0.0/csi_ug_node_crash.html" rel="nofollow noreferrer">here</a>.</p>
<p>My question is: Why does Kubernetes not provide a function to automatically terminate old PODs and resources like persistence volumes. Why do I have to intervene manually as an administrator? To me, this behavior seems not logical regarding to the promises that Kubernetes makes.</p>
<p>This is for example how my yaml file looks like:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
  labels: 
    app: db
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: db
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - env:
        - name: POSTGRES_DB
          value: office
        image: postgres:9.6.1
        name: db

        livenessProbe:
          tcpSocket:
            port: 5432
          initialDelaySeconds: 30
          periodSeconds: 10

        ports:
          - containerPort: 5432        
        resources: {}
        volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: dbdata
          subPath: postgres
      restartPolicy: Always
      volumes:
      - name: dbdata
        persistentVolumeClaim:
          claimName: office-demo-dbdata-pvc


# Storage
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: office-demo-dbdata-pv
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  claimRef:
    namespace: default
    name: office-demo-dbdata-pvc
  csi:
    driver: io.rancher.longhorn 
    fsType: ext4
    volumeHandle: office-demo-dbdata
  storageClassName: longhorn-durable
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: office-demo-dbdata-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-durable
  resources:
    requests:
      storage: 2Gi
  volumeName: &quot;office-demo-dbdata-pv&quot;
</code></pre>
<p>As explained the volume is created on Longhorn. But the attachement is not released even after kubernetes starts to reschedule the pod to another node.</p>
<p>The pod hanging in the termination status can be released if I manually delete the 'volumeattachment'</p>
<pre><code>$ kubectl delete volumeattachment csi-08d9842e.......
</code></pre>
<p>But in any case this is a manual action.</p>

<p>5 minutes is default eviction time set at control plane components of Kubernetes.If you want to customize that you can use <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions" rel="nofollow noreferrer">taint based evictions</a> and add below in the deployment yaml</p>

<pre><code>tolerations:
- key: "node.kubernetes.io/unreachable"
  operator: "Exists"
  effect: "NoExecute"
  tolerationSeconds: 60
</code></pre>

<p>Note that Kubernetes automatically adds a toleration for <code>node.kubernetes.io/not-ready</code> with <code>tolerationSeconds=300</code> unless the pod configuration provided by the user already has a toleration for <code>node.kubernetes.io/not-ready</code>. Likewise it adds a toleration for <code>node.kubernetes.io/unreachable</code> with <code>tolerationSeconds=300</code> unless the pod configuration provided by the user already has a toleration for <code>node.kubernetes.io/unreachable</code></p>
