<p>I recently noticed a big accumulation of pods with status 'Shutdown'. We have been using Kubernetes since October, 2020.</p>
<p>Production and staging is running on the same nodes except that staging uses preemtible nodes to cut the cost. The containers are also stable in staging. (Failures occur rarely as they are caught in testing before).</p>
<p>Service provider Google Cloud Kubernetes.</p>
<p>I familiarized myself with the docs and tried searching however neither I recognize neither google helps with this particular status. There are no errors in the logs.</p>
<p><a href="https://i.stack.imgur.com/pC7LW.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pC7LW.png" alt="example of bunch of shutdown pods" /></a>
<a href="https://i.stack.imgur.com/gEh6n.png" rel="noreferrer"><img src="https://i.stack.imgur.com/gEh6n.png" alt="pod description only says failed" /></a></p>
<p>I have no problem pods being stopped. Ideally I'd like K8s to automatically delete these shutdown pods. If I run <code>kubectl delete po redis-7b86cdccf9-zl6k9</code>, it goes away in a blink.</p>
<p><code>kubectl get pods | grep Shutdown | awk '{print $1}' | xargs kubectl delete pod</code> is manual temporary workaround.</p>
<p>PS. <code>k</code> is an alias to <code>kubectl</code> in my environment.</p>
<p>Final example: it happens across all namespaces // different containers.
<a href="https://i.stack.imgur.com/PXHY8.png" rel="noreferrer"><img src="https://i.stack.imgur.com/PXHY8.png" alt="enter image description here" /></a></p>
<p>I stumbled upon few related issues explaining the status
<a href="https://github.com/kubernetes/website/pull/28235" rel="noreferrer">https://github.com/kubernetes/website/pull/28235</a>
<a href="https://github.com/kubernetes/kubernetes/issues/102820" rel="noreferrer">https://github.com/kubernetes/kubernetes/issues/102820</a></p>
<p>&quot;When pods were evicted during the graceful node shutdown, they are marked as failed. Running <code>kubectl get pods</code> shows the status of the the evicted pods as <code>Shutdown</code>.&quot;</p>

<p>The evicted pods are not removed on purpose, as k8s team says here <a href="https://github.com/kubernetes/kubernetes/issues/54525#issuecomment-340035375" rel="noreferrer">1</a>, the evicted pods are nor removed in order to be inspected after eviction.</p>
<p>I believe here the best approach would be to create a cronjob <a href="https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/" rel="noreferrer">2</a> as already mentioned.</p>
<pre><code>apiVersion: batch/v1
kind: CronJob
metadata:
  name: del-shutdown-pods
spec:
  schedule: &quot;* 12 * * *&quot;
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - kubectl get pods | grep Shutdown | awk '{print $1}' | xargs kubectl delete pod
          restartPolicy: OnFailure
</code></pre>
