<p>I recently noticed a big accumulation of pods with status 'Shutdown'. We have been using Kubernetes since October, 2020.</p>
<p>Production and staging is running on the same nodes except that staging uses preemtible nodes to cut the cost. The containers are also stable in staging. (Failures occur rarely as they are caught in testing before).</p>
<p>Service provider Google Cloud Kubernetes.</p>
<p>I familiarized myself with the docs and tried searching however neither I recognize neither google helps with this particular status. There are no errors in the logs.</p>
<p><a href="https://i.stack.imgur.com/pC7LW.png" rel="noreferrer"><img src="https://i.stack.imgur.com/pC7LW.png" alt="example of bunch of shutdown pods" /></a>
<a href="https://i.stack.imgur.com/gEh6n.png" rel="noreferrer"><img src="https://i.stack.imgur.com/gEh6n.png" alt="pod description only says failed" /></a></p>
<p>I have no problem pods being stopped. Ideally I'd like K8s to automatically delete these shutdown pods. If I run <code>kubectl delete po redis-7b86cdccf9-zl6k9</code>, it goes away in a blink.</p>
<p><code>kubectl get pods | grep Shutdown | awk '{print $1}' | xargs kubectl delete pod</code> is manual temporary workaround.</p>
<p>PS. <code>k</code> is an alias to <code>kubectl</code> in my environment.</p>
<p>Final example: it happens across all namespaces // different containers.
<a href="https://i.stack.imgur.com/PXHY8.png" rel="noreferrer"><img src="https://i.stack.imgur.com/PXHY8.png" alt="enter image description here" /></a></p>
<p>I stumbled upon few related issues explaining the status
<a href="https://github.com/kubernetes/website/pull/28235" rel="noreferrer">https://github.com/kubernetes/website/pull/28235</a>
<a href="https://github.com/kubernetes/kubernetes/issues/102820" rel="noreferrer">https://github.com/kubernetes/kubernetes/issues/102820</a></p>
<p>&quot;When pods were evicted during the graceful node shutdown, they are marked as failed. Running <code>kubectl get pods</code> shows the status of the the evicted pods as <code>Shutdown</code>.&quot;</p>

Lukas
<p>You don't need any <em>grep</em> - just use selectors that kubectl provides. And, btw, you cannot call kubectl from the busybox image, because it doesn't have <em>kubectl</em> at all. I also created a service account with the right of pods deletion.</p>
<pre><code>apiVersion: batch/v1
kind: CronJob
metadata:
  name: del-shutdown-pods
spec:
  schedule: &quot;0 */2 * * *&quot;  
  concurrencyPolicy: Replace
  jobTemplate:
    metadata:
      name: shutdown-deleter
    spec:
      template:
        spec:
          serviceAccountName: deleter
          containers:
          - name: shutdown-deleter
            image: bitnami/kubectl
            imagePullPolicy: IfNotPresent
            command:
              - &quot;/bin/sh&quot;
            args:
              - &quot;-c&quot;
              - &quot;kubectl delete pods --field-selector status.phase=Failed -A --ignore-not-found=true&quot;
          restartPolicy: Never
</code></pre>
