<p>TL;DR - gke 1.20 preemptible nodes cause pods to zombie into Failed/Shutdown</p>
<p>We have been using GKE for a few years with clusters containing a mixture of both stable and preemptible node pools. Recently, since gke v1.20, we have started seeing preempted pods enter into a weird zombie state where they are described as:</p>
<p>Status:         Failed</p>
<p>Reason:         Shutdown</p>
<p>Message:        Node is shutting, evicting pods</p>
<p>When this started occurring we were convinced it was related to our pods failing to properly handle the SIGTERM at preemption. We decided to eliminate our service software as a source of a problem by boiling it down to a simple service that mostly sleeps:</p>
<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>/* eslint-disable no-console */
let exitNow = false

process.on( 'SIGINT', () =&gt; {
  console.log( 'INT shutting down gracefully' )
  exitNow = true
} )

process.on( 'SIGTERM', () =&gt; {
  console.log( 'TERM shutting down gracefully' )
  exitNow = true
} )

const sleep = ( seconds ) =&gt; {
  return new Promise( ( resolve ) =&gt; {
    setTimeout( resolve, seconds * 1000 )
  } )
}

const Main = async ( cycles = 120, delaySec = 5 ) =&gt; {
  console.log( `Starting ${cycles}, ${delaySec} second cycles` )

  for ( let i = 1; i &lt;= cycles &amp;&amp; !exitNow; i++ ) {
    console.log( `---&gt; ${i} of ${cycles}` )
    await sleep( delaySec ) // eslint-disable-line
  }

  console.log( '*** Cycle Complete - exiting' )
  process.exit( 0 )
}

Main()</code></pre>
</div>
</div>
</p>
<p>This code is built into a docker image using the tini init to spawn the pod process running under nodejs (fermium-alpine image). No matter how we shuffle the signal handling it seems the pods never really shutdown cleanly, even though the logs suggest they are.</p>
<p>Another oddity to this is that according to the Kubernetes Pod logs, we see the pod termination start and then gets cancelled:</p>
<p>2021-08-06 17:00:08.000 EDT Stopping container preempt-pod</p>
<p>2021-08-06 17:02:41.000 EDT Cancelling deletion of Pod preempt-pod</p>
<p>We have also tried adding a preStop 15 second delay just to see if that has any effect, but nothing we try seems to matter - the pods become zombies. New replicas are started on the other nodes that are available in the pool, so it always maintains the minimum number of successfully running pods on the system.</p>
<p>We are also testing the preemption cycle using a sim maintenance event:</p>
<p>gcloud compute instances simulate-maintenance-event node-id</p>

<p>After poking around various posts I finally relented to running a cronjob every 9 minutes to avoid the alertManager trigger that occurs after pods have been stuck in shutdown for 10+ minutes. This still feels like a hack to me, but it works, and it forced me to dig in to k8s cronjob and RBAC.</p>
<p>This post started me on the path:
<a href="https://stackoverflow.com/questions/68344231/how-to-remove-kubernetes-shutdown-pods">How to remove Kubernetes &#39;shutdown&#39; pods</a></p>
<p>And the resultant cronjob spec:</p>
<pre><code>apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-accessor-role
  namespace: default
rules:
- apiGroups: [&quot;&quot;] # &quot;&quot; indicates the core API group
  resources: [&quot;pods&quot;]
  verbs: [&quot;get&quot;, &quot;delete&quot;, &quot;watch&quot;, &quot;list&quot;]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-access
  namespace: default
subjects:
- kind: ServiceAccount
  name: cronjob-sa
  namespace: default
roleRef:
  kind: Role
  name: pod-accessor-role
  apiGroup: &quot;&quot;
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cronjob-sa
  namespace: default
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cron-zombie-killer
  namespace: default
spec:
  schedule: &quot;*/9 * * * *&quot;
  successfulJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        metadata:
          name: cron-zombie-killer
          namespace: default
        spec:
          serviceAccountName: cronjob-sa
          restartPolicy: Never
          containers:
          - name: cron-zombie-killer
            imagePullPolicy: IfNotPresent
            image: bitnami/kubectl
            command:
              - &quot;/bin/sh&quot;
            args:
              - &quot;-c&quot;
              - &quot;kubectl get pods -n default --field-selector='status.phase==Failed' -o name | xargs kubectl delete -n default 2&gt; /dev/null&quot;
status: {}
</code></pre>
<p>Note that the redirect of stderr to /dev/null is to simply avoid the error output from kubectl delete when the kubectl get doesn't find any pods in the failed state.</p>
<p><strong>Update</strong> added missing &quot;delete&quot; verb from the role, and added the missing RoleBinding</p>
<p><strong>Update</strong> added imagePullPolicy</p>
