title: k8s pods stuck in failed/shutdown state after preemption (gke v1.20)
body: <p>TL;DR - gke 1.20 preemptible nodes cause pods to zombie into Failed/Shutdown</p>
  <p>We have been using GKE for a few years with clusters containing a mixture of both stable and preemptible node pools. Recently, since gke v1.20, we have started seeing preempted pods enter into a weird zombie state where they are described as:</p>
  <p>Status:         Failed</p>
  <p>Reason:         Shutdown</p>
  <p>Message:        Node is shutting, evicting pods</p>
  <p>When this started occurring we were convinced it was related to our pods failing to properly handle the SIGTERM at preemption. We decided to eliminate our service software as a source of a problem by boiling it down to a simple service that mostly sleeps:</p>
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-js lang-js prettyprint-override"><code>/* eslint-disable no-console */
  let exitNow = false
  
  process.on( 'SIGINT', () =&gt; {
    console.log( 'INT shutting down gracefully' )
    exitNow = true
  } )
  
  process.on( 'SIGTERM', () =&gt; {
    console.log( 'TERM shutting down gracefully' )
    exitNow = true
  } )
  
  const sleep = ( seconds ) =&gt; {
    return new Promise( ( resolve ) =&gt; {
      setTimeout( resolve, seconds * 1000 )
    } )
  }
  
  const Main = async ( cycles = 120, delaySec = 5 ) =&gt; {
    console.log( `Starting ${cycles}, ${delaySec} second cycles` )
  
    for ( let i = 1; i &lt;= cycles &amp;&amp; !exitNow; i++ ) {
      console.log( `---&gt; ${i} of ${cycles}` )
      await sleep( delaySec ) // eslint-disable-line
    }
  
    console.log( '*** Cycle Complete - exiting' )
    process.exit( 0 )
  }
  
  Main()</code></pre>
  </div>
  </div>
  </p>
  <p>This code is built into a docker image using the tini init to spawn the pod process running under nodejs (fermium-alpine image). No matter how we shuffle the signal handling it seems the pods never really shutdown cleanly, even though the logs suggest they are.</p>
  <p>Another oddity to this is that according to the Kubernetes Pod logs, we see the pod termination start and then gets cancelled:</p>
  <p>2021-08-06 17:00:08.000 EDT Stopping container preempt-pod</p>
  <p>2021-08-06 17:02:41.000 EDT Cancelling deletion of Pod preempt-pod</p>
  <p>We have also tried adding a preStop 15 second delay just to see if that has any effect, but nothing we try seems to matter - the pods become zombies. New replicas are started on the other nodes that are available in the pool, so it always maintains the minimum number of successfully running pods on the system.</p>
  <p>We are also testing the preemption cycle using a sim maintenance event:</p>
  <p>gcloud compute instances simulate-maintenance-event node-id</p>