<p>Is there is a way to automatically delay <em>all</em> Kubernetes pod deletion requests such that the endpoint deregistration is signaled, but the pod's SIGTERM is delayed by several seconds?</p>

<p>It would be preferable, but not required, if the delay only affected pods with an Endpoint/Service.</p>

<p>Background:</p>

<p>It is <a href="https://stackoverflow.com/questions/45083275/kubernetes-nginx-how-to-have-zero-downtime-deployments">well</a> <a href="https://stackoverflow.com/questions/40545581/do-kubernetes-pods-still-receive-requests-after-receiving-sigterm">established</a> that some traffic can continue to a Pod after a pod has been sent the SIGTERM termination signal due to the <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods" rel="nofollow noreferrer">asynchronous</a> nature of endpoint deregistration and the deletion signal.  The recommended <a href="https://freecontent.manning.com/handling-client-requests-properly-with-kubernetes/" rel="nofollow noreferrer">mitigation</a> is to introduce a few seconds delay in the pod's <code>preStop</code> lifecycle hook by invoking <code>sleep</code>.</p>

<p>The difficulty rapidly arises where the pod's deployment may be done via helm or other upstream source, or else there are large numbers of deployments and containers to be managed.  Modifying many deployments in such a way may be difficult, or even impossible (e.g. the container may not have a sleep binary, shell, or anything but the application executable).</p>

<p>I briefly explored a mutating admission controller, but that seems unworkable to dynamically add a <code>preStop</code> hook, as all images do not have a <code>/bin/sleep</code> or already have a <code>preStop</code> that could need image-specific knowledge to merge.</p>

<p>(Of course, all of this could be avoided if the K8S API made the endpoint deregistration synchronous with a timeout to avoid deadlock (hint, hint), but I haven't seen any discussions of such a change. Yes, there are tons of reasons why this isn't synchronous, but that doesn't mean something can't be done.)</p>

<p>Kubernetes lifecycle has following steps.</p>

<ul>
<li>Pod is set to the “Terminating” State and removed from the endpoints list of all Services</li>
<li>preStop hook is executed</li>
<li>SIGTERM signal is sent to the pod</li>
<li>Kubernetes waits for a grace period, default is 30 seconds</li>
<li>SIGKILL signal is sent to pod, and the pod is removed</li>
</ul>

<p>Grace period is what you need. 
It's important to node that this grace period is happening in parallel to the preStop hook and the SIGTERM signal.</p>

<blockquote>
  <p>A call to the preStop hook fails if the container is already in terminated or completed state. It is blocking, meaning it is synchronous, so it must complete before the call to delete the container can be sent.</p>
</blockquote>

<p>Here you can read more about <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/" rel="nofollow noreferrer">Container Lifecycle Hooks</a>.</p>

<p>So for example you could set the <code>terminationGracePeriodSeconds: 90</code> and this might look like the following:</p>

<pre><code>spec: 
   terminationGracePeriodSeconds: 90
   containers:
       - name: myApplication
</code></pre>

<p>You can read the Kubernetes docs regarding <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods" rel="nofollow noreferrer">Termination of Pods</a>. I also recommend great blog post <a href="https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-terminating-with-grace" rel="nofollow noreferrer">Kubernetes best practices: terminating with grace</a>.</p>
