<p>I have a system where some long running tasks are done by processing messages from a message queue. The actual tasks are doing some significant processing on large videos.</p>
<p>Here is the problem in the following steps:</p>
<ol>
<li>Process in pod takes message off queue, start processing video, this takes minutes.</li>
<li>Developer makes change, releases, and a Kubernetes Deployment starts.</li>
<li>During the deployment, the long running process gets killed and replaced by new node, which loses all work.</li>
</ol>
<p>Is there a mechanism to work around this in Kubernetes? Some kind of check to ensure that the worker in the pod is in a state that it can be destroyed safely?  Almost something like a destroyProbe (the opposite of a readinessProbe)</p>

<p>Calling <code>preStop</code> hook before container is terminated should help you perform grace shutdown. <code>preStop</code> hook is configured at container level and allows you to run a custom command before the <code>SIGTERM</code> will be sent (please note that the termination grace period countdown actually starts before invoking the <code>preStop</code> hook and not once the <code>SIGTERM</code> signal will be sent).</p>
<blockquote>
<p>This hook is called immediately before a container is terminated due
to an API request or management event such as liveness probe failure,
preemption, resource contention and others. A call to the preStop hook
fails if the container is already in terminated or completed state. It
is blocking, meaning it is synchronous, so it must complete before the
call to delete the container can be sent. No parameters are passed to
the handler.</p>
</blockquote>
<p>Setting also appropriate <code>terminationGracePeriod</code> also matters since
Kubernetes' management of the Container blocks until the preStop handler completes, unless the Pod's grace period expires. This means that termination grace period countdown starts before invoking the <code>preStop</code> hook and not once the <code>SIGTERM</code> signal is sent.</p>
<p>Check  <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/%29" rel="nofollow noreferrer">lifecycle hooks</a> and <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="nofollow noreferrer">pod termination</a> documents for more information.</p>
