<p>I couldn't find anything in the official kubernetes docs about this.  What's the actual low-level process for replacing a long-running cron job?  I'd like to understand this so my application can handle it properly.</p>
<ul>
<li>Is it a clean <code>SIGHUP</code>/<code>SIGTERM</code> signal that gets sent to the app that's running?</li>
<li>Is there a waiting period after that signal gets sent, so the app has time to cleanup/shutdown before it potentially gets killed?  If so, what's that timeout in seconds?  Or does it wait forever?</li>
</ul>
<p>For reference, here's the <code>Replace</code> policy explanation in the docs:</p>
<p><a href="https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/</a></p>
<blockquote>
<p>Concurrency Policy</p>
<ul>
<li><strong>Replace: If it is time for a new job run and the previous job run hasn’t finished yet, the cron job replaces the currently running job run with a new job run</strong></li>
</ul>
</blockquote>

<p>A CronJob has just another Pod underneath.</p>

<p>When a Cronjob with a concurrency policy of "Replace" is still active, the <a href="https://github.com/kubernetes/kubernetes/blob/6a19261e96b12c83e6c69d15e2dcca8089432838/pkg/controller/cronjob/cronjob_controller.go#L324" rel="nofollow noreferrer">Job will be deleted</a>, which also deletes the Pod. </p>

<p>When a Pod is <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods" rel="nofollow noreferrer">deleted</a> the Linux container/s will be sent a <code>SIGTERM</code> and then a <code>SIGKILL</code> after a grace period, defaulted to 30 seconds.  The <code>terminationGracePeriodSeconds</code> property in a <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#podspec-v1-core" rel="nofollow noreferrer">PodSpec</a> can be set to override that default.</p>

<p>Due to the <a href="https://github.com/kubernetes/kubernetes/blob/3b440dfd5550b13278e40975a7cba13a3f129475/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L441-L443" rel="nofollow noreferrer">flag</a> added to the <a href="https://github.com/kubernetes/kubernetes/blob/933f9a469feadcb1904e9d2b7258102ea89f6c7a/pkg/controller/cronjob/injection.go#L120-L123" rel="nofollow noreferrer"><code>DeleteJob</code></a> call, it sounds like this delete is only deleting values from the kube key/value store. Which would mean the new Job/Pod could be created while the current Job/Pod is still terminating. You could confirm with a Job that doesn't respect a <code>SIGTERM</code> and has a <code>terminationGracePeriodSeconds</code> set to a few times your clusters scheduling speed.</p>
