<p>I'm working on a Kubernetes cron job which represents an integration test; it is Go test binary that is compiled with <code>go test -c</code> and copied into a Docker container run by the cron job. The Kubernetes YAML is starts similar to the following:</p>

<pre><code>apiVersion: batch/v1beta1
kind: CronJob
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 7
  failedJobsHistoryLimit: 7
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
</code></pre>

<p>At some point, the integration test starting failing (exiting with code 1). I can see that the job has the same duration as its age:</p>

<pre><code>$ kubectl get jobs -l app=integration-test
NAME                          COMPLETIONS   DURATION   AGE
integration-test-1592457300   0/1           7m20s      7m20s
</code></pre>

<p>The <code>kubectl get pods</code> commands shows that pods being created more frequently than every 15 minutes as I would expect from the cron schedule:</p>

<pre><code>$ kubectl get pods -l app=integration-test
NAME                                READY   STATUS   RESTARTS   AGE
integration-test-1592457300-224x8   0/1     Error    0          92s
integration-test-1592457300-5f8sz   0/1     Error    0          7m33s
integration-test-1592457300-9zvjq   0/1     Error    0          3m57s
integration-test-1592457300-th7sf   0/1     Error    0          6m26s
integration-test-1592457300-vhbr2   0/1     Error    0          5m17s
</code></pre>

<p>This behavior of spinning up new pods is problematic because it contributes to the running pod count on the node - essentially, it consumes resources.</p>

<p>How can I make it such that the cron job doesn't keep spinning up new pods, but only does one every 15 minutes, and doesn't continue to consume resources if the job fails?</p>

<h3>Update</h3>

<p>A simplified example of this uses a Kubernetes YAML adapted from <a href="https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/</a>:</p>

<pre><code>$ cat cronjob.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster; exit 1
          restartPolicy: Never
</code></pre>

<p>Note that it exits with code 1. If I run this using <code>kubernetes apply -f cronjob.yaml</code> and then check the pods, I see</p>

<pre><code>$ kubectl get pods
NAME                                                    READY   STATUS      RESTARTS   AGE
hello-1592459760-fnvcw                                  0/1     Error       0          30s
hello-1592459760-w75lt                                  0/1     Error       0          31s
hello-1592459760-xzhwn                                  0/1     Error       0          20s
</code></pre>

<p>The ages of the pods are less than a minute apart; in other words, pods are spun up before the cron interval has elapsed. How can I prevent this?</p>

Kurt Peek
<p>It's quite specific scenario and it's hard to guess what you want to achievie and if it will work for you.</p>

<p><a href="https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#concurrency-policy" rel="noreferrer">concurrencyPolicy: Forbid</a> prevents to create another <code>job</code> if previous was not <code>completed</code>. But I think that is not the case here.</p>

<p><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy" rel="noreferrer">restartPolicy</a> applies to <code>pod</code> (however in <code>Job template</code> you can use only <code>OnFailure</code> and <code>Never</code>). If you will set <code>restartPolicy</code> to <code>Never</code>, <code>job</code> will create automatically new <code>pods</code> till completion.</p>

<blockquote>
  <p>A Job creates one or more Pods and ensures that a specified number of them successfully terminate. As pods successfully complete, the Job tracks the successful completions. </p>
</blockquote>

<p>If you set <code>restartPolicy: Never</code> it will be creating pods till it will reach <a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy" rel="noreferrer">backoffLimit</a>, however those <code>pods</code> will be still visible in your cluster with <code>Error</code> status as each pod exit with <code>status 1</code>. You would need to remove it manually.
If you would set <code>restartPolicy: OnFailure</code> it will be restarting one <code>pod</code> and will not create more.</p>

<p>But there is another way. What is considered as <code>completed</code> job?</p>

<p><strong>Examples:</strong></p>

<p><strong>1. <code>restartPolicy: OnFailure</code></strong></p>

<pre><code>$ kubectl get po,jobs,cronjob
NAME                         READY   STATUS             RESTARTS   AGE
pod/hello-1592495280-w27mt   0/1     CrashLoopBackOff   5          5m21s
pod/hello-1592495340-tzc64   0/1     CrashLoopBackOff   5          4m21s
pod/hello-1592495400-w8cm6   0/1     CrashLoopBackOff   5          3m21s
pod/hello-1592495460-jjlx5   0/1     CrashLoopBackOff   4          2m21s
pod/hello-1592495520-c59tm   0/1     CrashLoopBackOff   3          80s
pod/hello-1592495580-rrdzw   0/1     Error              2          20s
NAME                         COMPLETIONS   DURATION   AGE
job.batch/hello-1592495220   0/1           6m22s      6m22s
job.batch/hello-1592495280   0/1           5m22s      5m22s
job.batch/hello-1592495340   0/1           4m22s      4m22s
job.batch/hello-1592495400   0/1           3m22s      3m22s
job.batch/hello-1592495460   0/1           2m22s      2m22s
job.batch/hello-1592495520   0/1           81s        81s
job.batch/hello-1592495580   0/1           21s        21s
NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/hello   */1 * * * *   False     6        25s             15m
</code></pre>

<p>Each <code>job</code> will create only 1 <code>pod</code> which will be restarted till <code>job</code> will be <code>finished</code> or will be considered as <code>completed</code> by <code>CronJob</code>.</p>

<p>If you will describe <code>CronJob</code> in <code>Event</code> section you can find.</p>

<pre><code>Events:
  Type    Reason            Age                  From                Message
  ----    ------            ----                 ----                -------
  Normal  SuccessfulCreate  18m                  cronjob-controller  Created job hello-1592494740
  Normal  SuccessfulCreate  17m                  cronjob-controller  Created job hello-1592494800
  Normal  SuccessfulCreate  16m                  cronjob-controller  Created job hello-1592494860
  Normal  SuccessfulCreate  15m                  cronjob-controller  Created job hello-1592494920
  Normal  SuccessfulCreate  14m                  cronjob-controller  Created job hello-1592494980
  Normal  SuccessfulCreate  13m                  cronjob-controller  Created job hello-1592495040
  Normal  SawCompletedJob   12m                  cronjob-controller  Saw completed job: hello-1592494740
  Normal  SuccessfulCreate  12m                  cronjob-controller  Created job hello-1592495100
  Normal  SawCompletedJob   11m                  cronjob-controller  Saw completed job: hello-1592494800
  Normal  SuccessfulDelete  11m                  cronjob-controller  Deleted job hello-1592494740
  Normal  SuccessfulCreate  11m                  cronjob-controller  Created job hello-1592495160
  Normal  SawCompletedJob   10m                  cronjob-controller  Saw completed job: hello-1592494860
</code></pre>

<p>Why job <code>hello-1592494740</code> was considered as <code>Completed</code>? <code>Cronjob</code> default value of <code>.spec.backoffLimit</code> is 6 (this information can be found in docs). If <code>job</code> will fail 6 times (pod will fail to restart 6 times) <code>Cronjob</code> will consider this <code>job</code> as <code>Completed</code> and it will remove it. As <code>job</code> was removed, also <code>pod</code> will be removed.</p>

<p>However, in your example, <code>pod</code> was created, pod executed date and echo command and then exit with code 1. Even if <code>pod</code> is Crashing it wrote information. As last command was <code>exit 1</code> so it will be Crashing till it reach limit. As per example below:</p>

<pre><code>$ kubectl get pods
NAME                     READY   STATUS             RESTARTS   AGE
hello-1592495400-w8cm6   0/1     Terminating        6          5m51s
hello-1592495460-jjlx5   0/1     CrashLoopBackOff   5          4m51s
hello-1592495520-c59tm   0/1     CrashLoopBackOff   5          3m50s
hello-1592495580-rrdzw   0/1     CrashLoopBackOff   4          2m50s
hello-1592495640-nbq59   0/1     CrashLoopBackOff   4          110s
hello-1592495700-p6pcx   0/1     Error              3          50s
user@cloudshell:~ (project)$ kubectl logs hello-1592495520-c59tm
Thu Jun 18 15:55:13 UTC 2020
Hello from the Kubernetes cluster
</code></pre>

<p><strong>2. <code>restartPolicy: Never</code> and <code>backoffLimit: 0</code></strong></p>

<p>YAML below was used:</p>

<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster; exit 1
          restartPolicy: Never
      backoffLimit: 0
</code></pre>

<p><strong>Output</strong></p>

<pre><code>$ kubectl get po,jobs,cronjob
NAME                         READY   STATUS   RESTARTS   AGE
pod/hello-1592497320-svd6k   0/1     Error    0          44s
NAME                         COMPLETIONS   DURATION   AGE
job.batch/hello-1592497320   0/1           44s        44s
NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/hello   */1 * * * *   False     0        51s             11m

$ kubectl describe cronjob
...
Events:
  Type    Reason            Age                  From                Message
  ----    ------            ----                 ----                -------
  Normal  SuccessfulCreate  12m                  cronjob-controller  Created job hello-1592496720
  Normal  SawCompletedJob   11m                  cronjob-controller  Saw completed job: hello-1592496720
  Normal  SuccessfulCreate  11m                  cronjob-controller  Created job hello-1592496780
  Normal  SawCompletedJob   10m                  cronjob-controller  Saw completed job: hello-1592496780
  Normal  SuccessfulDelete  10m                  cronjob-controller  Deleted job hello-1592496720
  Normal  SuccessfulCreate  10m                  cronjob-controller  Created job hello-1592496840
  Normal  SuccessfulDelete  9m55s                cronjob-controller  Deleted job hello-1592496780
  Normal  SawCompletedJob   9m55s                cronjob-controller  Saw completed job: hello-1592496840
  Normal  SuccessfulCreate  9m5s                 cronjob-controller  Created job hello-1592496900
  Normal  SawCompletedJob   8m55s                cronjob-controller  Saw completed job: hello-1592496900
  Normal  SuccessfulDelete  8m55s                cronjob-controller  Deleted job hello-1592496840
  Normal  SuccessfulCreate  8m5s                 cronjob-controller  Created job hello-1592496960
  Normal  SawCompletedJob   7m55s                cronjob-controller  Saw completed job: hello-1592496960
  Normal  SuccessfulDelete  7m55s                cronjob-controller  Deleted job hello-1592496900
  Normal  SuccessfulCreate  7m4s                 cronjob-controller  Created job hello-1592497020
</code></pre>

<p>This way only one <code>job</code> and one <code>pod</code> will be at the same time running (there might be 10 seconds gap when there will be 2 jobs and 2 pods).</p>

<pre><code>$ kubectl get po,job
NAME                         READY   STATUS   RESTARTS   AGE
pod/hello-1592497440-twzlf   0/1     Error    0          70s
pod/hello-1592497500-2q7fq   0/1     Error    0          10s

NAME                         COMPLETIONS   DURATION   AGE
job.batch/hello-1592497440   0/1           70s        70s
job.batch/hello-1592497500   0/1           10s        10s
user@cloudshell:~ (project)$ kk get po,job
NAME                         READY   STATUS   RESTARTS   AGE
pod/hello-1592497500-2q7fq   0/1     Error    0          11s

NAME                         COMPLETIONS   DURATION   AGE
job.batch/hello-1592497500   0/1           11s        11s
</code></pre>

<p>I hope it cleared a bit. If you would like more precise answer, please give more information about your scenario.</p>
