<p>I have a Kubernetes Cronjob that runs on GKE and runs Cucumber JVM tests. In case a Step fails due to assertion failure, some resource being unavailable, etc., Cucumber rightly throws an exception which leads the Cronjob job to fail and the Kubernetes pod's status changes to <code>ERROR</code>. This leads to creation of a new pod that tries to run the same Cucumber tests again, which fails again and retries again.</p>

<p>I don't want any of these retries to happen. If a Cronjob job fails, I want it to remain in the failed status and not retry at all. Based on <a href="https://stackoverflow.com/questions/51657105/how-to-ensure-kubernetes-cronjob-does-not-restart-on-failure">this</a>, I have already tried setting <code>backoffLimit: 0</code> in combination with <code>restartPolicy: Never</code> in combination with <code>concurrencyPolicy: Forbid</code>, but it still retries by creating new pods and running the tests again. </p>

<p>What am I missing? Here's my kube manifest for the Cronjob:</p>

<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: quality-apatha
  namespace: default
  labels:
    app: quality-apatha
spec:
  schedule: "*/1 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          containers:
            - name: quality-apatha
              image: FOO-IMAGE-PATH
              imagePullPolicy: "Always"
              resources:
                limits:
                  cpu: 500m
                  memory: 512Mi
              env:
                - name: FOO
                  value: BAR
              volumeMounts:
                - name: FOO
                  mountPath: BAR
              args:
                - java
                - -cp
                - qe_java.job.jar:qe_java-1.0-SNAPSHOT-tests.jar
                - org.junit.runner.JUnitCore
                - com.liveramp.qe_java.RunCucumberTest
          restartPolicy: Never
          volumes:
            - name: FOO
              secret:
                secretName: BAR
</code></pre>

<p>Is there any other Kubernetes <code>Kind</code> I can use to stop the retrying?</p>

<p>Thank you!</p>

<p>To make things as simple as possible I tested it using <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#example" rel="noreferrer">this</a> example from the official kubernetes documentation, applying to it minor modifications to illustrate what really happens in different scenarios.</p>

<p>I can confirm that when <code>backoffLimit</code> is set to <code>0</code> and <code>restartPolicy</code> to <code>Never</code> <strong>everything works exactly as expected and there are no retries</strong>. Note that every single run of your <code>Job</code> which in your example is scheduled to run <strong>at intervals of 60 seconds</strong> (<code>schedule: "*/1 * * * *"</code>) <strong>IS NOT considerd a retry</strong>.</p>

<p>Let's take a closer look at the following example (base <code>yaml</code> avialable <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#example" rel="noreferrer">here</a>):</p>

<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - non-existing-command
          restartPolicy: Never
</code></pre>

<p>It spawns new cron job <code>every 60 seconds</code> according to the <code>schedule</code>, no matter if it fails or runs successfully. In this particular example it is configured to fail as we are trying to run <code>non-existing-command</code>.</p>

<p>You can check what's happening by running:</p>

<pre><code>$ kubectl get pods
NAME                     READY   STATUS              RESTARTS   AGE
hello-1587558720-pgqq9   0/1     Error               0          61s
hello-1587558780-gpzxl   0/1     ContainerCreating   0          1s
</code></pre>

<p>As you can see there are <strong>no retries</strong>. Although the first <code>Pod</code> failed, the new one is spawned exactly 60 seconds later according to our specification. I'd like to emphasize it again. <strong>This is not a retry.</strong></p>

<p>On the other hand when we modify the above example and set <code>backoffLimit: 3</code>, we can observe the <strong>retries</strong>. As you can see, now new <code>Pods</code> are created <strong>much more often than every 60 seconds</strong>. <strong>This are retries.</strong></p>

<pre><code>$ kubectl get pods
NAME                     READY   STATUS   RESTARTS   AGE
hello-1587565260-7db6j   0/1     Error    0          106s
hello-1587565260-tcqhv   0/1     Error    0          104s
hello-1587565260-vnbcl   0/1     Error    0          94s
hello-1587565320-7nc6z   0/1     Error    0          44s
hello-1587565320-l4p8r   0/1     Error    0          14s
hello-1587565320-mjnb6   0/1     Error    0          46s
hello-1587565320-wqbm2   0/1     Error    0          34s
</code></pre>

<p>What we can see above are <strong>3 retries</strong> (<code>Pod</code> creation attempts), related with <code>hello-1587565260</code> <strong>job</strong> and <strong>4 retries</strong> (including the orignal <strong>1st try</strong> not counted in <code>backoffLimit: 3</code>) related with <code>hello-1587565320</code> <strong>job</strong>.</p>

<p>As you can see the <strong>jobs</strong> themselves are still run according to the schedule, <strong>at 60 second intervals</strong>:</p>

<pre><code>kubectl get jobs
NAME               COMPLETIONS   DURATION   AGE
hello-1587565260   0/1           2m12s      2m12s
hello-1587565320   0/1           72s        72s
hello-1587565380   0/1           11s        11s
</code></pre>

<p>However due to our <code>backoffLimit</code> set this time to <code>3</code>, every time the <code>Pod</code> responsible for running the job fails, <strong>3 additional retries occur</strong>.</p>

<p>I hope this helped to dispel any possible confusions about running <code>cronJobs</code> in <strong>kubernetes</strong>.</p>

<p>If you are rather interested in running something just once, not at regular intervals, take a look at simple <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#running-an-example-job" rel="noreferrer">Job</a> instead of <code>CronJob</code>.</p>

<p>Also consider changing your <a href="https://en.wikipedia.org/wiki/Cron" rel="noreferrer">Cron</a> configuration if you still want to run this particular job on regular basis but let's say once in 24 h, not every minute.</p>
