<p>Not sure what I am doing wrong, but I am experiencing an issue where CronJobs stop scheduling new Jobs. It <em>seems</em> like this happens only after a couple of failures to launch a new Job. In my specific case, Jobs were not able to start due an inability to pull the container image.</p>

<p>I'm not really finding any settings that would lead to this, but I'm no expert on Kubernetes CronJobs. Configuration below:</p>

<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  labels:
    app.kubernetes.io/instance: cron-deal-report
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: cron
    helm.sh/chart: cron-0.1.0
  name: cron-deal-report
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        spec:
          containers:
          - args:
            - -c
            - npm run script
            command:
            - /bin/sh
            env:
            image: &lt;redacted&gt;
            imagePullPolicy: Always
            name: cron
            resources: {}
            securityContext:
              runAsUser: 1000
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: 0/15 * * * *
  successfulJobsHistoryLimit: 3
  suspend: false
status: {}
</code></pre>

Randy L
<h2>How kubernetes jobs handle failures</h2>

<p>As per <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#handling-pod-and-container-failures" rel="noreferrer">Jobs - Run to Completion - Handling Pod and Container Failures</a>:</p>

<blockquote>
  <p>An entire Pod can <s>also</s> fail, for a number of reasons, such as when the
  pod is kicked off the node (node is upgraded, rebooted, deleted,
  etc.), <strong>or if a container of the Pod fails and the
  <code>.spec.template.spec.restartPolicy = "Never"</code></strong>. When a Pod fails, then
  the Job controller starts a new Pod.</p>
</blockquote>

<p>You are using <code>restartPolicy: Never</code> for your <code>jobTemplate</code>, so, see the next quote on <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy" rel="noreferrer">Pod backoff failure policy</a>:</p>

<blockquote>
  <p>There are situations where you want to fail a Job after some amount of
  retries due to a logical error in configuration etc. To do so, set
  <code>.spec.backoffLimit</code> to specify the number of retries before considering
  a Job as failed. <strong>The back-off limit is set by default to 6</strong>. The back-off count is reset if no new failed Pods appear before the Job’s next status check.</p>
</blockquote>

<p>The <code>.spec.backoffLimit</code> is not defined in your <code>jobTemplate</code>, so it's using the default (<code>6</code>).</p>

<p>Following, as per <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#job-termination-and-cleanup" rel="noreferrer">Job Termination and Cleanup</a>:</p>

<blockquote>
  <p>By default, a Job will run uninterrupted unless a Pod fails, at which
  point the Job defers to the <code>.spec.backoffLimit</code> described above. Another way to terminate a Job is by setting an active deadline. Do this by setting the <code>.spec.activeDeadlineSeconds</code> field of the Job to a number of seconds. </p>
</blockquote>

<p>That's your case: <strong>If your containers fail to pull the image six consecutive times, your Job will be considered as failed.</strong></p>

<hr>

<h2>Cronjobs</h2>

<p>As per <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations" rel="noreferrer">Cron Job Limitations</a>:</p>

<blockquote>
  <p>A cron job creates a job object about once per execution time of its schedule [...]. The Cronjob is
  only responsible for creating Jobs that match its schedule, and the
  Job in turn is responsible for the management of the Pods it
  represents.</p>
</blockquote>

<p>This means that all pod/container failures should be handled by the Job Controller (i.e., adjusting the <code>jobTemplate</code>).</p>

<p><strong>"Retrying" a Job:</strong></p>

<p>You do not need to recreate a Cronjob in case its Job of fails. You only need to wait for the next schedule.</p>

<p>If you want to run a new Job before the next schedule, you can use the Cronjob template to create a Job manually with:</p>

<pre><code>kubectl create job --from=cronjob/my-cronjob-name my-manually-job-name
</code></pre>

<hr>

<h2>What you should do:</h2>

<p>If your containers are unable to download the images constantly, you have the following options:</p>

<ul>
<li>Explicit set and tune <code>backoffLimit</code> to a higher value.</li>
<li>Use <code>restartPolicy: OnFailure</code> for your containers, so the Pod will stay on the node, and only the container will be re-run.</li>
<li><em>Consider using <a href="https://kubernetes.io/docs/concepts/containers/images/#updating-images" rel="noreferrer"><code>imagePullPolicy: IfNotPresent</code></a>. If you are not retagging your images, there is no need to force a re-pull for every job start.</em></li>
</ul>
