<p>We have a Kubernetes cluster of web scraping cron jobs set up. All seems to go well until a cron job starts to fail (e.g., when a site structure changes and our scraper no longer works). It looks like every now and then a few failing cron jobs will continue to retry to the point it brings down our cluster. Running <code>kubectl get cronjobs</code> (prior to a cluster failure) will show too many jobs running for a failing job.</p>

<p>I've attempted following the note described <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy" rel="noreferrer">here</a> regarding a known issue with the pod backoff failure policy; however, that does not seem to work.</p>

<p>Here is our config for reference:</p>

<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: scrape-al
spec:
  schedule: '*/15 * * * *'
  concurrencyPolicy: Allow
  failedJobsHistoryLimit: 0
  successfulJobsHistoryLimit: 0
  jobTemplate:
    metadata:
      labels:
        app: scrape
        scrape: al
    spec:
      template:
        spec:
          containers:
            - name: scrape-al
              image: 'govhawk/openstates:1.3.1-beta'
              command:
                - /opt/openstates/openstates/pupa-scrape.sh
              args:
                - al bills --scrape
          restartPolicy: Never
      backoffLimit: 3
</code></pre>

<p>Ideally we would prefer that a cron job would be terminated after N retries (e.g., something like <code>kubectl delete cronjob my-cron-job</code> after <code>my-cron-job</code> has failed 5 times). Any ideas or suggestions would be much appreciated. Thanks!</p>

<p>You can tell your Job to stop retrying using <code>backoffLimit</code>.</p>

<blockquote>
  <p>Specifies the number of retries before marking this job failed.</p>
</blockquote>

<p>In your case</p>

<pre><code>spec:
  template:
    spec:
      containers:
        - name: scrape-al
          image: 'govhawk/openstates:1.3.1-beta'
          command:
            - /opt/openstates/openstates/pupa-scrape.sh
          args:
            - al bills --scrape
      restartPolicy: Never
  backoffLimit: 3
</code></pre>

<p>You set 3 as<code>backoffLimit</code> of your Job. That means when a Job is created by CronJob, It will retry 3 times if fails. This controls Job, not CronJob</p>

<p>When Job is failed, another Job will be created again as your scheduled period. </p>

<p><strong>You want:</strong>
If I am not wrong, <em>you want to stop scheduling new Job, when your scheduled Jobs are failed for 5 times</em>. Right?</p>

<p><strong>Answer:</strong> 
In that case, <em>this is not possible automatically</em>. </p>

<p><strong>Possible solution:</strong>
You need to <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#suspend" rel="noreferrer">suspend</a> CronJob so than it stop scheduling new Job.</p>

<pre><code>Suspend: true
</code></pre>

<p>You can do this manually. If you do not want to do this manually, you need to setup a watcher, that will watch your CronJob status, and will update CronJob to suspend if necessary.</p>
