<p>I’ve created a <code>Cronjob</code> in kubernetes with schedule(<code>8 * * * *</code>), with job’s <code>backoffLimit</code> defaulting to 6 and pod’s <code>RestartPolicy</code> to <code>Never</code>, the pods are deliberately configured to FAIL. As I understand, (for podSpec with <code>restartPolicy : Never</code>) Job controller will try to create <code>backoffLimit</code> number of pods and then it marks the job as <code>Failed</code>, so, I expected that there would be 6 pods in <code>Error</code> state.</p>

<p>This is the actual Job’s status:</p>

<pre><code>status:
  conditions:
  - lastProbeTime: 2019-02-20T05:11:58Z
    lastTransitionTime: 2019-02-20T05:11:58Z
    message: Job has reached the specified backoff limit
    reason: BackoffLimitExceeded
    status: "True"
    type: Failed
  failed: 5
</code></pre>

<p>Why were there only 5 failed pods instead of 6? Or is my understanding about <code>backoffLimit</code> in-correct?</p>

goutham
<p>In short: You might not be seeing all created pods because period of schedule in the cronjob is too short.</p>
<p>As described in <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy" rel="noreferrer">documentation</a>:</p>
<blockquote>
<p>Failed Pods associated with the Job are recreated by the Job
controller with an exponential back-off delay (10s, 20s, 40s …) capped
at six minutes. The back-off count is reset if no new failed Pods
appear before the Job’s next status check.</p>
</blockquote>
<p>If new job is scheduled before Job controller has a chance to recreate a pod (having in mind the delay after previous failure), Job controller starts counting from one again.</p>
<p>I reproduced your issue in GKE using following <code>.yaml</code>:</p>
<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hellocron
spec:
  schedule: &quot;*/3 * * * *&quot; #Runs every 3 minutes
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hellocron
            image: busybox
            args:
            - /bin/cat
            - /etc/os
          restartPolicy: Never
      backoffLimit: 6
  suspend: false
</code></pre>
<p>This job will fail because file <code>/etc/os</code> doesn't exist.</p>
<p>And here is an output of <code>kubectl describe</code> for one of the jobs:</p>
<pre><code>Name:           hellocron-1551194280
Namespace:      default
Selector:       controller-uid=b81cdfb8-39d9-11e9-9eb7-42010a9c00d0
Labels:         controller-uid=b81cdfb8-39d9-11e9-9eb7-42010a9c00d0
                job-name=hellocron-1551194280
Annotations:    &lt;none&gt;
Controlled By:  CronJob/hellocron
Parallelism:    1
Completions:    1
Start Time:     Tue, 26 Feb 2019 16:18:07 +0100
Pods Statuses:  0 Running / 0 Succeeded / 6 Failed
Pod Template:
  Labels:  controller-uid=b81cdfb8-39d9-11e9-9eb7-42010a9c00d0
           job-name=hellocron-1551194280
  Containers:
   hellocron:
    Image:      busybox
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Args:
      /bin/cat
      /etc/os
    Environment:  &lt;none&gt;
    Mounts:       &lt;none&gt;
  Volumes:        &lt;none&gt;
Events:
  Type     Reason                Age   From            Message
  ----     ------                ----  ----            -------
  Normal   SuccessfulCreate      26m   job-controller  Created pod: hellocron-1551194280-4lf6h
  Normal   SuccessfulCreate      26m   job-controller  Created pod: hellocron-1551194280-85khk
  Normal   SuccessfulCreate      26m   job-controller  Created pod: hellocron-1551194280-wrktb
  Normal   SuccessfulCreate      26m   job-controller  Created pod: hellocron-1551194280-6942s
  Normal   SuccessfulCreate      25m   job-controller  Created pod: hellocron-1551194280-662zv
  Normal   SuccessfulCreate      22m   job-controller  Created pod: hellocron-1551194280-6c6rh
  Warning  BackoffLimitExceeded  17m   job-controller  Job has reached the specified backoff limit
</code></pre>
<p>Note the delay between creation of pods <code>hellocron-1551194280-662zv</code> and <code>hellocron-1551194280-6c6rh</code>.</p>
