<p>I found a failed cronjob in my Kubernetes cluster running on Google GKE. I only have information about the failure state and failure reason of the job.</p>
<p>Although I located the failed job and learned that it reached the backoff limit, I would like to access the actual logs or stdout of the pod to identify the code error that caused the failure.</p>
<p>However, when I tried to use a command to check the log, I discovered that the pod had already been deleted, and I could not locate the failed pod.</p>
<p>But from GKE UI, I can found there are logs showing errors in my code and the information is what I am looking for.</p>
<p>I am wondering whether it is GKE implementing a logging system so I can find the logs?</p>
<p>Because I found the following information in the doc:</p>
<blockquote>
<p>Note: If your job has restartPolicy = &quot;OnFailure&quot;, keep in mind that your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = &quot;Never&quot; when debugging the Job or using a logging system to ensure output from failed Jobs is not lost inadvertently.</p>
</blockquote>
<p>Does this mean that if my job's restartPolicy is set to OnFailure, the corresponding pod will be deleted upon failure, and there will be no way to check the pod's logs?</p>
<p>Did I find logs in GKE because it has its own logging system, or is there another action I can take to access these logs?</p>
<p>Thanks.</p>
<p>Following is the snippet of my cronjob.</p>
<pre><code>spec:
  concurrencyPolicy: Allow
  failedJobsHistoryLimit: 3
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        metadata:
          creationTimestamp: null
        spec:
          containers:
          - command:
            - $cmd
            image: $img
            imagePullPolicy: Always
            name: $job_name
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: OnFailure
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: 0 4 * * *
  successfulJobsHistoryLimit: 3
  suspend: false
</code></pre>

Wilson.Wang
<p>CronJobs have a history value:</p>
<p><code>.spec.successfulJobsHistoryLimit</code> and <code>.spec.failedJobsHistoryLimit</code></p>
<p>Set these to how many successful/failed jobs you wish to keep, and Kubernetes will keep that many failed or successful pods. I generally keep 3 failed, 1 successful, but if debugging, I'll keep more (6 failed, 3 successful)</p>
<hr />
<p>Edited version of your yaml</p>
<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  concurrencyPolicy: Allow
  failedJobsHistoryLimit: 3
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        metadata:
          creationTimestamp: null
        spec:
          containers:
            - image: busybox
              # command:
              # - $cmd
              command:
                - &quot;sh&quot;
                - &quot;-c&quot;
                - &quot;exit 1&quot;
              imagePullPolicy: Always
              name: crontest
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: OnFailure
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: 0 4 * * *
  successfulJobsHistoryLimit: 3
  suspend: false
</code></pre>
<p>I was able to reproduce your issue with this. I saw this message in logs:</p>
<pre><code>Events:                                                                                                                                                                                    │
│   Type     Reason                Age    From            Message                                                                                                                            │
│   ----     ------                ----   ----            -------                                                                                                                            │
│   Normal   SuccessfulCreate      9m26s  job-controller  Created pod: my-app-manual-xnj-fpvzq                                                                                               │
│   Normal   SuccessfulDelete      3m18s  job-controller  Deleted pod: my-app-manual-xnj-fpvzq                                                                                               │
│   Warning  BackoffLimitExceeded  3m18s  job-controller  Job has reached the specified backoff limit                                                                                        │
│ 
</code></pre>
<p>Change your yaml to change the <code>restartPolicy</code>:</p>
<pre><code>apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  concurrencyPolicy: Allow
  failedJobsHistoryLimit: 3
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        metadata:
          creationTimestamp: null
        spec:
          containers:
            - image: busybox
              # command:
              # - $cmd
              command:
                - &quot;sh&quot;
                - &quot;-c&quot;
                - &quot;exit 1&quot;
              imagePullPolicy: Always
              name: crontest
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: Never ### &lt;--- this is changed
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: 0 4 * * *
  successfulJobsHistoryLimit: 3
  suspend: false

</code></pre>
<p>Then produces this:</p>
<pre><code>$ kubectl get pods
NAME                      READY   STATUS   RESTARTS   AGE
my-app-manual-29l-27x8v   0/1     Error    0          2m9s
my-app-manual-29l-64th2   0/1     Error    0          110s
my-app-manual-29l-shfk7   0/1     Error    0          2m4s
my-app-manual-29l-tj9k7   0/1     Error    0          84s
my-app-manual-29l-tzlwd   0/1     Error    0          40s
</code></pre>
<p>So in short, yes, the <code>restartPolicy</code> is the reason you're not seeing the failed pods.</p>
