<p>I have a few cron jobs on GKE.</p>

<p>One of the pods did terminate and now I am trying to access the logs.</p>

<pre><code>➣ $ kubectl get events
LAST SEEN   TYPE      REASON             KIND      MESSAGE
23m         Normal    SuccessfulCreate   Job       Created pod: virulent-angelfish-cronjob-netsuite-proservices-15622200008gc42
22m         Normal    SuccessfulDelete   Job       Deleted pod: virulent-angelfish-cronjob-netsuite-proservices-15622200008gc42
22m         Warning   DeadlineExceeded   Job       Job was active longer than specified deadline
23m         Normal    Scheduled          Pod       Successfully assigned default/virulent-angelfish-cronjob-netsuite-proservices-15622200008gc42 to staging-cluster-default-pool-4b4827bf-rpnl
23m         Normal    Pulling            Pod       pulling image "gcr.io/my-repo/myimage:v8"
23m         Normal    Pulled             Pod       Successfully pulled image "gcr.io/my-repo/my-image:v8"
23m         Normal    Created            Pod       Created container
23m         Normal    Started            Pod       Started container
22m         Normal    Killing            Pod       Killing container with id docker://virulent-angelfish-cronjob:Need to kill Pod
23m         Normal    SuccessfulCreate   CronJob   Created job virulent-angelfish-cronjob-netsuite-proservices-1562220000
22m         Normal    SawCompletedJob    CronJob   Saw completed job: virulent-angelfish-cronjob-netsuite-proservices-1562220000
</code></pre>

<p>So at least one CJ run.</p>

<p>I would like to see the pod's logs, but there is nothing there</p>

<pre><code>➣ $ kubectl get pods
No resources found.
</code></pre>

<p>Given that in my cj definition, I have:</p>

<pre><code>failedJobsHistoryLimit: 1
successfulJobsHistoryLimit: 3
</code></pre>

<p>shouldn't <strong>at least</strong> one pod be there for me to do forensics?</p>

<p><strong>Your pod is crashing or otherwise unhealthy</strong></p>

<p>First, take a look at the logs of the current container:</p>

<pre><code>kubectl logs ${POD_NAME} ${CONTAINER_NAME}
</code></pre>

<p>If your container has previously crashed, you can access the previous container’s crash log with:</p>

<pre><code>kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}
</code></pre>

<p>Alternately, you can run commands inside that container with exec:</p>

<pre><code>kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- ${CMD} ${ARG1} ${ARG2} ... ${ARGN}
</code></pre>

<p>Note: <code>-c ${CONTAINER_NAME}</code> is optional. You can omit it for pods that only contain a single container.</p>

<p>As an example, to look at the logs from a running Cassandra pod, you might run:</p>

<pre><code>kubectl exec cassandra -- cat /var/log/cassandra/system.log
</code></pre>

<p>If none of these approaches work, you can find the host machine that the pod is running on and SSH into that host.</p>

<p>Finaly, check Logging on Google StackDriver.</p>

<p><strong>Debugging Pods</strong></p>

<p>The first step in debugging a pod is taking a look at it. Check the current state of the pod and recent events with the following command:</p>

<pre><code>kubectl describe pods ${POD_NAME}
</code></pre>

<p>Look at the state of the containers in the pod. Are they all Running? Have there been recent restarts?</p>

<p>Continue debugging depending on the state of the pods.</p>

<p><strong>Debugging ReplicationControllers</strong></p>

<p>ReplicationControllers are fairly straightforward. They can either create pods or they can’t. If they can’t create pods, then please refer to the instructions above to debug your pods.</p>

<p>You can also use <code>kubectl describe rc ${CONTROLLER_NAME}</code> to inspect events related to the replication controller.</p>

<p>Hope it helps you to find exactly problem.</p>
