<p>I have a pod in a state of <code>CrashLoopBackOff</code>, the logs I'm seeing from <code>kubectl logs &lt;pod-name&gt; -p</code> present only a partial picutre. Other logs are found in other files (e.g. <code>/var/log/something/something.log</code>).</p>

<p>Since this pod is crashed, I can't <code>kubectl exec</code> into a shell there and look at the files.</p>

<p>How can I look at the log files produced by a container that is no longer running?</p>

<p>To be more specific, I'm looking for log files file under <code>$HOME/logs/es.log</code> (in the container that failed)</p>

<p>I was so frustrated from finding no solution to this seemingly common problem that I built a docker image that tails log files and sends them to stdout, to be used as a sidecar container.</p>

<hr>

<p>Here's what I did:</p>

<ol>
<li>I added a volume with <code>emptyDir{}</code> to the pod</li>
<li>I mounted that volume to my main container, with the <code>mountPath</code> being the directory to which it writes the logs to</li>
<li>I added another container to the pod, called "logger", with the image being the log tracker I wrote (<code>lutraman/logger-sidecar:v2</code>), and mounted the same volume to <code>/logs</code> (I programmed the script to read the logs from this directory)</li>
</ol>

<p>then, all the logs written to that directory, can be accessed by <code>kubectl logs &lt;pod-name&gt; -c logger</code></p>

<hr>

<p>Here is an example yaml:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy
  labels:
    app: dummy
spec:
  selector:
    matchLabels:
      app: dummy
  template:
    metadata:
      labels:
        app: dummy
    spec:
      volumes:
        - name: logs
          emptyDir: {}
      containers:
        - name: dummy-app # the app that writes logs to files
          image: lutraman/dummy:v2
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          env:
            - name: MESSAGE
              value: 'hello-test'
            - name: LOG_FILE
              value: '/var/log/app.log'
          volumeMounts:
            - name: logs
              mountPath: /var/log
        - name: logger # the sidecar container tracking logs and sending them to stdout
          image: lutraman/logger-sidecar:v2
          volumeMounts:
            - name: logs
              mountPath: /logs
</code></pre>

<hr>

<p>For anyone who is interested, here is how I made the sidecar container:</p>

<p>Dockerfile:</p>

<pre><code>FROM alpine:3.9

RUN apk add bash --no-cache

COPY addTail /addTail
COPY logtrack.sh /logtrack.sh

CMD ["./logtrack.sh"]

</code></pre>

<p>addTail:</p>

<pre class="lang-sh prettyprint-override"><code>#!/bin/sh

(exec tail -F logs/$3 | sed "s/^/$3: /" ) &amp;
echo $! &gt;&gt; /tmp/pids
</code></pre>

<p>logtrack.sh:</p>

<pre class="lang-sh prettyprint-override"><code>#!/bin/bash

trap cleanup INT

function cleanup() {
  while read pid; do kill $pid; echo killed $pid; done &lt; /tmp/pids
}

: &gt; /tmp/pids

for log in $(ls logs); do
  ./addTail n logs $log
done

inotifyd ./addTail `pwd`/logs:n 
</code></pre>
