<p>I have updated a running cluster with a new image which unfortunately it's crashing. I want to log into the pod to look at logs. What is the way to do so?</p>
<pre class="lang-sh prettyprint-override"><code>manuchadha25@cloudshell:~ (copper-frame-262317)$ kubectl get pods
NAME                             READY   STATUS             RESTARTS   AGE
codingjediweb-7c45484669-czcpk   0/1     CrashLoopBackOff   6          9m34s
codingjediweb-7c45484669-qn4m5   0/1     CrashLoopBackOff   6          9m32s
</code></pre>
<p>The application does not generate much console logs. The main logs are in a file. How can I access that file?</p>
<pre class="lang-sh prettyprint-override"><code>manuchadha25@cloudshell:~ (copper-frame-262317)$ kubectl logs codingjediweb-7c45484669-czcpk
Oops, cannot start the server.
play.api.libs.json.JsResult$Exception: {&quot;obj&quot;:[{&quot;msg&quot;:[&quot;Unable to connect with database&quot;],&quot;args&quot;:[]}]}
manuchadha25@cloudshell:~ (copper-frame-262317)$ kubectl logs codingjediweb-7c45484669-qn4m5
Oops, cannot start the server.
play.api.libs.json.JsResult$Exception: {&quot;obj&quot;:[{&quot;msg&quot;:[&quot;Unable to connect with database&quot;],&quot;args&quot;:[]}]}
</code></pre>
<p>UPDATE
I tried to implement Christoph's suggestion of using two containers in a pod - one for main application and the other for logging. I switched back to the stable version of my application to be sure that the application is up/running and is generating logs. This would help test that the pattern works. It looks that the logging application keeps existing/crashing.</p>
<p>yaml file</p>
<pre><code>manuchadha25@cloudshell:~ (copper-frame-262317)$ cat codingjediweb-nodes.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: codingjediweb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: codingjediweb
  template:
    metadata:
      labels:
        app: codingjediweb
    spec:
      volumes:
      - name: shared-logs
        emptyDir: {}
      containers:
      - name: codingjediweb
        image: docker.io/manuchadha25/codingjediweb:03072020v2
        volumeMounts:
        - name: shared-logs
          mountPath: /deploy/codingjediweb-1.0/logs/
        env:
        - name: db.cassandraUri
          value: cassandra://xx.yy.xxx.238:9042
        - name: db.password
          value: 9__something
        - name: db.keyspaceName
          value: something2
        - name: db.username
          value: superawesomeuser
        ports:
        - containerPort: 9000
      - name: logging
        image: busybox
        volumeMounts:
        - name: shared-logs
          mountPath: /deploy/codingjediweb-1.0/logs/
        command: [&quot;tail -f /deploy/codingjediweb-1.0/logs/*.log&quot;]
</code></pre>
<p>When I apply the configuration then only one container stays up</p>
<pre><code>manuchadha25@cloudshell:~ (copper-frame-262317)$ kubectl get pods
NAME                             READY   STATUS             RESTARTS   AGE
busybox                          1/1     Running            1          10h
codingjediweb-857c6d584b-n4njp   1/2     CrashLoopBackOff   6          8m46s
codingjediweb-857c6d584b-s2hg2   1/2     CrashLoopBackOff   6          8m46s
</code></pre>
<p>further inspection shows that the main application is up</p>
<pre><code>manuchadha25@cloudshell:~ (copper-frame-262317)$ kubectl exec -it codingjediweb-857c6d584b-s2hg2 -c logging -- bash
error: unable to upgrade connection: container not found (&quot;logging&quot;)
manuchadha25@cloudshell:~ (copper-frame-262317)$ kubectl exec -it codingjediweb-857c6d584b-s2hg2 -c codingjediweb -- bash
</code></pre>
<p>And the application is generating logs at the right path</p>
<pre><code>root@codingjediweb-857c6d584b-s2hg2:/deploy# tail -f /deploy/codingjediweb-1.0/logs/*.log
2020-07-07 06:40:37,385 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/34.91.191.238:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2020-07-07 06:40:37,389 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/34.91.191.238:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2020-07-07 06:41:07,208 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/34.91.191.238:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2020-07-07 06:41:07,210 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/34.91.191.238:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2020-07-07 06:41:07,271 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/10.44.1.4:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2020-07-07 06:41:07,274 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/10.44.1.4:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2020-07-07 06:41:07,332 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/10.44.2.5:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2020-07-07 06:41:07,337 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/10.44.2.5:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2020-07-07 06:41:07,392 [DEBUG] from com.datastax.driver.core.Connection in codingJediCluster-nio-worker-0 - Connection[/34.91.191.238:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
</code></pre>

<p>You can create a second container in your pod for logging. Just mount the log directory in both containers. The second container can be based on busybox image and do a <code>tail -f xyz.log</code>.</p>
<pre><code>volumes:
  - name: shared-logs
    emptyDir: {}
  containers:
  - name: main-app
    image: &lt;your-image&gt;
    volumeMounts:
    - name: shared-logs
      mountPath: &lt;path/to/logs&gt;
  - name: logging
    image: busybox
    volumeMounts:
    - name: shared-logs
      mountPath: &lt;path/to/logs&gt;
    command: [&quot;tail -f &lt;path/to/logs/*.log&gt;&quot;]
</code></pre>
<p>Now you can view logs with</p>
<pre><code>Kubectl logs &lt;podname&gt; -c logging -f
</code></pre>
<p><strong>Update</strong></p>
<p>Since your log file name is dynamic, you can use the busybox container with an infinite sleep comand, something like <code>while true; do sleep 86400; done </code> should to. Now the logging container does nothing, but still mounts your log file. The other container will crash but the logging container should be running. You can than ssh into it:</p>
<pre><code>kubectl exec -it &lt;pod name&gt; -c logging -- sh
</code></pre>
<p>Or another idea:</p>
<p>You can copy the log directory of the pod to your local system and investigate the logs.</p>
<pre><code>kubectl cp &lt;some-pod&gt;:/path/to/logs -c logging /path/to/local/dir 
</code></pre>
