<p>I run three services in three different containers. The logs for these services are sent to the system so if I run these on a Linux server, I can see the logs with journalctl.</p>
<p>Also, if I run the services in Docker containers, I can gather the logs with docker logs &lt;container_name&gt; or from /var/lib/docker/containers directory. But when I move to Kubernetes (Microk8s), I cannot retrieve them with kubectl logs command, and there are also no logs in /var/log/containers or /var/log/pods.</p>
<p>If I login to the pods, I can see that the processes are running, but without logs I couldn't say if there are running correctly. Also, I tried to change the runtime of microk8s kubelet from containerd to docker, but still I can't get any  logs.</p>
<pre><code># kubectl get po -o wide
NAME                              READY   STATUS        RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
amf-deployment-7785db9758-h24kz   1/1     Running       0          72s   10.1.243.237   ubuntu   &lt;none&gt;

# kubectl describe po amf-deployment-7785db9758-h24kz
Events:
Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
Normal  Scheduled       87s   default-scheduler  Successfully assigned default/amf-deployment-7785db9758-h24kz to ubuntu
Normal  AddedInterface  86s   multus             Add eth0 [10.1.243.237/32]
Normal  Pulled          86s   kubelet            Container image &quot;amf:latest&quot; already present on machine
Normal  Created         86s   kubelet            Created container amf
Normal  Started         86s   kubelet            Started container amf

# kubectl logs amf-deployment-7785db9758-h24kz
# kubectl logs -f amf-deployment-7785db9758-h24kz
^C
</code></pre>
<p>You can see in the following screenshot the difference of running the same container with Docker and running it with Kubernetes. The behaviour seems very strange, since the logs can be gathered if the application run as an independent Docker container, but not when it is running with Kubernetes.<a href="https://i.stack.imgur.com/mdRHo.png" rel="nofollow noreferrer">enter image description here</a></p>

Robert Botez
<p>In traditional server environments, application logs are written to a file such as <code>/var/log/app.log</code>. However, when working with Kubernetes, you need to collect logs for multiple transient pods (applications), across multiple nodes in the cluster, making this log collection method less than optimal. Instead, the default Kubernetes logging framework recommends capturing the standard output (<code>stdout</code>) and standard error output (<code>stderr</code>) from each container on the node to a log file. If you can't see you apps logs when using <code>kubectl logs</code> command it most likely means that your app is not writing logs in the right place. The official <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#basic-logging-in-kubernetes" rel="nofollow noreferrer">Logging Architecture</a> docs explain this topic in more detail. There is also an example of <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#basic-logging-in-kubernetes" rel="nofollow noreferrer">Basic logging in Kubernetes</a>:</p>
<blockquote>
<p>This example uses a Pod specification with a container to write text
to the standard output stream once per second.</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while true; do echo &quot;$i: $(date)&quot;; i=$((i+1)); sleep 1; done']
</code></pre>
<p>To run this pod, use the following command:</p>
<pre><code>kubectl apply -f https://k8s.io/examples/debug/counter-pod.yaml
</code></pre>
<p>The output is:</p>
<pre><code>pod/counter created
</code></pre>
<p>To fetch the logs, use the kubectl logs command, as follows:</p>
<pre><code>kubectl logs counter
</code></pre>
<p>The output is:</p>
<pre><code>0: Mon Jan  1 00:00:00 UTC 2001
1: Mon Jan  1 00:00:01 UTC 2001
2: Mon Jan  1 00:00:02 UTC 2001
...
</code></pre>
<p>You can use <code>kubectl logs --previous</code> to retrieve logs from a previous
instantiation of a container. If your pod has multiple containers,
specify which container's logs you want to access by appending a
container name to the command. See the <a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs" rel="nofollow noreferrer">kubectl logs documentation</a>
for more details.</p>
</blockquote>
<p>You can compare it with your <code>Pod</code>/app configs to see if there are any mistakes.</p>
<hr />
<p>Having that knowledge in mind you now have several option to <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/" rel="nofollow noreferrer">Debug Running Pods</a> such as:</p>
<ul>
<li><p><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/#debugging-pods" rel="nofollow noreferrer">Debugging Pods</a> by executing <code>kubectl describe pods ${POD_NAME}</code> and checking the reason behind it's failure.</p>
</li>
<li><p><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#examine-pod-logs" rel="nofollow noreferrer">Examining pod logs</a>: with <code>kubectl logs ${POD_NAME} ${CONTAINER_NAME}</code> or <code>kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}</code></p>
</li>
<li><p><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#container-exec" rel="nofollow noreferrer">Debugging with container exec</a>: by running commands inside a specific container with <code>kubectl exec</code></p>
</li>
<li><p><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#ephemeral-container" rel="nofollow noreferrer">Debugging with an ephemeral debug container</a>: Ephemeral containers are useful for interactive troubleshooting when <code>kubectl exec</code> is insufficient because a container has crashed or a container image doesn't include debugging utilities, such as with <a href="https://github.com/GoogleContainerTools/distroless" rel="nofollow noreferrer">distroless images</a>.</p>
</li>
<li><p><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#node-shell-session" rel="nofollow noreferrer">Debugging via a shell on the node</a>: If none of these approaches work, you can find the host machine that the pod is running on and SSH into that host</p>
</li>
</ul>
<hr />
<p><strong>To sum up:</strong></p>
<ul>
<li><p>Make sure your logging is in place</p>
</li>
<li><p>Debug with the options listed above</p>
</li>
</ul>
