<p>I have installed a Kubernetes cluster using <a href="https://kind.sigs.k8s.io/" rel="nofollow noreferrer">kind k8s</a> as it was easier to setup and run in my local VM. I also installed Docker separately. I then created a docker image for Spring boot application I built for printing messages to the stdout.  It was then added to <strong>kind k8s</strong> local registry. Using this newly created local image, I created a deployment in the kubernetes cluster using <strong>kubectl apply -f config.yaml</strong> CLI command. Using similar method I've also deployed <strong>fluentd</strong> hoping to collect logs from <code>/var/log/containers</code> that would be mounted to fluentD container.</p>
<p>I noticed <code>/var/log/containers/</code> symlink link doesn't exist. However there is <code>/var/lib/docker/containers/</code> and it has folders for some containers that were created in the past. None of the new container IDs doesn't seem to exist in <code>/var/lib/docker/containers/</code> either.</p>
<p>I can see logs in the console when I run <code>kubectl logs pod-name</code> even though I'm unable to find the logs in the local storage.</p>
<p>Following the answer in another <a href="https://stackoverflow.com/questions/67931771/logs-of-pods-missing-from-var-logs-in-kubernetes">thread</a> given by stackoverflow member, I was able to get some information but not all.</p>
<p>I have confirmed Docker is configured with json logging driver by running the following command.
<code>docker info | grep -i logging</code></p>
<p>When I run the following command (found in the thread given above) I can get the image ID.
<code>kubectl get pod pod-name -ojsonpath='{.status.containerStatuses[0].containerID}'</code></p>
<p>However I cannot use it to inspect the docker image using <code>docker inspect</code> as Docker is not aware of such image which I assume it due to the fact it is managed by <strong>kind</strong> control plane.</p>
<p>Appreciate if the experts in the forum can assist to identify where the logs are written and recreate the <code>/var/log/containers</code> symbolink link to access the container logs.</p>

Jason Nanay
<p>It's absolutely normal that your local installed Docker doesn't have containers running in pod created by kind Kubernetes. Let me explain why.</p>
<p>First, we need to figure out, why kind Kubernetes actually needs Docker. It needs it <strong>not</strong> for running containers inside pods. It needs Docker to <strong><a href="https://kind.sigs.k8s.io/" rel="nofollow noreferrer">create container which will be Kubernetes node</a></strong> - and on this container you will have pods which will have containers that are you looking for.</p>
<blockquote>
<p><a href="https://sigs.k8s.io/kind" rel="nofollow noreferrer">kind</a> is a tool for running local Kubernetes clusters using Docker container “nodes”.</p>
</blockquote>
<p>So basically the layers are : your VM -&gt; container hosted on yours VM's docker  which is acting as Kubernetes node -&gt; on this container there are pods -&gt;  in those pods are containers.</p>
<p>In kind <a href="https://kind.sigs.k8s.io/docs/user/quick-start/" rel="nofollow noreferrer">quickstart section</a> you can find more detailed information about image used by kind:</p>
<blockquote>
<p>This will bootstrap a Kubernetes cluster using a pre-built  <a href="https://kind.sigs.k8s.io/docs/design/node-image" rel="nofollow noreferrer">node image</a>. Prebuilt images are hosted at<a href="https://hub.docker.com/r/kindest/node/" rel="nofollow noreferrer"><code>kindest/node</code></a>, but to find images suitable for a given release currently you should check the  <a href="https://github.com/kubernetes-sigs/kind/releases" rel="nofollow noreferrer">release notes</a>  for your given kind version (check with  <code>kind version</code>) where you'll find a complete listing of images created for a kind release.</p>
</blockquote>
<p>Back to your question, let's find missing containers!</p>
<p>On my local VM, I setup <a href="https://kind.sigs.k8s.io/docs/user/quick-start/#installation" rel="nofollow noreferrer">kind Kubernetes</a> and I have installed <code>kubectl</code> tool Then, I created an example <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment" rel="nofollow noreferrer">nginx-deployment</a>. By running <code>kubectl get pods</code> I can confirm pods are working.</p>
<p>Let's find container which is acting as node by running <code>docker ps -a</code>:</p>
<pre><code>CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                        NAMES
1d2892110866   kindest/node:v1.21.1   &quot;/usr/local/bin/entr…&quot;   50 minutes ago   Up 49 minutes   127.0.0.1:43207-&gt;6443/tcp   kind-control-plane
</code></pre>
<p>Okay, now we can exec into it and find containers. Note that <code>kindest/node</code> image is not using docker as the container runtime but <a href="https://github.com/kubernetes-sigs/cri-tools" rel="nofollow noreferrer"><code>crictl</code></a>.</p>
<p>Let's exec into node: <code>docker exec -it 1d2892110866 sh</code>:</p>
<pre><code># ls
bin  boot  dev  etc  home  kind  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
# 
</code></pre>
<p>Now we are in node - time to check if containers are here:</p>
<pre><code># crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID
135c7ad17d096       295c7be079025       47 minutes ago      Running             nginx                     0                   4e5092cab08f6
ac3b725061e12       295c7be079025       47 minutes ago      Running             nginx                     0                   6ecda41b665da
a416c226aea6b       295c7be079025       47 minutes ago      Running             nginx                     0                   17aa5c42f3512
455c69da57446       296a6d5035e2d       57 minutes ago      Running             coredns                   0                   4ff408658e04a
d511d62e5294d       e422121c9c5f9       57 minutes ago      Running             local-path-provisioner    0                   86b8fcba9a3bf
116b22b4f1dcc       296a6d5035e2d       57 minutes ago      Running             coredns                   0                   9da6d9932c9e4
2ebb6d302014c       6de166512aa22       57 minutes ago      Running             kindnet-cni               0                   6ef310d8e199a
2a5e0a2fbf2cc       0e124fb3c695b       57 minutes ago      Running             kube-proxy                0                   54342daebcad8
1b141f55ce4b2       0369cf4303ffd       57 minutes ago      Running             etcd                      0                   32a405fa89f61
28c779bb79092       96a295389d472       57 minutes ago      Running             kube-controller-manager   0                   2b1b556aeac42
852feaa08fcc3       94ffe308aeff9       57 minutes ago      Running             kube-apiserver            0                   487e06bb5863a
36771dbacc50f       1248d2d503d37       58 minutes ago      Running             kube-scheduler            0                   85ec6e38087b7
</code></pre>
<p>Here they are. You can also notice that there are other container which are acting as <a href="https://kubernetes.io/docs/concepts/overview/components/" rel="nofollow noreferrer">Kubernetes Components</a>.</p>
<p>For further debugging containers I would suggest reading documentation about <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/" rel="nofollow noreferrer">debugging Kubernetes nodes with crictl</a>.</p>
<p>Please also note that on your local VM there is file <code>~/.kube/config</code> which has information needed for <code>kubectl</code> to communicate between your VM and the Kubernetes cluster (in case of kind Kubernetes - docker container running locally).</p>
<p>Hope It will help you. Feel free to ask any question.</p>
<p><strong>EDIT - ADDED INFO HOW TO SETUP MOUNT POINTS</strong></p>
<p>Answering question from comment about mounting directory from node to local VM. We need to setup <a href="https://kind.sigs.k8s.io/docs/user/configuration/#extra-mounts" rel="nofollow noreferrer">&quot;Extra Mounts&quot;</a>.  Let's create a definition needed for kind Kubernetes:</p>
<pre><code>kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  # add a mount from /path/to/my/files on the host to /files on the node
  extraMounts:
  - hostPath: /tmp/logs/
    containerPath: /var/log/pods
    # optional: if set, the mount is read-only.
    # default false
    readOnly: false
    # optional: if set, the mount needs SELinux relabeling.
    # default false
    selinuxRelabel: false
    # optional: set propagation mode (None, HostToContainer or Bidirectional)
    # see https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
    # default None
    propagation: Bidirectional
</code></pre>
<p>Note that I'm using <code>/var/log/pods</code> instead of <code>/var/log/containers/</code> - it is because on the cluster created by kind Kubernetes <code>containers</code> directory has only symlinks to logs in <code>pod</code> directory.</p>
<p>Save this <code>yaml</code>, for example as <code>cluster-with-extra-mount.yaml</code> , then create a cluster using this (create a directory <code>/tmp/logs</code> before applying this command!):</p>
<pre><code>kind create cluster --config=/tmp/cluster-with-extra-mount.yaml
</code></pre>
<p>Then all containers logs will be in <code>/tmp/logs</code> on your VM.</p>
