<p>I'm trying to configure EFK stack in my local minikube setup. I have followed <a href="https://www.metricfire.com/blog/logging-for-kubernetes-fluentd-and-elasticsearch/" rel="nofollow noreferrer">this tutorial</a>.</p>
<p>Everything is working fine (I can see all my console logs in kibana and Elasticsearch). But I have another requirement. I have Node.js application which is logs as files to custom path <code>/var/log/services/dev</code> inside the pod.</p>
<p>File Tree:</p>
<pre><code>/var/log/services/dev/# ls -l
total 36
-rw-r--r--    1 root     root         28196 Nov 27 18:09 carts-service-dev.log.2021-11-27T18.1
-rw-r--r--    1 root     root          4483 Nov 27 18:09 carts-service-dev.log.2021-11-27T18
</code></pre>
<p>How to configure my Fluentd to read all my console logs and also read logs from the custom path configured?</p>
<p>My App Deployment File:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: carts
spec:
  replicas: 1
  selector:
    matchLabels:
      app: carts
  template:
    metadata:
      labels:
        app: carts
    spec:
      containers:
        - name: app
          image: carts-service
          resources:
            limits:
              memory: &quot;1024Mi&quot;
              cpu: &quot;500m&quot;
          ports:
            - containerPort: 4000

</code></pre>
<p>My Fluentd DaemonSet File:</p>
<pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: &quot;elasticsearch.default&quot;
            - name: FLUENT_ELASTICSEARCH_PORT
              value: &quot;9200&quot;

</code></pre>
<p>I do know that log files written into custom path <code>/var/log/services/dev</code> will be deleted if pod crashes. So I have to use persistent volume to mount this path.</p>
<p>But I lack the experience of how to create persistent volume for that path and also link Fluentd to read from it.</p>
<p>Thanks in advance.</p>

Sathish
<p>If a pod crashes, all logs still will be accessible in <code>efk</code>. No need to add a persistent volume to the pod with your application only for storing log file.</p>
<p>Main question is how to get logs from this file. There are <strong>two main approaches</strong> which are suggested and based on kubernetes documentation:</p>
<ol>
<li><p><strong>Use a sidecar container</strong>.</p>
<p>Containers in pod have the same file system and <code>sidecar</code> container will be streaming logs from file to <code>stdout</code> and/or <code>stderr</code> (depends on implementation) and after logs will be picked up by kubelet.</p>
<p>Please find <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#streaming-sidecar-container" rel="nofollow noreferrer">streaming sidecar container</a> and example how it works.</p>
</li>
<li><p><strong>Use a sidecar container with a logging agent</strong>.</p>
<p>Please find <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-logging-agent" rel="nofollow noreferrer">Sidecar container with a logging agent</a> and configuration example using <code>fluentd</code>. In this case logs will be collected by <code>fluentd</code> and they won't be available by <code>kubectl logs</code> commands since <code>kubelet</code> is not responsible for these logs.</p>
</li>
</ol>
