<p><strong>Facing: fluentd log unreadable. it is excluded and would be examined next time</strong></p>

<p>I have a simple configuration for fluentD daemon set running in kubernetes setup.</p>

<p>Fluentd version: <strong>fluentd-0.12.43</strong></p>

<p>Below is my configuration.</p>

<pre><code>  &lt;source&gt;
    @type tail
    path /var/log/containers/sample*.log
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    tag sample.*
    format json
    read_from_head true
  &lt;/source&gt;
  &lt;match sample.**&gt;
    @type forward
    heartbeat_type tcp
    send_timeout 60s
    recover_wait 10s
    hard_timeout 60s
    &lt;server&gt;
      name worker-node2
      host 10.32.0.15
      port 24224
      weight 60
    &lt;/server&gt;
  &lt;/match&gt;
</code></pre>

<p>Getting below warning and NO logs are forwarded</p>

<blockquote>
  <p>2018-08-03 06:36:53 +0000 [warn]:
  /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5d6328e7d.log
  unreadable. It is excluded and would be examined next time.</p>
  
  <p>2018-08-03 06:37:53 +0000 [warn]:
  /var/log/containers/samplelog-79bd66868b-t7xn9_logging1_fluentd-70e85c5bc89ab24.log
  unreadable. It is excluded and would be examined next time.</p>
</blockquote>

<p><strong>Permission for log file:</strong></p>

<pre><code>[root@k8s-master fluentd-daemonset]# ls -lrt **/var/log/containers/**

**lrwxrwxrwx** Jun 25 06:25 sample-77g68_kube-system_kube-proxy-9f3c3951c32ee.log 
-&gt; /var/log/pods/aa1f8d5b-746f-11e8-95c0-005056b9ff3a/sample/7.log
</code></pre>

<p><strong>YAML file for daemon set have mount instructions:</strong></p>

<pre><code>apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging1
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    -----
    -----
    -----

        volumeMounts:
        - name: fluentd-config
          mountPath: /fluentd/etc/ 
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
        - name: varlogcontainers
          mountPath: /var/log/containers
          readOnly: true
        - name: varlibdocker
          mountPath: /var/lib/docker
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: fluentd-config
        configMap:
          name: fluentd-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: varlogcontainers
        hostPath:
          path: /var/log/containers
      - name: varlibdocker
        hostPath:
          path: /var/lib/docker
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers 
</code></pre>

<p>Getting no clue even when <strong>permission is correct</strong>, <strong>fluentD version is correct</strong>, <strong>Mount instruction are their in kubernetes daemonset</strong>, why I am getting this warning.</p>

<p>I faced a similar issue.
So, what actually happens is -<br><br>
 1. Fluentd creates a symbolic links in /var/log/containers/ which are further a symbolic link of files that are in /var/log/pods/ like - </p>

<pre class="lang-sh prettyprint-override"><code>root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log 
lrwxrwxrwx. 1 root root 83 Oct 30 07:42 /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log -&gt; /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1628.log
</code></pre>

<p><br>
2. /var/log/pods are symbolic links of the log files mounted at host. In my setup I am using /data/ directory of host/node to store docker data. </p>

<pre class="lang-sh prettyprint-override"><code>root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log 
lrwxrwxrwx. 1 root root 162 Oct 30 07:47 /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log -&gt; /data/docker/containers/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b-json.log
</code></pre>

<p><br></p>

<p>So, in my deployment.yaml I had to mount /data/docker/containers rather /var/lib/containers/ to solve the issue i.e </p>

<pre class="lang-sh prettyprint-override"><code>        volumeMounts:
        - mountPath: /var/log
          name: varlog
        - mountPath: /data/docker/containers
          name: datadockercontainers
          readOnly: true
        - mountPath: /fluentd/etc
          name: config-path
</code></pre>
