<p>I have a Kubernetes Cluster and I've been trying to forward logs to Splunk with this <a href="https://github.com/splunk/splunk-connect-for-kubernetes#prerequisites" rel="nofollow noreferrer">splunk-connect-for-kubernetes</a> repo which is essentially Splunk's own kubernetes-oriented configuration of fluentd.</p>
<p>I initially could see logs in Splunk but they appeared to just be related to the system components but not from the pods that I needed from.</p>
<p>I think I tracked down to the problem in the global <code>values.yaml</code> file. I experimented a bit with the <code>fluentd path</code> and <code>containers path</code> and found that I likely needed to update the <code>containers pathDest</code> to the same file path as the pods logs.</p>
<p>It looks like something like this now:</p>
<pre><code>fluentd:
    # path of logfiles, default /var/log/containers/*.log
    path: /var/log/containers/*.log
    # paths of logfiles to exclude. object type is array as per fluentd specification:
    # https://docs.fluentd.org/input/tail#exclude_path
    exclude_path:
    #  - /var/log/containers/kube-svc-redirect*.log
    #  - /var/log/containers/tiller*.log
    #  - /var/log/containers/*_kube-system_*.log (to exclude `kube-system` namespace)

  # Configurations for container logs
  containers:
    # Path to root directory of container logs
    path: /var/log
    # Final volume destination of container log symlinks
    pathDest: /app/logs
</code></pre>
<p>But now I can see in my the logs for my <code>splunk-connect</code> repeated logs like</p>
<pre><code>[warn]: #0 [containers.log] /var/log/containers/application-0-tcc-broker-0_application-0bb08a71919d6b.log unreadable. It is excluded and would be examined next time.
</code></pre>

<p>I had a very similar problem once and changing the path in the values.yaml file helped to solve the problem. It is perfectly described in <a href="https://community.splunk.com/t5/All-Apps-and-Add-ons/splunk-connect-for-kubernetes-var-log-containers-log-unreadable/m-p/473943" rel="nofollow noreferrer">this thread</a>:</p>
<blockquote>
<p>Found the solution for my question -</p>
<p>./splunk-connect-for-kubernetes/charts/splunk-kubernetes-logging/values.yaml: path: <code>/var/log/containers/*.log </code>
Changed to:<br />
path: <code>/var/log/pods/*.log</code> works to me.</p>
</blockquote>
<p>The cited answer may not be readable. Just try changing <code>/var/log/containers/*.log</code> to <code>/var/log/pods/*.log</code> in your file.</p>
<p>See also <a href="https://stackoverflow.com/questions/51671212/fluentd-log-unreadable-it-is-excluded-and-would-be-examined-next-time">this similar question on stackoverflow</a>.</p>
