<p>At my company the kubernetes cluster is managed by a team, we must provision a namespace and then create our resources. We cannot use features such as <code>hostPath</code> volumes and we cannot create new roles or namespaces, etc.</p>
<p>So looking at an example implementation of the <code>fluentd-elasticsearch</code> container as a <code>DaemonSet</code>, they all appear to be using hostPath volume mounting but I'm not sure why.</p>
<p>For example, I ran through this:
<a href="https://www.howtoforge.com/create-a-daemonset-in-kubernetes/" rel="nofollow noreferrer">https://www.howtoforge.com/create-a-daemonset-in-kubernetes/</a></p>
<p>And created this:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-fluentd-elasticsearch-daemonset
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
</code></pre>
<p>But got this error:</p>
<pre><code>Error creating: pods &quot;fluentd-elasticsearch-&quot; is forbidden: unable to validate against any pod 
security policy: [spec.volumes[0]: Invalid value: &quot;hostPath&quot;: hostPath volumes are not allowed 
to be used spec.volumes[1]: Invalid value: &quot;hostPath&quot;: hostPath volumes are not allowed to be 
used]
</code></pre>
<p>So I have a couple of questions:</p>
<ol>
<li>Is fluentd mounting volumes then reading files in those volumes that get pushed out to elasticsearch?</li>
<li>Can I just remove the volume mounting or is that essential to it functioning?</li>
<li>Is fluentd using the kubernetes API at all?</li>
<li>Are there any non-daemonset containers which would just use the kubernetes API to get the pods then use log api to forward to a log db?</li>
</ol>

justin.m.chase
<blockquote>
<p>Is fluentd mounting volumes then reading files in those volumes that get pushed out to elasticsearch?</p>
</blockquote>
<p>Docker is storing logs on the node's disk. Fluentd needs to acces this log files somehow; this is why its running as daemonset, you need it to run on every node with hostpath to access log files.</p>
<blockquote>
<p>Can I just remove the volume mounting or is that essential to it functioning?</p>
</blockquote>
<p>No, you can't &quot;just remove&quot; volume mounting (hostpath) because fluentd will loose access to the log files that docker keeps on nodes.</p>
<blockquote>
<p>Is fluentd using the kubernetes API at all?</p>
</blockquote>
<p>There is no straightforward answer to this question. There are plugins I have found that can access k8s metadata using k8s api but I haven't found any plugin that would use k8s api to pull logs.</p>
<blockquote>
<p>Are there any non-daemonset containers which would just use the kubernetes API to get the pods then use log api to forward to a log db?</p>
</blockquote>
<p>Some similar to this is describled in k8s documentation: <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent" rel="nofollow noreferrer">sidecar container with a logging agent</a></p>
<p>So yes, you could possibly deploy fluentd as a sidecar to gather and forward logs to the db. Check the docs for more details.</p>
