<p>I'm trying to configure Fluentbit in Kubernetes to get Logs from application PODs/Docker Containers and send this log messages to Graylog using GELF format, but this is not working.</p>

<p>See my stack below:</p>

<h2>INPUT</h2>

<ul>
<li>Docker version 1.13.1

<ul>
<li>Docker Log format =>  JSON</li>
<li>Docker Log Driver => Journald => systemd</li>
</ul></li>
<li>Fluent-bit 1.3 running as Daemonset in Kubernetes</li>
<li>Kubernetes 1.17</li>
<li>OS Host: CentOS 7</li>
</ul>

<h2>OUTPUT</h2>

<ul>
<li>Message output format: GELF 1.1</li>
<li>Centralized log => Graylog 3</li>
</ul>

<p>The problem is the fluentbit not read the log from systemd  I'm not get any log in both outputs(Systemd,Stdout), the STDOUT is just to help in troubleshooting.</p>

<p>I don't know why I'm not able to read from systemd.
I followed the documentation exactly
<a href="https://docs.fluentbit.io/manual/input/systemd" rel="nofollow noreferrer">https://docs.fluentbit.io/manual/input/systemd</a></p>

<p>My K8S configurations:</p>

<p>fluent-bit-configmap.yaml</p>

<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: log
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
      Flush         1
      Log_Level     debug
      Daemon        off

    @INCLUDE input-systemd.conf
    @INCLUDE output-stdout.conf

  input-systemd.conf: |
    [INPUT]
      Name            systemd
      Tag             host.*
      Parser          json
      Systemd_Filter  _SYSTEMD_UNIT=docker.service

  output-graylog.conf: |
    [OUTPUT]
      Name          gelf
      Match         *
      Host          10.142.15.214
      Port          12201
      Mode          tcp
      Gelf_Short_Message_Key log

  output-stdout.conf: |
    [OUTPUT]
      Name   stdout
      Match  *
</code></pre>

<p>fluent-bit-ds.yaml</p>

<pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: log
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.3.5
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"
</code></pre>

<p>fluent-bit-role-binding.yaml</p>

<pre><code>apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: log
</code></pre>

<p>fluent-bit-role.yaml</p>

<pre><code>apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
</code></pre>

<p>fluent-bit-service-account.yaml</p>

<pre><code>apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: log
</code></pre>

<p>My Fluentbit OUTPUT(STDOUT) just for debug:</p>

<pre><code>$ kubectl logs -f fluent-bit-2bzxb -n log


[2020/02/20 18:54:23] [Warning] [config] I cannot open /fluent-bit/etc/..2020_02_20_18_54_22.252769193/parsers_custom.conf file
[2020/02/20 18:54:23] [ info] [storage] initializing...
[2020/02/20 18:54:23] [ info] [storage] in-memory
[2020/02/20 18:54:23] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/02/20 18:54:23] [ info] [engine] started (pid=1)
[2020/02/20 18:54:23] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2020/02/20 18:54:23] [ info] [filter_kube] local POD info OK
[2020/02/20 18:54:23] [ info] [filter_kube] testing connectivity with API server...
[2020/02/20 18:54:23] [ info] [filter_kube] API server connectivity OK
[2020/02/20 18:54:23] [ info] [sp] stream processor started
</code></pre>

<p>The problem is I'm not getting any log from systemd with this configuration</p>

rafambbr
<p>Thank you @edsiper I fix my Daemonset adding "path: /run/log"</p>

<pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: log
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.3.5
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: systemdlog
          mountPath: /run/log
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: systemdlog
        hostPath:
          path: /run/log
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"
</code></pre>
