<p>I have configured an <a href="https://github.com/elastic/cloud-on-k8s/blob/main/config/recipes/beats/filebeat_autodiscover.yaml" rel="nofollow noreferrer">Elastic ECK Beat with autodiscover</a> enabled for all pod logs, but I need to add logs from a specific pod log file too; from this path <code>/var/log/traefik/access.log</code> inside the container. I've tried with module and log config but still nothing works.</p>
<p>The access.log file exists on the pods and contains data.
The filebeat index does not show any data from this log.file.path</p>
<p>Here is the Beat yaml:</p>
<pre><code>---
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: elastic
spec:
  type: filebeat
  version: 8.3.1
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
  config:
    filebeat:
      autodiscover:
        providers:
          - type: kubernetes
            node: ${NODE_NAME}
            hints:
              enabled: true
              default_config:
                type: container
                paths:
                  - /var/log/containers/*${data.kubernetes.container.id}.log
            templates:
              - condition.contains:
                  kubernetes.pod.name: traefik
                config:
                  - module: traefik
                    access:
                      enabled: true
                      var.paths: [ &quot;/var/log/traefik/*access.log*&quot; ]
    processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
          - name: filebeat
            securityContext:
              runAsUser: 0
              # If using Red Hat OpenShift uncomment this:
              #privileged: true
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
              - name: varlog
                mountPath: /var/log
            env:
              - name: NODE_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: spec.nodeName
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers
          - name: varlog
            hostPath:
              path: /var/log
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  namespace: elastic
rules:
  - apiGroups: [&quot;&quot;] # &quot;&quot; indicates the core API group
    resources:
      - namespaces
      - pods
      - nodes
    verbs:
      - get
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: elastic
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
  namespace: elastic
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: elastic
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
</code></pre>
<p>Here is the module loaded from Filebeat Logs:</p>
<pre><code>...
{&quot;log.level&quot;:&quot;info&quot;,&quot;@timestamp&quot;:&quot;2022-08-18T19:58:55.337Z&quot;,&quot;log.logger&quot;:&quot;esclientleg&quot;,&quot;log.origin&quot;:{&quot;file.name&quot;:&quot;eslegclient/connection.go&quot;,&quot;file.line&quot;:291},&quot;message&quot;:&quot;Attempting to connect to Elasticsearch version 8.3.1&quot;,&quot;service.name&quot;:&quot;filebeat&quot;,&quot;ecs.version&quot;:&quot;1.6.0&quot;}
{&quot;log.level&quot;:&quot;info&quot;,&quot;@timestamp&quot;:&quot;2022-08-18T19:58:55.352Z&quot;,&quot;log.logger&quot;:&quot;modules&quot;,&quot;log.origin&quot;:{&quot;file.name&quot;:&quot;fileset/modules.go&quot;,&quot;file.line&quot;:108},&quot;message&quot;:&quot;Enabled modules/filesets: traefik (access)&quot;,&quot;service.name&quot;:&quot;filebeat&quot;,&quot;ecs.version&quot;:&quot;1.6.0&quot;}
{&quot;log.level&quot;:&quot;info&quot;,&quot;@timestamp&quot;:&quot;2022-08-18T19:58:55.353Z&quot;,&quot;log.logger&quot;:&quot;input&quot;,&quot;log.origin&quot;:{&quot;file.name&quot;:&quot;log/input.go&quot;,&quot;file.line&quot;:172},&quot;message&quot;:&quot;Configured paths: [/var/log/traefik/*access.log*]&quot;,&quot;service.name&quot;:&quot;filebeat&quot;,&quot;input_id&quot;:&quot;fa247382-c065-40ca-974e-4b69f14c3134&quot;,&quot;ecs.version&quot;:&quot;1.6.0&quot;}
{&quot;log.level&quot;:&quot;info&quot;,&quot;@timestamp&quot;:&quot;2022-08-18T19:58:55.355Z&quot;,&quot;log.logger&quot;:&quot;modules&quot;,&quot;log.origin&quot;:{&quot;file.name&quot;:&quot;fileset/modules.go&quot;,&quot;file.line&quot;:108},&quot;message&quot;:&quot;Enabled modules/filesets: traefik (access)&quot;,&quot;service.name&quot;:&quot;filebeat&quot;,&quot;ecs.version&quot;:&quot;1.6.0&quot;}
{&quot;log.level&quot;:&quot;info&quot;,&quot;@timestamp&quot;:&quot;2022-08-18T19:58:55.355Z&quot;,&quot;log.logger&quot;:&quot;input&quot;,&quot;log.origin&quot;:{&quot;file.name&quot;:&quot;log/input.go&quot;,&quot;file.line&quot;:172},&quot;message&quot;:&quot;Configured paths: [/var/log/traefik/*access.log*]&quot;,&quot;service.name&quot;:&quot;filebeat&quot;,&quot;input_id&quot;:&quot;6883d753-f149-4a68-9499-fe039e0de899&quot;,&quot;ecs.version&quot;:&quot;1.6.0&quot;}
{&quot;log.level&quot;:&quot;info&quot;,&quot;@timestamp&quot;:&quot;2022-08-18T19:58:55.437Z&quot;,&quot;log.origin&quot;:{&quot;file.name&quot;:&quot;input/input.go&quot;,&quot;file.line&quot;:134},&quot;message&quot;:&quot;input ticker stopped&quot;,&quot;service.name&quot;:&quot;filebeat&quot;,&quot;ecs.version&quot;:&quot;1.6.0&quot;}
{&quot;log.level&quot;:&quot;info&quot;,&quot;@timestamp&quot;:&quot;2022-08-18T19:58:55.439Z&quot;,&quot;log.logger&quot;:&quot;input&quot;,&quot;log.origin&quot;:{&quot;file.name&quot;:&quot;log/input.go&quot;,&quot;file.line&quot;:172},&quot;message&quot;:&quot;Configured paths: [/var/log/containers/*9a1680222e867802388f649f0a296e076193242962b28eb7e0e575bf68826d85.log]&quot;,&quot;service.name&quot;:&quot;filebeat&quot;,&quot;input_id&quot;:&quot;3c1fffae-0213-4889-b0e7-5dda489eeb51&quot;,&quot;ecs.version&quot;:&quot;1.6.0&quot;}
...
</code></pre>

joepa37
<p>Docker logging is based on the stdout/stderr output of a container. If you only write into a log file inside a container it will never be picked up by Docker logging and can therefore also not be processed by your Filebeat setup.</p>
<p>Instead ensure that all logs generated by your containers are sent to stdout. Which would mean in your example start the Traeffic pod with <code>--accesslogsfile=/dev/stdout</code> to also send the access logs to stdout instead of the log file.</p>
