<p>I am trying to run the filebeat daemon set to get the log for particular app. There are basically two nodegroups:- eai and eai-staging. <code>eai</code> nodgroup have only single namespace by the <code>eai-staging</code> have multiple namespace.
I have following filebeat config:</p>

<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: log
      fields:
        app_type: "${NAMESPACE}". &lt;&lt;&lt;---- I want this app type to be different based on namespace
        log_type: secure
      fields_under_root: true

    output.logstash:
      hosts: ["${LOGSTASH_HOST}:${LOGSTASH_PORT}"]
      ttl: 1s
      pipelining: 0
    processors:
      - drop_fields:
          fields: ["beat", "host", "input", "offset", "source"]
</code></pre>

<p><strong>Filebeat Daemon set</strong></p>

<pre><code>apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  labels:
    app: filebeat
spec:
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      nodeSelector:
        nodegroup: eai
      priorityClassName: critical
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        imagePullPolicy: Always
        image: docker.elastic.co/beats/filebeat:6.5.4
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: LOGSTASH_HOST
          value: "logstash-headless.etl.svc.cluster.local"
        - name: LOGSTASH_PORT
          value: "5046"
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: app-log
          mountPath: /var/log/app/
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: filebeat-config
      - name: data
        hostPath:
          path: /var/lib/filebeat-data/eai/app-filebeat
          type: DirectoryOrCreate
      - name: app-log
        hostPath:
          path: /var/log/app/
          type: DirectoryOrCreate
</code></pre>

<p>Now, how can I get the particular namespace from where the app log is obtained by the filebeat.
I tried to deploy one daemon set in the eai namespace in the eai nodegroup. So I can get the namespace for that using <code>metadata.namespace</code>.</p>

<p>But, if I deployed the daemon set in the eai-staging node group in the particular namespace. I will always get the same namespace value.</p>

<p>Is there any way around. Or should I deploy the daemon set in each namespace?</p>

<p>P.S. I could not use the filebeat in the same container because if filebeat is down due to some reason, the pod cannot receive the request for the app</p>

programmingtech
<p>Deploy filebeat as daemonset in each node and filebeat will get logs from all containers in that node but you can add namespace, pod name, labels as metadata to each event. This way you will get to know from which namespace the event was originated.</p>

<p>The <code>add_kubernetes_metadata</code> processor annotates each event with relevant metadata based on which Kubernetes pod the event originated from. Each event is annotated with:</p>

<pre><code>Pod Name
Namespace
Labels
</code></pre>

<p><a href="https://www.elastic.co/guide/en/beats/filebeat/6.1/add-kubernetes-metadata.html" rel="nofollow noreferrer">https://www.elastic.co/guide/en/beats/filebeat/6.1/add-kubernetes-metadata.html</a></p>
