<p>I am setting up pipeline to send the kubernetes pods log to elastic cluster. I have installed filebeat as deamonset (stream: stdout) in my cluster and connected output to logstash. Beats is connected with logstash without an issue, now i want logs from application namespaces not from all namespaces in cluster. can someone guide me how to filter this in beat adn also how can to see the source message from json in es?</p>
<p>This is my config:</p>
<pre class="lang-yaml prettyprint-override"><code>data:
  kubernetes.yml: |-
    - type: docker
      containers:
        path: &quot;/var/lib/docker/containers&quot;
        stream: &quot;stdout&quot;
        ids: &quot;*&quot;
        multiline.pattern: '^\s'
        multiline.match: after
      fields:
         logtype: container
      multiline:
         pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
         negate: true
         match: after
      ignore_older: 1h
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
        - decode_json_fields:
            fields: [&quot;log&quot;]
            overwrite_keys: true
            target: &quot;&quot;

</code></pre>
<p>Output in kibana:</p>
<pre class="lang-json prettyprint-override"><code>
{
  &quot;_index&quot;: &quot;filebeat-6.8.4-2020.03.06&quot;,
  &quot;_type&quot;: &quot;doc&quot;,
  &quot;_id&quot;: &quot;vHkzsHABJ57Tsdxxxxx&quot;,
  &quot;_version&quot;: 1,
  &quot;_score&quot;: null,
  &quot;_source&quot;: {
    &quot;log&quot;: {
      &quot;file&quot;: {
        &quot;path&quot;: &quot;/var/lib/docker/containers/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c/sdnksdsdlsdnfsdlfslfnsdslfnsnlnflksdnflkdsfnsdflsdfndslffndslf-json.log&quot;
      }
    },
    &quot;tags&quot;: [
      &quot;beats_input_codec_plain_applied&quot;,
      &quot;_grokparsefailure&quot;
    ],
    &quot;input&quot;: {
      &quot;type&quot;: &quot;docker&quot;
    },
    &quot;@version&quot;: &quot;1&quot;,
    &quot;prospector&quot;: {
      &quot;type&quot;: &quot;docker&quot;
    },
    &quot;beat&quot;: {
      &quot;version&quot;: &quot;6.8.4&quot;,
      &quot;name&quot;: &quot;filebeat-vtp2f&quot;,
      &quot;hostname&quot;: &quot;filebeat-vtp2f&quot;
    },
    &quot;host&quot;: {
      &quot;name&quot;: &quot;filebeat-vtp2f&quot;
    },
    &quot;offset&quot;: 5798785,
    &quot;stream&quot;: &quot;stdout&quot;,
    &quot;fields&quot;: {
      &quot;logtype&quot;: &quot;container&quot;
    },
    &quot;kubernetes&quot;: {
      &quot;node&quot;: {
        &quot;name&quot;: &quot;k8-test-22313607-0&quot;
      },
      &quot;labels&quot;: {
        &quot;version&quot;: &quot;v1&quot;,
        &quot;kubernetes&quot;: {
          &quot;io/cluster-service&quot;: &quot;true&quot;
        },
        &quot;controller-revision-hash&quot;: &quot;6b56cfcb69&quot;,
        &quot;pod-template-generation&quot;: &quot;1&quot;,
        &quot;k8s-app&quot;: &quot;fluent&quot;
      },
      &quot;container&quot;: {
        &quot;name&quot;: &quot;fluentd&quot;
      },
      &quot;pod&quot;: {
        &quot;uid&quot;: &quot;72c50b54-5ef0-11ea-83e1-26018882335d&quot;,
        &quot;name&quot;: &quot;fluent-4lft2&quot;
      },
      &quot;namespace&quot;: &quot;fluentd&quot;
    },
    &quot;source&quot;: &quot;/var/lib/docker/containers/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c-json.log&quot;,
    &quot;@timestamp&quot;: &quot;2020-03-06T14:15:18.561Z&quot;
  },
  &quot;fields&quot;: {
    &quot;@timestamp&quot;: [
      &quot;2020-03-06T14:15:18.561Z&quot;
    ]
  },
  &quot;highlight&quot;: {
    &quot;prospector.type&quot;: [
      &quot;@kibana-highlighted-field@docker@/kibana-highlighted-field@&quot;
    ]
  },
  &quot;sort&quot;: [
    1583504118561
  ]
}
</code></pre>

<p>If you want Filebeat to only grab logs from certain namespaces you use a condition:</p>
<p><code>filebeat.yml</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>    logging.level: error
    logging.json: true
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
          - condition:
              equals:
                kubernetes.namespace: stage
            config:
              - type: container
                paths:
                 - /var/log/containers/*${data.kubernetes.container.id}.log
                multiline.pattern: '^[[:space:]]'
                multiline.negate: false
                multiline.match: after
                include_lines: ['^{']
</code></pre>
<p>Note this part:</p>
<pre class="lang-yaml prettyprint-override"><code>          templates:
          - condition:
              equals:
                kubernetes.namespace: stage
</code></pre>
<p>I do run a Filebeat as a Daemonset in each Namespace.  It's a bit of extra overhead but Filebeat can be finicky so that does help us work out issues in other logical environments first.</p>
