<p>I wish to filter Filebeat autodiscover using Kubernetes Namespaces.</p>
<p>Kubernetes is running on EKS v1.20.7</p>
<p>ECK versions:</p>
<ul>
<li>Elasticsearch v7.7.0</li>
<li>Kibana v7.7.0</li>
<li>Filebeat v7.10.0</li>
</ul>
<p>Filtering is not working based on the following config:</p>
<pre><code>filebeatConfig:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          templates:
          - condition.or:
              - equals:
                  kubernetes.namespace: &quot;ingress&quot;
              - equals:
                  kubernetes.namespace: &quot;kube-system&quot;
            config:
              - type: container
                paths:
                  - /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
                exclude_lines: [&quot;^\\s+[\\-`('.|_]&quot;]

    processors:
      - add_cloud_metadata: ~
      - add_host_metadata: ~

    cloud:
      id: '${ELASTIC_CLOUD_ID}'
    cloud:
      auth: '${ELASTIC_CLOUD_AUTH}'

    output:
      elasticsearch:
        enabled: true
        hosts: &quot;elasticsearch-es-http.monitoring.svc.cluster.local:9200&quot;
        username: '${ELASTICSEARCH_USERNAME}'
        password: '${ELASTICSEARCH_PASSWORD}'
        protocol: https
        ssl:
          verification_mode: &quot;none&quot;

    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: &quot;filebeat-testing-v8-%{[agent.version]}&quot;
    setup.ilm.pattern: &quot;{now/d}-000001&quot;
</code></pre>
<p>Then when the index is queried via the Dev Console it returns both the &quot;ingress&quot; and &quot;monitoring&quot; namespaces, which it shouldn't.</p>
<p>Query:</p>
<pre><code>GET filebeat-testing-v8-7.10.0-2021.09.29-000001/_search 
{ // Identify all Kubernetes namespaces in this Filebeat index
  &quot;size&quot;: 0, // Don't return any documents, just aggregation
  &quot;aggs&quot; : {
    &quot;distinctValues&quot; : { 
      &quot;terms&quot; : {
        &quot;field&quot; : &quot;kubernetes.namespace&quot;, // Field to be aggregated
        &quot;size&quot; : 5000000 // How many unique values to return
      }
    }
  }
}   
</code></pre>
<p>Response:</p>
<pre><code>{
  &quot;took&quot; : 3,
  &quot;timed_out&quot; : false,
  &quot;_shards&quot; : {
    &quot;total&quot; : 1,
    &quot;successful&quot; : 1,
    &quot;skipped&quot; : 0,
    &quot;failed&quot; : 0
  },
  &quot;hits&quot; : {
    &quot;total&quot; : {
      &quot;value&quot; : 10000,
      &quot;relation&quot; : &quot;gte&quot;
    },
    &quot;max_score&quot; : null,
    &quot;hits&quot; : [ ]
  },
  &quot;aggregations&quot; : {
    &quot;distinctValues&quot; : {
      &quot;doc_count_error_upper_bound&quot; : 0,
      &quot;sum_other_doc_count&quot; : 0,
      &quot;buckets&quot; : [
        {
          &quot;key&quot; : &quot;monitoring&quot;,
          &quot;doc_count&quot; : 25093
        },
        {
          &quot;key&quot; : &quot;ingress&quot;,
          &quot;doc_count&quot; : 6041
        },
        {
          &quot;key&quot; : &quot;kube-system&quot;,
          &quot;doc_count&quot; : 132
        },
        {
          &quot;key&quot; : &quot;vault-agent&quot;,
          &quot;doc_count&quot; : 9
        }
      ]
    }
  }
}
</code></pre>
<p>Any ideas or pointers on why this is not filtering correctly, would be much appreciated.</p>

Theo Sweeny
<p>The working config is :</p>
<pre><code>filebeatConfig:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          templates:
            - config:
                - type: container
                  paths:
                    - /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
                  exclude_lines: [&quot;^\\s+[\\-`('.|_]&quot;]
                  processors:
                    - drop_event.when.not.or:
                        - equals.kubernetes.namespace: &quot;ingress&quot;
                        - equals.kubernetes.namespace: &quot;kube-system&quot;

    processors:
      - add_cloud_metadata: ~
      - add_host_metadata: ~

    cloud:
      id: '${ELASTIC_CLOUD_ID}'
    cloud:
      auth: '${ELASTIC_CLOUD_AUTH}'

    output:
      elasticsearch:
        enabled: true
        hosts: &quot;elasticsearch-es-http.monitoring.svc.cluster.local:9200&quot;
        username: '${ELASTICSEARCH_USERNAME}'
        password: '${ELASTICSEARCH_PASSWORD}'
        protocol: https
        ssl:
          verification_mode: &quot;none&quot;

    setup.ilm.enabled: auto
    setup.ilm.rollover_alias: &quot;filebeat-testing-v8-%{[agent.version]}&quot;
    setup.ilm.pattern: &quot;{now/d}-000001&quot;
</code></pre>
