<p>I’m trying to collect logs from Kubernetes nodes using Filebeat and ONLY ship them to ELK IF the logs originate from a specific Kubernetes Namespace.</p>

<p>So far I’ve discovered that you can define <em>Processors</em> which I think accomplish this. However, no matter what I do I can not get the shipped logs to be constrained. Does this look right?</p>

<p>Hm, does this look correct then?</p>

<pre><code>filebeat.config:
  inputs:
    path: ${path.config}/inputs.d/*.yml
    reload.enabled: true
    reload.period: 10s
    when.contains:
      kubernetes.namespace: "NAMESPACE"
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false
  processors:
    - add_kubernetes_metadata:
      namespace: "NAMESPACE"
xpack.monitoring.enabled: true
output.elasticsearch:
  hosts: ['elasticsearch:9200']
</code></pre>

<p>Despite this configuration I still get logs from all of the namespaces. </p>

<p>Filebeat is running as a DaemonSet on Kubernetes. Here is an example of an expanded log entry: <a href="https://i.stack.imgur.com/HWMqF.png" rel="nofollow noreferrer">https://i.stack.imgur.com/HWMqF.png</a></p>

TJ Zimmerman
<p>You have number options to do it:</p>

<ol>
<li><a href="https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html" rel="nofollow noreferrer">Filter data by filebeat</a></li>
</ol>

<pre><code>processors:
 - drop_event:
     when:
        contains:
           source: "field"
</code></pre>

<ol start="2">
<li><a href="https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ingest-node.html" rel="nofollow noreferrer">Use ingest pipeline into elasticsearch</a>:</li>
</ol>

<pre><code>output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: my_pipeline_id
</code></pre>

<p>And then test events into <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/drop-processor.html" rel="nofollow noreferrer">pipeline</a>:</p>

<pre><code>{
  "drop": {
    "if" : "ctx['field'] == null "
  }
}
</code></pre>

<ol start="3">
<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html" rel="nofollow noreferrer">Use drop filter of logstash</a>:</li>
</ol>

<pre><code>filter {
  if ![field] {
    drop { }
  }
}
</code></pre>
