<p>I would like to annotate some kubernetes pods with an annotation like <code>please_log_with_parser: myparser</code> and have something like <code>fluentbit</code> to <em>only</em> process the logs of those pods with the given parser to our elastic search cluster.</p>

<p>For fluentbit I'm only aware of annotation <code>fluentbit.io/exclude</code>, but that would result in having to annotate all other pods. - Is is possible to tell fluentbit to <em>only</em> process logs having a label, or otherwise discarding all messages without the proper annotation?</p>

<p>Is there maybe an alternative to fluentbit, which is able to do so?</p>

michas
<p>One option is using a combination of <code>kubernetes</code>, <code>nest</code>, and <code>grep</code> filter like in <a href="https://docs.fluentbit.io/manual/v/1.3/filter/grep#nested-fields-example" rel="nofollow noreferrer">this example</a>.</p>

<p>You use the <code>kubernetes</code> filter to create metadata like this:</p>

<pre><code>{
    "kubernetes": {
        "pod_name": "myapp-0",
        "namespace_name": "default",
        "pod_id": "216cd7ae-1c7e-11e8-bb40-000c298df552",
        "annotations": {
            "please_log_with_parser": "myparser"
        },
        "host": "minikube",
        "container_name": "myapp",
        "docker_id": "370face382c7603fdd309d8c6aaaf434fd98b92421ce7c7c8aafe7697d4aa362"
    }
}
</code></pre>

<p>Then use <code>nest</code> to lift the annotation to the top level.
Then use <code>grep</code> to discard only keep the record if that annotation is present.</p>

<p>My helm values now look like this:</p>

<pre><code>rawConfig: |-
 @INCLUDE fluent-bit-service.conf
 @INCLUDE fluent-bit-input.conf
 @INCLUDE fluent-bit-filter.conf
 [FILTER]
     Name         nest
     Match        *
     Operation    lift
     Nested_under kubernetes
     Add_prefix   kubernetes_
 [FILTER]
     Name         nest
     Match        *
     Operation    lift
     Nested_under kubernetes_annotations
     Add_prefix   kubernetes_annotations_
 [FILTER]
     Name    grep
     Match   *
     Regex   kubernetes_annotations_fluentbit.io/parser .*
 [FILTER]
     Name         nest
     Match        *
     Operation    nest
     Wildcard     kubernetes_annotations_*
     Nest_under   kubernetes_annotations
     Remove_prefix   kubernetes_annotations_
 [FILTER]
     Name         nest
     Match        *
     Operation    nest
     Wildcard     kubernetes_*
     Nest_under   kubernetes
     Remove_prefix  kubernetes_
 @INCLUDE fluent-bit-output.conf
</code></pre>
