<p>After shifting from Docker to containerd as docker engine used by our kubernetes, we are not able to show the multiline logs in a proper way by our visualization app (Grafana) as some details prepended to the container/pod logs by the containerd itself (i.e. timestamp, stream &amp; log severity to be specific it is appending something like the following and as shown in the below sample: <strong>2022-07-25T06:43:17.20958947Z stdout F</strong>  ) which make some confusion for the developers and the application owners.</p>
<p>I am showing here a dummy sample of the logs generated by the application and how it got printed in the nodes of kuberenetes'nodes after containerd prepended the mentioned details.</p>
<p>The following logs generated by the application (kubectl logs ):</p>
<pre><code>2022-07-25T06:43:17,309ESC[0;39m dummy-[txtThreadPool-2] ESC[39mDEBUGESC[0;39m
  ESC[36mcom.pkg.sample.ComponentESC[0;39m - Process message meta {
  timestamp: 1658731397308720468
  version {
      major: 1
      minor: 0
      patch: 0
  }
}
</code></pre>
<p>when I check the logs in the filesystem (/var/log/container/ABCXYZ.log) :</p>
<pre><code>2022-07-25T06:43:17.20958947Z stdout F 2022-07-25T06:43:17,309ESC[0;39m dummy-[txtThreadPool-2]
ESC[39mDEBUGESC[0;39m
ESC[36mcom.pkg.sample.ComponentESC[0;39m - Process message meta {
2022-07-25T06:43:17.20958947Z stdout F timestamp: 1658731449723010774
2022-07-25T06:43:17.209593379Z stdout F version {
2022-07-25T06:43:17.209595933Z stdout F major: 14
2022-07-25T06:43:17.209598466Z stdout F minor: 0
2022-07-25T06:43:17.209600712Z stdout F patch: 0
2022-07-25T06:43:17.209602926Z stdout F }
2022-07-25T06:43:17.209605099Z stdout F }
</code></pre>
<p>I am able to parse the multiline logs with fluentbit but the problem is I am not able to remove the details injected by containerd ( &gt;&gt; 2022-07-25T06:43:17.209605099Z stdout F .......). So is there anyway to configure containerd to not prepend these details somehow in the logs and print them as they are generated from the application/container ?</p>
<p>On the other hand is there any plugin to remove such details from fluentbit side .. as per the existing <a href="https://docs.fluentbit.io/manual/concepts/data-pipeline" rel="nofollow noreferrer">plugins</a> none of them can manipulate or change the logs (which is logical  as the log agent should not do any change on the logs).</p>
<p>Thanks in advance.</p>

k.elgohary
<p>This is the workaround I followed to show the multiline log lines in Grafana by applying extra fluentbit filters and multiline parser.</p>
<p>1- First I receive the stream by tail input which parse it by a multiline parser (multilineKubeParser).</p>
<p>2- Then another filter will intercept the stream to do further processing by a regex parser (kubeParser).</p>
<p>3- After that another filter will remove the details added by the containerd by a lua parser ().</p>
<pre><code>  fluent-bit.conf: |-
    [SERVICE]
        HTTP_Server    On
        HTTP_Listen    0.0.0.0
        HTTP_PORT      2020
        Flush          1
        Daemon         Off
        Log_Level      warn
        Parsers_File   parsers.conf
    [INPUT]
        Name           tail
        Tag            kube.*
        Path           /var/log/containers/*.log
        multiline.Parser         multilineKubeParser
        Exclude_Path   /var/log/containers/*_ABC-logging_*.log
        DB             /run/fluent-bit/flb_kube.db
        Mem_Buf_Limit  5MB
    [FILTER]
        Name           kubernetes
        Match          kube.*
        Kube_URL       https://kubernetes.default.svc:443
        Merge_Log On
        Merge_Parser   kubeParser
        K8S-Logging.Parser Off
        K8S-Logging.Exclude On
    [FILTER]
        Name           lua
        Match          kube.*
        call remove_dummy
        Script filters.lua
    [Output]
        Name grafana-loki
        Match kube.*
        Url http://loki:3100/api/prom/push
        TenantID &quot;&quot;
        BatchWait 1
        BatchSize 1048576
        Labels {job=&quot;fluent-bit&quot;}
        RemoveKeys kubernetes
        AutoKubernetesLabels false
        LabelMapPath /fluent-bit/etc/labelmap.json
        LineFormat json
        LogLevel warn
  labelmap.json: |-
    {
      &quot;kubernetes&quot;: {
        &quot;container_name&quot;: &quot;container&quot;,
        &quot;host&quot;: &quot;node&quot;,
        &quot;labels&quot;: {
          &quot;app&quot;: &quot;app&quot;,
          &quot;release&quot;: &quot;release&quot;
        },
        &quot;namespace_name&quot;: &quot;namespace&quot;,
        &quot;pod_name&quot;: &quot;instance&quot;
      },
      &quot;stream&quot;: &quot;stream&quot;
    }
  parsers.conf: |-
    [PARSER]
        Name        kubeParser
        Format      regex
        Regex       /^([^ ]*).* (?&lt;timeStamp&gt;[^a].*) ([^ ].*)\[(?&lt;requestId&gt;[^\]]*)\] (?&lt;severity&gt;[^ ]*) (?&lt;message&gt;[^ ].*)$/
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep   On
        Time_Offset +0200
    [MULTILINE_PARSER]
        name          multilineKubeParser
        type          regex
        flush_timeout 1000           
        rule      &quot;start_state&quot;   &quot;/[^ ]* stdout .\s+\W*\w+\d\d\d\d-\d\d-\d\d \d\d\:\d\d\:\d\d,\d\d\d.*$/&quot;  &quot;cont&quot;
        rule      &quot;cont&quot;          &quot;/[^ ]* stdout .\s+(?!\W+\w+\d\d\d\d-\d\d-\d\d \d\d\:\d\d\:\d\d,\d\d\d).*$/&quot;   &quot;cont&quot;

  filters.lua: |-

    function remove_dummy(tag, timestamp, record)
       new_log=string.gsub(record[&quot;log&quot;],&quot;%d+-%d+-%d+T%d+:%d+:%d+.%d+Z%sstdout%sF%s&quot;,&quot;&quot;)
       new_record=record
       new_record[&quot;log&quot;]=new_log
       return 2, timestamp, new_record
    end
</code></pre>
<p>As I mentioned this is a workaround till I can find any other/better solution.</p>
