<p>I'm working with Filebeat 7.9.3 as a daemonset on k8s.
I'm not able to parse docker container logs of a Springboot app that writes logs to stdout in json.
The fact is that the every row of the Springboot app logs is written in this way:</p>
<pre><code>{ &quot;@timestamp&quot;: &quot;2020-11-16T13:39:57.760Z&quot;, &quot;log.level&quot;: &quot;INFO&quot;, &quot;message&quot;: &quot;Checking comment 'se' done = true&quot;, &quot;service.name&quot;: &quot;conduit-be-moderator&quot;, &quot;event.dataset&quot;: &quot;conduit-be-moderator.log&quot;, &quot;process.thread.name&quot;: &quot;http-nio-8081-exec-2&quot;, &quot;log.logger&quot;: &quot;it.koopa.app.ModeratorController&quot;, &quot;transaction.id&quot;: &quot;1ed5c62964ff0cc2&quot;, &quot;trace.id&quot;: &quot;20b4b28a3817c9494a91de8720522972&quot;} 
</code></pre>
<p>But the corresponding docker log file under /var/log/containers/ writes log in this way:</p>
<pre><code>{
&quot;log&quot;: &quot;{\&quot;@timestamp\&quot;:\&quot;2020-11-16T11:27:32.273Z\&quot;, \&quot;log.level\&quot;: \&quot;INFO\&quot;, \&quot;message\&quot;:\&quot;Checking comment 'a'\&quot;, \&quot;service.name\&quot;:\&quot;conduit-be-moderator\&quot;,\&quot;event.dataset\&quot;:\&quot;conduit-be-moderator.log\&quot;,\&quot;process.thread.name\&quot;:\&quot;http-nio-8081-exec-4\&quot;,\&quot;log.logger\&quot;:\&quot;it.koopa.app.ModeratorController\&quot;,\&quot;transaction.id\&quot;:\&quot;9d3ad972dba65117\&quot;,\&quot;trace.id\&quot;:\&quot;8373edba92808d5e838e07c7f34af6c7\&quot;}\n&quot;,
&quot;stream&quot;: &quot;stdout&quot;,
&quot;time&quot;: &quot;2020-11-16T11:27:32.274816903Z&quot; 
} 
</code></pre>
<p>I always receive this on filebeat logs</p>
<pre><code>Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
</code></pre>
<p>This is my filebeat config that tries to parse json log message from docker logs where I'm using decode_json_fields to try to catch Elasticsearch standard fields (I'm using <a href="https://github.com/elastic/ecs-logging-java/blob/master/logback-ecs-encoder/README.md" rel="nofollow noreferrer">co.elastic.logging.logback.EcsEncoder</a>)</p>
<pre><code>filebeat.yml: |-
filebeat.inputs:
- type: container

  #json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.message_key: log

  paths:
    - /var/log/containers/*.log

  include_lines: &quot;conduit-be-moderator&quot;

  processors:
    - decode_json_fields:
        fields: [&quot;log&quot;]
        overwrite_keys: true
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        in_cluster: true
        matchers:
          - logs_path:
              logs_path: &quot;/var/log/containers/&quot;

processors:
  - add_cloud_metadata:
  - add_host_metadata:
</code></pre>
<p>How can I do this???</p>

<p>As processors are applied before the JSON parser of the input, you will need to first configure the <code>decode_json_fields</code> processors which will allow you to decode your json.log field. You will then be able to apply the json configuration fo the inputs on the <code>message</code> fields. Something like:</p>
<pre class="lang-yaml prettyprint-override"><code>filebeat.yml: |-
filebeat.inputs:
- type: container

  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.message_key: message

  paths:
    - /var/log/containers/*.log

  include_lines: &quot;conduit-be-moderator&quot;

  processors:
    - decode_json_fields:
        fields: ['log']
        expand_keys: true
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        in_cluster: true
        matchers:
          - logs_path:
              logs_path: &quot;/var/log/containers/&quot;

processors:
  - add_cloud_metadata:
  - add_host_metadata:
</code></pre>
<p>This configuration assumes that all your logs use JSON format. Else you will probably need to add an exclude or include regex pattern.</p>
