<p>I have a kubernetes cluster with few microservices . The microservices are logging/exporting the logs as JSON format. So my ELK stack is logging messages and visulaize in kibana (I am using version 7.3) and I am struggling to further de-serialize the message filed coming from my microservices, so that I can see individual fields further extracted in "message" field. For example, the JSON has "app" and "logger" fields - these should appear similarly how it displays kubernetes fields currently. I have configured the filebeat as follows but I cant see the further de-serialize the message filed. Could you help me on this. please refer my File beat configuration. </p>

<p>=======================</p>

<pre><code>kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    app: filebeat
data:
  filebeat.yml: |-

    # Enable filebeat config reloading
    filebeat.config:
      modules:
        enabled: true
        #path: ${path.config}/modules.d/*.yml
        reload.enabled: true
        reload.period: 10s

    # Available log levels are: error, warning, info, debug
    logging.level: warning

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          json.message_key: message
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
          templates:
            config:
              json.keys_under_root: true
              json.add_error_key: false

    processors:
      - drop_event:
          when:
              or:
                - equals:
                    kubernetes.namespace: "monitoring"
                - contains:
                    message: "NETWORK"
                - contains:
                    message: "INFO"
      - add_cloud_metadata:
      - add_kubernetes_metadata:
      - add_host_metadata:
</code></pre>

Dileeka Fernando
<p>I see you want to use filebeat.yml for getting the JSON decoding. This might be the approach you wanna use. I sent this as a separate answer because this is a completely different approach.</p>

<pre><code>filebeat.inputs:
- type: log

  # Change to true to enable this prospector configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - \PATH\TO\LOG\FILES\*

  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: message
</code></pre>

<blockquote>
  <p>PS: Please be attentive to the proper indentation for the json
  parameters.</p>
</blockquote>
