<p>I have an nginx 1.16.1 running in docker in vanilla K8s (bare metal).</p>

<p>Logs are forwarded to stdout by <code>ln -sf /dev/stdout /var/log/nginx/access.log</code> &amp; then are transferred to Elasticsearch <code>docker.elastic.co/elasticsearch/elasticsearch:7.2.0</code> by Fluentd <code>fluentd-kubernetes-daemonset:v1.10-debian-elasticsearch7-1</code></p>

<p>I have the following in my nginx.conf:</p>

<pre><code>http {
...
        log_format json_combined escape=json
          '{' 
            '"time_local":"$time_local",'
            '"remote_addr":"$remote_addr",'
            '"remote_user":"$remote_user",'
            '"request":"$request",'
            '"status": "$status",'
            '"body_bytes_sent":"$body_bytes_sent",'
            '"request_time":"$request_time",'
            '"http_referrer":"$http_referer",'
            '"http_user_agent":"$http_user_agent"'
          '}';

    access_log /var/log/nginx/access.log json_combined;
...
</code></pre>

<p>My fluentd uses configmap created from original kubernetes.conf by <code>kubectl create configmap fluentd-kubernetes-conf --from-file=kubernetes.conf --namespace=kube-system</code> (providing it fully, just in case):</p>

<pre><code># AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/kubernetes.conf.erb

&lt;label @FLUENT_LOG&gt;
  &lt;match fluent.**&gt;
    @type null
  &lt;/match&gt;
&lt;/label&gt;

&lt;source&gt;
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag "#{ENV['FLUENT_CONTAINER_TAIL_TAG'] || 'kubernetes.*'}"
  exclude_path "#{ENV['FLUENT_CONTAINER_TAIL_EXCLUDE_PATH'] || use_default}"
  read_from_head true
  &lt;parse&gt;
    @type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json'}"
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_minion
  path /var/log/salt/minion
  pos_file /var/log/fluentd-salt.pos
  tag salt
  &lt;parse&gt;
    @type regexp
    expression /^(?&lt;time&gt;[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?&lt;severity&gt;[^ \]]*) *\] (?&lt;message&gt;.*)$/
    time_format %Y-%m-%d %H:%M:%S
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_startupscript
  path /var/log/startupscript.log
  pos_file /var/log/fluentd-startupscript.log.pos
  tag startupscript
  &lt;parse&gt;
    @type syslog
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_docker
  path /var/log/docker.log
  pos_file /var/log/fluentd-docker.log.pos
  tag docker
  &lt;parse&gt;
    @type regexp
    expression /^time="(?&lt;time&gt;[^)]*)" level=(?&lt;severity&gt;[^ ]*) msg="(?&lt;message&gt;[^"]*)"( err="(?&lt;error&gt;[^"]*)")?( statusCode=($&lt;status_code&gt;\d+))?/
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_etcd
  path /var/log/etcd.log
  pos_file /var/log/fluentd-etcd.log.pos
  tag etcd
  &lt;parse&gt;
    @type none
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_kubelet
  multiline_flush_interval 5s
  path /var/log/kubelet.log
  pos_file /var/log/fluentd-kubelet.log.pos
  tag kubelet
  &lt;parse&gt;
    @type kubernetes
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_kube_proxy
  multiline_flush_interval 5s
  path /var/log/kube-proxy.log
  pos_file /var/log/fluentd-kube-proxy.log.pos
  tag kube-proxy
  &lt;parse&gt;
    @type kubernetes
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_kube_apiserver
  multiline_flush_interval 5s
  path /var/log/kube-apiserver.log
  pos_file /var/log/fluentd-kube-apiserver.log.pos
  tag kube-apiserver
  &lt;parse&gt;
    @type kubernetes
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_kube_controller_manager
  multiline_flush_interval 5s
  path /var/log/kube-controller-manager.log
  pos_file /var/log/fluentd-kube-controller-manager.log.pos
  tag kube-controller-manager
  &lt;parse&gt;
    @type kubernetes
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_kube_scheduler
  multiline_flush_interval 5s
  path /var/log/kube-scheduler.log
  pos_file /var/log/fluentd-kube-scheduler.log.pos
  tag kube-scheduler
  &lt;parse&gt;
    @type kubernetes
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_rescheduler
  multiline_flush_interval 5s
  path /var/log/rescheduler.log
  pos_file /var/log/fluentd-rescheduler.log.pos
  tag rescheduler
  &lt;parse&gt;
    @type kubernetes
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_glbc
  multiline_flush_interval 5s
  path /var/log/glbc.log
  pos_file /var/log/fluentd-glbc.log.pos
  tag glbc
  &lt;parse&gt;
    @type kubernetes
  &lt;/parse&gt;
&lt;/source&gt;

&lt;source&gt;
  @type tail
  @id in_tail_cluster_autoscaler
  multiline_flush_interval 5s
  path /var/log/cluster-autoscaler.log
  pos_file /var/log/fluentd-cluster-autoscaler.log.pos
  tag cluster-autoscaler
  &lt;parse&gt;
    @type kubernetes
  &lt;/parse&gt;
&lt;/source&gt;

# Example:
# 2017-02-09T00:15:57.992775796Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" ip="104.132.1.72" method="GET" user="kubecfg" as="&lt;self&gt;" asgroups="&lt;lookup&gt;" namespace="default" uri="/api/v1/namespaces/default/pods"
# 2017-02-09T00:15:57.993528822Z AUDIT: id="90c73c7c-97d6-4b65-9461-f94606ff825f" response="200"
&lt;source&gt;
  @type tail
  @id in_tail_kube_apiserver_audit
  multiline_flush_interval 5s
  path /var/log/kubernetes/kube-apiserver-audit.log
  pos_file /var/log/kube-apiserver-audit.log.pos
  tag kube-apiserver-audit
  &lt;parse&gt;
    @type multiline
    format_firstline /^\S+\s+AUDIT:/
    # Fields must be explicitly captured by name to be parsed into the record.
    # Fields may not always be present, and order may change, so this just looks
    # for a list of key="\"quoted\" value" pairs separated by spaces.
    # Unknown fields are ignored.
    # Note: We can't separate query/response lines as format1/format2 because
    #       they don't always come one after the other for a given query.
    format1 /^(?&lt;time&gt;\S+) AUDIT:(?: (?:id="(?&lt;id&gt;(?:[^"\\]|\\.)*)"|ip="(?&lt;ip&gt;(?:[^"\\]|\\.)*)"|method="(?&lt;method&gt;(?:[^"\\]|\\.)*)"|user="(?&lt;user&gt;(?:[^"\\]|\\.)*)"|groups="(?&lt;groups&gt;(?:[^"\\]|\\.)*)"|as="(?&lt;as&gt;(?:[^"\\]|\\.)*)"|asgroups="(?&lt;asgroups&gt;(?:[^"\\]|\\.)*)"|namespace="(?&lt;namespace&gt;(?:[^"\\]|\\.)*)"|uri="(?&lt;uri&gt;(?:[^"\\]|\\.)*)"|response="(?&lt;response&gt;(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
    time_format %Y-%m-%dT%T.%L%Z
  &lt;/parse&gt;
&lt;/source&gt;

&lt;filter kubernetes.**&gt;
  @type kubernetes_metadata
  @id filter_kube_metadata
  kubernetes_url "#{ENV['FLUENT_FILTER_KUBERNETES_URL'] || 'https://' + ENV.fetch('KUBERNETES_SERVICE_HOST') + ':' + ENV.fetch('KUBERNETES_SERVICE_PORT') + '/api'}"
  verify_ssl "#{ENV['KUBERNETES_VERIFY_SSL'] || true}"
  ca_file "#{ENV['KUBERNETES_CA_FILE']}"
&lt;/filter&gt;
</code></pre>

<p>Unfortunately I see the following in Kibana in JSON:</p>

<pre><code>  "_source": {
    "log": "{\"time_local\":\"25/Apr/2020:00:06:36 +0000\",\"remote_addr\":\"10.244.1.1\",\"remote_user\":\"\",\"request\":\"GET /health HTTP/1.1\",\"status\": \"200\",\"body_bytes_sent\":\"2\",\"request_time\":\"0.000\",\"http_referrer\":\"\",\"http_user_agent\":\"kube-probe/1.18\"}\n",
    "stream": "stdout",
</code></pre>

<p>(with "\" instead of properly parsed log).</p>

<p>The logs look ok in stdout <code>kubectl get logs podname</code> &amp; I've also tried to set <code>escape=none</code> in nginx.conf (no difference) so it looks like it's a fluentd's configuration issue.</p>

<p>Can you please instruct me on what exactly should I change/add in the fluentd's kubernetes.conf to get a properly parsed json <code>log</code>?</p>

<p>The "problem" indeed was in the fluentd config in <code>kubernetes.conf</code><a href="https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v0.12/debian-elasticsearch/conf/kubernetes.conf" rel="nofollow noreferrer">https://github.com/fluent/fluentd-kubernetes-daemonset...</a></p>

<p>I've changed <code>parse</code> type in <code>source</code> from <code>json</code> to <code>json_in_json</code> which allowed me to get all the logs parsed properly!</p>

<pre><code>&lt;source&gt;
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag "#{ENV['FLUENT_CONTAINER_TAIL_TAG'] || 'kubernetes.*'}"
  exclude_path "#{ENV['FLUENT_CONTAINER_TAIL_EXCLUDE_PATH'] || use_default}"
  read_from_head true
  &lt;parse&gt;
    @type "#{ENV['FLUENT_CONTAINER_TAIL_PARSER_TYPE'] || 'json_in_json'}"
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  &lt;/parse&gt;
&lt;/source&gt;
</code></pre>

<p><a href="https://github.com/alikhil/fluent-plugin-json-in-json/" rel="nofollow noreferrer">https://github.com/alikhil/fluent-plugin-json-in-json/</a></p>

<p>What DIDN'T help:
<code>key_name log</code>
<code>reserve_data true</code>
<code>hash_value_field log</code>
<a href="https://docs.fluentd.org/v/0.12/filter/parser" rel="nofollow noreferrer">https://docs.fluentd.org/v/0.12/filter/parser</a></p>
