<p>I'm using Fluentd with Elasticsearch for logs from Kubernetes but I noticed that some JSON logs cannot be correctly indexed because JSON is stored as string.</p>

<p>Logs from kubectl logs  look like:</p>

<pre><code>{"timestamp":"2016-11-03T15:48:12.007Z","level":"INFO","thread":"cromwell-system-akka.actor.default-dispatcher-4","logger":"akka.event.slf4j.Slf4jLogger","message":"Slf4jLogger started","context":"default"}
</code></pre>

<p>But logs saved in file in /var/log/containers/... have escaped quotes and makes them string instead of JSON which spoil indexing:</p>

<pre><code>{"log":"{\"timestamp\":\"2016-11-03T15:45:07.976Z\",\"level\":\"INFO\",\"thread\":\"cromwell-system-akka.actor.default-dispatcher-4\",\"logger\":\"akka.event.slf4j.Slf4jLogger\",\"message\":\"Slf4jLogger started\",\"context\":\"default\"}\n","stream":"stdout","time":"2016-11-03T15:45:07.995443479Z"}
</code></pre>

<p>I'm trying to get logs looking like:</p>

<pre><code>{
    "log": {
        "timestamp": "2016-11-03T15:45:07.976Z",
        "level": "INFO",
        "thread": "cromwell-system-akka.actor.default-dispatcher-4",
        "logger": "akka.event.slf4j.Slf4jLogger",
        "message": "Slf4jLogger started",
        "context": "default"
    },
    "stream": "stdout",
    "time": "2016-11-03T15: 45: 07.995443479Z"
}
</code></pre>

<p>Can you suggest me how to do it?</p>

<p>I ran into the same issue, however I'm using <code>fluent-bit</code>, the "C" version of <code>fluentd</code> (Ruby). Since this is an older issue, I'm answering for the benefit of others who find this.</p>

<p>In <code>fluent-bit</code> v0.13, they addressed this issue. You can now specify the parser to use through annotations. The parser can be configured to decode the log as json.</p>

<ul>
<li><a href="https://github.com/fluent/fluent-bit/issues/615" rel="nofollow noreferrer">fluent-bit issue detailing problem</a></li>
<li><a href="https://www.linux.com/blog/event/kubecon/2018/4/fluent-bit-flexible-logging-kubernetes" rel="nofollow noreferrer">blog post about annotations for specifying the parser</a></li>
<li><a href="https://fluentbit.io/documentation/0.14/parser/json.html" rel="nofollow noreferrer">json parser documentation</a> - The docker container's logs come out as json. However, your logs are <em>also</em> json. So an <a href="https://fluentbit.io/documentation/0.14/parser/decoder.html" rel="nofollow noreferrer">additional decoder</a> is needed.</li>
</ul>

<p>The final parser with decoder looks like this:</p>

<pre><code>[PARSER]
    Name        embedded-json
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On
    # Command      |  Decoder         | Field | Optional Action
    # =============|==================|=======|=========
    Decode_Field_As   escaped           log     do_next
    Decode_Field_As   json              log
</code></pre>
