<p>I currently have this nginx log output.</p>
<pre><code>      log_format json_logs escape=json
                            '{'
                            '&quot;time_local&quot;:&quot;$time_local&quot;,'
                            '&quot;remote_addr&quot;:&quot;$remote_addr&quot;,'
                            '&quot;remote_user&quot;:&quot;$remote_user&quot;,'
                            '&quot;request&quot;:&quot;$request&quot;,'
                            '&quot;status&quot;: &quot;$status&quot;,'
                            '&quot;body_bytes_sent&quot;:&quot;$body_bytes_sent&quot;,'
                            '&quot;request_time&quot;:&quot;$request_time&quot;,'
                            '&quot;http_referrer&quot;:&quot;$http_referer&quot;,'
                            '&quot;http_user_agent&quot;:&quot;$http_user_agent&quot;'
                            '}';
      access_log /var/log/nginx/access.log json_logs;
</code></pre>
<p>However, when outputted and collected by Fluentd it is prefixed with the timestamp and stdout.</p>
<p>For example..</p>
<pre><code>2022-06-18T19:05:15.014296769Z stdout F {\&quot;time_local\&quot;:\&quot;18/Jun/2022:19:05:15       +0000\&quot;,\&quot;remote_addr\&quot;:\&quot;10.106.0.5\&quot;,\&quot;remote_user\&quot;:\&quot;\&quot;,\&quot;request\&quot;:\&quot;GET /       HTTP/1.1\&quot;,\&quot;status\&quot;:       \&quot;304\&quot;,\&quot;body_bytes_sent\&quot;:\&quot;0\&quot;,\&quot;request_time\&quot;:\&quot;0.000\&quot;,\&quot;http_referrer\&quot;:\&quot;\&quot;,\&quot;htt      p_user_agent\&quot;:\&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36    (KHTML, like Gecko) Chrome/102.0.5005.99 Safari/537.36\&quot;}
</code></pre>
<p>I can't parse it correctly, i temporarily set it to</p>
<pre><code> &lt;source&gt;
      @type tail
      path /var/log/containers/*nginx*.log
      pos_file /var/log/nginx.log.pos
      tag nginx.access
      &lt;parse&gt;
        @type nginx
        expression ^(?&lt;somenginxstuff&gt;.*)$
        time_key logtime
        time_format %d/%b/%Y:%H:%M:%S.%z
      &lt;/parse&gt;
      
    &lt;/source&gt;
</code></pre>
<p>to dump it all in elastic/kibana so i can check the outputs.</p>
<p>Questions is - what is the best/easiest way to do this? I assume it would be very common usecase?</p>
<p>Also, i've seen mention of plugins and i'm using the base <code>fluent/fluentd-kubernetes-daemonset:v1.14.6-debian-elasticsearch7-1.0</code> image. How do i add these (if they help)?</p>
<p>Many thanks in advance</p>

Matt The Ninja
<p>I ended up doing this by parsing it with json then filtering the field to parse as json like follows.</p>
<p>For following log output...</p>
<pre><code>2022-06-18T19:05:15.014296769Z stdout F {\&quot;time_local\&quot;:\&quot;18/Jun/2022:19:05:15       +0000\&quot;,\&quot;remote_addr\&quot;:\&quot;10.106.0.5\&quot;,\&quot;remote_user\&quot;:\&quot;\&quot;,\&quot;request\&quot;:\&quot;GET /       HTTP/1.1\&quot;,\&quot;status\&quot;:       \&quot;304\&quot;,\&quot;body_bytes_sent\&quot;:\&quot;0\&quot;,\&quot;request_time\&quot;:\&quot;0.000\&quot;,\&quot;http_referrer\&quot;:\&quot;\&quot;,\&quot;htt      p_user_agent\&quot;:\&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36    (KHTML, like Gecko) Chrome/102.0.5005.99 Safari/537.36\&quot;}
</code></pre>
<p>And this configuration</p>
<pre><code>&lt;source&gt;
  @type tail
  path /var/log/containers/*nginx*.log
  pos_file /var/log/nginx.log.pos
  tag nginx.access
  &lt;parse&gt;
    @type regexp
    expression ^(?&lt;timestamp&gt;[^ ]*) [^ ]*[ ][^ ] (?&lt;data&gt;.*).*$
   &lt;/parse&gt;
&lt;/source&gt;

 &lt;filter nginx.access&gt;
  @type parser
  key_name data
  &lt;parse&gt;
    @type json
  &lt;/parse&gt;
&lt;/filter&gt;
</code></pre>
