<p>I have a sort of ELK stack, with fluentd instead of logstash, running as a DaemonSet on a Kubernetes cluster and sending all logs from all containers, in logstash format, to an Elasticsearch server.</p>

<p>Out of the many containers running on the Kubernetes cluster some are nginx containers which output logs of the following format:</p>

<pre><code>121.29.251.188 - [16/Feb/2017:09:31:35 +0000] host="subdomain.site.com" req="GET /data/schedule/update?date=2017-03-01&amp;type=monthly&amp;blocked=0 HTTP/1.1" status=200 body_bytes=4433 referer="https://subdomain.site.com/schedule/2589959/edit?location=23092&amp;return=monthly" user_agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0" time=0.130 hostname=webapp-3188232752-ly36o
</code></pre>

<p>The fields visible in Kibana are as per this screenshot:</p>

<p><a href="https://i.stack.imgur.com/rqk6K.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/rqk6K.png" alt="kibana nginx log"></a></p>

<p>Is it possible to extract fields from this type of log after it was indexed?</p>

<p>The fluentd collector is configured with the following source, which handles all containers, so enforcing a format at this stage is not possible due to the very different outputs from different containers:</p>

<pre><code>&lt;source&gt;
  type tail
  path /var/log/containers/*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.*
  format json
  read_from_head true
&lt;/source&gt;
</code></pre>

<p>In an ideal situation, I would like to enrich the fields visible in the screenshot above with the meta-fields in the "log" field, like "host", "req", "status" etc.</p>

<p>After a few days of research and getting accustomed to the <a href="https://github.com/so0k/docker-efk" rel="noreferrer">EFK stack</a>, I arrived to an EFK specific solution, as opposed to that in Darth_Vader's answer, which is only possible on the ELK stack.</p>

<p>So to summarize, I am using Fluentd instead of Logstash, so any grok solution would work if you also install the <a href="https://github.com/fluent/fluent-plugin-grok-parser" rel="noreferrer">Fluentd Grok Plugin</a>, which I decided not to do, because:</p>

<p>As it turns out, Fluentd has its own field extraction functionality through the use of <a href="http://docs.fluentd.org/v0.12/articles/parser-plugin-overview" rel="noreferrer">parser filters</a>. To solve the problem in my question, right before the <code>&lt;match **&gt;</code> line, so after the log line object was already enriched with kubernetes metadata fields and labels, I added the following:</p>

<pre><code>&lt;filter kubernetes.var.log.containers.webapp-**.log&gt;
  type parser
  key_name log
  reserve_data yes
  format /^(?&lt;ip&gt;[^-]*) - \[(?&lt;datetime&gt;[^\]]*)\] host="(?&lt;hostname&gt;[^"]*)" req="(?&lt;method&gt;[^ ]*) (?&lt;uri&gt;[^ ]*) (?&lt;http_version&gt;[^"]*)" status=(?&lt;status_code&gt;[^ ]*) body_bytes=(?&lt;body_bytes&gt;[^ ]*) referer="(?&lt;referer&gt;[^"]*)" user_agent="(?&lt;user_agent&gt;[^"]*)" time=(?&lt;req_time&gt;[^ ]*)/
&lt;/filter&gt;
</code></pre>

<p><strong>To explain:</strong></p>

<p><code>&lt;filter kubernetes.var.log.containers.webapp-**.log&gt;</code> - apply the block on all the lines matching this label; in my case the containers of the web server component are called webapp-{something}</p>

<p><code>type parser</code> - tells fluentd to apply a parser filter</p>

<p><code>key_name log</code> - apply the pattern only on the <code>log</code> property of the log line, not the whole line, which is a json string</p>

<p><code>reserve_data yes</code> - very important, if not specified the whole log line object is replaced by only the properties extracted from <code>format</code>, so if you already have other properties, like the ones added by the <code>kubernetes_metadata</code> filter, these are removed when not adding the <code>reserve_data</code> option</p>

<p><code>format</code> - a regex that is applied on the value of the <code>log</code> key to extract named properties</p>

<p><strong>Please note</strong> that I am using Fluentd 1.12, so this syntax is not fully compatible with the newer 1.14 syntax, but the principle will work with minor tweaks to the parser declaration.</p>
