title: How to extract a json and write them to a specific file using fluentd?
body: <p>I have an application that writes logs to the container log files. I want to use fluentd to filter these logs based on a string and write them to a file.</p>
  <p>A single line from the container logs look like:</p>
  <pre class="lang-json prettyprint-override"><code>{
    &quot;stream&quot;: &quot;stdout&quot;,
    &quot;time&quot;: &quot;2022-01-21T07:54:18.14263612Z&quot;,
    &quot;log&quot;: &quot;{\&quot;MATCH_TEXT\&quot;:{\&quot;id\&quot;:\&quot;001\&quot;,\&quot;timestamp\&quot;:1642751532000,\&quot;Details\&quot;:{\&quot;key1\&quot;:{\&quot;key2\&quot;:\&quot;val2\&quot;,\&quot;key-3\&quot;:\&quot;val-3\&quot;,\&quot;status\&quot;:\&quot;done\&quot;}}}}&quot;,
    &quot;docker&quot;: {
      &quot;container_id&quot;: &quot;container_id&quot;
    },
    &quot;kubernetes&quot;: {
      &quot;container_name&quot;: &quot;test-container&quot;,
      &quot;namespace_name&quot;: &quot;test-ns&quot;,
      &quot;pod_name&quot;: &quot;test-pod&quot;,
      &quot;container_image&quot;: &quot;test-container_image:tag&quot;,
      &quot;pod_id&quot;: &quot;POD_ID&quot;,
      &quot;pod_ip&quot;: &quot;1.1.1.1&quot;,
      &quot;host&quot;: &quot;host&quot;
    }
  }
  </code></pre>
  <p>I am trying to find all logs containing the text <code>MATCH_TEXT</code> and log them to a file without the key <code>MATCH_TEXT</code>.</p>
  <p>So my expected output becomes:</p>
  <pre class="lang-json prettyprint-override"><code>{\&quot;id\&quot;:\&quot;001\&quot;,\&quot;timestamp\&quot;:1642751532000,\&quot;Details\&quot;:{\&quot;key1\&quot;:{\&quot;key2\&quot;:\&quot;val2\&quot;,\&quot;key-3\&quot;:\&quot;val-3\&quot;,\&quot;status\&quot;:\&quot;done\&quot;}}}
  </code></pre>
  <p>So far I was able to add filters to bring it down to this.</p>
  <pre class="lang-xml prettyprint-override"><code># this filter matches logs containing the text MATCH_TEXT and tags them to new tag required_logs
   &lt;match logs&gt;
    @type copy
    &lt;store&gt;
      @type rewrite_tag_filter
      &lt;rule&gt;
        key $.log
        pattern MATCH_TEXT
        tag required_events
      &lt;/rule&gt;
    &lt;/store&gt;
  &lt;/match&gt; 
  
  #this filter removes all the fields expect log field.
  &lt;filter required_events&gt;
    @type record_transformer
    renew_record true
    keep_keys log
  &lt;/filter&gt;
  
  #this filter removes the log key from the message
  &lt;filter required_events&gt;
    @type parser
    key_name $.log
    reserve_data true
    remove_key_name_field true
    &lt;parse&gt;
      @type json
    &lt;/parse&gt;
  &lt;/filter&gt;
  
  # now were left with message:
  # {&quot;MATCH_TEXT&quot;:{&quot;id&quot;:&quot;001&quot;,&quot;timestamp&quot;:1642751532000,&quot;Details&quot;:{&quot;key1&quot;:{&quot;key2&quot;:&quot;val2&quot;,&quot;key-3&quot;:&quot;val-3&quot;,&quot;status&quot;:&quot;done&quot;}}}}
  </code></pre>
  <p>This is where I start getting problems. I tried to further filter the message using filter:</p>
  <pre class="lang-xml prettyprint-override"><code>&lt;filter required_events&gt;
    @type parser
    key_name $.MATCH_TEXT
    reserve_data true
    remove_key_name_field true
    &lt;parse&gt;
      @type json
    &lt;/parse&gt;
  &lt;/filter&gt;
  </code></pre>
  <p>But this makes fluentd give me an error:</p>
  <pre><code>2022-01-24 08:35:20 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=&quot;pattern not matched with data '{\&quot;MATCH_TEXT\&quot;:{\&quot;id\&quot;:\&quot;001\&quot;,\&quot;timestamp\&quot;:1642751532000,\&quot;Details\&quot;:{\&quot;key1\&quot;:{\&quot;key2\&quot;:\&quot;val2\&quot;,\&quot;key-3\&quot;:\&quot;val-3\&quot;,\&quot;status\&quot;:\&quot;done\&quot;}}}}&quot;'&quot; location=nil tag=&quot;required_events&quot; time=2022-01-24 08:35:20.955622000 +0000 record={&quot;MATCH_TEXT&quot;=&gt;{&quot;id&quot;:&quot;001&quot;,&quot;timestamp&quot;:1642751532000,&quot;Details&quot;:{&quot;key1&quot;:{&quot;key2&quot;:&quot;val2&quot;,&quot;key-3&quot;:&quot;val-3&quot;,&quot;status&quot;:&quot;done&quot;}}}}
  </code></pre>
  <p>I am new to fluentd. Please help me figure out what I'm doing wrong here. I would like to do it without installing any new plugin if possible. Thanks!</p>