<p>I have a service, deployed into a kubernetes cluster, with fluentd set as a daemon set. And i need to diversify logs it receives so they end up in different s3 buckets. 
One bucket would be for all logs, generated by kubernetes and our debug/error handling code, and another bucket would be a subset of logs, generated by the service, parsed by structured logger and identified by a specific field in json. Think of it one bucket is for machine state and errors, another is for "user_id created resource image_id at ts" description of user actions</p>

<p>The service itself is ignorant of the fluentd, so i cannot manually set the tag for logs based on which s3 bucket i want them to end in.
Now, the fluentd.conf i use sets s3 stuff like this:</p>

<pre><code>&lt;match **&gt;
  # docs: https://docs.fluentd.org/v0.12/articles/out_s3
  # note: this configuration relies on the nodes have an IAM instance profile with access to your S3 bucket
  type copy
  &lt;store&gt;
    type s3
    log_level info
    s3_bucket "#{ENV['S3_BUCKET_NAME']}"
    s3_region "#{ENV['S3_BUCKET_REGION']}"
    aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}"
    aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
    s3_object_key_format %{path}%{time_slice}/cluster-log-%{index}.%{file_extension}
    format json
    time_slice_format %Y/%m/%d
    time_slice_wait 1m
    flush_interval 10m
    utc
    include_time_key true
    include_tag_key true
    buffer_chunk_limit 128m
    buffer_path /var/log/fluentd-buffers/s3.buffer
  &lt;/store&gt;
  &lt;store&gt;
  ...
  &lt;/store&gt;
&lt;/match&gt;
</code></pre>

<p>So, what i would like to do is to have something like a grep plugin</p>

<pre><code>&lt;store&gt;
  type grep
  &lt;regexp&gt;
    key type
    pattern client-action
  &lt;/regexp&gt;
&lt;/store&gt;
</code></pre>

<p>Which would send logs into a separate s3 bucket to the one defined for all logs</p>

dgmt
<p>I am assuming that user action logs are generated by your service and system logs include docker, kubernetes and systemd logs from the nodes.
I found your example yaml file at the <a href="https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.2/debian-s3/conf/fluent.conf" rel="nofollow noreferrer">official fluent github repo</a>.
If you check out the folder in that link, you'll see two more files called <code>kubernetes.conf</code> and <code>systemd.conf</code>. These files have got <code>source</code> sections where they <code>tag</code> their data.</p>

<p>The <code>match</code> section in fluent.conf is matching <code>**</code>, i.e. all logs and sending them to s3. You want to split your log types here.
Your container logs are being tagged <code>kubernetes.*</code> in <a href="https://github.com/fluent/fluentd-kubernetes-daemonset/blob/4c01c443f8cfefba17d683fa10501ca775943578/docker-image/v1.2/debian-s3/conf/kubernetes.conf#L13" rel="nofollow noreferrer">kubernetes.conf on this line</a>.</p>

<p>so your above config turns into</p>

<pre><code>&lt;match kubernetes.* &gt;
@type s3
# user log s3 bucket
...
</code></pre>

<p>and for system logs <code>match</code> every other tag except kubernetes.*</p>
