title: How to make fluentd send logs faster to Elasticsearch?
body: <p>I've setup some docker containers, where the fluentd container send JSON formatted logs to an Elasticsearch container (which in turn are read by Kibana). I've setup fluentd to receive logs via UDP, since our apps are logging by sending UDP messages. It works, but for some reason, there's a big delay and fluentd logs only reach ES after 5 minutes. </p>
  
  <p>My fluentd config is:</p>
  
  <pre><code>&lt;system&gt;
    log_level debug
  &lt;/system&gt;
  
  &lt;source&gt;
    @type udp
    @label @udp_stream
    tag ma.udp_events
    &lt;parse&gt;
      @type json
      time_key $.log.@timestamp
      keep_time_key true
    &lt;/parse&gt;
    port 20001
    bind 0.0.0.0
    message_length_limit 1MB
    source_hostname_key client_host
    source_address_key  client_addr
  &lt;/source&gt;
  
  &lt;label @udp_stream&gt;
    &lt;match **&gt;
      @type copy
      &lt;store&gt;
        @type elasticsearch
        host elastic
        port 9200
        index_name logs.${app_environment}.${app_category}.${app_area}.${app_name}
        include_tag_key true
        reload_connections true
        reconnect_on_error false
        reload_on_failure false
        &lt;buffer tag, app_environment, app_category, app_area, app_name&gt;
          flush_at_shutdown true
          flush_mode immediate
          flush_thread_count 8
          flush_thread_interval 1
          flush_thread_burst_interval 1
          retry_forever true
          retry_type exponential_backoff
          retry_max_interval 30
          retry_wait 1
          chunk_limit_size 1K
          queue_limit_length 8
        &lt;/buffer&gt;
      &lt;/store&gt;
    &lt;/match&gt;
  &lt;/label&gt;
  </code></pre>
  
  <p>Apart from these settings, there's the same source for TCP, and another source for tail. I've tried removing them, but the problem seems to be some other thing. I've read about this 5 minutes delay regarding buffers, but I hoped that a more detailed flush policy would help.</p>
  
  <p>Logs are only 1Kb to 2Kb, and I'm testing by manually sending logs, so there's no overhead there, I just send a few per minute.
  I'd like fluentd flushing its buffers as soon as possible. After some research on SO and some blogs, I've tweaked the buffer section, to no avail. For sure I'm making a mistake, but I cannot find it. Any help would be much appreciated.</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>It seems that I cannot get logs from fluentd to ES in less than 5 mins. However, I tried removing the buffer section, expecting it to be faster. While I see fluentd logs about flushing and purging chunks, somehow they're not immediately delivered. My testing configuration is a bit different than the original post, because I've set a new key on the JSON (which I've called "@es_index_name"), and also removed the buffer section.</p>
  
  <p>I've read <a href="https://support.treasuredata.com/hc/en-us/articles/360000687108-Overview-of-Server-Side-Agent-td-agent-" rel="nofollow noreferrer">here</a>:</p>
  
  <p>"td-agent continuously uploads logs every 5 minutes. You can force td-agent to flush the buffered logs into the cloud by sending a SIGUSR1 signal."</p>
  
  <p>In any case, my current testing configuration is:</p>
  
  <pre><code>&lt;label @udp_stream&gt;
    &lt;match **&gt;
      @type copy
      &lt;store&gt;
        @type elasticsearch
        host elastic
        port 9200
        prefer_oj_serializer true
        target_index_key @es_index_name
        include_tag_key true
        reload_connections true
        reconnect_on_error false
        reload_on_failure false
        flush_interval 5s
      &lt;/store&gt;
      &lt;store&gt;
        @type stdout
      &lt;/store&gt;
    &lt;/match&gt;
  &lt;/label&gt;
  </code></pre>
  
  <p>Apart from the lines I've removed while trying to simplify, the important line here is:</p>
  
  <p><strong>target_index_key @es_index_name</strong></p>
  
  <p><strong>EDIT 2:</strong> For some reason, after I've deployed this container as a Kubernetes workload, logs began to be flushed as expected. I don't know why this is happening, but in the end, I wanted this to run like this. Now one of our apps send an UDP log to our Fluentd, and it takes just a few seconds for Kibana to show it in its dashboards.</p>