<p>I have configured a <code>fluentd</code> sidecar container to send logs to <code>kafka</code>.</p>
<p>Here is my <code>fluent.conf</code> file:</p>
<pre><code>&lt;source&gt;
  @type forward
  bind 127.0.0.1
  port 24224
&lt;/source&gt;

&lt;match log.default&gt;
@type kafka2

  # list of seed brokers
  brokers node1:9092, node2:9092, node3:9092
  use_event_time true

  # buffer settings
  buffer_type file
  buffer_path /var/log/td-agent/buffer/kafka
  flush_interval 3s

  # topic settings
  default_topic app_logs

  # data type settings
  &lt;format&gt;
    @type json
  &lt;/format&gt;

  # producer settings
  max_send_retries 1
  required_acks -1

  #deubg log
  @log_level debug
&lt;/match&gt;
</code></pre>
<p>The broker hosts are reachable from the pod where the <code>fluentd</code> sidecar container is running.
However, the sidecar container is not able to connect to the brokers. It returns the following log:</p>
<pre><code>2020-10-23 11:41:51 +0000 [info]: starting fluentd-1.10.4 pid=8 ruby=&quot;2.5.8&quot;
2020-10-23 11:41:51 +0000 [info]: spawn command to main:  cmdline=[&quot;/usr/bin/ruby&quot;, &quot;-Eascii-8bit:ascii-8bit&quot;, &quot;/usr/bin/fluentd&quot;, &quot;-c&quot;, &quot;/fluentd/etc/fluent.conf&quot;, &quot;-p&quot;, &quot;/fluentd/plugins&quot;, &quot;--under-supervisor&quot;]
2020-10-23 11:41:55 +0000 [info]: adding match pattern=&quot;log.default&quot; type=&quot;kafka2&quot;
2020-10-23 11:41:56 +0000 [info]: #0 brokers has been set: [&quot;node1:9092&quot;, &quot;node2:9092&quot;, &quot;node3:9092&quot;]
2020-10-23 11:41:56 +0000 [info]: adding source type=&quot;forward&quot;
2020-10-23 11:41:56 +0000 [info]: #0 starting fluentd worker pid=22 ppid=8 worker=0
2020-10-23 11:41:56 +0000 [debug]: #0 buffer started instance=47608408649900 stage_size=0 queue_size=0
2020-10-23 11:41:56 +0000 [info]: #0 initialized kafka producer: fluentd
2020-10-23 11:41:56 +0000 [info]: #0 listening port port=24224 bind=&quot;127.0.0.1&quot;
2020-10-23 11:41:56 +0000 [info]: #0 fluentd worker is now running worker=0
2020-10-23 11:41:57 +0000 [debug]: #0 flush_thread actually running
2020-10-23 11:41:57 +0000 [debug]: #0 enqueue_thread actually running
2020-10-23 11:43:02 +0000 [warn]: #0 Send exception occurred: Could not connect to any of the seed brokers:
- kafka://node1:9092: Connection error Errno::ECONNRESET: Connection reset by peer
- kafka://node2:9092: Connection error Errno::ECONNRESET: Connection reset by peer
- kafka://node3:9092: Connection error Errno::ECONNRESET: Connection reset by peer
2020-10-23 11:43:02 +0000 [warn]: #0 Exception Backtrace : /usr/lib/ruby/gems/2.5.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:448:in `fetch_cluster_info'
</code></pre>
<p>How to get rid of this issue? Could it be an issue from the broker nodes side?</p>

<p>From the fluentd host, try using the <a href="https://github.com/edenhill/kafkacat" rel="nofollow noreferrer">https://github.com/edenhill/kafkacat</a> to debug Kafka connectivity/access. I'm sure this is a network/firewall issue, not the fluentd.</p>
