title: Spring for Apache Kafka: KafkaTemplate Behavior with Async Requests, Batching, and Max In-flight of 1
body: <p><strong>Scenario/Use Case:</strong>
  I have a Spring Boot application using Spring for Kafka to send messages to Kafka topics.  Upon completion of a specific event (triggered by http request) a new thread is created (via Spring @Async) which calls kafkatemplate.send() and has a callback for the ListenableFuture that it returns.  The original thread which handled the http request returns a response to the calling client and is freed.</p>
  
  <p><strong>Normal Behavior:</strong>
  Under normal application load I've verified that the individual messages are all published to the topic as desired (application log entries upon callback success or failure as well as viewing the message in the topic on the kafka cluster).  If I bring down all kafka brokers for 3-5 minutes and then bring the cluster back online the application's publisher immediately re-establishes connection to kafka and proceeds with publishing messages.</p>
  
  <p><strong>Problem Behavior:</strong>
  However, when performing load testing, if I bring down all kafka brokers for 3-5 minutes and then bring the cluster back online, the Spring application's publisher continues to show failures for all publish attempts.  This continues for approximately 7 hours at which time the publisher is able to successfully re-establish communication with kafka again (usually this is preceeded by a broken pipe exception but not always).</p>
  
  <p><strong>Current Findings:</strong>
  While performing the load test, for approx. 10 minutes, I connected to the the application using JConsole and monitored the <a href="https://kafka.apache.org/documentation/#producer_monitoring" rel="nofollow noreferrer">producer metrics</a> exposed via kafka.producer.  Within the first approx. 30 seconds of heavy load, <strong>buffer-available-bytes</strong> continues to decrease until it reaches 0 and stays at 0.  <strong>waiting-threads</strong> remains between 6 and 10 (alternates everytime I hit refresh) and buffer-available-bytes remains at 0 for approx. 6.5 hours.  After that buffer-available-bytes shows all of the originally allocated memory restored but kafka publish attempts continue failing for approx. another 30 minutes before finally the desired behavior restores.</p>
  
  <p><strong>Current Producer Configuration</strong></p>
  
  <pre><code>request.timeout.ms=3000
  max.retry.count=2
  max.inflight.requests=1
  max.block.ms=10000
  retry.backoff.ms=3000
  </code></pre>
  
  <p>All other properties are using <a href="https://kafka.apache.org/documentation/#producerconfigs" rel="nofollow noreferrer">their default values</a></p>
  
  <p><strong>Questions:</strong></p>
  
  <ol>
  <li>Given my use case would altering <strong>batch.size</strong> or <strong>linger.ms</strong> have any positive impact in terms of eliminating the issue encountered when under heavy load?</li>
  <li>Given that I have separate threads all calling kafkatemplate.send() with separate messages and callbacks and I have<strong>max.in.flight.requests.per.connection</strong> set to 1, are batch.size and linger.ms ignored beyond limiting the size of each message?  My understanding is that no batching is actually occurring in this scenario and that each message is sent as a separate request.</li>
  <li>Given that I have max.block.ms set to 10 seconds, why does buffer memory remain utilized for so long and why do all messages continue to fail to be published for so many hours.  My understanding is that after 10 seconds each new publish attempt should fail and return the failure callback which in turn frees up the associated thread</li>
  </ol>
  
  <p><strong>Update:</strong>
  To try and clarify thread usage.  I'm using the single producer instance as recommended in the JavaDocs.  There are threads such as https-jsse-nio-22443-exec-* which are handling incoming https requests. When a request comes in some processing occurs and once all non-kafka related logic completes a call is made to a method in another class decorated with @Async.  This method makes the call to kafkatemplate.send().  The response back to the client is shown in the logs before the publish to kafka is performed (this is how Im verifying its being performed via separate thread as the service doesn't wait to publish before returning a response).
  There are task-scheduler-* threads which appear to be handling the callbacks from kafkatemplate.send().  My guess is that the single kafka-producer-network-thread handles all of the publishing.</p>