<p><strong>Problem</strong></p>

<p>I have a Kafka setup with three brokers in Kubernetes, set up according to the guide at <a href="https://github.com/Yolean/kubernetes-kafka" rel="nofollow noreferrer">https://github.com/Yolean/kubernetes-kafka</a>. The following error message appears when producing messages from a Java client.</p>

<pre><code>2018-06-06 11:15:44.103 ERROR 1 --- [ad | producer-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='[...redacted...]':
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for topicname-0: 30001 ms has passed since last append
</code></pre>

<p><strong>Detailed setup</strong></p>

<p>The listeners are set up to allow SSL producers/consumers from the outside world:</p>

<pre><code>advertised.host.name = null
advertised.listeners = OUTSIDE://kafka-0.mydomain.com:32400,PLAINTEXT://:9092
advertised.port = null
listener.security.protocol.map = PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,OUTSIDE:SSL
listeners = OUTSIDE://:9094,PLAINTEXT://:9092
inter.broker.listener.name = PLAINTEXT
host.name =
port.name = 9092
</code></pre>

<p>The OUTSIDE listeners are listening on kafka-0.mydomain.com, kafka-1.mydomain.com, etc. The plaintext listeners are listening on any IP, since they are cluster-local to Kubernetes. </p>

<p>The producer settings:</p>

<pre><code>kafka:
  bootstrap-servers: kafka.mydomain.com:9092
  properties:
    security.protocol: SSL
   producer:
    batch-size: 16384
    buffer-memory: 1048576 # 1MB
    retries: 1
    ssl:
      key-password: redacted
      keystore-location: file:/var/private/ssl/kafka.client.keystore.jks
      keystore-password: redacted
      truststore-location: file:/var/private/ssl/kafka.client.truststore.jks
      truststore-password: redacted
</code></pre>

<p>In addition I set <code>linger.ms</code> to 100 in code, which forces messages to be transmitted within 100ms. Linger time is set intentionally low, because the use case requires minimal delays.</p>

<p><strong>Analysis</strong></p>

<ul>
<li>The errors started appearing when the broker was moved moved to SSL. </li>
<li>On the server side everything is running as expected, there are no errors in the log and I can connect to the broker manually with a Kafka client tool. </li>
<li>The errors appear intermittently: sometimes it sends 30+ messages per second, sometimes it sends nothing at all. It may work like a charm for hours and then just spam timeouts for a little while.</li>
<li>Clocks for the client and server are in sync (UTC).</li>
<li>CPU is consistently around 20% for both the producing and server side.</li>
</ul>

<p>What could it be?</p>

<p>This problem normally occurs when the producer is faster than the brokers, the reason why this happens with your setup seems to be that the SSL needs extra CPU and that may slow down the brokers. But anyway check the following:</p>

<ul>
<li>Check if you are producing message at the same speed, according what you are saying seems that you are having spikes.</li>
<li>Another possibility is that other kafka clients in the cluster (producer or consumers), which not necessarily uses the same topic, makes this to happen because overloads the brokers (check brokers cpu/network).</li>
</ul>

<p>To minimize whatever causes this retention you should increase the <code>buffer-memory</code> to more than 32MB, think that 32MB is the default and you are setting this lower. The lower you have, the easy is that the buffer gets full and if this happens it will block at most <code>max.block.ms</code>, and a request will timeout after <code>request.timeout.ms</code>.</p>

<p>Another parameter that you should increase is <code>batch-size</code>, this parameter is in bytes, not in number of messages. Also linger.ms should be increased, in case this producer messages are created in user request time, do not increase very much, a good choice could be 1-4 ms.</p>

<p>Messages will be send when the <code>batch.size</code> gets full or takes longer than <code>linger.ms</code> to have more data than the <code>batch.size</code>. Big batches increases the throughput in normal cases, but if the linger is too low it doesn't help, because you will send before you have enough data to get the <code>batch.size</code>.</p>

<p>Also recheck on producer logs that the properties are loaded correctly.</p>
