title: Why could GC time increase in Kafka brokers with constant number of records, producers and consumers?
body: <p>I use Kafka 2.1.0.</p>
  
  <p>We have a Kafka cluster with 5 brokers (r5.xlarge machines). We often observe that the GC timings increase too much without any change in the rate of incoming messages severely impacting the performance of the cluster. Now, I don't understand what could be causing much sudden increase in GC time. </p>
  
  <p>I have tried a few things with little improvement but I don't really understand the reason behind them. </p>
  
  <pre><code>export KAFKA_HEAP_OPTS="-Xmx10G -Xms1G"
  export KAFKA_JVM_PERFORMANCE_OPTS="-XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"
  </code></pre>
  
  <p>I would like to understand the most important parameters when tuning GC in a Kafka broker.
  Seeing the configuration above, where am I going wrong? What can be done to rectify this?</p>
  
  <p>All the producers and consumers are working fine, and the rate of incoming messages remains fairly constant. Till now, we have not been able to figure out any pattern behind the sudden increase in GC times, it seems random.</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>After some further analysis, It turns out there was indeed some increase in the amount of data per sec. One of the topics had increased message input from around 10 KBps to 200 KBps. But I believed that Kafka could easily handle this much of data.</p>
  
  <p>Is there something I am missing??</p>
  
  <p>Grafana Snapshot
  <a href="https://i.stack.imgur.com/A8ZZh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/A8ZZh.png" alt="enter image description here"></a></p>