title: What is the performance impact of having a Kafka consumer subscribed to multiple topics of varying load?
body: <p>I have a Spring Boot application where I am consuming from around 10 different Kafka topics using a single KafkaConsumer object. This application runs on Kubernetes, so there are many pods running in parallel. These topics vary significantly in the amount of messages that come through them. I am finding that the highest-load Kafka topic is often backlogged while the other topics are always caught up.</p>
  
  <p>I understand that it is expected that the high-load topic will get backlogged without enough consumer throughput, but I am wondering if the KafkaConsumer class is working on these topics in parallel or not. Or, is each topic taking an exact fraction of the polling time, giving equal resources to all topics, regardless of their rate of new messages?</p>
  
  <p>Here is some code similar to my implementation, removing anything confidential:</p>
  
  <pre class="lang-java prettyprint-override"><code>@Autowired
  KafkaConsumer&lt;String, String&gt; kafkaConsumer;
  @Audowired
  CustomProcessor customProcessor;
  
  public void consume() {
      kafkaConsumer.subscribe(Arrays.asList("topic1", "topic2, "topic3", "topic4", "topic5");
      while (true) {
          ConsumerRecords&lt;String, String&gt; records = kafkaConsumer.poll(100);
          for (ConsumerRecord&lt;String, String&gt; record : records) {
              customProcessor.processMessage(record.value());
          }
      }
  }
  </code></pre>
  
  <p>Let's say <strong>topic1</strong> has 1000 messages/sec, and every other topic has 50 messages/sec. What is the impact to the throughput of consuming the first topic due to waiting on the other topics? Would I be better off instantiating a separate consumer object for the high-load topic?</p>