title: Why is Kafka distribution over partitions far from even when no key is used?
body: <p>I'm trying to get my head around Kafka. One of the Hello World-like apps I've made is where a Producer doesn't use a key with the messages. All the documentation and tutorials say that the messages should be spread over the consumers in a round-robin fashion, but my tests show otherwise. It uses the default sample solution in Visual Studio (with the weather forecasts). Everytime the API is queried, the generated forecasts are also published to Kafka.</p>
  <p>The producer is nothing more than the following, in an API controller class (every time this is called, there are 5 weather forecasts in the array):</p>
  <pre><code>ProducerConfig config = new ProducerConfig
  {
      BootstrapServers = &quot;localhost:9092&quot;,
      ClientId = $&quot;KafkaResearch-{Dns.GetHostName()}&quot;
  };
  
  using (IProducer&lt;string, string&gt; producer = new ProducerBuilder&lt;string, string&gt;(config).Build())
  {
      foreach (var forecast in weatherForecasts)
      {
          string value = JsonSerializer.Serialize(forecast);
          Message&lt;string, string&gt; message = new Message&lt;string, string&gt; { Value = value };
      
          producer.Produce(&quot;weather-forecasts&quot;, message, DeliveryHandler);
      }
      producer.Flush();
  }
  </code></pre>
  <p>The messages are all different. They contain randomly generated data, every time:</p>
  <pre><code>weatherForecasts = Enumerable.Range(1, 5).Select(index =&gt; new WeatherForecast
  {
      Date = DateTime.Today.AddDays(index),
      TemperatureC = rng.Next(-20, 55),
      Summary = Summaries[rng.Next(Summaries.Length)]
  })
  .ToArray();
  </code></pre>
  <p>The consumer is a very simple console application:</p>
  <pre><code>static void Main(string[] args)
  {
      Console.WriteLine(&quot;Press any key to abort&quot;);
  
      var config = new ConsumerConfig
      {
          BootstrapServers = &quot;localhost:9092&quot;,
          GroupId = &quot;foo&quot;,
          AutoOffsetReset = AutoOffsetReset.Latest                
      };
  
      using (var consumer = new ConsumerBuilder&lt;string, string&gt;(config).Build())
      {
          consumer.Subscribe(&quot;weather-forecasts&quot;);
  
          while (!Console.KeyAvailable)
          {
              var consumeResult = consumer.Consume(TimeSpan.FromMilliseconds(500));
  
              if (consumeResult != null)
              {
                  Console.WriteLine($&quot;Message received on partition {consumeResult.Partition}, key: {consumeResult.Message.Key ?? &quot;&lt;--null--&gt;&quot;}&quot;);
                  Console.WriteLine($&quot;  {consumeResult.Message.Value}&quot;);
              }
          }
  
          consumer.Close();
      }
  }
  </code></pre>
  <p>The topic is created with 12 partitions:</p>
  <pre><code>kafka-topics --bootstrap-server localhost:9092  --create --partitions 12 --topic weather-forecasts
   
  </code></pre>
  <p>When I spam the API with a browser I would expect a (sort of) equal distribution over the partitions, but it is far from that. Only 4 out of 12 partitions get messages, so when using 5 consumers, one of them is idle.</p>
  <p>If I send all the weather forecasts in an array in one message (so without the <code>foreach</code> in the producer) all messages are assigned to the same partition, every single time.</p>
  <p>As requested in the comments: the code looks like this then:</p>
  <pre><code>ProducerConfig config = new ProducerConfig
  {
      BootstrapServers = &quot;localhost:9092&quot;,
      ClientId = $&quot;KafkaResearch-{Dns.GetHostName()}&quot;
  };
  
  using (IProducer&lt;string, string&gt; producer = new ProducerBuilder&lt;string, string&gt;(config).Build())
  {
      string value = JsonSerializer.Serialize(weatherForecasts);
  
      Message&lt;string, string&gt; message = new Message&lt;string, string&gt; { Value = value };
      producer.Produce(&quot;weather-forecasts&quot;, message, DeliveryHandler);
      producer.Flush();
  }
  </code></pre>
  <p>When I add a random key to the message, the distribution over all the consumer is pretty good.</p>
  <p>I've tried specifying a <code>Partitioner</code> in the <code>ProducerConfig</code>. That changes to which partition the message is assigned, but still produces only to very few partitions (bonus question: why is the Partitioner a setting on the Producer instead of the topic? I'd expect that all producers have to use the same method for partitioning...).</p>
  <pre><code>kafka-topics --version
  2.7.1 (Commit:61dbce85d0d41457)
  </code></pre>
  <p>Using Confluent.Kafka Nuget package (by Confluent Inc., Andreas Heider) version 1.5.3</p>
  <p>Thanks for any insights,</p>
  <p>Quido.</p>