title: Kafka: Consume partition with manual batching - Messages are being skipped
body: <p>I am using <a href="https://github.com/confluentinc/confluent-kafka-dotnet" rel="nofollow noreferrer">Confluent Kafka .NET</a> to create a consumer for a partitioned topic.</p>
  <p>As Confluent Kafka .NET does not support consuming in batches, I built a function that consumes messages until the batch size is reached. The idea of this function is to build batches with messages from the same partition only, that is why I stop building the batch once I consume a result that has a different partition and return whatever number of messages I was able to consume up to that point.</p>
  <p><strong>Goal or Objective:</strong> I want to be able to process the messages I returned in the batch, and commit the offsets for those messages only. i.e:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Message Consumed From Partition</th>
  <th>Offset</th>
  <th>Stored in Batch</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>0</td>
  <td>0</td>
  <td>Yes</td>
  </tr>
  <tr>
  <td>0</td>
  <td>1</td>
  <td>Yes</td>
  </tr>
  <tr>
  <td>2</td>
  <td>0</td>
  <td>No</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>From the table above I would like to process both messages I got from partition 0. Message from partition 2 would be ignored and (hopefully) PICKED UP LATER in another call to <strong>ConsumeBatch</strong>.</p>
  <p>To commit I simply call the synchronous <code>Commit</code> function passing the offset of the latest message I processed as parameter. In this case I would pass the offset of the second message of the batch shown in the table above (Partition 0 - Offset 1).</p>
  <p><strong>ISSUE:</strong></p>
  <p>The problem is that for some reason, when I build a batch like the one shown above, the messages I decide not to process because of validations are being ignored forever. i.e: Message 0 of partition 2 will never be picked up by the consumer again.</p>
  <p>As you can see in the consumer configuration below, I have set both <strong>EnableAutoCommit</strong> and <strong>EnableAutoOffsetStore</strong> as false. I think this would be enough for the consumer to not do anything with the offsets and be able to pick up ignored messages in another <code>Consume</code> call, but it isn't. The offset is somehow increasing up to the latest consumed message for each partition, regardless of my configuration.</p>
  <p>Can anybody give me some light on what am I missing here to achieve the desired behavior if possible?</p>
  <p><strong>Simplified version of the function to build the batch:</strong></p>
  <pre><code>public IEnumerable&lt;ConsumeResult&lt;string, string&gt;&gt; ConsumeBatch(int batchSize)
  {
      List&lt;ConsumeResult&lt;string, string&gt;&gt; consumedMessages = new List&lt;ConsumeResult&lt;string, string&gt;&gt;();
  
      int latestPartition = -1; // The partition from where we consumed the last message
  
      for (int i = 0; i &lt; batchSize; i++)
      {
          var result = _consumer.Consume(100);
          
          if (result != null)
          {
              if (latestPartition == -1 || result.Partition.Value == latestPartition)
              {
                  consumedMessages.Add(result);
                  latestPartition = result.Partition.Value;
              }
              else
                  break;
          }
          else
              break;
      }
  
      return consumedMessages;
  }
  </code></pre>
  <p><strong>ConsumerConfig used to instantiate my consumer client:</strong></p>
  <pre><code>_consumerConfig = new ConsumerConfig
          {
              BootstrapServers = _bootstrapServers,
              EnableAutoCommit = false,
              AutoCommitIntervalMs = 0,
              GroupId = &quot;WorkerConsumers&quot;,
              AutoOffsetReset = AutoOffsetReset.Earliest,
              EnableAutoOffsetStore = false,
          };
  </code></pre>
  <p><em><strong>Additional Information:</strong></em>
  This is being tested with:</p>
  <ul>
  <li>1 topic with 6 partitions and replication factor of 2</li>
  <li>3 brokers</li>
  <li>1 single-threaded consumer client that belongs to a consumer group</li>
  <li>Local environment with wsl2 on Windows 10</li>
  </ul>