title: Kafka Streams does not increment offset by 1 when producing to topic
body: <p>I have implemented a simple Kafka Dead letter record processor.</p>
  
  <p>It works perfectly when using records produced from the Console producer.</p>
  
  <p>However I find that our Kafka Streams applications do not guarantee that producing records to the sink topics that the offsets will be incremented by 1 for each record produced.</p>
  
  <p>Dead Letter Processor Background:</p>
  
  <p>I have a scenario where records may be received before all data required to process it is published. 
  When records are not matched for processing by the streams app they are move to a Dead letter topic instead of continue to flow down stream. When new data is published we dump the latest messages from the Dead letter topic back in to the stream application's source topic for reprocessing with the new data.</p>
  
  <p>The Dead Letter processor:</p>
  
  <ul>
  <li>At the start of the run application records the ending offsets of each partition</li>
  <li>The ending offsets marks the point to stop processing records for a given Dead Letter topic to avoid infinite loop if reprocessed records return to Dead Letter topic.</li>
  <li>Application resumes from the last Offsets produced by the previous run via consumer groups.</li>
  <li>Application is using transactions and <code>KafkaProducer#sendOffsetsToTransaction</code> to commit the last produced offsets.</li>
  </ul>
  
  <p>To track when all records in my range are processed for a topic's partition my service compares its last produced offset from the producer to the the consumers saved map of ending offsets. When we reach the ending offset the consumer pauses that partition via <code>KafkaConsumer#pause</code> and when all partitions are paused (meaning they reached the saved Ending offset)then calls it exits.</p>
  
  <p>The <a href="https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html" rel="nofollow noreferrer">Kafka Consumer API</a> States:</p>
  
  <blockquote>
    <p>Offsets and Consumer Position
    Kafka maintains a numerical offset for each record in a partition. This offset acts as a unique identifier of a record within that partition, and also denotes the position of the consumer in the partition. For example, a consumer which is at position 5 has consumed records with offsets 0 through 4 and will next receive the record with offset 5.</p>
  </blockquote>
  
  <p>The <a href="https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html#sendOffsetsToTransaction-java.util.Map-java.lang.String-" rel="nofollow noreferrer">Kafka Producer API</a> references the next offset is always +1 as well.</p>
  
  <blockquote>
    <p>Sends a list of specified offsets to the consumer group coordinator, and also marks those offsets as part of the current transaction. These offsets will be considered committed only if the transaction is committed successfully. The committed offset should be the next message your application will consume, i.e. lastProcessedMessageOffset + 1.</p>
  </blockquote>
  
  <p>But you can clearly see in my debugger that the records consumed for a single partition are anything but incremented 1 at a time...
  <a href="https://i.stack.imgur.com/owkWX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/owkWX.png" alt="enter image description here"></a></p>
  
  <p>I thought maybe this was a Kafka configuration issue such as <code>max.message.bytes</code> but none really made sense.
  Then I thought perhaps it is from joining but didn't see any way that would change the way the producer would function.</p>
  
  <p>Not sure if it is relevant or not but all of our Kafka applications are using Avro and Schema Registry...</p>
  
  <p>Should the offsets always increment by 1 regardless of method of producing or is it possible that using Kafka streams API does not offer the same guarantees as the normal Producer Consumer clients?</p>
  
  <p>Is there just something entirely that I am missing?</p>