title: How to control sequence of Kafka Messages sent Asynchronously
body: <p>I have developed a <code>Kafka version : 0.9.0.1</code> application.</p>
  
  <p>One very important aspect of this application is that messages need to be consumed in the correct sequence. This is because I am propagating Database rows between two databases. This means I need to ensure all records within each Unit Of Work are sent as well as UOW-Insert arriving before UOW-Update.</p>
  
  <p>If I use asynchronous message production, how can I guarantee the messages are consumed in the correct sequence?</p>
  
  <p>I will employ <code>Kafka Producer</code> send with <code>Callback</code> to be notified whether or not each message was successfuly sent.</p>
  
  <p>I will have <code>acks=all</code>, <code>retries=0</code>, <code>batch.size=16384</code> and my Kafka topics have a single partition.</p>
  
  <p>My consumer can cope with duplicate messages, e.g. in the case of retries being necessary, however consumers cannot cope with messages out of sequence.</p>
  
  <p>My retry approach will be to Fail Fast, e.g. as soon as a messages fails to send I report the records Log Record Sequence Number (LRSN) or Relative Byte Address (RBA) and stop sending any messages.</p>
  
  <p>Then reset the source database logs to the reported LRSN or RBA and restart my message production.</p>
  
  <p>for example I send messages</p>
  
  <pre><code>Message               UOW
  M1                    uow-0000 
  M2                    uow-0000 
  M3                    uow-0000 
  M4                    uow-0001 
  M5                    uow-0001 
  M6                    uow-0001 
  M7                    uow-0001 
  M8                    uow-0002 
  </code></pre>
  
  <p>When message M5 failed to send successfully, I would stop sending anymore messages. However I have an issue that the consumer will have received messages M1, M2, M3, M4, M6, M7, &amp; M8.</p>
  
  <p>To recover this situation I will reset my source database log reader to the reported LRSN or RBA of M5 and start resending messages from that point.</p>
  
  <p>Now the consumer receives</p>
  
  <pre><code>Message               UOW                    Duplicate
  M5                    uow-0001                  No
  M6                    uow-0001                  Yes
  M7                    uow-0001                  Yes
  M8                    uow-0002                  Yes
  </code></pre>
  
  <p>With this approach I get the speed of Asynchronous messaging and "hopefully" all messages consumed in the desired sequence.</p>