title: Kafka Streams Fault Tolerance with Offset Management in Parellel
body: <p><strong>Description</strong> :</p>
  <p>I have one Kafka Stream application which is consuming from a topic.
  The events are coming at high volumes.</p>
  <p>KafkaStream will consume the events as a terminal operation and club the events in a bunch say 1000 events and writes it to AWS S3.</p>
  <p>I have threads that are writing to s3 in parallel after consuming events from Kafka topic.</p>
  <p>Not using kafka-connector-s3 due to some business application logics and processings.</p>
  <p><strong>Problem</strong> ::</p>
  <p>I want the application to be fault-tolerant don't want to loose messages.</p>
  <p>--&gt; CRASH SCENARIO</p>
  <p>Suppose the application has 10 threads all are running and trying to put the events in S3, and a crash happens, in that case, since the KafkaStream has ( enable.auto.commit = false )and we cannot commit the offset manually and all the threads have consumed messages from Kafka topic.
  In this case, KafkaStreams has already committed the offset after reading but it could not have processed the events to S3.</p>
  <p>I need a mechanism so that I can be sure of that what was the last offset till the events get written to the S3 file successfully.
  And In crash scenarios, how should I deal with this and how to manage the Kafka offsets in Kafka Streams as I am using say 10 threads. What if some failed to write to s3 and some are passed. How do I ensure the ordering of offset getting successfully processed to s3 or not?</p>
  <p>Let me know if I am not clear to describe my problem statement.</p>
  <p>Thanks!</p>