title: Avro Records -> Kafka -> Kafka Connect Sink -> Amazon S3 Storage. Idempotency?
body: <p>So my employer has Avro messages going through Kafka. We want to archive all messages to Amazon S3. Ideally, they would be stored in S3-like-directories by day and use a path structure something like:</p>
  
  <p>s3://my-bucket/data/day=2016-03-04/data.avro</p>
  
  <p>Is there a reference or best practices for how to do this well?</p>
  
  <p>My one question is <strong>idempotency</strong>: How do I provide write idempotency, where a record may be sent to my sink writer more than once yet only be stored on S3 once.</p>
  
  <p>Am I correct that I need idempotency? If I implement a simple append (non-idempotent), Kafka Connect may send the same records twice and they may get stored redundantly?</p>