title: How to achieve exactly-once write guaranty with foreachBatch sink in Spark Structured Streaming
body: <p>From the <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#foreachbatch" rel="nofollow noreferrer">docs</a>:</p>
  <blockquote>
  <p>By default, foreachBatch provides only at-least-once write guarantees.
  However, you can use the batchId provided to the function as way to
  deduplicate the output and get an exactly-once guarantee.</p>
  </blockquote>
  <p>Does this mean that unless we go the extra mile, we wouldn't achieve exactly-once write guaranty <strong>even if we use checkpointing in the main writeStream operation?</strong></p>
  <p>If yes, what should be done to achieve exactly-once write guaranty? What is meant in the docs by:</p>
  <blockquote>
  <p>using the batchId
  ?</p>
  </blockquote>
  <p><em>PS: The original question was specific for Kafka but I generalized it as the solution would apply to anything within the foreachBatch block.</em></p>