title: How to purge the checkpoint directory in spark structured streaming
body: <p>I am new to spark structured streaming. I have set up a streaming pipeline from kafka and also set up the checkpoint directory.</p>
  <p>Now the offsets are being written to the checkpoint directory and it works fine.</p>
  <p>I could see the checkpoint directory keeps increasing. Is there any property in spark to purge this automatically. If not what is a better solution to do this.</p>
  <p>I tried to use the property spark.cleaner.referenceTracking.cleanCheckpoints = true. But this does not seem to help.</p>