title: Spark Structured Streaming won't pull the final batch from Kafka
body: <p>I noticed that SSS won't process a waiting batch if there are no batches after that. What I saw is that Spark must always leave one batch on Kafka waiting to be consumed when it is writing Parquet to HDFS.</p>
  <p>This is to do with the way Spark cleans up after each batch, apparently.</p>
  <p>There is a JIRA for it at:</p>
  <p><a href="https://issues.apache.org/jira/browse/SPARK-24156" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/SPARK-24156</a></p>
  <p>that says it's resolved in 2.4.0 but my code is using 2.4.2 yet I still see Spark reluctant to consume another batch from Kafka if it means there is nothing else waiting to be processed in the topic.</p>
  <p>Do I have to do something special to exploit the behaviour that SPARK-24156 says it has introduced?</p>