title: Limit kafka batch size when using Spark Structured Streaming
body: <p>We have some historical data queued up on our topics, we don't want to process all this data in a single batch as that is harder to do (and if it fails it has to start again!).</p>
  
  <p>Also, knowing how to control the batch size would be quite helpful in tuning jobs.</p>
  
  <p>When using <code>DStreams</code> the way to control the size of the batch as exactly as possible is <a href="https://stackoverflow.com/questions/39981650/limit-kafka-batches-size-when-using-spark-streaming/46156603#46156603">Limit Kafka batches size when using Spark Streaming</a></p>
  
  <p>The same approach i.e. setting <code>maxRatePerPartition</code> and then tuning <code>batchDuration</code> is extremely cumbersome but works with <code>DStream</code> it doesn't work at all with Structured Streaming.</p>
  
  <p>Ideally I'd like to know of a config like <code>maxBatchSize</code> and <code>minBatchSize</code>, where I can simply set the number of records I'd like.</p>