title: Processing unpartitioned data with AWS Glue using bookmarking
body: <p>I have data being written from Kafka to a directory in s3 with a structure like this:</p>
  
  <pre><code>s3://bucket/topics/topic1/files1...N
  s3://bucket/topics/topic2/files1...N
  .
  .
  s3://bucket/topics/topicN/files1...N
  </code></pre>
  
  <p>There is already a lot of data in this bucket and I want to use AWS Glue to transform it into parquet and partition it, but there is way too much data to do it all at once. I was looking into bookmarking and it seems like you can't use it to only read the most recent data or to process data in chunks. Is there a recommended way of processing data like this so that bookmarking will work for when new data comes in? </p>
  
  <p>Also, does bookmarking require that spark or glue has to scan my entire dataset each time I run a job to figure out which files are greater than the last runs max_last_modified timestamp? That seems pretty inefficient especially as the data in the source bucket continues to grow.</p>