title: Apache Spark/AWS EMR and tracking of processed files
body: <p>I have AWS S3 folder where the big number of JSON files is stored. I need to ETL these files with AWS EMR over Spark and store the transformation into AWS RDS.</p>
  
  <p>I have implemented the Spark job for this purpose on Scala and everything is working fine. I plan to execute this job once a week.</p>
  
  <p>From time to time the external logic can add a new files to AWS S3 folder so the next time when my Spark job is starting I'd like to process only the new(unprocessed) JSON files.</p>
  
  <p>Right now I don't know where to store the information about the processed JSON files so the Spark job can decide what files/folders to process. Could you please advise me what is the best practice(and how) to track this changes with Spark/AWS?</p>