title: Efficiently downloading files from S3 periodically using python boto3
body: <p>I'm trying to download the last 24hours of new files added to an S3 bucket - however, the S3 bucket contains a large number of files.</p>
  
  <p>From my understanding s3 buckets use a flat structure where files are stored alphabetically based on the key name. </p>
  
  <p>I've written a script to pull all the data stored on the bucket using threading. However, now I have all the files on my local system I want to update the database every 24hours with any new files that have been uploaded to S3. </p>
  
  <p>Most forums recommend using 'last modified' to search for the correct files and then download the files that match the data specified.</p>
  
  <p>Firstly, does downloading a file from the s3 bucket change the 'last modified'? Seems like this could cause problems.</p>
  
  <p>Secondly, this seems like a really in-efficient process - searching through the entire bucket for files with the correct 'last modified' each time, then downloading... especially since the bucket contains a huge number of files. Is there a better way to achieve this?</p>
  
  <p>Finally, does the pre-fix filter make this process any more efficient? or does this also require searching through all files.</p>
  
  <p>Thanks in advance!</p>