title: How to iterate over files in an S3 bucket?
body: <p>I have a large number of files (>1,000) stored in an S3 bucket, and I would like to iterate over them (e.g. in a <code>for</code> loop) to extract data from them using <code>boto3</code>.</p>
  
  <p>However, I notice that in accordance with <a href="http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects" rel="noreferrer">http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects</a>, the <code>list_objects()</code> method of the <code>Client</code> class only lists up to 1,000 objects:</p>
  
  <pre><code>In [1]: import boto3
  
  In [2]: client = boto3.client('s3')
  
  In [11]: apks = client.list_objects(Bucket='iper-apks')
  
  In [16]: type(apks['Contents'])
  Out[16]: list
  
  In [17]: len(apks['Contents'])
  Out[17]: 1000
  </code></pre>
  
  <p>However, I would like to list <strong>all</strong> the objects, even if there are more than 1,000. How could I achieve this?</p>