title: Move millions of files accross s3 buckets with boto3
body: <p>I have this code to copy files between S3 buckets.</p>
  <pre><code>def move_file(key):
  file_name = key.split(&quot;/&quot;)[-1]
  dest_key = f&quot;{S3_OBJECT_DEST_PREFIX}/{file_name}&quot;
  copy_source = {
  'Bucket': S3_SOURCE_BUCKET_NAME,
  'Key': key
  }
  try:
      if check_exists(dest_key):
          response = s3.copy_object(
          Bucket=S3_DEST_BUCKET_NAME,
          Key=dest_key,
          CopySource=copy_source
          )
          logging.info(f&quot;Successfully copied file {dest_key}&quot;)
          insert_db_row(dest_key, key)
      else:
          print(f&quot;The asset {dest_key} already exists&quot;)
  except Exception as e:
      logging.error(f&quot;Error while copying file: {e}&quot;)
      sys.exit(1)
  </code></pre>
  <p>I'm currently looping through all the files in my bucket and copying each of them. Do you think this method is efficient for millions of files</p>