title: Requests + s3 painfully slow
body: <p>I'm using Python's <code>requests</code> with <code>smart_open</code> to download a file (a <code>.tif</code> file, in case that helps), and upload it to an S3 bucket without saving any temporary file. I'm looping through a few thousands of URLs per request. This is the function I wrote: </p>
  
  <pre class="lang-py prettyprint-override"><code>def stream_download_s3(url,
                         aws_key,
                         aws_secret,
                         aws_bucket_name,
                         path,
                         auth):
      """
      Stream files from request to S3
      """
  
      headers = {'Authorization': f'Bearer {auth}',
                 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15'}
      session = boto3.Session(
          aws_access_key_id=aws_key,
          aws_secret_access_key=aws_secret
      )
  
      bucket_path_strip = path[5:]
      bucket_name_strip = aws_bucket_name[5:]
  
      with requests.Session() as s:
          s.headers.update(headers)
          try:
              with s.get(url) as r:
                  if r.status_code == requests.codes.ok:
                      soup = BeautifulSoup(r.content)
                      download_files = [link.contents[0] for link in
                                        soup.find_all('a') if '.tif' in
                                       link.contents[0]]
  
                      for file_name in download_files:
                          save_file = os.path.join(path,
                                                   file_name)
  
                          if check_s3_exists(session, 
                                             bucket_name_strip,
                                             os.path.join(bucket_path_strip, file_name)):
                              print(f'S3: {os.path.join(path, file_name)} already exists. Skipping download')
                          else:
                              with s.get(url + file_name) as file_request:
                                  if file_request.status_code == requests.codes.ok:
                                      with smart_open.open(save_file, 'wb', transport_params=dict(session=session)) as so:
                                          so.write(file_request.content)
  
                  else:
                      print(f'Request GET failed with {r.content} [{r.url}]')
  
          except requests.exceptions.HTTPError as err:
              print(f'{err}')
  </code></pre>
  
  <p>This function is making a first request to scrape all the available images URLs (that's the <code>bs4</code> part), and then looping through all the retrieved URLs and download their content. The returned request content it's a binary  that I send to the <code>smart_open</code>'s <code>open</code> function to upload to S3. </p>
  
  <p>The whole process is taking ~150 minutes for 510 images (less than 2 Gb), whereas  a combination of <code>wget</code> and <code>aws s3 ls</code> does the same in ~86 mins (<code>wget</code> took <code>1h 26m 46s</code>, and <code>s3 cp</code> took seconds). </p>
  
  <h3>Some options in consideration:</h3>
  
  <ul>
  <li>I'm working in an AWS machine, although some APIs would ban AWS-like IPs, this seems not to be the case. If it is making the download slower, I don't know. Also, S3 and EC2 are in the same region. </li>
  <li>I know that <code>stream=True</code> in the <code>requests.get()</code> is an alternative, but as far as I know, that is mainly useful for streaming big files without filling up the memory. Does that might change anything? </li>
  <li>A similar implementation with <code>io.BytesIO</code> give similar results. Am I doing something wrong there? </li>
  <li>I use <code>requests</code> because I like the API (<em>a lot</em>!), but if there's any other alternative, I am open to try :-) </li>
  </ul>