title: Ridiculously low download speed with Python requests module
body: <h1>Problem:</h1>
  
  <p>I have been trying to make a simple anime downloader using Python's request module. I am tracking the progress using the progressbar2 module. While trying to download, I'm getting speed of 0.x B/s. I assumed the problem is about choosing the <code>chunk_size</code> based on <a href="https://stackoverflow.com/questions/37571466/increase-download-speed-of-requests">this question</a>. But I am getting the same negligible speeds irrespective of chunk size.</p>
  
  <h1>Specs and info:</h1>
  
  <ol>
  <li>I am using Windows 10, Python 3.5, latest <code>requests</code> module (2.18.4) and have a decent internet with speed of 40Mbps.</li>
  <li>I can download the file from the link through browser(Chrome) and <em>Free 
  Download Manager</em> in about 1 minute.</li>
  <li>The link is perfectly working and I have no firewall conflicts.</li>
  </ol>
  
  <h1>Code:</h1>
  
  <pre><code>import os
  import requests
  import progressbar
  from progressbar import *
  
  os.chdir('D:\\anime\\ongoing')
  
  widgets = ['Downloading: ', Percentage(), ' ', Bar(marker='#',left='[',right=']'),
             ' ', ETA(), FileTransferSpeed()]
  
  url = 'https://lh3.googleusercontent.com/AtkUe87GbrINzTJS_Fj4W08CGqlOg9anwEF7n5-eKXcyS1RsaB8LdzRVaXloiJwiaX2IX1xqUiA=m22?title=(720P%20-%20mp4)Net-juu%20no%20Susume%20Episode%207'
  r = requests.get(url,stream=True)
  remotesize = r.headers['content-length']
  
  print("Downloading {}.mp4!\n\n".format(url.split('title=')[1]))
  pbar = ProgressBar(max_value=int(remotesize),widgets=widgets).start()
  i = 0
  with open('./tempy/tempy_file.mp4', 'wb') as f:
     for chunk in r.iter_content(chunk_size=5*1024*1024): 
        if chunk:
           i = i + len(chunk)
           f.write(chunk)
           pbar.update(int(i/int(remotesize) * 100))
  pbar.finish()         
  print("Successfully downloaded!\n\n")
  </code></pre>
  
  <h1>Screenshot:</h1>
  
  <p><a href="https://i.stack.imgur.com/BnS4A.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BnS4A.png" alt="The speed is just ridiculous."></a></p>
  
  <h1>Expected Solution:</h1>
  
  <p><em>Not sure if this <a href="https://github.com/requests/requests/issues/2015" rel="nofollow noreferrer">Github Issue</a> was fixed.</em></p>
  
  <ol>
  <li>It would be preferable to find a solution within requests module but I am open to any answers within the scope of Python that can get me a good speed.</li>
  <li>I want the download to be chunk-wise because I want to see the progress via the progressbar. So <code>shutil.copyfileobj(r.raw)</code> isn't what I'm looking for.</li>
  <li>I did try using multiple threads but it only complicated things and didn't help. I think the problem is with writing the chunk to the buffer itself and splitting this task between threads doesn't help.  </li>
  </ol>
  
  <h1>Edit:</h1>
  
  <p>As per suggestion, I tried it by including random user agents as shown:</p>
  
  <pre><code>desktop_agents = ['Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',
                   'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',
                   'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',
                   'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14',
                   'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36',
                   'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
                   'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
                   'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36',
                   'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',
                   'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0']
  
  def random_headers():
      return {'User-Agent': choice(desktop_agents),'Accept':'text/html,video/mp4,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'}
  </code></pre>
  
  <p>and sending the request with header as <code>r = requests.get(url,stream=True,headers=random_headers())</code></p>
  
  <p>However, it made no difference. :(</p>
  
  <h1>Edit no. 2:</h1>
  
  <p>Tried it with a sample video from "<a href="http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_5mb.mp4" rel="nofollow noreferrer">http://www.sample-videos.com/video/mp4/720/big_buck_bunny_720p_5mb.mp4</a>". Same problem persists. :/</p>