title: Downloading thousands of files using python
body: <p>I'm trying to download about 500k small csv files (5kb-1mb) from a list of urls but it is been taking too long to get this done. With the code bellow, I am lucky if I get 10k files a day.</p>
  
  <p>I have tried using the multiprocessing package and a pool to download multiple files simultaneously. This seems to be effective for the first few thousand downloads, but eventually the overall speed goes down. I am no expert, but I assume that the decreasing speed indicates that the server I am trying to download from cannot keep up with this number of requests. Does that makes sense? </p>
  
  <p>To be honest, I am quite lost here and was wondering if there is any piece of advice on how to speed this up.  </p>
  
  <pre><code>import urllib2
  import pandas as pd
  import csv
  from multiprocessing import Pool
  
  #import url file
  df = pd.read_csv("url_list.csv")
  
  #select only part of the total list to download 
  list=pd.Series(df[0:10000])
  
  
  #define a job and set file name as the id string under urls
  def job(url):
      file_name = str("test/"+url[46:61])+".csv"
      u = urllib2.urlopen(url)
      f = open(file_name, 'wb')
      f.write(u.read())
      f.close()
  
  #run job
  pool = Pool()
  url = [ "http://" + str(file_path) for file_path in list]
  pool.map(job, url)
  </code></pre>