title: Downloading a LOT of files using python
body: <p>Is there a good way to download a lot of files en masse using python? This code is speedy enough for downloading about 100 or so files. But I need to download 300,000 files. Obviously they are all very small files (or I wouldn't be downloading 300,000 of them :) ) so the real bottleneck seems to be this loop. Does anyone have any thoughts? Maybe using MPI or threading? </p>
  
  <p>Do I just have to live with the bottle neck? Or is there a faster way, maybe not even using python?</p>
  
  <p>(I included the full beginning of the code just for completeness sake)</p>
  
  <pre><code>from __future__ import division
  import pandas as pd
  import numpy as np
  import urllib2
  import os
  import linecache 
  
  #we start with a huge file of urls
  
  data= pd.read_csv("edgar.csv")
  datatemp2=data[data['form'].str.contains("14A")]
  datatemp3=data[data['form'].str.contains("14C")]
  
  #data2 is the cut-down file
  
  data2=datatemp2.append(datatemp3)
  flist=np.array(data2['filename'])
  print len(flist)
  print flist
  
  ###below we have a script to download all of the files in the data2 database
  ###here you will need to create a new directory named edgar14A14C in your CWD
  
  original=os.getcwd().copy()
  os.chdir(str(os.getcwd())+str('/edgar14A14C'))
  
  
  for i in xrange(len(flist)):
      url = "ftp://ftp.sec.gov/"+str(flist[i])
      file_name = str(url.split('/')[-1])
      u = urllib2.urlopen(url)
      f = open(file_name, 'wb')
      f.write(u.read())
      f.close()
      print i
  </code></pre>