title: Urllib urlopen/urlretrieve too many open files error
body: <h3>Problem</h3>
  <p>I'm trying to download &gt;100.000 files from a ftp server in parallel (using threads). I previously tried it with urlretrieve as answered <a href="https://stackoverflow.com/questions/16140921/how-to-download-a-few-files-simultaneusly-from-ftp-in-python">here</a>, however this gave me the following error: <code>URLError(OSError(24, 'Too many open files'))</code>. Apparently this problem is a bug (cannot find the reference anymore), so I tried to use <code>urlopen</code> in combination with <code>shutil</code> and then write it to file which I could close myself, as described <a href="https://stackoverflow.com/questions/7243750/download-file-from-web-in-python-3">here</a>. This seemed to work fine, but then I got the same error again: <code>URLError(OSError(24, 'Too many open files'))</code>. I thought whenever writing to a file is incomplete or will fail the <code>with</code> statement will cause to file to close itself, but seemingly the files still keep open and will eventually cause the script to halt.</p>
  <h3>Question</h3>
  <p>How can I prevent this error, i.e. make sure that every files get closed?</p>
  <h3>Code</h3>
  <pre><code>import csv
  import urllib.request
  import shutil
  from multiprocessing.dummy import Pool
  
  def url_to_filename(url):
      filename = 'patric_genomes/' + url.split('/')[-1]
      return filename
  
  def download(url):
      url = url.strip()
      try:
          with urllib.request.urlopen(url) as response, open(url_to_filename(url), 'wb') as out_file:
              shutil.copyfileobj(response, out_file)
      except Exception as e:
          return None, e
  
  def build_urls(id_list):
      base_url = 'ftp://some_ftp_server/'
      urls = []
      for some_id in id_list:
          url = base_url + some_id + '/' + some_id + '.fna'
          print(url)
          urls.append(url)
      return urls
  
  
  if __name__ == &quot;__main__&quot;:
      with open('full_data/genome_ids.txt') as inFile:
          reader = csv.DictReader(inFile, delimiter = '\t')
          ids = {row['some_id'] for row in reader}
          urls = build_urls(ids)
          p = Pool(100)
          print(p.map(download, urls)) 
  </code></pre>