title: Python requests stops working mid-file
body: <p>I've got a csv file with URL's and I need to scrape metadata from those website. I'm using python requests for that reasons with code below:</p>
  
  <pre><code>from tempfile import NamedTemporaryFile
  import shutil
  import csv
  from bs4 import BeautifulSoup
  import requests
  import re
  import html5lib
  import sys
  #import logging
  
  filename = 'TestWTF.csv'
  #logging.basicConfig(level=logging.DEBUG)
  
  #Get filename (with extension) from terminal
  #filename = sys.argv[1]
  tempfile = NamedTemporaryFile(delete=False)
  read_timeout = 1.0
  
  
  #Does actual scraping done, returns metaTag data
  def getMetadata (url, metaTag):
      r = requests.get("http://" + url, timeout=2)
      data = r.text
      soup = BeautifulSoup(data, 'html5lib')
      metadata = soup.findAll(attrs={"name":metaTag})
      return metadata
  
  #Gets either keyword or description
  def addDescription ( row ):
      scrapedKeywordsData = getMetadata(row,  'keywords')
      if not scrapedKeywordsData:
          print row + ' NO KEYWORDS'
          scrapedKeywordsData = getMetadata(row,  'description')
          if not scrapedKeywordsData:
              return ''
      return scrapedKeywordsData[0]
  
  def prepareString ( data ):
      output = data
      #Get rid of opening meta content
      if output.startswith( '&lt;meta content="' ):
          output = data[15:]
  
      #Get rid of closing meta content (keywords)
      if output.endswith( '" name="keywords"/&gt;' ):
          output = output[:-19]
  
      #Get rid of closing meta content (description)
      if output.endswith( '" name="description"/&gt;' ):
          output = output[:-22]
  
      return output
  
  def iterator():
      with open(filename, 'rb') as csvFile, tempfile:
          reader = csv.reader(csvFile, delimiter=',', quotechar='"')
          writer = csv.writer(tempfile, delimiter=',', quotechar='"')
  
          i = 0
          for row in reader:
              try:
                  data = str(addDescription (row[1] ))
                  row[3] = prepareString( data )
              except requests.exceptions.RequestException as e:
                  print e
              except requests.exceptions.Timeout as e:
                  print e
              except requests.exceptions.ReadTimeout as e:
                  print "lol"
              except requests.exceptions.ConnectionError as e:
                  print "These aren't the domains we're looking for."
              except requests.exceptions.ConnectTimeout as e:
                  print "Too slow Mojo!"
  
  
  
          writer.writerow(row)
          i = i + 1
          print i
      shutil.move(tempfile.name, filename)
  
  def main():
      iterator()
  
  
  #Defining main function
  if __name__ == '__main__':
      main()
  </code></pre>
  
  <p>It works just fine but at some URL's (out of 3000 let's say maybe 2-3) it would just suddenly stop working and not progress to next one after timeout time.. So I have to kill it using Ctr+C which results in file not being saved.</p>
  
  <p>I know it's a problem of catching exceptions but I cannot figure out which one or what to do with that problem.. I'm more than happy to simply ignore the one which is stuck on..</p>
  
  <p>EDIT:</p>
  
  <p>Added traceback:</p>
  
  <pre><code>^CTraceback (most recent call last):
    File "blacklist.py", line 90, in &lt;module&gt;
      main()
    File "blacklist.py", line 85, in main
      iterator()
    File "blacklist.py", line 62, in iterator
      data = str(addDescription (row[1] ))
    File "blacklist.py", line 30, in addDescription
      scrapedKeywordsData = getMetadata(row,  'keywords')
    File "blacklist.py", line 25, in getMetadata
      metadata = soup.findAll(attrs={"name":metaTag})
    File "/Library/Python/2.7/site-packages/bs4/element.py", line 1259, in find_all
      return self._find_all(name, attrs, text, limit, generator, **kwargs)
    File "/Library/Python/2.7/site-packages/bs4/element.py", line 537, in _find_all
      found = strainer.search(i)
    File "/Library/Python/2.7/site-packages/bs4/element.py", line 1654, in search
      found = self.search_tag(markup)
    File "/Library/Python/2.7/site-packages/bs4/element.py", line 1626, in search_tag
      if not self._matches(attr_value, match_against):
    File "/Library/Python/2.7/site-packages/bs4/element.py", line 1696, in _matches
      if isinstance(markup, Tag):
  KeyboardInterrupt 
  </code></pre>
  
  <p>EDIT 2:</p>
  
  <p>Example website for which script doesn't work: miniusa.com</p>