title: Requests library crashing on Python 2 and Python 3 with
body: <p>I am trying to parse arbitrary webpages with the <code>requests</code> and <code>BeautifulSoup</code> libraries with this code:</p>
  
  <pre><code>try:
      response = requests.get(url)
  except Exception as error:
      return False
  
  if response.encoding == None:
      soup = bs4.BeautifulSoup(response.text) # This is line 809
  else:
      soup = bs4.BeautifulSoup(response.text, from_encoding=response.encoding)
  </code></pre>
  
  <p>On most webpages this works fine. However, on some arbitrary pages (&lt;1%) I get this crash:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/dotancohen/code/parser.py", line 155, in has_css
      soup = bs4.BeautifulSoup(response.text)
    File "/usr/lib/python3/dist-packages/requests/models.py", line 809, in text
      content = str(self.content, encoding, errors='replace')
    TypeError: str() argument 2 must be str, not None
  </code></pre>
  
  <p>For reference, this is the relevent method of the requests library:</p>
  
  <pre><code>@property
  def text(self):
      """Content of the response, in unicode.
  
      if Response.encoding is None and chardet module is available, encoding
      will be guessed.
      """
  
      # Try charset from content-type
      content = None
      encoding = self.encoding
  
      # Fallback to auto-detected encoding.
      if self.encoding is None:
          if chardet is not None:
              encoding = chardet.detect(self.content)['encoding']
  
      # Decode unicode from given encoding.
      try:
          content = str(self.content, encoding, errors='replace') # This is line 809
      except LookupError:
          # A LookupError is raised if the encoding was not found which could
          # indicate a misspelling or similar mistake.
          #
          # So we try blindly encoding.
          content = str(self.content, errors='replace')
  
      return content
  </code></pre>
  
  <p>As can be seen, I am not passing in an encoding when this error is thrown. <strong>How am I using the library incorrectly, and what can I do to prevent this error?</strong> This is on Python 3.2.3, but I can also get the same results with Python 2.</p>