title: UnicodeEncodeError when fetching url
body: <p>I have this issue trying to get all the text nodes in an HTML document using lxml but I get an UnicodeEncodeError: <code>'ascii' codec can't encode character u'\xe9' in position 8995: ordinal not in range(128)</code>. However, when I try to find out the type of encoding of this page (<code>encoding = chardet.detect(response)['encoding']</code>), it says it's <code>utf-8</code>. It seems weird that a single page has utf-8 and ascii. Actually, this:</p>
  
  <pre><code>fromstring(response).text_content().encode('ascii', 'replace')
  </code></pre>
  
  <p>solves the problem.</p>
  
  <p>Here it's my code:</p>
  
  <pre><code>from lxml.html import fromstring
  import urllib2
  import chardet
  request = urllib2.Request(my_url)
  request.add_header('User-Agent',
                     'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)')   
  request.add_header("Accept-Language", "en-us")
  response = urllib2.urlopen(request).read()
  
  print encoding
  print fromstring(response).text_content()
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>utf-8
  UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 8995: ordinal not in range(128)
  </code></pre>
  
  <p>What can I do to solve this issue?. Keep in mind that I want to do this with a few other pages, so I don't want to encode on an individual basis.</p>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>Maybe there is something else going on here. When I run this script on the terminal, I get a correct output but when a run it inside SublimeText, I get UnicodeEncodeError... ¿?</p>
  
  <p><strong>UPDATE2:</strong></p>
  
  <p>It's also happening when I create a file with this output. <code>.encode('ascii', 'replace')</code> is working but I'd like to have a more general solution.</p>
  
  <p>Regards</p>