title: Python reading a file into unicode strings
body: <p>I am having some troubles with understanding the correct way to handle unicode strings in Python. I have read many questions about it but it is still unclear what should I do to avoid problems when reading and writing files.</p>
  
  <p>My goal is to read some huge (up to 7GB) files efficiently line by line. I was doing it with the simple <code>with open(filename) as f:</code> but it I ended up with an error in ASCII decoding.</p>
  
  <p>Then I read the correct way of doing it would be to write: </p>
  
  <pre><code>with codecs.open(filename, 'r', encoding='utf-8') as logfile:
  </code></pre>
  
  <p>However this ends up in: </p>
  
  <pre><code>UnicodeDecodeError: 'utf8' codec can't decode byte 0x88 in position 13: invalid start byte
  </code></pre>
  
  <p>Frankly I haven't understood why this exception is raised. </p>
  
  <p>I have found a working solution doing:</p>
  
  <pre><code>with open(filename) as f:
      for line in logfile:
          line = unicode(line, errors='ignore')
  </code></pre>
  
  <p>But this approach ended up being incredibly slow.
  Therefore my question is:</p>
  
  <p>Is there a correct way of doing this, and what is the fastest way?
  Thanks</p>