title: python failing to encode strange characters
body: <p>I am attempting to parse txt file > 2GB by running following script:</p>
  
  <pre><code>#!/usr/bin/env python
  import json
  
  def convert2json(filename):
      with open(filename) as I:
          for line in I:
              d = {"data": line}
              print(json.dumps(d, ensure_ascii=False))
  
  if __name__ == "__main__":
      import sys
  
      convert2json(sys.argv[1])
  </code></pre>
  
  <p>Script throws error: </p>
  
  <pre><code>Traceback (most recent call last):
    File "ori.py", line 13, in &lt;module&gt;
      convert2json(sys.argv[1])
    File "ori.py", line 8, in convert2json
      print(json.dumps(d))
    File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
      return _default_encoder.encode(obj)
    File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
      chunks = self.iterencode(o, _one_shot=True)
    File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
      return _iterencode(o, 0)
  UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 31: invalid continuation byte
  </code></pre>
  
  <p>and it fails (I believe) when processing special characters:</p>
  
  <pre><code>&lt;E0&gt;ทำคาม:
  :&lt;E9&gt;皇甫
  :&lt;E9&gt;皇甫:&lt;E9&gt;皇甫:&lt;E9&gt;皇甫
  </code></pre>
  
  <p>How can I make the script to just ignore the lines that are causing problems?</p>
  
  <p>When I go to the file that I am parsing and copy a big chunk of the lines that cannot be processed, create a new file and run the script again - than it works. I am doing that  by copying lines from <code>less</code> and than to a file using <code>vi</code>. Am I doing something when copying the lines with the encoding itself?</p>