title: Unicode file with python and fileinput
body: <p>I am becoming more and more convinced that the business of file encodings is made as confusing as possible on purpose. I have a problem with reading a file in <code>utf-8</code> encoding that contains  just one line:</p>
  
  <pre><code>“blabla this is some text”
  </code></pre>
  
  <p>(note that the quotation marks are some <em>fancy</em> version of the standard quotation marks).</p>
  
  <p>Now, I run this piece of <code>Python</code> code on it:</p>
  
  <pre><code>import fileinput
  def charinput(paths):
      with open(paths) as fi:
          for line in fi:
              for char in line:
                  yield char
  i = charinput('path/to/file.txt')
  for item in i:
      print(item)
  </code></pre>
  
  <p>with two results:
  If i run my python code from command prompt, the result is some strange characters, followed by an error mesage:</p>
  
  <pre><code>ď
  »
  ż
  â
  Traceback (most recent call last):
    File "krneki.py", line 11, in &lt;module&gt;
      print(item)
    File "C:\Python34\lib\encodings\cp852.py", line 19, in encode
      return codecs.charmap_encode(input,self.errors,encoding_map)[0]
  UnicodeEncodeError: 'charmap' codec can't encode character '\u20ac' in position
  0: character maps to &lt;undefined&gt;
  </code></pre>
  
  <p>I get the idea that the problem comes from the fact that Python tries to read a "wrongly" encoded document, but is there a way to order <code>fileinput.input</code> to read <code>utf-8</code>?</p>
  
  <hr>
  
  <p>EDIT: Some really weird stuff is happening and I have <strong>NO</strong> idea how any of it works. After saving the same file as before in <code>notepad++</code>, the python code now runs within IDLE and results in the following output (newlines removed):</p>
  
  <pre><code>ď»żâ€śblabla this is some textâ€ť
  </code></pre>
  
  <p>while I can get the command prompt to not crash if I first input <code>chcp 65001</code>. Running the file then results in</p>
  
  <pre><code>ÄÂ»Å¼Ã¢â‚¬Å›blabla this is some text Ã¢â‚¬Å¥
  </code></pre>
  
  <p>Any ideas? This is a horrible mess, if you ask me, but it is vital I understand it...</p>