title: Unicode characters output from python I/O to files
body: <p>I don't know if this is my misunderstanding of UTF-8 or of python, but I'm having trouble understanding how python writes Unicode characters to a file. I'm on a Mac under OSX by the way, if that makes a difference.</p>
  
  <p>Let's say I have the following unicode string</p>
  
  <p><code>foo=u'\x93Stuff in smartquotes\x94\n'</code></p>
  
  <p>Here \x93 and \x94 are those awful smart-quotes.</p>
  
  <p>Then I write it to a file:</p>
  
  <p><code>with open('file.txt','w') as file:
      file.write(foo.encode('utf8'))</code></p>
  
  <p>When I open the file in a text editor like TextWrangler or in a web browser <code>file.txt</code> seems like it was written as</p>
  
  <blockquote>
    <p>\xc2\x93**Stuff in smartquotes\xc2\x94\n</p>
  </blockquote>
  
  <p>The text editor properly understands the file to be UTF8 encoded, but it renders \xc2\x93 as garbage. If I go in and manually strip out the \xc2 part, I get what I expect, and TextWrangler and Firefox render the utf characters as smartquotes.</p>
  
  <p>This is exactly what I get when I read the file back into python without decoding it as 'utf8'. However, when I do read it in with the <code>read().decode('utf8')</code> method, I get back what I originally put in, without the \xc2 bit.</p>
  
  <p>This is driving me bonkers, because I'm trying to parse a bunch of html files into text and the incorrect rendering of these unicode characters is screwing up a bunch of stuff.</p>
  
  <p>I also tried it in python3 using the read/write methods normally, and it has the same behavior.</p>
  
  <p><strong>edit</strong>: Regarding stripping out the \xc2 manually, it turns out that it was rendering correctly when I did that because the browser and text editors were defaulting to Latin encoding.</p>
  
  <p>Also, as a follow up, Filefox renders the text as </p>
  
  <blockquote>
    <p>☐Stuff in smartquotes☐</p>
  </blockquote>
  
  <p>where the boxes are empty unicode values, while Chrome renders the text as </p>
  
  <blockquote>
    <p>Stuff in smartquotes</p>
  </blockquote>