title: Python doesn't interpret UTF8 correctly
body: <p>I know similar questions have been asked a million times, but despite reading through many of them I can't find a solution that applies to my situation.</p>
  <p>I have a django application, in which I've created a management script. This script reads some text files, and outputs them to the terminal (it will do more useful stuff with the contents later, but I'm still testing it out) and the characters come out with escape sequences like <code>\xc3\xa5</code> instead of the intended <code>å</code>. Since that escape sequence means <code>Ã¥</code>, which is a common misinterpretation of <code>å</code> because of encoding problems, I suspect there are at least two places where this is going wrong. However, I can't figure out where - I've checked all the possible culprits I can think of:</p>
  <ul>
  <li>The terminal encoding is UTF-8; <code>echo $LANG</code> gives <code>en_US.UTF-8</code></li>
  <li>The text files are encoded in UTF-8; <code>file *</code> in the directory where they reside results in all entries being listed as &quot;UTF-8 Unicode text&quot; except one, which does not contain any non-ASCII characters and is listed as &quot;ASCII text&quot;. Running <code>iconv -f ascii -t utf8 thefile.txt &gt; utf8.txt</code> on that file yields another file with ASCII text encoding.</li>
  <li>The Python scripts are all UTF-8 (or, in several cases, ASCII with no non-ASCII characters). I tried inserting a comment in my management script with some special characters to force it to save as UTF-8, but it did not change the behavior. The above observations on the text files apply on all Python script files as well.</li>
  <li>The Python script that handles the text files has <code># -*- encoding: utf-8 -*-</code> at the top; the only line preceding that is <code>#!/usr/bin/python3</code>, but I've tried both changing to <code>.../python</code> for Python 2.7 or removing it entirely to leave it up to Django, without results.</li>
  <li>According to <a href="https://docs.djangoproject.com/en/dev/ref/unicode/" rel="nofollow noreferrer">the documentation</a>, &quot;Django natively supports Unicode data&quot;, so I &quot;can safely pass around Unicode strings&quot; anywhere in the application.</li>
  </ul>
  <p>I really can't think of anywhere else to look for a non-UTF-8 link in the chain. Where could I possibly have missed a setting to change to UTF-8?</p>
  <p>For completeness: I'm reading from the files with <code>lines = file.readlines()</code> and printing with the standard <code>print()</code> function. No manual encoding or decoding happens at either end.</p>
  <h3>UPDATE:</h3>
  <p>In response to quiestions in comments:</p>
  <ul>
  <li><code>print(sys.getdefaultencoding(), sys.stdout.encoding, f.encoding)</code> yields <code>('ascii', 'UTF-8', None)</code> for all files.</li>
  <li>I started compiling an SSCCE, and quickly found that the problem is only there if I try to print the value in a tuple. In other words, <code>print(lines[0].strip())</code> works fine, but <code>print(lines[0].strip(), lines[1].strip())</code> does not. Adding <code>.decode('utf-8')</code> yields a tuple where both strings are marked with a prepending <code>u</code> and <code>\xe5</code> (the correct escape sequence for <code>å</code>) instead of the odd characters before - but I can't figure out how to print them as regular strings, with no escape characters. I've tested another call to <code>.decode('utf-8')</code> as well as wrapping in <code>str()</code> but both fail with <code>UnicodeEncodeError</code> complaining that <code>\xe5</code> can't be encoded in ascii. Since a single string works correctly, I don't know what else to test.</li>
  </ul>
  <p><strong>SSCCE:</strong></p>
  <pre><code># -*- coding: utf-8 -*-
  
  import os, sys
  
  for root,dirs,files in os.walk('txt-songs'):
      for filename in files:
          with open(os.path.join(root,filename)) as f:
              print(sys.getdefaultencoding(), sys.stdout.encoding, f.encoding)
  
              lines = f.readlines()
              print(lines[0].strip()) # works
              print(lines[0].strip(), lines[1].strip()) # does not work
  </code></pre>