title: How to handle undecodable filenames in Python?
body: <p>I'd really like to have my Python application deal exclusively with Unicode strings internally. This has been going well for me lately, but I've run into an issue with handling paths. The POSIX API for filesystems isn't Unicode, so it's possible (and actually somewhat common) for files to have "undecodable" names: filenames that aren't encoded in the filesystem's stated encoding.</p>
  
  <p>In Python, this manifests as a mixture of <code>unicode</code> and <code>str</code> objects being returned from <code>os.listdir()</code>.</p>
  
  <pre><code>&gt;&gt;&gt; os.listdir(u'/path/to/foo')
  [u'bar', 'b\xe1z']
  </code></pre>
  
  <p>In that example, the character <code>'\xe1'</code> is encoded in Latin-1 or somesuch, even when the (hypothetical) filesystem reports <code>sys.getfilesystemencoding() == 'UTF-8'</code> (in UTF-8, that character would be the two bytes <code>'\xc3\xa1'</code>). For this reason, you'll get <code>UnicodeError</code>s all over the place if you try to use, for example, <code>os.path.join()</code> with Unicode paths, because the filename can't be decoded.</p>
  
  <p>The <a href="http://docs.python.org/py3k/howto/unicode.html" rel="noreferrer">Python Unicode HOWTO</a> offers this advice about unicode pathnames:</p>
  
  <blockquote>
    <p>Note that in most occasions, the Unicode APIs should be used. The bytes APIs should only be used on systems where undecodable file names can be present, i.e. Unix systems.</p>
  </blockquote>
  
  <p>Because I mainly care about Unix systems, does this mean I should restructure my program to deal only with bytestrings for paths? (If so, how can I maintain Windows compatibility?) Or are there other, better ways of dealing with undecodable filenames? Are they rare enough "in the wild" that I should just ask users to rename their damn files?</p>
  
  <p>(If it is best to just deal with bytestrings internally, I have a followup question: How do I store bytestrings in SQLite for one column while keeping the rest of the data as friendly Unicode strings?)</p>