title: Python: unicode in system commands
body: <p>Suppose I have a mysterious unicode string in Python (2.7) that I want to feed to a command line program such as imagemagick (or really just get it <em>out</em> of Python in any way). The strings might be:</p>
  
  <ul>
  <li>Adolfo López Mateos</li>
  <li>Stanisława Walasiewicz</li>
  <li>Jörgen Jönsson</li>
  </ul>
  
  <p>So in Python I might make a little command like this:</p>
  
  <pre><code>cmd = u'convert -pointsize 24 label:"%s" "%s.png"' % (name, name)
  </code></pre>
  
  <p>If I just print <code>cmd</code> and get <code>convert -pointsize 24 label:"Jörgen Jönsson" "Jörgen Jönsson.png"</code> and then run it myself, everything is fine.</p>
  
  <ul>
  <li>Adolfo López Mateos.png </li>
  <li><a href="http://4u.jeffcrouse.info/stackoverflow/A-01.png" rel="nofollow noreferrer">example 1 http://4u.jeffcrouse.info/stackoverflow/A-01.png</a></li>
  <li>Stanisława Walasiewicz.png </li>
  <li><a href="http://4u.jeffcrouse.info/stackoverflow/A-02.png" rel="nofollow noreferrer">example 2 http://4u.jeffcrouse.info/stackoverflow/A-02.png</a></li>
  </ul>
  
  <p>But if I do <code>os.system( cmd )</code>, I get this:</p>
  
  <ul>
  <li>Adolfo L√≥pez Mateos.png</li>
  <li><a href="http://4u.jeffcrouse.info/stackoverflow/B-01.png" rel="nofollow noreferrer">example 4 http://4u.jeffcrouse.info/stackoverflow/B-01.png</a></li>
  <li>Stanis≈Çawa Walasiewicz.png</li>
  <li><a href="http://4u.jeffcrouse.info/stackoverflow/B-02.png" rel="nofollow noreferrer">example 5 http://4u.jeffcrouse.info/stackoverflow/B-02.png</a></li>
  </ul>
  
  <p>I know it's not an imagemagick problem because the filenames are messed up too. I know that Python is converting the command to ascii when it passes it off to os.system, but why is it getting the encoding so wrong?  Why is it interpreting each non-ASCII character as 2 characters?  <a href="http://www.joelonsoftware.com/articles/Unicode.html" rel="nofollow noreferrer">According to</a> <a href="http://docs.python.org/2/howto/unicode.html" rel="nofollow noreferrer">a few</a> <a href="http://packages.python.org/kitchen/unicode-frustrations.html" rel="nofollow noreferrer">articles</a> <a href="http://blog.lumino.so/2012/08/20/fix-unicode-mistakes-with-python/" rel="nofollow noreferrer">that I've read</a>, it might be because it's encoded as latin-1 but it's being read as utf-8, but I've tried encoding it back and forth between them and it's not helping.</p>
  
  <p>I get Unicode exceptions when I try to just encode it manually as ascii without a replacement argument, but if I do name.encode('ascii','xmlcharrefreplace'), I get the following:</p>
  
  <ul>
  <li><a href="http://4u.jeffcrouse.info/stackoverflow/C-01.png" rel="nofollow noreferrer">example 4 http://4u.jeffcrouse.info/stackoverflow/C-01.png</a></li>
  <li><a href="http://4u.jeffcrouse.info/stackoverflow/C-02.png" rel="nofollow noreferrer">example 5 http://4u.jeffcrouse.info/stackoverflow/C-02.png</a></li>
  </ul>
  
  <p>I'm hoping that someone recognizes this particular kind of encoding problem and can offer some advice, because I'm about out of ideas.</p>
  
  <p>Thanks!</p>