title: SQL Server (SQLCMD), Python and encoding issue when using non ascii chars
body: <p>i'm facing an encoding issue with my python code, when asking data that are in SQL Server 2005.</p>
  
  <p>(because i was <a href="https://stackoverflow.com/questions/7914108/error-unable-to-find-vcvarsall-bat-when-installing-pymssql-2-0-0b1-what-am-i">unable to compile PyMSSQL-2.0.0b1</a>) i'm using this <a href="http://code.activestate.com/recipes/144183-python-database-interface-for-ms-sql-server/" rel="nofollow noreferrer">piece of code</a> and i am able to do some select but now i stick with the issue that i do not know what SQLCMD is output-ting to me :(</p>
  
  <p>(i had to work with European language contained in table, so i had to face other encodings with accent and so on)</p>
  
  <p>for example : <br/></p>
  
  <ul>
  <li>when i read it (select) from the Ms SQLServer Management Studio i have this country name :  'Ceská republika' (note the first a is with acute on it)<br/></li>
  <li>when using it from SQLCMD from command line (Powershell in Windows 7), it is still ok, i can see the "Cesk'a with acute'"<br/></li>
  <li><p>now when using Python with the os.popen trick from the <a href="http://code.activestate.com/recipes/144183-python-database-interface-for-ms-sql-server/" rel="nofollow noreferrer">recipe</a>, that is with this connection string :<br/></p>
  
  <p>sqlcmd -U adminname -P password -S servername -d dbname /w 8192  -u</p></li>
  </ul>
  
  <p>i get this string : 'Cesk\xa0 republika'</p>
  
  <p>notice the \xa0 that i do know what encoding it is, and how i can pass from this \xa0 to {a with acute}...</p>
  
  <p>if i test from Python, and unicode i should have this one '\xe1'</p>
  
  <pre><code>&gt;&gt;&gt; unicode('Cesk\xa0 republika')
  
  Traceback (most recent call last):
    File "&lt;pyshell#13&gt;", line 1, in &lt;module&gt;
      unicode('Cesk\xa0 republika')
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 4: ordinal not in range(128)
  
  &gt;&gt;&gt; unicode_a_with_acute = u'\N{LATIN SMALL LETTER A WITH ACUTE}'
  &gt;&gt;&gt; unicode_a_with_acute
  u'\xe1'
  &gt;&gt;&gt; print unicode_a_with_acute
  á
  &gt;&gt;&gt; print unicode_a_with_acute.encode('cp1252')
  á
  &gt;&gt;&gt; unicode_a_with_acute.encode('cp1252')
  '\xe1'
  &gt;&gt;&gt; print 'Cesk\xa0 republika'.decode('cp1252')
  Cesk  republika
  &gt;&gt;&gt; print 'Cesk\xa0 republika'.decode('utf8')
  
  Traceback (most recent call last):
    File "&lt;pyshell#21&gt;", line 1, in &lt;module&gt;
      print 'Cesk\xa0 republika'.decode('utf8')
    File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
      return codecs.utf_8_decode(input, errors, True)
   UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 4: invalid start byte
  </code></pre>
  
  <p>so what SQLCMD is giving to me? How should i force it and/or os.popen and others to be sure that i have understandable utf8 for Python?</p>
  
  <p>(notice, i have tried both with and without the -u ending on the os.popen cmd for SQLCMD and that should stand for asking to SQLCMD to answer in unicode, with no effect, also i have tried to feed it with a "select" python string encoded in utf8 with no more success :</p>
  
  <pre><code> sqlstr = unicode('select * from table_pays where country_code="CZ"')
   cu = c.cursor
   lst = cu.execute(sqlstr)
   rows = cu.fetchall()
   for x in rows:
        print x
  
   ( 'CZ          ', 'Cesk\xa0 republika       ')
  </code></pre>
  
  <p>)</p>
  
  <p>another point : from what i googl-ed, about "sqlcmd.exe", there are also these parameters that could may be help : </p>
  
  <pre><code>[ -f &lt; codepage &gt; | i: &lt; codepage &gt; [ &lt; , o: &lt; codepage &gt; ] ]
  </code></pre>
  
  <p>but i was unable to specify the right one, i do not know what are the possible values, BTW using (or not using) the :</p>
  
  <pre><code>[ -u unicode output]
  </code></pre>
  
  <p>dit not help me also...</p>