Take the 2-minute tour ×

I was hunting down a bug in my code today and found something particularly worrying. In python:

wim@wim-acer:~$ python
[GCC 4.5.2] on linux2
>>> x = u'$€%'
>>> x.find('%')
>>> len(x)

Whereas in ipython:

|3>x = u'$€%'

[4] 4

[5] 5

Could anyone enlighten me as to what's going on here?

edit: with some additional info requested from the comments below:


|53>import sys, locale

<module 'sys' (built-in)>



|57>x = u'$€%'


|59>print x



>>> import sys, locale
>>> reload(sys)
<module 'sys' (built-in)>
>>> sys.setdefaultencoding(locale.getdefaultlocale()[1])
>>> sys.getdefaultencoding()
>>> x = u'$€%'
>>> x
>>> print x
>>> len(x)
share|improve this question
What happens if you type x <Enter> on the command line in CPython? What character encodings are your shells using? –  Tim Pietzcker Sep 29 '11 at 6:53
print x gives $€%, whereas just x gives u'$\u20ac%' –  wim Sep 29 '11 at 6:57
sys.stdin.encoding is 'UTF-8' –  wim Sep 29 '11 at 8:23
This is certainly a bug in the ipython shell. It shouldn't affect your running programs though. Broken Unicode terminals are quite common in general (just look at the Windows console...) so relying on being able to type and print Unicode to console, in any language, is generally pretty iffy. It will work fine inside scripts themselves and for other methods of IO. –  bobince Sep 29 '11 at 19:49

2 Answers 2

up vote 5 down vote accepted

@nye17 It's officially not a good idea to ever call setdefaultencoding() (it is removed from sys after first use for a reason). One common culprit is gtk, which causes all kinds of problems, so if IPython has imported gtk, sys.getdefaultencoding() will return utf8. IPython does not set the default encoding itself.

@wim can I ask what version of IPython you are using? Part of the major overhaul in 0.11 was fixing many unicode bugs, but more do crop up (mostly on Windows, now).

I ran your test case in IPython 0.11, and the behavior of IPython and Python do appear to be the same, so I think this bug is fixed.

Relevant values:

  • sys.stdin.encoding = utf8
  • sys.getdefaultencoding() = ascii
  • platforms tested: Ubuntu 10.04+Python2.6.5, OSX 10.7+Python2.7.1

As for an explanation, essentially IPython didn't recognize that input could be unicode. In IPython 0.10, the multibyte utf8 input is not being respected, so each byte = 1 character, which you can see with:

In [1]: x = '$€%'

In [2]: x
Out[2]: '$\xe2\x82\xac%'

In [3]: y = u'$€%'

In [4]: y
Out[4]: u'$\xe2\x82\xac%'# wrong!

Whereas, what should happen, and what does happen in 0.11, is that y == x.decode(sys.stdin.encoding), not repr(y) == 'u'+repr(x).

share|improve this answer
you're correct, it was a ipython bug.. i built 0.12.dev and it's right now. oh, the new inline qtconsole is very cool too!! –  wim Sep 30 '11 at 9:24

if you do

import sys

I think you will get different results in python an ipython, possible one ascii, and the other one being utf-8, so it should only be a matter of which default encoding each one is choosing.

The other test you can do is to type the following to enfore it as your default locale,

import sys, locale

then try the test of x in your question.

share|improve this answer
Both ipython and python give me 'ascii'. –  wim Sep 29 '11 at 7:06
@wim that is disturbing. I just tested in my ipython and python, the len and print give the same output as in your questin, but my defaultencodings are different, one ascii, the one utf-8. –  nye17 Sep 29 '11 at 7:10
@wim can you try the test I edited in the answer? basically to enforce the same encoding in both places to see whether strings are behaving in the same way. –  nye17 Sep 29 '11 at 7:25
In the Notebook, the sys module doesn't have the setdefaultencoding() method, nor is the builtin reload() method available. so i have no idea how to get unicode working in the IPy Notebook –  taylor May 26 '13 at 17:32

Your Answer


