Take the 2-minute tour ×

Good day

Can somebody explain what is going on behind the Django manage.py shell console? The problem is following. I'm developing a Django app, which is using an urllib to parse some html pages to get some info from them. And that info is in russian language, so it should be unicode (this is address string in this case). Next, my script feeds this to some other third-party module which falls, because it is not valid unicode string (I'm trying to geodecode point from address). I tried to print the string (parsed address in this case) to console with print address command but it fails:

... some useless stacktrace ...    
    print address

Now comes the interesting part.
I have 2 computers: workstation with Ubuntu and Python 2.7.2 and Debian Lenny VPS with Python 2.7.2. I start parser the same way on both machines: by executing python manage.py shell and calling my function from it.
First I got the same error on both installations, but then I noticed that my python encoding is set to 'ascii' (import sys; sys.getdefaultencoding()). And when I put

import sys; reload(sys).setdefaultencoding('utf-8')

into settings.py the problem solves for Ubuntu. Now I get proper print on it, e.g. г. Челябинск, ул. Кирова, д. 27, КТК Набережный, but this is not working for Debian.

If i delete this print address string than, I get non-readable geolocation errors, but again - only on Debian. Ubuntu is working just fine:

Failed to geodecode address [Ð³. Ð§ÐµÐ»ÑÐ±Ð¸Ð½ÑÐº, ÑÐ». 1-Ð¾Ð¹ ÐÑÑÐ¸Ð»ÐµÑÐºÐ¸, 17/1, ÑÑÐ½Ð¾Ðº ÐÑÑÐ°Ðº, 1-Ð·]

No amount of unicode(address).encode('utf-8') magic can help this.

So I just can't get it. What's the differences between machines that cause me so much trouble?

share|improve this question
That is interesting. Ubuntu is Debian, for all intents and purposes. –  Chris Pratt Sep 15 '11 at 14:41
My thought exactly –  gorodechnyj Sep 15 '11 at 16:18
add comment

1 Answer

up vote 2 down vote accepted

If you run the following python script, you'll see what's happening:

# -*- coding: utf-8 -*-
a = r"Челябинск"
print "Encode from UTF-8 to UTF-8:",unicode(a,'utf-8').encode('utf-8')
print "Encode from ISO8859-1 to UTF-8:",unicode(a,'iso8859-1').encode('utf-8')

The output is:

Encode from ISO8859-1 to UTF-8: Челябинск

Encode from ISO8859-1 to UTF-8: Ð§ÐµÐ»ÑÐ±Ð¸Ð½ÑÐº

In essence you're taking a string encoded (already) as UTF-8 and re-encoding it (a second time, as if it were ISO8859-1) into UTF-8.

It's worth checking what the default encoding of the machine is in each case.

If anyone can add to this answer then please do.

share|improve this answer
Thanks for the response. On later inspection I and my boss found out that it was lxml module who was causing so much trouble. We tried to screen our code from system encoding by proper use of unicode strings. But than lxml got an html page it decoded it with the use of system encoding and on server it guessed wrong. We already fixed this by parsing corresponding META of html page and encoding its content properly. But thanks for the answer - it is very enlightening. –  gorodechnyj Oct 3 '11 at 13:13
add comment

Your Answer


