Take the 2-minute tour ×

I am trying to convert an incoming byte string that contains non-ascii characters into a valid utf-8 string such that I can dump is as json.

b = '\x80'
u8 = b.encode('utf-8')
j = json.dumps(u8)

I expected j to be '\xc2\x80' but instead I get:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)

In my situation, 'b' is coming from mysql via google protocol buffers and is filled out with some blob data.

Any ideas?

EDIT: I have ethernet frames that are stored in a mysql table as a blob (please, everyone, stay on topic and keep from discussing why there are packets in a table). The table collation is utf-8 and the db layer (sqlalchemy, non-orm) is grabbing the data and creating structs (google protocol buffers) which store the blob as a python 'str'. In some cases I use the protocol buffers directly with out any issue. In other cases, I need to expose the same data via json. What I noticed is that when json.dumps() does its thing, '\x80' can be replaced with the invalid unicode char (\ufffd iirc)

share|improve this question
You need to give a code snippet that shows what "use the protocol buffers directly without any issue" means. You need to show with a code snippet what you do with the protocol buffer to make json.dumps produce \ufffd. You need to say exactly what the consumer of this JSONised packet is expected to do to recover the original packet. –  John Machin Mar 7 '12 at 20:23
add comment

3 Answers

You need to examine the documentation for the software API that you are using. BLOB is an acronym: BINARY Large Object.

If your data is in fact binary, the idea of decoding it to Unicode is of course a nonsense.

If it is in fact text, you need to know what encoding to use to decode it to Unicode.

Then you use json.dumps(a_Python_object) ... if you encode it to UTF-8 yourself, json will decode it back again:

>>> import json
>>> json.dumps(u"\u0100\u0404")
>>> json.dumps(u"\u0100\u0404".encode('utf8'))

UPDATE about latin1:

u'\x80' is a useless meaningless C1 control character -- the encoding is extremely unlikely to be Latin-1. Latin-1 is "a snare and a delusion" -- all 8-bit bytes are decoded to Unicode without raising an exception. Don't confuse "works" and "doesn't raise an exception".

share|improve this answer
interesting. i guess i can keep it simple: print json.dumps('\x80'.decode('latin1')) –  kung-foo Mar 7 '12 at 16:19
@kung-foo: You have no evidence that latin1 is the correct encoding. –  John Machin Mar 7 '12 at 16:22
Then what is the method for encoding a string of bytes into utf-8? –  kung-foo Mar 7 '12 at 16:23
Where do BLOBs come into this? –  Marcin Mar 7 '12 at 16:30
@kung-foo: The method is a_string_of_bytes.decode('some_encoding').encode('utf8') ... however you don't need to encode into utf8 to be able to use json.dumps; you DO need to establish if your data is text and if so, what some_encoding is, and latin1 is unlikely ... ummm in other words: just re-read my answer slowly and carefully –  John Machin Mar 7 '12 at 16:34
show 7 more comments

I think what you are trying to do is decode the string object of some encoding. Do you know what that encoding is? To get the unicode object.

unicode_b = b.decode('some_encoding')

and then re-encoding the unicode object using the utf_8 encoding back to a string object.

b = unicode_b.encode('utf_8')

Using the unicode object as a translator, without knowing what the original encoding of the string is I can't know for certain but there is the possibility that the conversion will not go as expected. The unicode object is not meant for converting strings of one encoding to another. I would work with the unicode object assuming you know what the encoding is, if you don't know what the encoding is then there really isn't a way to find out without trial and error, and then convert back to the encoded string when you want a string object back.

share|improve this answer
add comment

Use b.decode('name of source encoding') to get a unicode version. This was surprising to me when I learned it. eg:

In [123]: 'foo'.decode('latin-1')
Out[123]: u'foo'
share|improve this answer
Remember: decode goes from bytes to unicode. encode goes from unicode to bytes. –  Daniel Roseman Mar 7 '12 at 16:00
@DanielRoseman Yes, that is why this is an answer to the question. –  Marcin Mar 7 '12 at 16:02
Sure, I wasn't arguing, just providing some extra explanation for the OP. –  Daniel Roseman Mar 7 '12 at 16:04
I had come up with the same thing but it seemed so inefficient. I am surprised there isn't a way to directly utf-8 encode a byte string. –  kung-foo Mar 7 '12 at 16:15
@kung-foo What do you mean? In what way is this not "direct"? –  Marcin Mar 7 '12 at 16:18
show 6 more comments

Your Answer


