{"action":"created","issue":{"url":"https://api.github.com/repos/micropython/micropython/issues/4397","repository_url":"https://api.github.com/repos/micropython/micropython","labels_url":"https://api.github.com/repos/micropython/micropython/issues/4397/labels{/name}","comments_url":"https://api.github.com/repos/micropython/micropython/issues/4397/comments","events_url":"https://api.github.com/repos/micropython/micropython/issues/4397/events","html_url":"https://github.com/micropython/micropython/issues/4397","id":395856347,"node_id":"MDU6SXNzdWUzOTU4NTYzNDc=","number":4397,"title":"ujson.dumps() not handling non-ascii characters correctly","user":{"login":"martijnthe","id":193881,"node_id":"MDQ6VXNlcjE5Mzg4MQ==","avatar_url":"https://avatars3.githubusercontent.com/u/193881?v=4","gravatar_id":"","url":"https://api.github.com/users/martijnthe","html_url":"https://github.com/martijnthe","followers_url":"https://api.github.com/users/martijnthe/followers","following_url":"https://api.github.com/users/martijnthe/following{/other_user}","gists_url":"https://api.github.com/users/martijnthe/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnthe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnthe/subscriptions","organizations_url":"https://api.github.com/users/martijnthe/orgs","repos_url":"https://api.github.com/users/martijnthe/repos","events_url":"https://api.github.com/users/martijnthe/events{/privacy}","received_events_url":"https://api.github.com/users/martijnthe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2019-01-04T09:14:08Z","updated_at":"2019-01-04T10:34:44Z","closed_at":null,"author_association":"NONE","body":"`ujson.dumps` does not appear to properly escape non-ascii characters and generated JSON that is not up to the spec (and fails to parse in a web browser context for example).\r\n\r\n```\r\n$ ./micropython\r\nMicroPython v1.9.4-771-gb33f108cd-dirty on 2019-01-04; darwin version\r\nUse Ctrl-D to exit, Ctrl-E for paste mode\r\n>>> from ujson import dumps\r\n>>> dumps(\"\\U0001F602\")\r\n'\"\\U0001f602\"'\r\n>>> dumps(\"\\u2082\")\r\n'\"\\u2082\"'\r\n>>> print(\"\\u2082\")\r\n₂\r\n>>> print(\"\\U0001F602\")\r\n😂\r\n```\r\n\r\nSee source code: https://github.com/micropython/micropython/blob/5a91fce9f868eeba37a0b1cf6c3b4435ed5eecec/py/objstr.c#L95\r\n\r\nFor the emoji example (outside the BMP), the code point should be escaped as a surrogate pair `\"\\uD83D\\uDE02\"`.\r\nThe subscript two should be escaped as `\"\\u2082\"`.\r\n\r\nI'm happy to fix it, but some guidance would be nice around these questions:\r\n1. IMHO, JSON unicode escaping should happen, even if `MICROPY_PY_BUILTINS_STR_UNICODE` is not defined. Agree?\r\n2. There's already some utf-8 decoding code in the loop in `uni_print_quoted`, but it's not purely utf-8 decoding that is happening there (it's mixed with somewhat python specific backslash-escaping stuff). So I'm not sure if it's worth extracting that. Since it's just a few lines of duplication, I'm thinking of basically copy the relevant parts over, plus adding the additional handling for codepoints outside of the BMP and emit surrogate pairs.\r\n3. Why is there no test expectation for `ujson_dumps.py`? Can I add one?"},"comment":{"url":"https://api.github.com/repos/micropython/micropython/issues/comments/451408995","html_url":"https://github.com/micropython/micropython/issues/4397#issuecomment-451408995","issue_url":"https://api.github.com/repos/micropython/micropython/issues/4397","id":451408995,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MTQwODk5NQ==","user":{"login":"martijnthe","id":193881,"node_id":"MDQ6VXNlcjE5Mzg4MQ==","avatar_url":"https://avatars3.githubusercontent.com/u/193881?v=4","gravatar_id":"","url":"https://api.github.com/users/martijnthe","html_url":"https://github.com/martijnthe","followers_url":"https://api.github.com/users/martijnthe/followers","following_url":"https://api.github.com/users/martijnthe/following{/other_user}","gists_url":"https://api.github.com/users/martijnthe/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnthe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnthe/subscriptions","organizations_url":"https://api.github.com/users/martijnthe/orgs","repos_url":"https://api.github.com/users/martijnthe/repos","events_url":"https://api.github.com/users/martijnthe/events{/privacy}","received_events_url":"https://api.github.com/users/martijnthe/received_events","type":"User","site_admin":false},"created_at":"2019-01-04T10:34:44Z","updated_at":"2019-01-04T10:34:44Z","author_association":"NONE","body":"@pfalcon thanks for the quick reply.\r\n\r\nMy bad, I misinterpreted the [spec] / missed crucial parts... TIL, thanks!\r\n\r\n[spec]: https://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf"}}
{"actor":{"display_login":"martijnthe"}}