{"action":"created","issue":{"url":"https://api.github.com/repos/golang/go/issues/16282","repository_url":"https://api.github.com/repos/golang/go","labels_url":"https://api.github.com/repos/golang/go/issues/16282/labels{/name}","comments_url":"https://api.github.com/repos/golang/go/issues/16282/comments","events_url":"https://api.github.com/repos/golang/go/issues/16282/events","html_url":"https://github.com/golang/go/issues/16282","id":164187839,"number":16282,"title":"encoding/json: unexpected handling of Unicode characters","user":{"login":"adityats","id":7434202,"avatar_url":"https://avatars.githubusercontent.com/u/7434202?v=3","gravatar_id":"","url":"https://api.github.com/users/adityats","html_url":"https://github.com/adityats","followers_url":"https://api.github.com/users/adityats/followers","following_url":"https://api.github.com/users/adityats/following{/other_user}","gists_url":"https://api.github.com/users/adityats/gists{/gist_id}","starred_url":"https://api.github.com/users/adityats/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adityats/subscriptions","organizations_url":"https://api.github.com/users/adityats/orgs","repos_url":"https://api.github.com/users/adityats/repos","events_url":"https://api.github.com/users/adityats/events{/privacy}","received_events_url":"https://api.github.com/users/adityats/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/golang/go/milestones/42","html_url":"https://github.com/golang/go/milestones/Go1.8Maybe","labels_url":"https://api.github.com/repos/golang/go/milestones/42/labels","id":1748849,"number":42,"title":"Go1.8Maybe","description":"","creator":{"login":"bradfitz","id":2621,"avatar_url":"https://avatars.githubusercontent.com/u/2621?v=3","gravatar_id":"","url":"https://api.github.com/users/bradfitz","html_url":"https://github.com/bradfitz","followers_url":"https://api.github.com/users/bradfitz/followers","following_url":"https://api.github.com/users/bradfitz/following{/other_user}","gists_url":"https://api.github.com/users/bradfitz/gists{/gist_id}","starred_url":"https://api.github.com/users/bradfitz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bradfitz/subscriptions","organizations_url":"https://api.github.com/users/bradfitz/orgs","repos_url":"https://api.github.com/users/bradfitz/repos","events_url":"https://api.github.com/users/bradfitz/events{/privacy}","received_events_url":"https://api.github.com/users/bradfitz/received_events","type":"User","site_admin":false},"open_issues":64,"closed_issues":3,"state":"open","created_at":"2016-05-04T23:40:26Z","updated_at":"2016-07-07T21:11:30Z","due_on":"2017-02-01T08:00:00Z","closed_at":null},"comments":8,"created_at":"2016-07-06T22:18:25Z","updated_at":"2016-07-07T21:15:05Z","closed_at":null,"body":"\r\n1. What version of Go are you using (`go version`)?\r\n1.6.2\r\n\r\n2. What operating system and processor architecture are you using (`go env`)?\r\nGOARCH=\"amd64\"\r\nGOBIN=\"\"\r\nGOEXE=\"\"\r\nGOHOSTARCH=\"amd64\"\r\nGOHOSTOS=\"linux\"\r\nGOOS=\"linux\"\r\nGOPATH=\"*****/src/avi/go\"\r\nGORACE=\"\"\r\nGOROOT=\"/usr/local/go\"\r\nGOTOOLDIR=\"/usr/local/go/pkg/tool/linux_amd64\"\r\nGO15VENDOREXPERIMENT=\"1\"\r\nCC=\"gcc\"\r\nGOGCCFLAGS=\"-fPIC -m64 -pthread -fmessage-length=0\"\r\nCXX=\"g++\"\r\nCGO_ENABLED=\"1\"\r\n\r\n3. What did you do?\r\nI am reporting the problem from the surface and if need be will provide more details (which are complicated and hard to list here).\r\n\r\nI have a protobuf call Pool (It is in a file called pool.pb.go).\r\nStorage of this protobuf goes something like this:\r\n{\"obj\": <Serialized/Marshalled string of the Pool protobuf>} and this <map> is serialized and stored as one big JSON blob:\r\n\r\n\"{\\\"obj\\\":\\\"\\\\n)pool-6b9bad47-65ac-477c-a32a-6fd5c7a98c91\\\\u0012\\\\ttest-pool(P@\\\\u0001H\\\\nP\\\\u0000Z2healthmonitor-665dbe8c-aecb-49df-97b1-a87214e65270Z2healthmonitor-f6052e41-616d-46d6-989d-0fdfe0c5a403b1\\\\n\\\\u0011\\\\n\\\\r10.160.161.11\\\\u0010\\\\u0000\\\\u0010P\\\\u001A\\\\r10.160.161.11 \\\\u0001(\\\\u0001X\\\\u0000h\\\\u0000r\\\\u0000\\\\u0080\\\\u0001\\\\u0000h\\\\u0001\\\\u00A0\\\\u0001\\\\u0001\\\\u00B0\\\\u0002\\\\u0001\\\\u00B8\\\\u0002\\\\u0000\\\\u00C2\\\\u0002\\\\u0002\\\\b\\\\u0003\\\\u00D0\\\\u0002\\\\u0000\\\\u00F0\\\\u0002\\\\u0000\\\\u00A2\\\\u0003/vrfcontext-bfec9a97-7df7-4cfd-bd6d-ee84415ed1ab\\\\u00B0\\\\u0003\\\\n\\\\u00B8\\\\u0003\\\\u0001\\\\u00E8\\\\u0003\\\\u0000\\\\u00F0\\\\u0003\\\\u0080\\\\u0001\\\\u0082\\\\u0004\\\\u0006\\\\b\\\\u0000\\\\u0018\\\\u0004 \\\\u0000\\\\u0098\\\\u0004\\\\u0000\\\\u00A8\\\\u0004\\\\u0001\\\\u00A2\\\\u0006\\\\u0005admin\\\\u00AA\\\\u0006*cloud-2c2c062a-9d10-4d3b-a2e5-6ac9af08a4c4\\\\u0082\\\\u00A6\\\\u001D\\\\npool.proto\\\\u008A\\\\u00A6\\\\u001D\\\\u0004Pool\\\"}\"\r\n\r\n\r\n4. What did you expect to see?\r\nWhen I UnMarshal the above blob, I expect to see this:\r\n\r\n \"obj\": \"\\n)pool-6b9bad47-65ac-477c-a32a-6fd5c7a98c91\\x12\\ttest-pool(P@\\x01H\\nP\\x00Z2healthmonitor-665dbe8c-aecb-49df-97b1-a87214e65270Z2healthmonitor-f6052e41-616d-46d6-989d-0fdfe0c5a403b1\\n\\x11\\n\\r10.160.161.11\\x10\\x00\\x10P\\x1a\\r10.160.161.11 \\x01(\\x01X\\x00h\\x00r\\x00\\x80\\x01\\x00h\\x01\\xa0\\x01\\x01\\xb0\\x02\\x01\\xb8\\x02\\x00\\xc2\\x02\\x02\\x08\\x03\\xd0\\x02\\x00\\xf0\\x02\\x00\\xa2\\x03/vrfcontext-bfec9a97-7df7-4cfd-bd6d-ee84415ed1ab\\xb0\\x03\\n\\xb8\\x03\\x01\\xe8\\x03\\x00\\xf0\\x03\\x80\\x01\\x82\\x04\\x06\\x08\\x00\\x18\\x04 \\x00\\x98\\x04\\x00\\xa8\\x04\\x01\\xa2\\x06\\x05admin\\xaa\\x06*cloud-2c2c062a-9d10-4d3b-a2e5-6ac9af08a4c4\\x82\\xa6\\x1d\\npool.proto\\x8a\\xa6\\x1d\\x04Pool\"\r\n\r\nAll Unicode characters are decoded to corresponding hex versions correctly.\r\n\r\n\r\n5. What did you see instead?\r\n\r\n\"obj\": \"\\n)pool-6b9bad47-65ac-477c-a32a-6fd5c7a98c91\\x12\\ttest-pool(P@\\x01H\\nP\\x00Z2healthmonitor-665dbe8c-aecb-49df-97b1-a87214e65270Z2healthmonitor-f6052e41-616d-46d6-989d-0fdfe0c5a403b1\\n\\x11\\n\\r10.160.161.11\\x10\\x00\\x10P\\x1a\\r10.160.161.11 \\x01(\\x01X\\x00h\\x00r\\x00\\u0080\\x01\\x00h\\x01\\u00a0\\x01\\x01°\\x02\\x01¸\\x02\\x00Â\\x02\\x02\\b\\x03Ð\\x02\\x00ð\\x02\\x00¢\\x03/vrfcontext-bfec9a97-7df7-4cfd-bd6d-ee84415ed1ab°\\x03\\n¸\\x03\\x01è\\x03\\x00ð\\x03\\u0080\\x01\\u0082\\x04\\x06\\b\\x00\\x18\\x04 \\x00\\u0098\\x04\\x00¨\\x04\\x01¢\\x06\\x05adminª\\x06*cloud-2c2c062a-9d10-4d3b-a2e5-6ac9af08a4c4\\u0082¦\\x1d\\npool.proto\\u008a¦\\x1d\\x04Pool\"\r\n\r\nOnly Unicode characters lesser than (u\\\\0080) are decoded to corresponding hex versions correctly. The rest of them are kept as is (look for u\\\\ in the above string, whereas in the correct case all of these should be decoded to the relevant hex values). This looks like the Unmarshal method in encoding/json decodes only if the input fits in 2 bytes?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/golang/go/issues/comments/231209572","html_url":"https://github.com/golang/go/issues/16282#issuecomment-231209572","issue_url":"https://api.github.com/repos/golang/go/issues/16282","id":231209572,"user":{"login":"robpike","id":4324516,"avatar_url":"https://avatars.githubusercontent.com/u/4324516?v=3","gravatar_id":"","url":"https://api.github.com/users/robpike","html_url":"https://github.com/robpike","followers_url":"https://api.github.com/users/robpike/followers","following_url":"https://api.github.com/users/robpike/following{/other_user}","gists_url":"https://api.github.com/users/robpike/gists{/gist_id}","starred_url":"https://api.github.com/users/robpike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robpike/subscriptions","organizations_url":"https://api.github.com/users/robpike/orgs","repos_url":"https://api.github.com/users/robpike/repos","events_url":"https://api.github.com/users/robpike/events{/privacy}","received_events_url":"https://api.github.com/users/robpike/received_events","type":"User","site_admin":false},"created_at":"2016-07-07T21:15:05Z","updated_at":"2016-07-07T21:15:05Z","body":"Please provide a complete working example. Without one it's difficult to understand the problem you are seeing.\r\n\r\nI trust you know that \\u0098 is a Unicode character that when stored as text will not be the bytes \\x00\\x98 but instead the UTF-8 encoding \\xc2\\x98. When you say \"ideally\" \\u0098 will become \\x00\\x98 you appear to be missing this point.\r\n\r\nTo put it another way, in a string \\uNNNN does not represent a byte, it represents the byte sequence necessary to represent the Unicode code point NNNN."}}
{"actor":{"display_login":"robpike"}}