{"action":"created","issue":{"url":"https://api.github.com/repos/pugjs/js-stringify/issues/4","repository_url":"https://api.github.com/repos/pugjs/js-stringify","labels_url":"https://api.github.com/repos/pugjs/js-stringify/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/pugjs/js-stringify/issues/4/comments","events_url":"https://api.github.com/repos/pugjs/js-stringify/issues/4/events","html_url":"https://github.com/pugjs/js-stringify/issues/4","id":181026526,"number":4,"title":"String arrives with different characters in Firefox","user":{"login":"mk-pmb","id":3442602,"avatar_url":"https://avatars.githubusercontent.com/u/3442602?v=3","gravatar_id":"","url":"https://api.github.com/users/mk-pmb","html_url":"https://github.com/mk-pmb","followers_url":"https://api.github.com/users/mk-pmb/followers","following_url":"https://api.github.com/users/mk-pmb/following{/other_user}","gists_url":"https://api.github.com/users/mk-pmb/gists{/gist_id}","starred_url":"https://api.github.com/users/mk-pmb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mk-pmb/subscriptions","organizations_url":"https://api.github.com/users/mk-pmb/orgs","repos_url":"https://api.github.com/users/mk-pmb/repos","events_url":"https://api.github.com/users/mk-pmb/events{/privacy}","received_events_url":"https://api.github.com/users/mk-pmb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-10-04T22:58:56Z","updated_at":"2016-10-05T22:01:06Z","closed_at":null,"body":"I wrote a node.js program to generate HTML code, the core:\r\n```javascript\r\nconsole.log(['\\uFEFF<!DOCTYPE html><html><body><p></p><script>',\r\n  String(charCodes),\r\n  'console.log(charCodes(' + jsStringify(animals) + '));',\r\n  'console.log(' + charCodes(jsStringify(animals)) + ');',\r\n  '</script></body></html>'].join('\\n'));\r\n```\r\n\r\nI [gisted](https://gist.github.com/mk-pmb/191436f9963fb82d7c9025324c76accb) the full source, also the generated HTML and what Firefox prints in its console when loading the HTML:\r\n\r\n```javascript\r\n{ \"dog\": \"[d83d][dc15]\", \"cow\": \"[d83d][dc04]\", \"halfbreed\": \"[fffd][fffd]\" }\r\n{ \"dog\": \"[d83d][dc15]\", \"cow\": \"[d83d][dc04]\", \"halfbreed\": \"[dc15][d83d]\" }\r\n```\r\n\r\nAs you can see, in the first expression, the halfbreed characters both became U+FFFD. Probably some Unicode interpolation is messing with JavaScript's UCS-2 characters. Is this a bug? If a feature instead, it should be documented more prominently."},"comment":{"url":"https://api.github.com/repos/pugjs/js-stringify/issues/comments/251812455","html_url":"https://github.com/pugjs/js-stringify/issues/4#issuecomment-251812455","issue_url":"https://api.github.com/repos/pugjs/js-stringify/issues/4","id":251812455,"user":{"login":"mk-pmb","id":3442602,"avatar_url":"https://avatars.githubusercontent.com/u/3442602?v=3","gravatar_id":"","url":"https://api.github.com/users/mk-pmb","html_url":"https://github.com/mk-pmb","followers_url":"https://api.github.com/users/mk-pmb/followers","following_url":"https://api.github.com/users/mk-pmb/following{/other_user}","gists_url":"https://api.github.com/users/mk-pmb/gists{/gist_id}","starred_url":"https://api.github.com/users/mk-pmb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mk-pmb/subscriptions","organizations_url":"https://api.github.com/users/mk-pmb/orgs","repos_url":"https://api.github.com/users/mk-pmb/repos","events_url":"https://api.github.com/users/mk-pmb/events{/privacy}","received_events_url":"https://api.github.com/users/mk-pmb/received_events","type":"User","site_admin":false},"created_at":"2016-10-05T22:01:06Z","updated_at":"2016-10-05T22:01:06Z","body":"Looks like Node.js's stdout cannot write those UCS-2 characters:\r\n\r\n```text\r\n$ nodejs -p '\"\\uDC15\\uD83D\"' | hd\r\n00000000  ef bf bd ef bf bd 0a                              |.......|\r\n00000007\r\n$ nodejs -e 'process.stdout.write(\"\\uDC15\\uD83D\")' | hd\r\n00000000  ef bf bd ef bf bd                                 |......|\r\n00000006\r\n```\r\n\r\n… and I assume it's not a node bug but the UCS-2 chars just _cannot_ be represented in stdout's default encoding, UTF-8. So what if I change the encoding:\r\n\r\n```text\r\n$ nodejs -e 'process.stdout.write(\"\\uDC15\\uD83D\", \"UCS-2\");' | hd\r\n00000000  15 dc 3d d8                                       |..=.|\r\n```\r\n\r\nThat works! So I change it in the HTML generator:\r\n```javascript\r\nhtml = ['\\uFEFF<!DOCTYPE html><html><head>',\r\n  // […]\r\n  '</script></body></html>'].join('\\n');\r\nprocess.stdout.write(html, 'UCS-2');\r\n```\r\n\r\nThanks to the BOM at start of HTML, Firefox now correctly detects `UTF-16LE`, ignores my meta tag, and… still reports `\"[fffd][fffd]\"`.\r\nSo even if I'd manage to re-configure all of my web stack to use UCS-2 instead of UTF-8, probably no luck. Also I think it would be overkill to convert all my templates, database replies etc. to UCS-2, when we have a lot of good infrastructure modules for UTF-8, all just to transmit some minor script tag correctly.\r\n\r\nThere has to be an easier way. How about I have js-stringify escape most data and then escape the non-UTF-8 chars myself?\r\n\r\n```javascript\r\nconsole.log(['\\uFEFF<!DOCTYPE html><html><head>',\r\n  '<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">',\r\n  '</head><body><p></p><script>',\r\n  String(charCodes),\r\n  'console.log(document.characterSet);',\r\n\r\n  'console.log(charCodes(' +\r\n    require('surrog8').uHHHH(jsStringify(animals)) +\r\n    '));',\r\n\r\n  'console.log(' + charCodes(jsStringify(animals)) + ');',\r\n  '</script></body></html>'].join('\\n'));\r\n```\r\n\r\nYeah! That one works! Firefox detects UTF-8 __and__ both objects arrive correctly! … because the HTML says:\r\n\r\n```javascript\r\nconsole.log(charCodes(\"{ \\\"dog\\\": \\\"\\uD83D\\uDC15\\\", \\\"cow\\\": \\\"\\uD83D\\uDC04\\\", \\\"halfbreed\\\": \\\"\\uDC15\\uD83D\\\" }\"));\r\n```\r\n\r\nSo now we know how to fix the encoding, and we just have to clarify whether data integrity is part of js-stringify's \"safely\" claim:\r\n\r\n> Stringify an object so it can be safely inlined in JavaScript code \r\n\r\n… or I shall make a new module to combine that with my need for \"verbatim\"."}}
{"actor":{"display_login":"mk-pmb"}}