{"action":"created","issue":{"url":"https://api.github.com/repos/whatwg/encoding/issues/75","repository_url":"https://api.github.com/repos/whatwg/encoding","labels_url":"https://api.github.com/repos/whatwg/encoding/issues/75/labels{/name}","comments_url":"https://api.github.com/repos/whatwg/encoding/issues/75/comments","events_url":"https://api.github.com/repos/whatwg/encoding/issues/75/events","html_url":"https://github.com/whatwg/encoding/issues/75","id":180760626,"number":75,"title":"Confusion between Big5 and Big5-HKSCS encodings","user":{"login":"bhaible","id":1690467,"avatar_url":"https://avatars.githubusercontent.com/u/1690467?v=3","gravatar_id":"","url":"https://api.github.com/users/bhaible","html_url":"https://github.com/bhaible","followers_url":"https://api.github.com/users/bhaible/followers","following_url":"https://api.github.com/users/bhaible/following{/other_user}","gists_url":"https://api.github.com/users/bhaible/gists{/gist_id}","starred_url":"https://api.github.com/users/bhaible/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bhaible/subscriptions","organizations_url":"https://api.github.com/users/bhaible/orgs","repos_url":"https://api.github.com/users/bhaible/repos","events_url":"https://api.github.com/users/bhaible/events{/privacy}","received_events_url":"https://api.github.com/users/bhaible/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-10-03T22:13:36Z","updated_at":"2016-10-31T08:21:13Z","closed_at":null,"body":"The current draft maps the labels \"big5\" and \"big5-hkscs\" to a single encoding, and the mapping table that it uses (index-big5.txt) does not match either of the widely used mapping tables for BIG5 and BIG5-HKSCS.\n\nIn detail:\n\nGlibc and other software consider BIG5 and BIG5-HKSCS to be different.\nBIG5 is mainly used in Taiwan, and evolves over time without formal standards. But Microsoft is a strong player in this area, as it provides the Windows CP950 encoding.\nBIG5-HKSCS is mainly used in Hong Kong, and evolves through regular updates of the standard. The current version BIG5-HKSCS:2008 is found in http://www.ogcio.gov.hk/en/business/tech_promotion/ccli/terms/doc/e_hkscs_2008.pdf .\n\nThe mapping in index-big5.txt differs from the common BIG5 mapping (Microsoft CP950): It adds 5085 characters, at the positions 0x8862, 0x8864, 0x88A3, 0x88A5, 0x8740..0xA0FE, 0xA3C0..0xA3E0, 0xC6A1..0xC8FE, 0xFA40..0xFEFE.\n\nThe mapping in index-big5.txt differs from the one in BIG5-HKSCS:2008:\nIt has additional mappings for 0x8E69, 0x8E6F, 0x8E7E, 0x8EAB, 0x8EB4, 0x8ECD, 0x8ED0, 0x8F57, 0x8F69, 0x8F6E, 0x8FCB..0x8FCC, 0x8FFE, 0x906D, 0x907A, 0x90DC, 0x90F1, 0x91BF, 0x9244, 0x92AF..0x92B2, 0x92C8, 0x92D1, 0x9447, 0x94CA, 0x95D9, 0x9644, 0x96ED, 0x96FC, 0x9B76, 0x9B78, 0x9B7B, 0x9BC6, 0x9BDE, 0x9BEC, 0x9BF6, 0x9C42, 0x9C53, 0x9C62, 0x9C68, 0x9C6B, 0x9C77, 0x9CBC, 0x9CBD, 0x9CD0, 0x9D57, 0x9D5A, 0x9DC4, 0x9EA9, 0x9EEF, 0x9EFD, 0x9F60, 0x9F66, 0x9FCB, 0x9FD8, 0xA063, 0xA077, 0xA0D5, 0xA0DF, 0xA0E4, 0xA15A, 0xA1C3, 0xA1C5, 0xA1FE, 0xA240, 0xA2CC, 0xA2CE, 0xA3C0..0xA3E1, 0xC6CF, 0xC6D3, 0xC6D5, 0xC6D7, 0xC6DE..0xC6DF, 0xFA5F, 0xFA66, 0xFABD, 0xFAC5, 0xFAD5, 0xFB48, 0xFBB8, 0xFBF3, 0xFBF9, 0xFC4F, 0xFC6C, 0xFCB9, 0xFCE2, 0xFCF1, 0xFDB7, 0xFDB8, 0xFDBB, 0xFDF1, 0xFE52, 0xFE6F, 0xFEAA, 0xFEDD,\nand differs in the mappings of 0xA145, 0xA14E, 0xA1C2, 0xA1E3, 0xA1F2..0xA1F3, 0xA241..0xA242, 0xA244, 0xA246..0xA247.\n\nFor details about the mapping tables, see\nhttp://haible.de/bruno/charsets/conversion-tables/index.html\nhttp://haible.de/bruno/charsets/conversion-tables/Big5.html\nhttp://haible.de/bruno/charsets/conversion-tables/BIG5-HKSCS.html\n\nIn summary, it looks like this encoding is a clever merge between CP950 and BIG5-HKSCS, giving priority to the CP950 mapping value in those 11 positions where there is a conflict. This is very good for converting BIG5/BIG5-HKSCS texts of either kind to Unicode (the \"decoder\" part). However, when the \"encoder\" is used, it will likely generate HKSCS code points, which cannot be displayed in CP950. What shall the Windows users with this CP950 encoding do?\n"},"comment":{"url":"https://api.github.com/repos/whatwg/encoding/issues/comments/257236738","html_url":"https://github.com/whatwg/encoding/issues/75#issuecomment-257236738","issue_url":"https://api.github.com/repos/whatwg/encoding/issues/75","id":257236738,"user":{"login":"hsivonen","id":478856,"avatar_url":"https://avatars.githubusercontent.com/u/478856?v=3","gravatar_id":"","url":"https://api.github.com/users/hsivonen","html_url":"https://github.com/hsivonen","followers_url":"https://api.github.com/users/hsivonen/followers","following_url":"https://api.github.com/users/hsivonen/following{/other_user}","gists_url":"https://api.github.com/users/hsivonen/gists{/gist_id}","starred_url":"https://api.github.com/users/hsivonen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hsivonen/subscriptions","organizations_url":"https://api.github.com/users/hsivonen/orgs","repos_url":"https://api.github.com/users/hsivonen/repos","events_url":"https://api.github.com/users/hsivonen/events{/privacy}","received_events_url":"https://api.github.com/users/hsivonen/received_events","type":"User","site_admin":false},"created_at":"2016-10-31T08:21:13Z","updated_at":"2016-10-31T08:21:13Z","body":"> It seems that there are some code points (I haven't worked out which, yet) that the Encoding Standard has Big5 encoder mappings for but glibc does not have Big5-HKSCS encoder mappings for. \r\n\r\nOops. I misread what I saw. This was about decode after all.\r\n\r\nglibc's iconv's big5-hkscs fails when decoding 0xC6 0xD3, which Encoding Standard decodes to U+65E0."}}
{"actor":{"display_login":"hsivonen"}}