{"action":"created","issue":{"url":"https://api.github.com/repos/nodejs/node-eps/issues/15","repository_url":"https://api.github.com/repos/nodejs/node-eps","labels_url":"https://api.github.com/repos/nodejs/node-eps/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/nodejs/node-eps/issues/15/comments","events_url":"https://api.github.com/repos/nodejs/node-eps/issues/15/events","html_url":"https://github.com/nodejs/node-eps/pull/15","id":149868469,"number":15,"title":"00X: Add 'icu' module","user":{"login":"jasnell","id":439929,"avatar_url":"https://avatars.githubusercontent.com/u/439929?v=3","gravatar_id":"","url":"https://api.github.com/users/jasnell","html_url":"https://github.com/jasnell","followers_url":"https://api.github.com/users/jasnell/followers","following_url":"https://api.github.com/users/jasnell/following{/other_user}","gists_url":"https://api.github.com/users/jasnell/gists{/gist_id}","starred_url":"https://api.github.com/users/jasnell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasnell/subscriptions","organizations_url":"https://api.github.com/users/jasnell/orgs","repos_url":"https://api.github.com/users/jasnell/repos","events_url":"https://api.github.com/users/jasnell/events{/privacy}","received_events_url":"https://api.github.com/users/jasnell/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-04-20T19:51:58Z","updated_at":"2016-05-04T16:34:44Z","closed_at":"2016-04-30T05:56:21Z","pull_request":{"url":"https://api.github.com/repos/nodejs/node-eps/pulls/15","html_url":"https://github.com/nodejs/node-eps/pull/15","diff_url":"https://github.com/nodejs/node-eps/pull/15.diff","patch_url":"https://github.com/nodejs/node-eps/pull/15.patch"},"body":"## Description\r\n\r\nThe ICU4C library that we use for internationalization contains a significant array of additional functionality not currently exposed by the EcmaScript 402 standard. Some of this additional functionality would be useful to expose via a new `'icu'` or (`'unicode'`) module.\r\n\r\n## Interface\r\n\r\nInitially, the `'icu'` module would provide methods for the following:\r\n\r\n### Character encoding detection\r\n\r\nICU includes code that is able to look at a stream of bytes and apply heuristics to detect the character encoding in use. This is not always an exact match but it does a reasonably good job. We can tune this detection to only look for the character encodings we support in Core (ascii, iso8859-1, utf8 and utf16-le). Two specific APIs would be exposed by the `'icu'` module for this capability:\r\n   \r\n```js\r\nconst icu = require('icu');\r\n\r\n// Detect the encoding for a given buffer or string.\r\n// Returns a string with the most likely match.\r\nicu.detectEncoding(myBuffer);\r\nicu.detectEncoding(myString);\r\n\r\n// Detect the encoding for a given buffer or string.\r\n// Returns an object whose keys are the detected\r\n// encodings and whose values are a confidence value\r\n// provided by ICU. The higher the confidence value,\r\n// the better the match.\r\nconst encs = icu.detectEncodings(myBuffer);\r\nconsole.log(encs);\r\n // Prints something like {'ascii': 90, 'utf8': 15}\r\n```\r\n\r\nThis mechanism is useful when working with data that might be in multiple character sets (such as filenames on Linux, or reading through multiple files in a directory).\r\n\r\n```js\r\nconst data = getDataSomehow();\r\nconst buffer = Buffer.from(data, icu.detectEncoding(data));\r\n```\r\n\r\n### One-Shot and Streaming Buffer re-encoding\r\n\r\nICU includes code for converting from one encoding to another. This is similar to what is provided by `iconv` but it is built in to ICU4C. The `'icu'` module would include converters for *only* the character encodings directly supported by core. Developers would continue to use `iconv` or `iconv-lite` for more exotic things.\r\n\r\n```js\r\nconst icu = require('icu');\r\n\r\n// One-shot conversion. Converts the entire Buffer in one go.\r\n// Assumes that the Buffer is properly aligned on UFT-8 boundaries\r\nconst myBuffer = Buffer.from(getUtf8DataSomehow(), 'utf8');\r\nconst newBuffer = icu.reencode(myBuffer, 'utf8', 'ucs2');\r\n\r\n\r\n// Streaming conversion\r\nconst convertStream = icu.createConverter('utf8', 'ucs2');\r\nconvertStream.on('data', (chunk) => {\r\n  // chunk is a UTF-16 (ucs2) encoded buffer\r\n});\r\n// Writing UTF-8 data\r\nconvertStream.write(getUtf8DataSomehow());\r\n```\r\n\r\nAdditional convenience methods would be attached to `Buffer.prototype`:\r\n\r\n```js\r\nconst myBuffer = Buffer.from(getUtf8DataShow(), 'uf8');\r\nconst newBuffer = myBuffer.reencode('utf8', 'ucs2');\r\n```\r\n\r\nAgain, this would ONLY support the encodings for which we already have built-in support in core (acsii, iso8859-1, utf8 and utf16). This does not expand the encoding support in core so `iconv` and `iconv-lite` would still be necessary.\r\n\r\n### UTF-8 and UTF-16 aware `codePointAt()` and `charAt()` methods for `Buffer`\r\n\r\nThis one is pretty straightforward. They would return either the Unicode codepoint or the character at the given byte offset even if the byte offset is not on a UTF-8 or UTF-16 lead byte. These are intended to be symmetrical with `String.prototype.codePointAt()` and `String.prototype.charAt()`\r\n\r\n```js\r\nconst icu = require('icu');\r\n\r\nconst myBuffer = Buffer.from('a€bc', 'utf8');\r\n\r\nconsole.log(icu.codePointAt(myBuffer, 1, 'utf8'));\r\n// or\r\nconsole.log(myBuffer.codePointAt(1, 'utf8'));\r\n\r\nconsole.log(icu.charAt(myBuffer, 1, 'utf8'));\r\n// or\r\nconsole.log(myBuffer.charAt(1, 'utf8'));\r\n```\r\n\r\n### UTF-16 and UTF-8 aware `slice()` for `Buffer`\r\n \r\nThis is similar to the existing `Buffer.prototype.slice()` except that, rather than byte offsets, the `start` and `end` are codepoint/character offsets This would make it symmetrical with `String.prototype.slice()` but for Buffers. The advantage is that this allows the Buffer to be sliced in a way that ensures proper alignment with UTF-8 or UTF-16 encodings.\r\n\r\n```js\r\nconst icu = require('icu');\r\n\r\nconst myBuffer = Buffer.from('a€bc', 'utf8');\r\n\r\nicu.slice(myBuffer, 'utf8', 1, 2); // returns a Buffer with €\r\n// or\r\nBuffer.slice(1, 2, 'utf8'); // returns a Buffer with €\r\n```\r\n\r\n*Passing in either `ascii` or `binary` would fallback to the current `Buffer.slice()` behavior.*\r\n\r\n## Proof-of-concept\r\n\r\nA proof-on-concept implementation that includes everything except the streaming converters can be found at: \r\n\r\n  https://github.com/jasnell/node/tree/icu-module\r\n\r\nThe module requires turning on the basic ICU converter switch which has a nominal impact on binary size but nothing significant. Since the conversions we would require in core are purely algorithmic, there is no additional ICU data requirement (the current small-icu is just fine).\r\n\r\n/cc @trevnorris @bnoordhuis "},"comment":{"url":"https://api.github.com/repos/nodejs/node-eps/issues/comments/216922176","html_url":"https://github.com/nodejs/node-eps/pull/15#issuecomment-216922176","issue_url":"https://api.github.com/repos/nodejs/node-eps/issues/15","id":216922176,"user":{"login":"trevnorris","id":289174,"avatar_url":"https://avatars.githubusercontent.com/u/289174?v=3","gravatar_id":"","url":"https://api.github.com/users/trevnorris","html_url":"https://github.com/trevnorris","followers_url":"https://api.github.com/users/trevnorris/followers","following_url":"https://api.github.com/users/trevnorris/following{/other_user}","gists_url":"https://api.github.com/users/trevnorris/gists{/gist_id}","starred_url":"https://api.github.com/users/trevnorris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trevnorris/subscriptions","organizations_url":"https://api.github.com/users/trevnorris/orgs","repos_url":"https://api.github.com/users/trevnorris/repos","events_url":"https://api.github.com/users/trevnorris/events{/privacy}","received_events_url":"https://api.github.com/users/trevnorris/received_events","type":"User","site_admin":false},"created_at":"2016-05-04T16:34:44Z","updated_at":"2016-05-04T16:34:44Z","body":"@jasnell Why was this closed?"}}