{"action":"created","issue":{"url":"https://api.github.com/repos/rust-lang/rust/issues/23344","repository_url":"https://api.github.com/repos/rust-lang/rust","labels_url":"https://api.github.com/repos/rust-lang/rust/issues/23344/labels{/name}","comments_url":"https://api.github.com/repos/rust-lang/rust/issues/23344/comments","events_url":"https://api.github.com/repos/rust-lang/rust/issues/23344/events","html_url":"https://github.com/rust-lang/rust/issues/23344","id":61165537,"number":23344,"title":"stdio: Handle unicode boundaries better on Windows","user":{"login":"alexcrichton","id":64996,"avatar_url":"https://avatars.githubusercontent.com/u/64996?v=3","gravatar_id":"","url":"https://api.github.com/users/alexcrichton","html_url":"https://github.com/alexcrichton","followers_url":"https://api.github.com/users/alexcrichton/followers","following_url":"https://api.github.com/users/alexcrichton/following{/other_user}","gists_url":"https://api.github.com/users/alexcrichton/gists{/gist_id}","starred_url":"https://api.github.com/users/alexcrichton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexcrichton/subscriptions","organizations_url":"https://api.github.com/users/alexcrichton/orgs","repos_url":"https://api.github.com/users/alexcrichton/repos","events_url":"https://api.github.com/users/alexcrichton/events{/privacy}","received_events_url":"https://api.github.com/users/alexcrichton/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rust-lang/rust/labels/A-io","name":"A-io","color":"d7e102"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-03-13T18:06:52Z","updated_at":"2016-06-23T22:46:43Z","closed_at":null,"body":"The stdio support on windows has two separate modes. One is when stdout is not a console and implies byte orientation, the other is when stdout is a console and implies [u16] orientation. In the stdout-is-a-console case we require that all input and output to be valid unicode so we can translate [u8] to [u16].\r\n\r\nIn this console case, however, our translation may not be quite right for a number of scenarios:\r\n\r\n* When reading data, the console will give us a block of u16's which we need to translate to UTF-8. In theory the console can give us half of a surrogate pair (with the next half available on the next call to read), but we do not handle this case well currently. This should be fixed by simply adding a \"buffer of size 1\" to hold half of a surrogate pair if necessary.\r\n\r\n* When reading data, we require the entire set of input to be valid UTF-16. We should instead attempt to read as much of the input as possible as valid UTF-16, only returning an error for the actual invalid elements. For example if we read 10 elements, 5 of which are valid UTF-16, the 6th is bad, and then the remaining are all valid UTF-16, we should probably return the first 5 on a call to `read`, then return an error, then return the remaining on the next call to `read`.\r\n\r\n* When writing data, we require that the entire block of input is valid UTF-8. We should instead take a similar approach as above where we try to interpret as much of the input as possible as UTF-8, but we simply ignore the rest for that one call to `read` (returning an error if the first byte is invalid UTF-8).\r\n\r\n* When writing data, we don't handle the case where a multibyte character straddles calls to `write`. Like the reading case, this could be alleviated with a 4-byte buffer for unwritten-but-valid-utf8 characters.\r\n\r\n* When writing, we translate a block of valid UTF-8 to a block of UTF-16. Upon writing this UTF-16, however, not all characters may be written. There are two problems here:\r\n  * We need to translate a length of UTF-16 characters to a number of UTF-8 bytes that were written.\r\n  * If half of a surrogate pair is written, the other half should be \"buffered\" and the length of UTF-8 written should include the half-written character (as the extra data is buffered).\r\n\r\nAt this time I'm not sure what sort of guarantees Windows actually gives us on these APIs. For example does windows never deal with only half of a surrogate pair on a read/write? If Windows were to guarantee various properties like this (or that it always writes the entire buffer, for example), our lives would be a lot easier! For now, though, it is unclear what we can rely on, so this issue is written as if we rely on nothing."},"comment":{"url":"https://api.github.com/repos/rust-lang/rust/issues/comments/228208528","html_url":"https://github.com/rust-lang/rust/issues/23344#issuecomment-228208528","issue_url":"https://api.github.com/repos/rust-lang/rust/issues/23344","id":228208528,"user":{"login":"retep998","id":666308,"avatar_url":"https://avatars.githubusercontent.com/u/666308?v=3","gravatar_id":"","url":"https://api.github.com/users/retep998","html_url":"https://github.com/retep998","followers_url":"https://api.github.com/users/retep998/followers","following_url":"https://api.github.com/users/retep998/following{/other_user}","gists_url":"https://api.github.com/users/retep998/gists{/gist_id}","starred_url":"https://api.github.com/users/retep998/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/retep998/subscriptions","organizations_url":"https://api.github.com/users/retep998/orgs","repos_url":"https://api.github.com/users/retep998/repos","events_url":"https://api.github.com/users/retep998/events{/privacy}","received_events_url":"https://api.github.com/users/retep998/received_events","type":"User","site_admin":false},"created_at":"2016-06-23T22:46:43Z","updated_at":"2016-06-23T22:46:43Z","body":"Sure you can write bytes to the console by calling `WriteConsoleA` instead of `WriteConsoleW`, but it'll be interpreted as whatever the current code page is. While you can change the console output code page using `SetConsoleOutputCP`, trying to use UTF-8 will still suffer as it also has issues with multi byte codepoints straddling write boundaries. Trying to use `SetConsoleCP` to read UTF-8 is even worse as multi byte codepoints simply do not work. Rust takes the approach of trying to be as unicode friendly as possible by using wide functions. If you don't like that default and think unicode is a waste of time, then by all means feel free to call the appropriate system functions and write to stdout yourself in whatever manner you feel like.\r\n\r\n> The simplest example here would be a program that wants to write arbitrary binary data to stdout (expecting it to be piped somewhere),\r\n\r\nIf stdout _is_ piped then Rust won't check that what you're outputting is UTF-8 and will gladly just throw those bytes through the pipe. The only time it matters is when you're interacting with the Windows Console which is a __terrible__ place to read/write arbitrary binary data, and if your program tries to spew binary data to the console then that's just bad design."}}
{"actor":{"display_login":"retep998"}}