{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/20937","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/20937/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/20937/comments","events_url":"https://api.github.com/repos/apache/spark/issues/20937/events","html_url":"https://github.com/apache/spark/pull/20937","id":309787757,"number":20937,"title":"[SPARK-23094][SPARK-23723][SPARK-23724][SQL] Support custom encoding for json files","user":{"login":"MaxGekk","id":1580697,"avatar_url":"https://avatars1.githubusercontent.com/u/1580697?v=4","gravatar_id":"","url":"https://api.github.com/users/MaxGekk","html_url":"https://github.com/MaxGekk","followers_url":"https://api.github.com/users/MaxGekk/followers","following_url":"https://api.github.com/users/MaxGekk/following{/other_user}","gists_url":"https://api.github.com/users/MaxGekk/gists{/gist_id}","starred_url":"https://api.github.com/users/MaxGekk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaxGekk/subscriptions","organizations_url":"https://api.github.com/users/MaxGekk/orgs","repos_url":"https://api.github.com/users/MaxGekk/repos","events_url":"https://api.github.com/users/MaxGekk/events{/privacy}","received_events_url":"https://api.github.com/users/MaxGekk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2018-03-29T14:54:59Z","updated_at":"2018-04-05T19:36:49Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/20937","html_url":"https://github.com/apache/spark/pull/20937","diff_url":"https://github.com/apache/spark/pull/20937.diff","patch_url":"https://github.com/apache/spark/pull/20937.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nI propose new option for JSON datasource which allows to specify encoding (charset) of input and output files. Here is an example of using of the option:\r\n\r\n```\r\nspark.read.schema(schema)\r\n  .option(\"multiline\", \"true\")\r\n  .option(\"encoding\", \"UTF-16LE\")\r\n  .json(fileName)\r\n```\r\n\r\nIf the option is not specified, charset auto-detection mechanism is used by default.\r\n\r\nThe option can be used for saving datasets to jsons. Currently Spark is able to save datasets into json files in `UTF-8` charset only. The changes allow to save data in any supported charset. Here is the approximate list of supported charsets by Oracle Java SE: https://docs.oracle.com/javase/8/docs/technotes/guides/intl/encoding.doc.html . An user can specify the charset of output jsons via the charset option like `.option(\"charset\", \"UTF-16BE\")`. By default the output charset is still `UTF-8` to keep backward compatibility.\r\n\r\nThe solution has the following restrictions for per-line mode (`multiline = false`):\r\n\r\n- If charset is different from UTF-8, the lineSep option must be specified. The option required because Hadoop LineReader cannot detect the line separator correctly. Here is the ticket for solving the issue: https://issues.apache.org/jira/browse/SPARK-23725\r\n\r\n- Encoding with [BOM](https://en.wikipedia.org/wiki/Byte_order_mark) are not supported. For example, the `UTF-16` and `UTF-32` encodings are blacklisted. The problem can be solved by https://github.com/MaxGekk/spark-1/pull/2\r\n\r\n## How was this patch tested?\r\n\r\nI added the following tests:\r\n- reads an json file in `UTF-16LE` encoding with BOM in `multiline` mode\r\n- read json file by using charset auto detection (`UTF-32BE` with BOM) \r\n- read json file using of user's charset (`UTF-16LE`)\r\n- saving in `UTF-32BE` and read the result by standard library (not by Spark)\r\n- checking that default charset is `UTF-8`\r\n- handling wrong (unsupported) charset\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/379052341","html_url":"https://github.com/apache/spark/pull/20937#issuecomment-379052341","issue_url":"https://api.github.com/repos/apache/spark/issues/20937","id":379052341,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-04-05T19:36:49Z","updated_at":"2018-04-05T19:36:49Z","author_association":"NONE","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}