{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13518","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13518/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13518/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13518/events","html_url":"https://github.com/apache/spark/pull/13518","id":158549193,"number":13518,"title":"[WIP][SPARK-15472][SQL] Add support for writing in `csv`, `json`, `text` formats in Structured Streaming","user":{"login":"lw-lin","id":15843379,"avatar_url":"https://avatars.githubusercontent.com/u/15843379?v=3","gravatar_id":"","url":"https://api.github.com/users/lw-lin","html_url":"https://github.com/lw-lin","followers_url":"https://api.github.com/users/lw-lin/followers","following_url":"https://api.github.com/users/lw-lin/following{/other_user}","gists_url":"https://api.github.com/users/lw-lin/gists{/gist_id}","starred_url":"https://api.github.com/users/lw-lin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lw-lin/subscriptions","organizations_url":"https://api.github.com/users/lw-lin/orgs","repos_url":"https://api.github.com/users/lw-lin/repos","events_url":"https://api.github.com/users/lw-lin/events{/privacy}","received_events_url":"https://api.github.com/users/lw-lin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-06-05T11:02:44Z","updated_at":"2016-06-06T12:52:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13518","html_url":"https://github.com/apache/spark/pull/13518","diff_url":"https://github.com/apache/spark/pull/13518.diff","patch_url":"https://github.com/apache/spark/pull/13518.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis patch adds support for writing in `csv`, `json`, `text` formats in Structured Streaming:\r\n\r\n**1. at a high level, this patch forms the following hierarchy**(`text` as an example):\r\n```\r\n                <OutputWriter>\r\n                      ↑\r\n             TextOutputWriterBase\r\n                 ↗          ↖\r\nBatchTextOutputWriter   StreamingTextOutputWriter\r\n```\r\n```\r\n                    <OutputWriterFactory>\r\n                        ↗          ↖\r\nBatchTextOutputWriterFactory   StreamingOutputWriterFactory\r\n                                              ↑\r\n                              StreamingTextOutputWriterFactory\r\n```\r\nThe `StreamingTextOutputWriter` and other 'streaming' output writers would write data **without** using an `OutputCommitter`. This was the same approach taken by [SPARK-14716](https://github.com/apache/spark/pull/12409).\r\n\r\n**2. to support compression, this patch attaches an extension to the path assigned by `FileStreamSink`**, which is slightly different from [SPARK-14716](https://github.com/apache/spark/pull/12409). For example, if we are writing out using the `gzip` compression and `FileStreamSink` assigns path `${uuid}` to a text writer, then in the end the file written out will be `${uuid}.txt.gz` -- so that when we read the file back, we'll correctly interpret it as `gzip` compressed.\r\n\r\n## How was this patch tested?\r\n\r\n`FileStreamSinkSuite` is expanded much more to cover the added `csv`, `json`, `text` formats:\r\n\r\n```scala\r\ntest(\" csv - unpartitioned data - codecs: none/gzip\")\r\ntest(\"json - unpartitioned data - codecs: none/gzip\")\r\ntest(\"text - unpartitioned data - codecs: none/gzip\")\r\n\r\ntest(\" csv - partitioned data - codecs: none/gzip\")\r\ntest(\"json - partitioned data - codecs: none/gzip\")\r\ntest(\"text - partitioned data - codecs: none/gzip\")\r\n\r\ntest(\" csv - unpartitioned writing and batch reading - codecs: none/gzip\")\r\ntest(\"json - unpartitioned writing and batch reading - codecs: none/gzip\")\r\ntest(\"text - unpartitioned writing and batch reading - codecs: none/gzip\")\r\n\r\ntest(\" csv - partitioned writing and batch reading - codecs: none/gzip\")\r\ntest(\"json - partitioned writing and batch reading - codecs: none/gzip\")\r\ntest(\"text - partitioned writing and batch reading - codecs: none/gzip\")\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/223949717","html_url":"https://github.com/apache/spark/pull/13518#issuecomment-223949717","issue_url":"https://api.github.com/repos/apache/spark/issues/13518","id":223949717,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-06-06T12:52:37Z","updated_at":"2016-06-06T12:52:37Z","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}