{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/23163","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/23163/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/23163/comments","events_url":"https://api.github.com/repos/apache/spark/issues/23163/events","html_url":"https://github.com/apache/spark/pull/23163","id":385188815,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM0MjE0MjQ1","number":23163,"title":"[SPARK-26164][SQL] Allow FileFormatWriter to write multiple partitions/buckets without sort","user":{"login":"c21","id":4629931,"node_id":"MDQ6VXNlcjQ2Mjk5MzE=","avatar_url":"https://avatars2.githubusercontent.com/u/4629931?v=4","gravatar_id":"","url":"https://api.github.com/users/c21","html_url":"https://github.com/c21","followers_url":"https://api.github.com/users/c21/followers","following_url":"https://api.github.com/users/c21/following{/other_user}","gists_url":"https://api.github.com/users/c21/gists{/gist_id}","starred_url":"https://api.github.com/users/c21/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/c21/subscriptions","organizations_url":"https://api.github.com/users/c21/orgs","repos_url":"https://api.github.com/users/c21/repos","events_url":"https://api.github.com/users/c21/events{/privacy}","received_events_url":"https://api.github.com/users/c21/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-11-28T10:01:48Z","updated_at":"2018-11-29T06:30:40Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/23163","html_url":"https://github.com/apache/spark/pull/23163","diff_url":"https://github.com/apache/spark/pull/23163.diff","patch_url":"https://github.com/apache/spark/pull/23163.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nCurrently spark always requires a local sort before writing to output table on partition/bucket columns (see `write.requiredOrdering` in `FileFormatWriter.scala`), which is unnecessary, and can be avoided by keeping multiple output writers concurrently in `FileFormatDataWriter.scala`.\r\n\r\nThis pr is first doing hash-based write, then falling back to sort-based write (current implementation) when number of opened writer exceeding a threshold (controlled by a config). Specifically:\r\n\r\n1. (hash-based write) Maintain mapping between file path and output writer, and re-use writer for writing input row. In case of the number of opened output writers exceeding a threshold (can be changed by a config), we go to 2.\r\n\r\n2. (sort-based write) Sort the rest of input rows (use the same sorter in SortExec). Then writing the rest of sorted rows, and we can close the writer on the fly, in case no more rows for current file path.\r\n\r\n## How was this patch tested?\r\n\r\nAdded unit test in `DataFrameReaderWriterSuite.scala`. Existing test like `SQLMetricsSuite.scala` would already exercise the code path of executor write metrics.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/442721804","html_url":"https://github.com/apache/spark/pull/23163#issuecomment-442721804","issue_url":"https://api.github.com/repos/apache/spark/issues/23163","id":442721804,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MjcyMTgwNA==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-11-29T06:30:40Z","updated_at":"2018-11-29T06:30:40Z","author_association":"NONE","body":"**[Test build #99432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99432/testReport)** for PR 23163 at commit [`a7ddb22`](https://github.com/apache/spark/commit/a7ddb2242faf1260c1104965363b1d7ab031623c)."}}
{"actor":{"display_login":"SparkQA"}}