{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16677","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16677/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16677/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16677/events","html_url":"https://github.com/apache/spark/pull/16677","id":202476118,"node_id":"MDExOlB1bGxSZXF1ZXN0MTAyNjc3OTYy","number":16677,"title":"[SPARK-19355][SQL] Use map output statistics to improve global limit's parallelism","user":{"login":"viirya","id":68855,"node_id":"MDQ6VXNlcjY4ODU1","avatar_url":"https://avatars1.githubusercontent.com/u/68855?v=4","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":121,"created_at":"2017-01-23T08:54:28Z","updated_at":"2018-08-09T00:12:58Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16677","html_url":"https://github.com/apache/spark/pull/16677","diff_url":"https://github.com/apache/spark/pull/16677.diff","patch_url":"https://github.com/apache/spark/pull/16677.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nA logical `Limit` is performed physically by two operations `LocalLimit` and `GlobalLimit`.\r\n\r\nMost of time, we gather all data into a single partition in order to run `GlobalLimit`. If we use a very big limit number, shuffling data causes performance issue also reduces parallelism.\r\n\r\nWe can avoid shuffling into single partition if we don't care data ordering. This patch implements this idea by doing a map stage during global limit. It collects the info of row numbers at each partition. For each partition, we locally retrieves limited data without any shuffling to finish this global limit.\r\n\r\nFor example, we have three partitions with rows (100, 100, 50) respectively. In global limit of 100 rows, we may take (34, 33, 33) rows for each partition locally. After global limit we still have three partitions.\r\n\r\nIf the data partition has certain ordering, we can't distribute required rows evenly to each partitions because it could change data ordering. But we still can avoid shuffling.\r\n\r\n## How was this patch tested?\r\n\r\nJenkins tests.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/411594882","html_url":"https://github.com/apache/spark/pull/16677#issuecomment-411594882","issue_url":"https://api.github.com/repos/apache/spark/issues/16677","id":411594882,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTU5NDg4Mg==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2018-08-09T00:12:58Z","updated_at":"2018-08-09T00:12:58Z","author_association":"NONE","body":"**[Test build #94452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94452/testReport)** for PR 16677 at commit [`69513d1`](https://github.com/apache/spark/commit/69513d166ee56587d7b039b5d9645299785dcb77).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}