{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/22010","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/22010/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/22010/comments","events_url":"https://api.github.com/repos/apache/spark/issues/22010/events","html_url":"https://github.com/apache/spark/pull/22010","id":348027412,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA2NDc1Mzkz","number":22010,"title":"[SPARK-21436][CORE] Take advantage of known partitioner for distinct on RDDs to avoid a shuffle","user":{"login":"holdenk","id":59893,"node_id":"MDQ6VXNlcjU5ODkz","avatar_url":"https://avatars1.githubusercontent.com/u/59893?v=4","gravatar_id":"","url":"https://api.github.com/users/holdenk","html_url":"https://github.com/holdenk","followers_url":"https://api.github.com/users/holdenk/followers","following_url":"https://api.github.com/users/holdenk/following{/other_user}","gists_url":"https://api.github.com/users/holdenk/gists{/gist_id}","starred_url":"https://api.github.com/users/holdenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holdenk/subscriptions","organizations_url":"https://api.github.com/users/holdenk/orgs","repos_url":"https://api.github.com/users/holdenk/repos","events_url":"https://api.github.com/users/holdenk/events{/privacy}","received_events_url":"https://api.github.com/users/holdenk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2018-08-06T18:08:15Z","updated_at":"2018-09-09T21:17:09Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22010","html_url":"https://github.com/apache/spark/pull/22010","diff_url":"https://github.com/apache/spark/pull/22010.diff","patch_url":"https://github.com/apache/spark/pull/22010.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSpecial case the situation where we know the partioner and the number of requested partions output is the same as the current partioner to avoid a shuffle and instead compute distinct inside of each partion.\r\n\r\n## How was this patch tested?\r\n\r\nNew unit test that verifies partitioner does not change if the partitioner is known and distinct is called with the same target # of partition."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/419745628","html_url":"https://github.com/apache/spark/pull/22010#issuecomment-419745628","issue_url":"https://api.github.com/repos/apache/spark/issues/22010","id":419745628,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTc0NTYyOA==","user":{"login":"holdenk","id":59893,"node_id":"MDQ6VXNlcjU5ODkz","avatar_url":"https://avatars1.githubusercontent.com/u/59893?v=4","gravatar_id":"","url":"https://api.github.com/users/holdenk","html_url":"https://github.com/holdenk","followers_url":"https://api.github.com/users/holdenk/followers","following_url":"https://api.github.com/users/holdenk/following{/other_user}","gists_url":"https://api.github.com/users/holdenk/gists{/gist_id}","starred_url":"https://api.github.com/users/holdenk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/holdenk/subscriptions","organizations_url":"https://api.github.com/users/holdenk/orgs","repos_url":"https://api.github.com/users/holdenk/repos","events_url":"https://api.github.com/users/holdenk/events{/privacy}","received_events_url":"https://api.github.com/users/holdenk/received_events","type":"User","site_admin":false},"created_at":"2018-09-09T21:17:09Z","updated_at":"2018-09-09T21:17:09Z","author_association":"CONTRIBUTOR","body":"@rxin So that RDD could not exist with a known partitioner (regardless of range-based or hash based the partitioner must be deterministic so two elements with the same key must go to the same partition & if two elements do not have same key they can not be duplicates of each other). Distinct looks at both the input k/v as one elem not just v (e.g an RDD of `[(1, 2), (2, 2), (2, 2)].distinct()` should produce `[(1,2), (2, 2)]`).\r\n"}}
{"actor":{"display_login":"holdenk"}}