{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13443","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13443/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13443/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13443/events","html_url":"https://github.com/apache/spark/pull/13443","id":157943443,"number":13443,"title":"[SPARK-15671] performance regression CoalesceRDD.pickBin with large #…","user":{"login":"tgravescs","id":4563792,"avatar_url":"https://avatars.githubusercontent.com/u/4563792?v=3","gravatar_id":"","url":"https://api.github.com/users/tgravescs","html_url":"https://github.com/tgravescs","followers_url":"https://api.github.com/users/tgravescs/followers","following_url":"https://api.github.com/users/tgravescs/following{/other_user}","gists_url":"https://api.github.com/users/tgravescs/gists{/gist_id}","starred_url":"https://api.github.com/users/tgravescs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgravescs/subscriptions","organizations_url":"https://api.github.com/users/tgravescs/orgs","repos_url":"https://api.github.com/users/tgravescs/repos","events_url":"https://api.github.com/users/tgravescs/events{/privacy}","received_events_url":"https://api.github.com/users/tgravescs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-06-01T15:54:53Z","updated_at":"2016-06-01T17:43:34Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13443","html_url":"https://github.com/apache/spark/pull/13443","diff_url":"https://github.com/apache/spark/pull/13443.diff","patch_url":"https://github.com/apache/spark/pull/13443.patch"},"body":"I was running a 15TB join job with 202000 partitions. It looks like the changes I made to CoalesceRDD in pickBin() are really slow with that large of partitions. The array filter with that many elements just takes to long.\r\nIt took about an hour for it to pickBins for all the partitions.\r\noriginal change:\r\nhttps://github.com/apache/spark/commit/83ee92f60345f016a390d61a82f1d924f64ddf90\r\n\r\nJust reverting the pickBin code back to get currpreflocs fixes the issue\r\n\r\nAfter reverting the pickBin code the coalesce takes about 10 seconds so for now it makes sense to revert those changes and we can look at further optimizations later.\r\n\r\nTested this via RDDSuite unit test and manually testing the very large job."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/223070105","html_url":"https://github.com/apache/spark/pull/13443#issuecomment-223070105","issue_url":"https://api.github.com/repos/apache/spark/issues/13443","id":223070105,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-06-01T17:43:34Z","updated_at":"2016-06-01T17:43:34Z","body":"**[Test build #59743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59743/consoleFull)** for PR 13443 at commit [`f630b64`](https://github.com/apache/spark/commit/f630b64a590bd85e88e02f8896fffd253a43bed5).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}