{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14239","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14239/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14239/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14239/events","html_url":"https://github.com/apache/spark/pull/14239","id":165981453,"number":14239,"title":"[SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage.","user":{"login":"f7753","id":5338550,"avatar_url":"https://avatars.githubusercontent.com/u/5338550?v=3","gravatar_id":"","url":"https://api.github.com/users/f7753","html_url":"https://github.com/f7753","followers_url":"https://api.github.com/users/f7753/followers","following_url":"https://api.github.com/users/f7753/following{/other_user}","gists_url":"https://api.github.com/users/f7753/gists{/gist_id}","starred_url":"https://api.github.com/users/f7753/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f7753/subscriptions","organizations_url":"https://api.github.com/users/f7753/orgs","repos_url":"https://api.github.com/users/f7753/repos","events_url":"https://api.github.com/users/f7753/events{/privacy}","received_events_url":"https://api.github.com/users/f7753/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-07-17T15:30:34Z","updated_at":"2016-08-23T05:59:10Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14239","html_url":"https://github.com/apache/spark/pull/14239","diff_url":"https://github.com/apache/spark/pull/14239.diff","patch_url":"https://github.com/apache/spark/pull/14239.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nAdded a pre-fetch mechanism for shuffle stage.\r\nThe map stage would load the blocks before the openBlock message arrives, then while the block data were transferred through the net, disks would no longer has nothing to do,  they would provide the data and let the server load to memory.\r\n This would make the time between the server received  openBlock message and ready to transfer the data even shorter.\r\n\r\n\r\n## How was this patch tested?\r\nAfter setting the params:`spark.shuffle.prepare.count ` > 0 & `spark.shuffle.prepare.open` to be `true`, this mechanism will work.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/241634710","html_url":"https://github.com/apache/spark/pull/14239#issuecomment-241634710","issue_url":"https://api.github.com/repos/apache/spark/issues/14239","id":241634710,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-08-23T05:59:10Z","updated_at":"2016-08-23T05:59:10Z","body":"**[Test build #64275 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64275/consoleFull)** for PR 14239 at commit [`09ab278`](https://github.com/apache/spark/commit/09ab2789baf95e47653a87c0bb35d9c979ecf5c7).\n * This patch **fails Scala style tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}