{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15178","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15178/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15178/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15178/events","html_url":"https://github.com/apache/spark/pull/15178","id":178282109,"number":15178,"title":"[SPARK-17556][SQL] Executor side broadcast for broadcast joins","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2016-09-21T08:39:46Z","updated_at":"2016-10-01T02:34:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15178","html_url":"https://github.com/apache/spark/pull/15178","diff_url":"https://github.com/apache/spark/pull/15178.diff","patch_url":"https://github.com/apache/spark/pull/15178.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThe mechanism of broadcast in Spark is to collect the result of an RDD and then broadcast it. This introduces some extra latency. We can broadcast directly from executors. This patch implements broadcast from executors, and applies it on broadcast join of Spark SQL.\r\n\r\nDesign document: https://issues.apache.org/jira/secure/attachment/12831201/executor-side-broadcast.pdf\r\n\r\n## Major changes\r\n\r\n* New API `broadcastRDDOnExecutor` in `SparkContext`\r\n\r\n  It takes two parameters `rdd: RDD[T]` and `mode: BroadcastMode[T]`. It will broadcast the content of the rdd between executors without collecting it back to the driver. `mode` is used to convert the content of the rdd to the broadcasted object.\r\n\r\n  Besides `T`, this API has another type parameter `U`, which is the type of the converted object.\r\n\r\n* New `Broadcast` implementation  `TorrentExecutorBroadcast`\r\n\r\n  Different to `TorrentBroadcast`, this implementation doesn't divide and store object data waiting to broadcast in the driver. The executors use local and remote fetches to fetch the blocks of the RDD and convert the rdd content to broadcasted object.\r\n\r\n* `BroadcastMode` is moved from `org.apache.spark.sql.catalyst.plans.physical` to `org.apache.spark.broadcast`\r\n\r\n  It is added a type parameter `T` now which is the converted type of the  broadcasted object on executors.\r\n\r\n\r\n## How was this patch tested?\r\n\r\nJenkins tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/250887489","html_url":"https://github.com/apache/spark/pull/15178#issuecomment-250887489","issue_url":"https://api.github.com/repos/apache/spark/issues/15178","id":250887489,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-10-01T02:34:19Z","updated_at":"2016-10-01T02:34:19Z","body":"**[Test build #66196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66196/consoleFull)** for PR 15178 at commit [`17b4470`](https://github.com/apache/spark/commit/17b4470e6564f58bc1b8c9c32640f08c262db91a).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `abstract class RowBroadcastMode extends BroadcastMode[InternalRow] `\n  * `case class BroadcastDistribution(mode: RowBroadcastMode) extends Distribution`\n  * `case class BroadcastPartitioning(mode: RowBroadcastMode) extends Partitioning `"}}
{"actor":{"display_login":"SparkQA"}}