{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11083","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11083/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11083/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11083/events","html_url":"https://github.com/apache/spark/pull/11083","id":131483078,"number":11083,"title":"[SPARK-13136][SQL] Create a dedicated Broadcast exchange operator","user":{"login":"hvanhovell","id":9616802,"avatar_url":"https://avatars.githubusercontent.com/u/9616802?v=3","gravatar_id":"","url":"https://api.github.com/users/hvanhovell","html_url":"https://github.com/hvanhovell","followers_url":"https://api.github.com/users/hvanhovell/followers","following_url":"https://api.github.com/users/hvanhovell/following{/other_user}","gists_url":"https://api.github.com/users/hvanhovell/gists{/gist_id}","starred_url":"https://api.github.com/users/hvanhovell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hvanhovell/subscriptions","organizations_url":"https://api.github.com/users/hvanhovell/orgs","repos_url":"https://api.github.com/users/hvanhovell/repos","events_url":"https://api.github.com/users/hvanhovell/events{/privacy}","received_events_url":"https://api.github.com/users/hvanhovell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":33,"created_at":"2016-02-04T21:47:28Z","updated_at":"2016-02-21T01:13:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11083","html_url":"https://github.com/apache/spark/pull/11083","diff_url":"https://github.com/apache/spark/pull/11083.diff","patch_url":"https://github.com/apache/spark/pull/11083.patch"},"body":"Quite a few Spark SQL join operators broadcast one side of the join to all nodes. The are a few problems with this:\r\n\r\n- This conflates broadcasting (a data exchange) with joining. Data exchanges should be managed by a different operator.\r\n- All these nodes implement their own (duplicate) broadcasting logic.\r\n- Re-use of indices is quite hard.\r\n\r\nThis PR defines both a ```BroadcastDistribution``` and ```BroadcastPartitioning```, these contain a `BroadcastMode`. The `BroadcastMode` defines the way in which we transform the Array of `InternalRow`'s into an index. We currently support the following `BroadcastMode`'s:\r\n\r\n- IdentityBroadcastMode: This broadcasts the rows in their original form.\r\n- HashSetBroadcastMode: This applies a projection to the input rows, deduplicates these rows and broadcasts the resulting `Set`.\r\n- HashedRelationBroadcastMode: This transforms the input rows into a `HashedRelation`, and broadcasts this index.\r\n\r\nTo match this distribution we implement a ```BroadcastExchange``` operator which will perform the broadcast for us, and have ```EnsureRequirements``` plan this operator. The old Exchange operator has been renamed into ShuffleExchange in order to clearly separate between Shuffled and Broadcasted exchanges.\r\n\r\ncc @rxin @davies "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/186714653","html_url":"https://github.com/apache/spark/pull/11083#issuecomment-186714653","issue_url":"https://api.github.com/repos/apache/spark/issues/11083","id":186714653,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-02-21T01:13:38Z","updated_at":"2016-02-21T01:13:38Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51605/\nTest PASSed."}}