{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/26044","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/26044/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/26044/comments","events_url":"https://api.github.com/repos/apache/spark/issues/26044/events","html_url":"https://github.com/apache/spark/pull/26044","id":503595960,"node_id":"MDExOlB1bGxSZXF1ZXN0MzI1NDE3NjU1","number":26044,"title":"[SPARK-29375][SQL] Exchange reuse across all subquery levels","user":{"login":"peter-toth","id":7253827,"node_id":"MDQ6VXNlcjcyNTM4Mjc=","avatar_url":"https://avatars1.githubusercontent.com/u/7253827?v=4","gravatar_id":"","url":"https://api.github.com/users/peter-toth","html_url":"https://github.com/peter-toth","followers_url":"https://api.github.com/users/peter-toth/followers","following_url":"https://api.github.com/users/peter-toth/following{/other_user}","gists_url":"https://api.github.com/users/peter-toth/gists{/gist_id}","starred_url":"https://api.github.com/users/peter-toth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peter-toth/subscriptions","organizations_url":"https://api.github.com/users/peter-toth/orgs","repos_url":"https://api.github.com/users/peter-toth/repos","events_url":"https://api.github.com/users/peter-toth/events{/privacy}","received_events_url":"https://api.github.com/users/peter-toth/received_events","type":"User","site_admin":false},"labels":[{"id":1405794576,"node_id":"MDU6TGFiZWwxNDA1Nzk0NTc2","url":"https://api.github.com/repos/apache/spark/labels/SQL","name":"SQL","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2019-10-07T17:45:27Z","updated_at":"2019-11-11T11:44:38Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/26044","html_url":"https://github.com/apache/spark/pull/26044","diff_url":"https://github.com/apache/spark/pull/26044.diff","patch_url":"https://github.com/apache/spark/pull/26044.patch"},"body":"### What changes were proposed in this pull request?\r\nThis PR:\r\n- enables exchange reuse across all subquery levels\r\n- adds minor optimization by using a map of canonicalized plans to look up reusable exchanges\r\n\r\nExample query:\r\n```\r\nSELECT\r\n  (SELECT max(a.key) FROM testData AS a JOIN testData AS b ON b.key = a.key),\r\n  a.key\r\nFROM testData AS a\r\nJOIN testData AS b ON b.key = a.key\r\n```\r\nPlan before this PR:\r\n```\r\n*(5) Project [Subquery scalar-subquery#270, [id=#605] AS scalarsubquery()#277, key#13]\r\n:  +- Subquery scalar-subquery#270, [id=#605]\r\n:     +- *(6) HashAggregate(keys=[], functions=[max(key#13)], output=[max(key)#276])\r\n:        +- Exchange SinglePartition, true, [id=#601]\r\n:           +- *(5) HashAggregate(keys=[], functions=[partial_max(key#13)], output=[max#281])\r\n:              +- *(5) Project [key#13]\r\n:                 +- *(5) SortMergeJoin [key#13], [key#273], Inner\r\n:                    :- *(2) Sort [key#13 ASC NULLS FIRST], false, 0\r\n:                    :  +- Exchange hashpartitioning(key#13, 5), true, [id=#584]\r\n:                    :     +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#13]\r\n:                    :        +- Scan[obj#12]\r\n:                    +- *(4) Sort [key#273 ASC NULLS FIRST], false, 0\r\n:                       +- Exchange hashpartitioning(key#273, 5), true, [id=#592]\r\n:                          +- *(3) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#273]\r\n:                             +- Scan[obj#12]\r\n+- *(5) SortMergeJoin [key#13], [key#271], Inner\r\n   :- *(2) Sort [key#13 ASC NULLS FIRST], false, 0\r\n   :  +- Exchange hashpartitioning(key#13, 5), true, [id=#617]\r\n   :     +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#13]\r\n   :        +- Scan[obj#12]\r\n   +- *(4) Sort [key#271 ASC NULLS FIRST], false, 0\r\n      +- Exchange hashpartitioning(key#271, 5), true, [id=#625]\r\n         +- *(3) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#271]\r\n            +- Scan[obj#12]\r\n```\r\nPlan after this PR:\r\n```\r\n*(5) Project [Subquery scalar-subquery#240, [id=#193] AS scalarsubquery()#247, key#13]\r\n:  +- Subquery scalar-subquery#240, [id=#193]\r\n:     +- *(6) HashAggregate(keys=[], functions=[max(key#13)], output=[max(key)#246])\r\n:        +- Exchange SinglePartition, true, [id=#189]\r\n:           +- *(5) HashAggregate(keys=[], functions=[partial_max(key#13)], output=[max#251])\r\n:              +- *(5) Project [key#13]\r\n:                 +- *(5) SortMergeJoin [key#13], [key#243], Inner\r\n:                    :- *(2) Sort [key#13 ASC NULLS FIRST], false, 0\r\n:                    :  +- Exchange hashpartitioning(key#13, 5), true, [id=#145]\r\n:                    :     +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#13]\r\n:                    :        +- Scan[obj#12]\r\n:                    +- *(4) Sort [key#243 ASC NULLS FIRST], false, 0\r\n:                       +- ReusedExchange [key#243], Exchange hashpartitioning(key#13, 5), true, [id=#145]\r\n+- *(5) SortMergeJoin [key#13], [key#241], Inner\r\n   :- *(2) Sort [key#13 ASC NULLS FIRST], false, 0\r\n   :  +- ReusedExchange [key#13], Exchange hashpartitioning(key#13, 5), true, [id=#145]\r\n   +- *(4) Sort [key#241 ASC NULLS FIRST], false, 0\r\n      +- ReusedExchange [key#241], Exchange hashpartitioning(key#13, 5), true, [id=#145]\r\n```\r\n\r\n### Why are the changes needed?\r\nPerformance improvement. \r\n\r\n### Does this PR introduce any user-facing change?\r\nNo.\r\n\r\n### How was this patch tested?\r\nAdded a new UT.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/552411102","html_url":"https://github.com/apache/spark/pull/26044#issuecomment-552411102","issue_url":"https://api.github.com/repos/apache/spark/issues/26044","id":552411102,"node_id":"MDEyOklzc3VlQ29tbWVudDU1MjQxMTEwMg==","user":{"login":"SparkQA","id":8108735,"node_id":"MDQ6VXNlcjgxMDg3MzU=","avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2019-11-11T11:44:38Z","updated_at":"2019-11-11T11:44:38Z","author_association":"NONE","body":"**[Test build #113577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/113577/testReport)** for PR 26044 at commit [`04654f3`](https://github.com/apache/spark/commit/04654f314117faf129716dc5a124ca587d1f5825).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}