{"action":"created","comment":{"url":"https://api.github.com/repos/prestodb/presto/pulls/comments/194316292","pull_request_review_id":127441324,"id":194316292,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5NDMxNjI5Mg==","diff_hunk":"@@ -200,23 +203,22 @@ private void assertExistsRewrittenToAggregationBelowJoin(@Language(\"SQL\") String\n \n     private void assertExistsRewrittenToAggregationAboveJoin(@Language(\"SQL\") String actual, @Language(\"SQL\") String expected, boolean extraAggregation)\n     {\n+        Consumer<Plan> singleAggregationValidator = plan -> assertEquals(countSingleAggregationNodes(plan), 1);\n+        Consumer<Plan> finalAggregationValidator = plan -> assertEquals(countFinalAggregationNodes(plan), extraAggregation ? 1 : 0);\n+\n         assertions.assertQueryAndPlan(actual, expected,\n                 anyTree(\n                         // FINAL\n-                        aggregation(ImmutableMap.of(\"COUNT\", functionCall(\"count\", ImmutableList.of(\"COUNT_PARTIAL\"))),\n-                                exchange(LOCAL, REPARTITION,\n-                                        // PARTIAL\n-                                        aggregation(ImmutableMap.of(\"COUNT_PARTIAL\", functionCall(\"count\", ImmutableList.of(\"NON_NULL\"))),\n-                                                anyTree(\n-                                                        node(JoinNode.class,\n-                                                                anyTree(\n-                                                                        node(ValuesNode.class)),\n-                                                                anyTree(\n-                                                                        node(ProjectNode.class,\n-                                                                                anyTree(\n-                                                                                        node(ValuesNode.class)))\n-                                                                                .withAlias(\"NON_NULL\", expression(\"true\"))))))))),\n-                plan -> assertEquals(countFinalAggregationNodes(plan), extraAggregation ? 2 : 1));\n+                        aggregation(ImmutableMap.of(\"COUNT\", functionCall(\"count\", ImmutableList.of(\"NON_NULL\"))),","path":"presto-main/src/test/java/com/facebook/presto/sql/query/TestSubqueries.java","position":41,"original_position":41,"commit_id":"d7be4f03e810de15779f034f8a42daa6a9b821a1","original_commit_id":"96c2c8cb68dd8d1f8497f44c73d85bfd58a8ae37","user":{"login":"sopel39","id":5989165,"node_id":"MDQ6VXNlcjU5ODkxNjU=","avatar_url":"https://avatars1.githubusercontent.com/u/5989165?v=4","gravatar_id":"","url":"https://api.github.com/users/sopel39","html_url":"https://github.com/sopel39","followers_url":"https://api.github.com/users/sopel39/followers","following_url":"https://api.github.com/users/sopel39/following{/other_user}","gists_url":"https://api.github.com/users/sopel39/gists{/gist_id}","starred_url":"https://api.github.com/users/sopel39/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sopel39/subscriptions","organizations_url":"https://api.github.com/users/sopel39/orgs","repos_url":"https://api.github.com/users/sopel39/repos","events_url":"https://api.github.com/users/sopel39/events{/privacy}","received_events_url":"https://api.github.com/users/sopel39/received_events","type":"User","site_admin":false},"body":"optional: we could explicitly test aggregation step here.\r\n","created_at":"2018-06-11T07:50:11Z","updated_at":"2018-06-11T07:52:36Z","html_url":"https://github.com/prestodb/presto/pull/10731#discussion_r194316292","pull_request_url":"https://api.github.com/repos/prestodb/presto/pulls/10731","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/prestodb/presto/pulls/comments/194316292"},"html":{"href":"https://github.com/prestodb/presto/pull/10731#discussion_r194316292"},"pull_request":{"href":"https://api.github.com/repos/prestodb/presto/pulls/10731"}}},"pull_request":{"url":"https://api.github.com/repos/prestodb/presto/pulls/10731","id":191649157,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkxNjQ5MTU3","html_url":"https://github.com/prestodb/presto/pull/10731","diff_url":"https://github.com/prestodb/presto/pull/10731.diff","patch_url":"https://github.com/prestodb/presto/pull/10731.patch","issue_url":"https://api.github.com/repos/prestodb/presto/issues/10731","number":10731,"state":"open","locked":false,"title":"Optimize aggregation in a correlated scalar subquery","user":{"login":"mbasmanova","id":27965151,"node_id":"MDQ6VXNlcjI3OTY1MTUx","avatar_url":"https://avatars3.githubusercontent.com/u/27965151?v=4","gravatar_id":"","url":"https://api.github.com/users/mbasmanova","html_url":"https://github.com/mbasmanova","followers_url":"https://api.github.com/users/mbasmanova/followers","following_url":"https://api.github.com/users/mbasmanova/following{/other_user}","gists_url":"https://api.github.com/users/mbasmanova/gists{/gist_id}","starred_url":"https://api.github.com/users/mbasmanova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbasmanova/subscriptions","organizations_url":"https://api.github.com/users/mbasmanova/orgs","repos_url":"https://api.github.com/users/mbasmanova/repos","events_url":"https://api.github.com/users/mbasmanova/events{/privacy}","received_events_url":"https://api.github.com/users/mbasmanova/received_events","type":"User","site_admin":false},"body":"Optimize aggregation in a correlated scalar subquery to remove unnecessary local exchange and partial aggregation and enable streaming mode.\r\n\r\nCorrelated subquery like\r\n\r\n`select name, (select max(name) from region where regionkey = nation.regionkey) from nation;`\r\n\r\nis rewritten to a left join with an aggregation over synthetic row ID column generated by AssignUniqueId node:\r\n\r\n```\r\n- Aggregation (unique, probe.*)\r\n  - LeftJoin\r\n     - AssignUniqueId (unique)\r\n        - probe\r\n     - build\r\n```\r\n\r\nSince \"unique\" column is unique, the aggregation input is partitioned on grouping keys and therefore aggregation can be executed in a single step without any exchanges. Furthermore, since the output of the join is grouped on \"unique\" column, the aggregation doesn't need to accumulate all of the input in memory before emitting results. It can generate results every time the value of the \"unique\" column changes. \r\n\r\nThis PR implements streaming support in `HashAggregationOperator`, updates `StreamPropertyDerivations` and `PropertyDerivations` to set `paritioned_on(unique)` and `grouped(unique)` properties for the output of the `AssignUniqueId` node, and uses these properties in `AddLocalExchanges` to enable streaming aggregation.\r\n\r\nFixes #8171 ","created_at":"2018-05-31T05:40:56Z","updated_at":"2018-06-11T07:52:36Z","closed_at":null,"merged_at":null,"merge_commit_sha":"65a042d42fa1962ea295d2f9c0ddec5ff2852d7f","assignee":null,"assignees":[],"requested_reviewers":[{"login":"findepi","id":144328,"node_id":"MDQ6VXNlcjE0NDMyOA==","avatar_url":"https://avatars0.githubusercontent.com/u/144328?v=4","gravatar_id":"","url":"https://api.github.com/users/findepi","html_url":"https://github.com/findepi","followers_url":"https://api.github.com/users/findepi/followers","following_url":"https://api.github.com/users/findepi/following{/other_user}","gists_url":"https://api.github.com/users/findepi/gists{/gist_id}","starred_url":"https://api.github.com/users/findepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/findepi/subscriptions","organizations_url":"https://api.github.com/users/findepi/orgs","repos_url":"https://api.github.com/users/findepi/repos","events_url":"https://api.github.com/users/findepi/events{/privacy}","received_events_url":"https://api.github.com/users/findepi/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":196992743,"node_id":"MDU6TGFiZWwxOTY5OTI3NDM=","url":"https://api.github.com/repos/prestodb/presto/labels/CLA%20Signed","name":"CLA Signed","color":"ededed","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/prestodb/presto/pulls/10731/commits","review_comments_url":"https://api.github.com/repos/prestodb/presto/pulls/10731/comments","review_comment_url":"https://api.github.com/repos/prestodb/presto/pulls/comments{/number}","comments_url":"https://api.github.com/repos/prestodb/presto/issues/10731/comments","statuses_url":"https://api.github.com/repos/prestodb/presto/statuses/d7be4f03e810de15779f034f8a42daa6a9b821a1","head":{"label":"mbasmanova:streaming-aggregation","ref":"streaming-aggregation","sha":"d7be4f03e810de15779f034f8a42daa6a9b821a1","user":{"login":"mbasmanova","id":27965151,"node_id":"MDQ6VXNlcjI3OTY1MTUx","avatar_url":"https://avatars3.githubusercontent.com/u/27965151?v=4","gravatar_id":"","url":"https://api.github.com/users/mbasmanova","html_url":"https://github.com/mbasmanova","followers_url":"https://api.github.com/users/mbasmanova/followers","following_url":"https://api.github.com/users/mbasmanova/following{/other_user}","gists_url":"https://api.github.com/users/mbasmanova/gists{/gist_id}","starred_url":"https://api.github.com/users/mbasmanova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbasmanova/subscriptions","organizations_url":"https://api.github.com/users/mbasmanova/orgs","repos_url":"https://api.github.com/users/mbasmanova/repos","events_url":"https://api.github.com/users/mbasmanova/events{/privacy}","received_events_url":"https://api.github.com/users/mbasmanova/received_events","type":"User","site_admin":false},"repo":{"id":100315926,"node_id":"MDEwOlJlcG9zaXRvcnkxMDAzMTU5MjY=","name":"presto","full_name":"mbasmanova/presto","owner":{"login":"mbasmanova","id":27965151,"node_id":"MDQ6VXNlcjI3OTY1MTUx","avatar_url":"https://avatars3.githubusercontent.com/u/27965151?v=4","gravatar_id":"","url":"https://api.github.com/users/mbasmanova","html_url":"https://github.com/mbasmanova","followers_url":"https://api.github.com/users/mbasmanova/followers","following_url":"https://api.github.com/users/mbasmanova/following{/other_user}","gists_url":"https://api.github.com/users/mbasmanova/gists{/gist_id}","starred_url":"https://api.github.com/users/mbasmanova/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbasmanova/subscriptions","organizations_url":"https://api.github.com/users/mbasmanova/orgs","repos_url":"https://api.github.com/users/mbasmanova/repos","events_url":"https://api.github.com/users/mbasmanova/events{/privacy}","received_events_url":"https://api.github.com/users/mbasmanova/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mbasmanova/presto","description":"Distributed SQL query engine for big data","fork":true,"url":"https://api.github.com/repos/mbasmanova/presto","forks_url":"https://api.github.com/repos/mbasmanova/presto/forks","keys_url":"https://api.github.com/repos/mbasmanova/presto/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mbasmanova/presto/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mbasmanova/presto/teams","hooks_url":"https://api.github.com/repos/mbasmanova/presto/hooks","issue_events_url":"https://api.github.com/repos/mbasmanova/presto/issues/events{/number}","events_url":"https://api.github.com/repos/mbasmanova/presto/events","assignees_url":"https://api.github.com/repos/mbasmanova/presto/assignees{/user}","branches_url":"https://api.github.com/repos/mbasmanova/presto/branches{/branch}","tags_url":"https://api.github.com/repos/mbasmanova/presto/tags","blobs_url":"https://api.github.com/repos/mbasmanova/presto/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mbasmanova/presto/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mbasmanova/presto/git/refs{/sha}","trees_url":"https://api.github.com/repos/mbasmanova/presto/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mbasmanova/presto/statuses/{sha}","languages_url":"https://api.github.com/repos/mbasmanova/presto/languages","stargazers_url":"https://api.github.com/repos/mbasmanova/presto/stargazers","contributors_url":"https://api.github.com/repos/mbasmanova/presto/contributors","subscribers_url":"https://api.github.com/repos/mbasmanova/presto/subscribers","subscription_url":"https://api.github.com/repos/mbasmanova/presto/subscription","commits_url":"https://api.github.com/repos/mbasmanova/presto/commits{/sha}","git_commits_url":"https://api.github.com/repos/mbasmanova/presto/git/commits{/sha}","comments_url":"https://api.github.com/repos/mbasmanova/presto/comments{/number}","issue_comment_url":"https://api.github.com/repos/mbasmanova/presto/issues/comments{/number}","contents_url":"https://api.github.com/repos/mbasmanova/presto/contents/{+path}","compare_url":"https://api.github.com/repos/mbasmanova/presto/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mbasmanova/presto/merges","archive_url":"https://api.github.com/repos/mbasmanova/presto/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mbasmanova/presto/downloads","issues_url":"https://api.github.com/repos/mbasmanova/presto/issues{/number}","pulls_url":"https://api.github.com/repos/mbasmanova/presto/pulls{/number}","milestones_url":"https://api.github.com/repos/mbasmanova/presto/milestones{/number}","notifications_url":"https://api.github.com/repos/mbasmanova/presto/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mbasmanova/presto/labels{/name}","releases_url":"https://api.github.com/repos/mbasmanova/presto/releases{/id}","deployments_url":"https://api.github.com/repos/mbasmanova/presto/deployments","created_at":"2017-08-14T22:58:12Z","updated_at":"2018-06-04T14:37:23Z","pushed_at":"2018-06-07T18:00:17Z","git_url":"git://github.com/mbasmanova/presto.git","ssh_url":"git@github.com:mbasmanova/presto.git","clone_url":"https://github.com/mbasmanova/presto.git","svn_url":"https://github.com/mbasmanova/presto","homepage":"https://prestodb.io","size":92128,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"prestodb:master","ref":"master","sha":"2a245384cb247d01153c897163a6b4b1f00422fc","user":{"login":"prestodb","id":6882181,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY4ODIxODE=","avatar_url":"https://avatars0.githubusercontent.com/u/6882181?v=4","gravatar_id":"","url":"https://api.github.com/users/prestodb","html_url":"https://github.com/prestodb","followers_url":"https://api.github.com/users/prestodb/followers","following_url":"https://api.github.com/users/prestodb/following{/other_user}","gists_url":"https://api.github.com/users/prestodb/gists{/gist_id}","starred_url":"https://api.github.com/users/prestodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prestodb/subscriptions","organizations_url":"https://api.github.com/users/prestodb/orgs","repos_url":"https://api.github.com/users/prestodb/repos","events_url":"https://api.github.com/users/prestodb/events{/privacy}","received_events_url":"https://api.github.com/users/prestodb/received_events","type":"Organization","site_admin":false},"repo":{"id":5349565,"node_id":"MDEwOlJlcG9zaXRvcnk1MzQ5NTY1","name":"presto","full_name":"prestodb/presto","owner":{"login":"prestodb","id":6882181,"node_id":"MDEyOk9yZ2FuaXphdGlvbjY4ODIxODE=","avatar_url":"https://avatars0.githubusercontent.com/u/6882181?v=4","gravatar_id":"","url":"https://api.github.com/users/prestodb","html_url":"https://github.com/prestodb","followers_url":"https://api.github.com/users/prestodb/followers","following_url":"https://api.github.com/users/prestodb/following{/other_user}","gists_url":"https://api.github.com/users/prestodb/gists{/gist_id}","starred_url":"https://api.github.com/users/prestodb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prestodb/subscriptions","organizations_url":"https://api.github.com/users/prestodb/orgs","repos_url":"https://api.github.com/users/prestodb/repos","events_url":"https://api.github.com/users/prestodb/events{/privacy}","received_events_url":"https://api.github.com/users/prestodb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/prestodb/presto","description":"Distributed SQL query engine for big data","fork":false,"url":"https://api.github.com/repos/prestodb/presto","forks_url":"https://api.github.com/repos/prestodb/presto/forks","keys_url":"https://api.github.com/repos/prestodb/presto/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prestodb/presto/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prestodb/presto/teams","hooks_url":"https://api.github.com/repos/prestodb/presto/hooks","issue_events_url":"https://api.github.com/repos/prestodb/presto/issues/events{/number}","events_url":"https://api.github.com/repos/prestodb/presto/events","assignees_url":"https://api.github.com/repos/prestodb/presto/assignees{/user}","branches_url":"https://api.github.com/repos/prestodb/presto/branches{/branch}","tags_url":"https://api.github.com/repos/prestodb/presto/tags","blobs_url":"https://api.github.com/repos/prestodb/presto/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prestodb/presto/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prestodb/presto/git/refs{/sha}","trees_url":"https://api.github.com/repos/prestodb/presto/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prestodb/presto/statuses/{sha}","languages_url":"https://api.github.com/repos/prestodb/presto/languages","stargazers_url":"https://api.github.com/repos/prestodb/presto/stargazers","contributors_url":"https://api.github.com/repos/prestodb/presto/contributors","subscribers_url":"https://api.github.com/repos/prestodb/presto/subscribers","subscription_url":"https://api.github.com/repos/prestodb/presto/subscription","commits_url":"https://api.github.com/repos/prestodb/presto/commits{/sha}","git_commits_url":"https://api.github.com/repos/prestodb/presto/git/commits{/sha}","comments_url":"https://api.github.com/repos/prestodb/presto/comments{/number}","issue_comment_url":"https://api.github.com/repos/prestodb/presto/issues/comments{/number}","contents_url":"https://api.github.com/repos/prestodb/presto/contents/{+path}","compare_url":"https://api.github.com/repos/prestodb/presto/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prestodb/presto/merges","archive_url":"https://api.github.com/repos/prestodb/presto/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prestodb/presto/downloads","issues_url":"https://api.github.com/repos/prestodb/presto/issues{/number}","pulls_url":"https://api.github.com/repos/prestodb/presto/pulls{/number}","milestones_url":"https://api.github.com/repos/prestodb/presto/milestones{/number}","notifications_url":"https://api.github.com/repos/prestodb/presto/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prestodb/presto/labels{/name}","releases_url":"https://api.github.com/repos/prestodb/presto/releases{/id}","deployments_url":"https://api.github.com/repos/prestodb/presto/deployments","created_at":"2012-08-09T01:03:37Z","updated_at":"2018-06-11T06:55:32Z","pushed_at":"2018-06-11T03:45:38Z","git_url":"git://github.com/prestodb/presto.git","ssh_url":"git@github.com:prestodb/presto.git","clone_url":"https://github.com/prestodb/presto.git","svn_url":"https://github.com/prestodb/presto","homepage":"https://prestodb.io","size":92321,"stargazers_count":7618,"watchers_count":7618,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2563,"mirror_url":null,"archived":false,"open_issues_count":1295,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":2563,"open_issues":1295,"watchers":7618,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/prestodb/presto/pulls/10731"},"html":{"href":"https://github.com/prestodb/presto/pull/10731"},"issue":{"href":"https://api.github.com/repos/prestodb/presto/issues/10731"},"comments":{"href":"https://api.github.com/repos/prestodb/presto/issues/10731/comments"},"review_comments":{"href":"https://api.github.com/repos/prestodb/presto/pulls/10731/comments"},"review_comment":{"href":"https://api.github.com/repos/prestodb/presto/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/prestodb/presto/pulls/10731/commits"},"statuses":{"href":"https://api.github.com/repos/prestodb/presto/statuses/d7be4f03e810de15779f034f8a42daa6a9b821a1"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"sopel39"}}