{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14865","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14865/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14865/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14865/events","html_url":"https://github.com/apache/spark/pull/14865","id":173836023,"number":14865,"title":"[SPARK-17289][SQL] Fix a bug to satisfy sort requirements in partial aggregations","user":{"login":"maropu","id":692303,"avatar_url":"https://avatars.githubusercontent.com/u/692303?v=3","gravatar_id":"","url":"https://api.github.com/users/maropu","html_url":"https://github.com/maropu","followers_url":"https://api.github.com/users/maropu/followers","following_url":"https://api.github.com/users/maropu/following{/other_user}","gists_url":"https://api.github.com/users/maropu/gists{/gist_id}","starred_url":"https://api.github.com/users/maropu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maropu/subscriptions","organizations_url":"https://api.github.com/users/maropu/orgs","repos_url":"https://api.github.com/users/maropu/repos","events_url":"https://api.github.com/users/maropu/events{/privacy}","received_events_url":"https://api.github.com/users/maropu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-08-29T18:11:39Z","updated_at":"2016-08-30T09:32:36Z","closed_at":"2016-08-30T08:46:16Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14865","html_url":"https://github.com/apache/spark/pull/14865","diff_url":"https://github.com/apache/spark/pull/14865.diff","patch_url":"https://github.com/apache/spark/pull/14865.patch"},"body":"## What changes were proposed in this pull request?\r\nPartial aggregations are generated in `EnsureRequirements`, but the planner fails to\r\ncheck if partial aggregation satisfies sort requirements.\r\nFor the following query:\r\n```\r\nval df2 = (0 to 1000).map(x => (x % 2, x.toString)).toDF(\"a\", \"b\").createOrReplaceTempView(\"t2\")\r\nspark.sql(\"select max(b) from t2 group by a\").explain(true)\r\n```\r\nNow, the SortAggregator won't insert Sort operator before partial aggregation, this will break sort-based partial aggregation.\r\n```\r\n== Physical Plan ==\r\nSortAggregate(key=[a#5], functions=[max(b#6)], output=[max(b)#17])\r\n+- *Sort [a#5 ASC], false, 0\r\n   +- Exchange hashpartitioning(a#5, 200)\r\n      +- SortAggregate(key=[a#5], functions=[partial_max(b#6)], output=[a#5, max#19])\r\n         +- LocalTableScan [a#5, b#6]\r\n```\r\nActually, a correct plan is:\r\n```\r\n== Physical Plan ==\r\nSortAggregate(key=[a#5], functions=[max(b#6)], output=[max(b)#17])\r\n+- *Sort [a#5 ASC], false, 0\r\n   +- Exchange hashpartitioning(a#5, 200)\r\n      +- SortAggregate(key=[a#5], functions=[partial_max(b#6)], output=[a#5, max#19])\r\n         +- *Sort [a#5 ASC], false, 0\r\n            +- LocalTableScan [a#5, b#6]\r\n```\r\n\r\n## How was this patch tested?\r\nAdded tests in `PlannerSuite`."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/243385240","html_url":"https://github.com/apache/spark/pull/14865#issuecomment-243385240","issue_url":"https://api.github.com/repos/apache/spark/issues/14865","id":243385240,"user":{"login":"maropu","id":692303,"avatar_url":"https://avatars.githubusercontent.com/u/692303?v=3","gravatar_id":"","url":"https://api.github.com/users/maropu","html_url":"https://github.com/maropu","followers_url":"https://api.github.com/users/maropu/followers","following_url":"https://api.github.com/users/maropu/following{/other_user}","gists_url":"https://api.github.com/users/maropu/gists{/gist_id}","starred_url":"https://api.github.com/users/maropu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maropu/subscriptions","organizations_url":"https://api.github.com/users/maropu/orgs","repos_url":"https://api.github.com/users/maropu/repos","events_url":"https://api.github.com/users/maropu/events{/privacy}","received_events_url":"https://api.github.com/users/maropu/received_events","type":"User","site_admin":false},"created_at":"2016-08-30T09:32:36Z","updated_at":"2016-08-30T09:32:36Z","body":"@liancheng okay, I'll check the discussion. Thanks!"}}
{"actor":{"display_login":"maropu"}}