{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/12719","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/12719/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/12719/comments","events_url":"https://api.github.com/repos/apache/spark/issues/12719/events","html_url":"https://github.com/apache/spark/pull/12719","id":151262663,"number":12719,"title":"[WIP][SPARK-14939][SQL] Add FoldablePropagation optimizer","user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":30,"created_at":"2016-04-26T23:26:23Z","updated_at":"2016-05-06T01:13:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/12719","html_url":"https://github.com/apache/spark/pull/12719","diff_url":"https://github.com/apache/spark/pull/12719.diff","patch_url":"https://github.com/apache/spark/pull/12719.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR aims to add new **FoldablePropagation** optimizer that propagates foldable expressions. Basically, `FoldablePropagation` replaces all attributes with the original foldable expression *except the following three cases*.\r\n\r\n1. Set queries: UNION/INTERSECT/EXCEPT should not be optimized.\r\n2. User-defined aliases: `Project.projectList` should keep the original names.\r\n3. Generated aliases: `Sort.aggregateExpression` should keep the original full names.\r\n\r\nAliases and ordinal expressions are the main target to be transformed after propagation. Other optimizations will take advantage of the propagated foldable expressions: e.g. `EliminateSorts` optimizer now can handle the following Case 2 and 3. (Case 1 is the previous implementation.)\r\n\r\n1. Literals and foldable expression, e.g. \"ORDER BY 1.0, 'abc', Now()\"\r\n2. Foldable ordinals, e.g. \"SELECT 1.0, 'abc', Now() ORDER BY 1, 2, 3\"\r\n3. Foldable aliases, e.g. \"SELECT 1.0 x, 'abc' y, Now() z ORDER BY x, y, z\"\r\n\r\nThis PR has been generalized much better based on @cloud-fan 's key idea; he should be credited for the work he did.\r\n\r\n**Before**\r\n```\r\nscala> sql(\"SELECT 1.0, Now() x ORDER BY 1, x\").explain\r\n== Physical Plan ==\r\nWholeStageCodegen\r\n:  +- Sort [1.0#5 ASC,x#0 ASC], true, 0\r\n:     +- INPUT\r\n+- Exchange rangepartitioning(1.0#5 ASC, x#0 ASC, 200), None\r\n   +- WholeStageCodegen\r\n      :  +- Project [1.0 AS 1.0#5,1461873043577000 AS x#0]\r\n      :     +- INPUT\r\n      +- Scan OneRowRelation[]\r\n```\r\n\r\n**After**\r\n```\r\nscala> sql(\"SELECT 1.0, Now() x ORDER BY 1, x\").explain\r\n== Physical Plan ==\r\nWholeStageCodegen\r\n:  +- Project [1.0 AS 1.0#5,1461873079484000 AS x#0]\r\n:     +- INPUT\r\n+- Scan OneRowRelation[]\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nPass the Jenkins tests including a new test case."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/217323300","html_url":"https://github.com/apache/spark/pull/12719#issuecomment-217323300","issue_url":"https://api.github.com/repos/apache/spark/issues/12719","id":217323300,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-05-06T01:13:44Z","updated_at":"2016-05-06T01:13:44Z","body":"**[Test build #57936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57936/consoleFull)** for PR 12719 at commit [`a8b9816`](https://github.com/apache/spark/commit/a8b981664cdc844e3e358d851bddf34159f92de9).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}