{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15502","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15502/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15502/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15502/events","html_url":"https://github.com/apache/spark/pull/15502","id":183215021,"number":15502,"title":"[SPARK-17892] [SQL] [2.0] Do Not Optimize Query in CTAS More Than Once #15048","user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars.githubusercontent.com/u/11567269?v=3","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-15T15:22:44Z","updated_at":"2016-10-17T07:35:54Z","closed_at":"2016-10-17T07:35:54Z","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15502","html_url":"https://github.com/apache/spark/pull/15502","diff_url":"https://github.com/apache/spark/pull/15502.diff","patch_url":"https://github.com/apache/spark/pull/15502.patch"},"body":"### What changes were proposed in this pull request?\r\nThis PR is to backport https://github.com/apache/spark/pull/15048 and https://github.com/apache/spark/pull/15459. \r\n\r\nHowever, in 2.0, we do not have a unified logical node `CreateTable` and the analyzer rule `PreWriteCheck` is also different. To minimize the code changes, this PR adds a new rule `AnalyzeCreateTableAsSelect`. Please treat it as a new PR to review. Thanks!\r\n\r\nAs explained in https://github.com/apache/spark/pull/14797:\r\n>Some analyzer rules have assumptions on logical plans, optimizer may break these assumption, we should not pass an optimized query plan into QueryExecution (will be analyzed again), otherwise we may some weird bugs.\r\nFor example, we have a rule for decimal calculation to promote the precision before binary operations, use PromotePrecision as placeholder to indicate that this rule should not apply twice. But a Optimizer rule will remove this placeholder, that break the assumption, then the rule applied twice, cause wrong result.\r\n\r\nWe should not optimize the query in CTAS more than once. For example, \r\n```Scala\r\nspark.range(99, 101).createOrReplaceTempView(\"tab1\")\r\nval sqlStmt = \"SELECT id, cast(id as long) * cast('1.0' as decimal(38, 18)) as num FROM tab1\"\r\nsql(s\"CREATE TABLE tab2 USING PARQUET AS $sqlStmt\")\r\ncheckAnswer(spark.table(\"tab2\"), sql(sqlStmt))\r\n```\r\nBefore this PR, the results do not match\r\n```\r\n== Results ==\r\n!== Correct Answer - 2 ==       == Spark Answer - 2 ==\r\n![100,100.000000000000000000]   [100,null]\r\n [99,99.000000000000000000]     [99,99.000000000000000000]\r\n```\r\nAfter this PR, the results match.\r\n```\r\n+---+----------------------+\r\n|id |num                   |\r\n+---+----------------------+\r\n|99 |99.000000000000000000 |\r\n|100|100.000000000000000000|\r\n+---+----------------------+\r\n```\r\n\r\nIn this PR, we do not treat the `query` in CTAS as a child. Thus, the `query` will not be optimized when optimizing CTAS statement. However, we still need to analyze it for normalizing and verifying the CTAS in the Analyzer. Thus, we do it in the analyzer rule `PreprocessDDL`, because so far only this rule needs the analyzed plan of the `query`.\r\n\r\n### How was this patch tested?"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/254133642","html_url":"https://github.com/apache/spark/pull/15502#issuecomment-254133642","issue_url":"https://api.github.com/repos/apache/spark/issues/15502","id":254133642,"user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars.githubusercontent.com/u/11567269?v=3","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"created_at":"2016-10-17T07:35:54Z","updated_at":"2016-10-17T07:35:54Z","body":"Thanks! Close it now. "}}
{"actor":{"display_login":"gatorsmile"}}