{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14797","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14797/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14797/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14797/events","html_url":"https://github.com/apache/spark/pull/14797","id":173061004,"number":14797,"title":"[SPARK-17230] [SQL] Should not pass optimized query into QueryExecution in DataFrameWriter","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-08-24T21:20:41Z","updated_at":"2016-09-01T19:18:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14797","html_url":"https://github.com/apache/spark/pull/14797","diff_url":"https://github.com/apache/spark/pull/14797.diff","patch_url":"https://github.com/apache/spark/pull/14797.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nSome analyzer rules have assumptions on logical plans, optimizer may break these assumption, we should not pass an optimized query plan into QueryExecution (will be analyzed again), otherwise we may some weird bugs.\r\n\r\nFor example, we have a rule for decimal calculation to promote the precision before binary operations, use PromotePrecision as placeholder to indicate that this rule should not apply twice. But a Optimizer rule will remove this placeholder, that break the assumption, then the rule applied twice, cause wrong result.\r\n\r\nIdeally, we should make all the analyzer rules all idempotent, that may require lots of effort to double checking them one by one (may be not easy).\r\n\r\nAn easier approach could be never feed a optimized plan into Analyzer, this PR fix the case for RunnableComand, they will be optimized, during execution, the passed `query` will also be passed into QueryExecution again. This PR make these `query` not part of the children, so they will not be optimized and analyzed again.\r\n\r\nRight now, we did not know a logical plan is optimized or not, we could introduce a flag for that, and make sure a optimized logical plan will not be analyzed again. \r\n\r\n## How was this patch tested?\r\n\r\nAdded regression tests."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/244183339","html_url":"https://github.com/apache/spark/pull/14797#issuecomment-244183339","issue_url":"https://api.github.com/repos/apache/spark/issues/14797","id":244183339,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-09-01T19:18:33Z","updated_at":"2016-09-01T19:18:33Z","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}