{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/77820548","id":77820548,"diff_hunk":"@@ -524,11 +524,16 @@ class Analyzer(\n           val newRight = right transformUp {\n             case r if r == oldRelation => newRelation\n           } transformUp {\n-            case other => other transformExpressions {\n-              case a: Attribute =>\n-                attributeRewrites.get(a).getOrElse(a).withQualifier(a.qualifier)\n-            }\n+            case other =>\n+              val transformed = other transformExpressions {\n+                case a: Attribute =>\n+                  attributeRewrites.get(a).getOrElse(a).withQualifier(a.qualifier)\n+              }\n+\n+              transformed.setPlanId(other.planId)\n+              transformed\n           }\n+          newRight.setPlanId(right.planId)","path":"sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala","position":17,"original_position":17,"commit_id":"ccf71fcc8c366eddb4f43b6a79105e47576d99d5","original_commit_id":"ccf71fcc8c366eddb4f43b6a79105e47576d99d5","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars.githubusercontent.com/u/3182036?v=3","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"body":"I don't quite understand this. If it's a self-join, and `left` and `right` is same plan(same plan id), then after `dedupRight`, `left` and `right` is not same plan but still have same plan id right? How do we resolve `UnresolvedAttribute` with plan id? I think it's still ambiguous.","created_at":"2016-09-07T13:25:22Z","updated_at":"2016-09-07T13:25:22Z","html_url":"https://github.com/apache/spark/pull/14719#discussion_r77820548","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/14719","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/77820548"},"html":{"href":"https://github.com/apache/spark/pull/14719#discussion_r77820548"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/14719"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14719","id":82011204,"html_url":"https://github.com/apache/spark/pull/14719","diff_url":"https://github.com/apache/spark/pull/14719.diff","patch_url":"https://github.com/apache/spark/pull/14719.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/14719","number":14719,"state":"open","locked":false,"title":"[SPARK-17154][SQL] Wrong result can be returned or AnalysisException can be thrown after self-join or similar operations","user":{"login":"sarutak","id":4736016,"avatar_url":"https://avatars.githubusercontent.com/u/4736016?v=3","gravatar_id":"","url":"https://api.github.com/users/sarutak","html_url":"https://github.com/sarutak","followers_url":"https://api.github.com/users/sarutak/followers","following_url":"https://api.github.com/users/sarutak/following{/other_user}","gists_url":"https://api.github.com/users/sarutak/gists{/gist_id}","starred_url":"https://api.github.com/users/sarutak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sarutak/subscriptions","organizations_url":"https://api.github.com/users/sarutak/orgs","repos_url":"https://api.github.com/users/sarutak/repos","events_url":"https://api.github.com/users/sarutak/events{/privacy}","received_events_url":"https://api.github.com/users/sarutak/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nWhen we join two DataFrames which are originated from a same DataFrame, operations to the joined DataFrame can fail.\r\n\r\nOne reproducible  example is as follows.\r\n\r\n```\r\nval df = Seq(\r\n  (1, \"a\", \"A\"),\r\n  (2, \"b\", \"B\"),\r\n  (3, \"c\", \"C\"),\r\n  (4, \"d\", \"D\"),\r\n  (5, \"e\", \"E\")).toDF(\"col1\", \"col2\", \"col3\")\r\n  val filtered = df.filter(\"col1 != 3\").select(\"col1\", \"col2\")\r\n  val joined = filtered.join(df, filtered(\"col1\") === df(\"col1\"), \"inner\")\r\n  val selected1 = joined.select(df(\"col3\"))\r\n```\r\n\r\nIn this case, AnalysisException is thrown.\r\n\r\nAnother example is as follows.\r\n\r\n```\r\nval df = Seq(\r\n  (1, \"a\", \"A\"),\r\n  (2, \"b\", \"B\"),\r\n  (3, \"c\", \"C\"),\r\n  (4, \"d\", \"D\"),\r\n  (5, \"e\", \"E\")).toDF(\"col1\", \"col2\", \"col3\")\r\n  val filtered = df.filter(\"col1 != 3\").select(\"col1\", \"col2\")\r\n  val rightOuterJoined = filtered.join(df, filtered(\"col1\") === df(\"col1\"), \"right\")\r\n  val selected2 = rightOuterJoined.select(df(\"col1\"))\r\n  selected2.show\r\n```\r\n\r\nIn this case, we will expect to get the answer like as follows.\r\n```\r\n1\r\n2\r\n3\r\n4\r\n5\r\n```\r\n\r\nBut the actual result is as follows.\r\n\r\n```\r\n1\r\n2\r\nnull\r\n4\r\n5\r\n```\r\n\r\nThe cause of the problems in the examples is that the logical plan related to the right side DataFrame and the expressions of its output are re-created in the analyzer (at ResolveReference rule) when a DataFrame has expressions which have a same exprId each other.\r\nRe-created expressions are equally to the original ones except exprId.\r\nThis will happen when we do self-join or similar pattern operations.\r\n\r\nIn the first example, `df(\"col3\")` returns a Column which includes an expression and the expression have an exprId (say id1 here).\r\nAfter join, the expresion which the right side DataFrame (`df`) has is re-created and the old and new expressions are equally but exprId is renewed (say id2 for the new exprId here).\r\nBecause of the mismatch of those exprIds, AnalysisException is thrown.\r\n\r\nIn the second example, `df(\"col1\")` returns a column and the expression contained in the column is assigned an exprId (say id3).\r\nOn the other hand, a column returned by `filtered(\"col1\")` has an expression which has the same exprId (id3).\r\nAfter join, the expressions in the right side DataFrame are re-created and the expression assigned id3 is no longer present in the right side but present in the left side.\r\nSo, referring `df(\"col1\")` to the joined DataFrame, we get col1 of right side which includes null.\r\n\r\nTo resolve this issue, I have introduced `LazilyDeterminedAttribute`.\r\nIt is returned when we refer a column like `df(\"expr\")` and determines which expression `df(\"expr\")` should point to lazily.\r\n\r\n## How was this patch tested?\r\n\r\nI added some test cases.","created_at":"2016-08-19T16:48:56Z","updated_at":"2016-09-07T13:25:22Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c02f2d620d821ebfaec42d1b2c5fdc121c386c7c","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/14719/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/14719/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/14719/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/ccf71fcc8c366eddb4f43b6a79105e47576d99d5","head":{"label":"sarutak:SPARK-17154","ref":"SPARK-17154","sha":"ccf71fcc8c366eddb4f43b6a79105e47576d99d5","user":{"login":"sarutak","id":4736016,"avatar_url":"https://avatars.githubusercontent.com/u/4736016?v=3","gravatar_id":"","url":"https://api.github.com/users/sarutak","html_url":"https://github.com/sarutak","followers_url":"https://api.github.com/users/sarutak/followers","following_url":"https://api.github.com/users/sarutak/following{/other_user}","gists_url":"https://api.github.com/users/sarutak/gists{/gist_id}","starred_url":"https://api.github.com/users/sarutak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sarutak/subscriptions","organizations_url":"https://api.github.com/users/sarutak/orgs","repos_url":"https://api.github.com/users/sarutak/repos","events_url":"https://api.github.com/users/sarutak/events{/privacy}","received_events_url":"https://api.github.com/users/sarutak/received_events","type":"User","site_admin":false},"repo":{"id":19300892,"name":"spark","full_name":"sarutak/spark","owner":{"login":"sarutak","id":4736016,"avatar_url":"https://avatars.githubusercontent.com/u/4736016?v=3","gravatar_id":"","url":"https://api.github.com/users/sarutak","html_url":"https://github.com/sarutak","followers_url":"https://api.github.com/users/sarutak/followers","following_url":"https://api.github.com/users/sarutak/following{/other_user}","gists_url":"https://api.github.com/users/sarutak/gists{/gist_id}","starred_url":"https://api.github.com/users/sarutak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sarutak/subscriptions","organizations_url":"https://api.github.com/users/sarutak/orgs","repos_url":"https://api.github.com/users/sarutak/repos","events_url":"https://api.github.com/users/sarutak/events{/privacy}","received_events_url":"https://api.github.com/users/sarutak/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/sarutak/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/sarutak/spark","forks_url":"https://api.github.com/repos/sarutak/spark/forks","keys_url":"https://api.github.com/repos/sarutak/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/sarutak/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/sarutak/spark/teams","hooks_url":"https://api.github.com/repos/sarutak/spark/hooks","issue_events_url":"https://api.github.com/repos/sarutak/spark/issues/events{/number}","events_url":"https://api.github.com/repos/sarutak/spark/events","assignees_url":"https://api.github.com/repos/sarutak/spark/assignees{/user}","branches_url":"https://api.github.com/repos/sarutak/spark/branches{/branch}","tags_url":"https://api.github.com/repos/sarutak/spark/tags","blobs_url":"https://api.github.com/repos/sarutak/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/sarutak/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/sarutak/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/sarutak/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/sarutak/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/sarutak/spark/languages","stargazers_url":"https://api.github.com/repos/sarutak/spark/stargazers","contributors_url":"https://api.github.com/repos/sarutak/spark/contributors","subscribers_url":"https://api.github.com/repos/sarutak/spark/subscribers","subscription_url":"https://api.github.com/repos/sarutak/spark/subscription","commits_url":"https://api.github.com/repos/sarutak/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/sarutak/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/sarutak/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/sarutak/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/sarutak/spark/contents/{+path}","compare_url":"https://api.github.com/repos/sarutak/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/sarutak/spark/merges","archive_url":"https://api.github.com/repos/sarutak/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/sarutak/spark/downloads","issues_url":"https://api.github.com/repos/sarutak/spark/issues{/number}","pulls_url":"https://api.github.com/repos/sarutak/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/sarutak/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/sarutak/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/sarutak/spark/labels{/name}","releases_url":"https://api.github.com/repos/sarutak/spark/releases{/id}","deployments_url":"https://api.github.com/repos/sarutak/spark/deployments","created_at":"2014-04-30T04:35:03Z","updated_at":"2014-07-08T00:24:24Z","pushed_at":"2016-09-01T21:54:23Z","git_url":"git://github.com/sarutak/spark.git","ssh_url":"git@github.com:sarutak/spark.git","clone_url":"https://github.com/sarutak/spark.git","svn_url":"https://github.com/sarutak/spark","homepage":null,"size":170966,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"e07baf14120bc94b783649dabf5fffea58bff0de","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-09-07T13:17:33Z","pushed_at":"2016-09-07T12:20:31Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":523515,"stargazers_count":9921,"watchers_count":9921,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":9241,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":472,"forks":9241,"open_issues":472,"watchers":9921,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/14719"},"html":{"href":"https://github.com/apache/spark/pull/14719"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/14719"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/14719/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/14719/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/14719/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/ccf71fcc8c366eddb4f43b6a79105e47576d99d5"}}}}
{"actor":{"display_login":"cloud-fan"}}