{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15523","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15523/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15523/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15523/events","html_url":"https://github.com/apache/spark/pull/15523","id":183556677,"number":15523,"title":"[SPARK-17981] [SPARK-17957] [SQL] Fix Incorrect Nullability Setting to False in FilterExec","user":{"login":"gatorsmile","id":11567269,"avatar_url":"https://avatars.githubusercontent.com/u/11567269?v=3","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-10-17T23:47:05Z","updated_at":"2016-10-20T04:25:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15523","html_url":"https://github.com/apache/spark/pull/15523","diff_url":"https://github.com/apache/spark/pull/15523.diff","patch_url":"https://github.com/apache/spark/pull/15523.patch"},"body":"### What changes were proposed in this pull request?\r\nWhen `FilterExec` contains `isNotNull`, which could be inferred and pushed down or users specified, we convert the nullability of the involved columns if the top-layer expression is null-intolerant. However, this is not correct, if the top-layer expression is not a leaf expression, it could still tolerate the null when it has null-tolerant child expressions. \r\n\r\nFor example, `cast(coalesce(a#5, a#15) as double)`. Although `cast` is a null-intolerant expression, but obviously` coalesce` is null-tolerant. Thus, it could eat null.\r\n\r\nWhen the nullability is wrong, we could generate incorrect results in different cases. For example, \r\n\r\n```Scala\r\n    val df1 = Seq((1, 2), (2, 3)).toDF(\"a\", \"b\")\r\n    val df2 = Seq((2, 5), (3, 4)).toDF(\"a\", \"c\")\r\n    val joinedDf = df1.join(df2, Seq(\"a\"), \"outer\").na.fill(0)\r\n    val df3 = Seq((3, 1)).toDF(\"a\", \"d\")\r\n    joinedDf.join(df3, \"a\").show\r\n```\r\n\r\nThe optimized plan is like \r\n```\r\nProject [a#29, b#30, c#31, d#42]\r\n+- Join Inner, (a#29 = a#41)\r\n   :- Project [cast(coalesce(cast(coalesce(a#5, a#15) as double), 0.0) as int) AS a#29, cast(coalesce(cast(b#6 as double), 0.0) as int) AS b#30, cast(coalesce(cast(c#16 as double), 0.0) as int) AS c#31]\r\n   :  +- Filter isnotnull(cast(coalesce(cast(coalesce(a#5, a#15) as double), 0.0) as int))\r\n   :     +- Join FullOuter, (a#5 = a#15)\r\n   :        :- LocalRelation [a#5, b#6]\r\n   :        +- LocalRelation [a#15, c#16]\r\n   +- LocalRelation [a#41, d#42]\r\n```\r\n\r\nWithout the fix, it returns an empty result. With the fix, it can return a correct answer:\r\n```\r\n+---+---+---+---+\r\n|  a|  b|  c|  d|\r\n+---+---+---+---+\r\n|  3|  0|  4|  1|\r\n+---+---+---+---+\r\n```\r\n\r\n### How was this patch tested?\r\nAdded test cases to verify the nullability changes in FilterExec. Also added a test case for verifying the reported incorrect result."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/255006001","html_url":"https://github.com/apache/spark/pull/15523#issuecomment-255006001","issue_url":"https://api.github.com/repos/apache/spark/issues/15523","id":255006001,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-10-20T04:25:53Z","updated_at":"2016-10-20T04:25:53Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67229/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}