{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/217433074","pull_request_review_id":155143541,"id":217433074,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxNzQzMzA3NA==","diff_hunk":"@@ -223,8 +223,8 @@ trait ExpressionEvalHelper extends GeneratorDrivenPropertyChecks with PlanTestBa\n           }\n         } else {\n           val lit = InternalRow(expected, expected)\n-          val expectedRow =\n-            UnsafeProjection.create(Array(expression.dataType, expression.dataType)).apply(lit)\n+          val dtAsNullable = expression.dataType.asNullable\n+          val expectedRow = UnsafeProjection.create(Array(dtAsNullable, dtAsNullable)).apply(lit)","path":"sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala","position":null,"original_position":7,"commit_id":"04a29880f1a0638b579beeb69577100217c8ded4","original_commit_id":"51aa9d58999a546678a8dff1660e3e9f6d73ec8b","user":{"login":"viirya","id":68855,"node_id":"MDQ6VXNlcjY4ODU1","avatar_url":"https://avatars1.githubusercontent.com/u/68855?v=4","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"body":"oh, yes. this is going to detect such failure.","created_at":"2018-09-13T15:36:56Z","updated_at":"2018-09-13T15:36:56Z","html_url":"https://github.com/apache/spark/pull/22375#discussion_r217433074","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/22375","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/217433074"},"html":{"href":"https://github.com/apache/spark/pull/22375#discussion_r217433074"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/22375"}},"in_reply_to_id":216679499},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/22375","id":214180192,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE0MTgwMTky","html_url":"https://github.com/apache/spark/pull/22375","diff_url":"https://github.com/apache/spark/pull/22375.diff","patch_url":"https://github.com/apache/spark/pull/22375.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/22375","number":22375,"state":"open","locked":false,"title":"[SPARK-25388][Test][SQL] Detect incorrect nullable of DataType in the result","user":{"login":"kiszk","id":1315079,"node_id":"MDQ6VXNlcjEzMTUwNzk=","avatar_url":"https://avatars2.githubusercontent.com/u/1315079?v=4","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR can correctly cause assertion failure when incorrect nullable of DataType in the result is generated by a target function to be tested.\r\n\r\nLet us think the following example. In the future, a developer would write incorrect code that returns unexpected result. We have to correctly cause fail in this test since `valueContainsNull=false` while `expr` includes `null`. However, without this PR, this test passes. This PR can correctly cause fail.\r\n\r\n\r\n```\r\ntest(\"test TARGETFUNCTON\") {\r\n  val expr = TARGETMAPFUNCTON()\r\n  // expr = UnsafeMap(3 -> 6, 7 -> null)\r\n  // expr.dataType = (IntegerType, IntegerType, false)\r\n\r\n  expected = Map(3 -> 6, 7 -> null)\r\n  checkEvaluation(expr, expected)\r\n```\r\n\r\nIn [`checkEvaluationWithUnsafeProjection`](https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala#L208-L235), the results are compared using `UnsafeRow`. When the given `expected` is [converted](https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala#L226-L227)) to `UnsafeRow` using the `DataType` of `expr`.\r\n```\r\nval expectedRow = UnsafeProjection.create(Array(expression.dataType, expression.dataType)).apply(lit)\r\n```\r\n\r\nIn summary, `expr` is `[0,1800000038,5000000038,18,2,0,700000003,2,0,6,18,2,0,700000003,2,0,6]` with and w/o this PR. `expected` is converted to\r\n\r\n* w/o  this PR, `[0,1800000038,5000000038,18,2,0,700000003,2,0,6,18,2,0,700000003,2,0,6]`\r\n* with this PR, `[0,1800000038,5000000038,18,2,0,700000003,2,2,6,18,2,0,700000003,2,2,6]`\r\n\r\nAs a result, w/o this PR, the test unexpectedly passes.\r\n\r\n\r\nThis is because, w/o this PR, based on given `dataType`, generated code of projection for `expected` avoids to set nullbit.\r\n```\r\n                    // tmpInput_2 is expected\r\n/* 155 */           for (int index_1 = 0; index_1 < numElements_1; index_1++) {\r\n/* 156 */             mutableStateArray_1[1].write(index_1, tmpInput_2.getInt(index_1));\r\n/* 157 */           }\r\n```\r\n\r\nWith this PR, generated code of projection for `expected` always checks whether nullbit should be set by `isNullAt`\r\n```\r\n                    // tmpInput_2 is expected\r\n/* 161 */           for (int index_1 = 0; index_1 < numElements_1; index_1++) {\r\n/* 162 */\r\n/* 163 */             if (tmpInput_2.isNullAt(index_1)) {\r\n/* 164 */               mutableStateArray_1[1].setNull4Bytes(index_1);\r\n/* 165 */             } else {\r\n/* 166 */               mutableStateArray_1[1].write(index_1, tmpInput_2.getInt(index_1));\r\n/* 167 */             }\r\n/* 168 */\r\n/* 169 */           }\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nExisting UTs","created_at":"2018-09-10T01:54:29Z","updated_at":"2018-09-13T15:36:56Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2fc56e59d8d1fca915cdebc844114d70d505bddd","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/22375/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/22375/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/22375/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/04a29880f1a0638b579beeb69577100217c8ded4","head":{"label":"kiszk:SPARK-25388","ref":"SPARK-25388","sha":"04a29880f1a0638b579beeb69577100217c8ded4","user":{"login":"kiszk","id":1315079,"node_id":"MDQ6VXNlcjEzMTUwNzk=","avatar_url":"https://avatars2.githubusercontent.com/u/1315079?v=4","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"repo":{"id":45008789,"node_id":"MDEwOlJlcG9zaXRvcnk0NTAwODc4OQ==","name":"spark","full_name":"kiszk/spark","owner":{"login":"kiszk","id":1315079,"node_id":"MDQ6VXNlcjEzMTUwNzk=","avatar_url":"https://avatars2.githubusercontent.com/u/1315079?v=4","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/kiszk/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/kiszk/spark","forks_url":"https://api.github.com/repos/kiszk/spark/forks","keys_url":"https://api.github.com/repos/kiszk/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kiszk/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kiszk/spark/teams","hooks_url":"https://api.github.com/repos/kiszk/spark/hooks","issue_events_url":"https://api.github.com/repos/kiszk/spark/issues/events{/number}","events_url":"https://api.github.com/repos/kiszk/spark/events","assignees_url":"https://api.github.com/repos/kiszk/spark/assignees{/user}","branches_url":"https://api.github.com/repos/kiszk/spark/branches{/branch}","tags_url":"https://api.github.com/repos/kiszk/spark/tags","blobs_url":"https://api.github.com/repos/kiszk/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kiszk/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kiszk/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/kiszk/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kiszk/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/kiszk/spark/languages","stargazers_url":"https://api.github.com/repos/kiszk/spark/stargazers","contributors_url":"https://api.github.com/repos/kiszk/spark/contributors","subscribers_url":"https://api.github.com/repos/kiszk/spark/subscribers","subscription_url":"https://api.github.com/repos/kiszk/spark/subscription","commits_url":"https://api.github.com/repos/kiszk/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/kiszk/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/kiszk/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/kiszk/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/kiszk/spark/contents/{+path}","compare_url":"https://api.github.com/repos/kiszk/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kiszk/spark/merges","archive_url":"https://api.github.com/repos/kiszk/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kiszk/spark/downloads","issues_url":"https://api.github.com/repos/kiszk/spark/issues{/number}","pulls_url":"https://api.github.com/repos/kiszk/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/kiszk/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/kiszk/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kiszk/spark/labels{/name}","releases_url":"https://api.github.com/repos/kiszk/spark/releases{/id}","deployments_url":"https://api.github.com/repos/kiszk/spark/deployments","created_at":"2015-10-27T01:25:07Z","updated_at":"2018-08-03T21:23:52Z","pushed_at":"2018-09-13T13:00:44Z","git_url":"git://github.com/kiszk/spark.git","ssh_url":"git@github.com:kiszk/spark.git","clone_url":"https://github.com/kiszk/spark.git","svn_url":"https://github.com/kiszk/spark","homepage":null,"size":272721,"stargazers_count":1,"watchers_count":1,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"77c996403d5c761f0dfea64c5b1cb7480ba1d3ac","user":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"node_id":"MDEwOlJlcG9zaXRvcnkxNzE2NTY1OA==","name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2018-09-13T15:23:45Z","pushed_at":"2018-09-13T15:23:01Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":281720,"stargazers_count":18760,"watchers_count":18760,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":16895,"mirror_url":null,"archived":false,"open_issues_count":490,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":16895,"open_issues":490,"watchers":18760,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/22375"},"html":{"href":"https://github.com/apache/spark/pull/22375"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/22375"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/22375/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/22375/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/22375/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/04a29880f1a0638b579beeb69577100217c8ded4"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"viirya"}}