{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/23265","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/23265/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/23265/comments","events_url":"https://api.github.com/repos/apache/spark/issues/23265/events","html_url":"https://github.com/apache/spark/pull/23265","id":388992101,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM3MTExNjgy","number":23265,"title":"[2.4][SPARK-26021][SQL][FOLLOWUP] only deal with NaN and -0.0 in UnsafeWriter","user":{"login":"cloud-fan","id":3182036,"node_id":"MDQ6VXNlcjMxODIwMzY=","avatar_url":"https://avatars3.githubusercontent.com/u/3182036?v=4","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-12-09T06:26:09Z","updated_at":"2018-12-09T08:05:20Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/23265","html_url":"https://github.com/apache/spark/pull/23265","diff_url":"https://github.com/apache/spark/pull/23265.diff","patch_url":"https://github.com/apache/spark/pull/23265.patch"},"body":"backport https://github.com/apache/spark/pull/23239 to 2.4\r\n\r\n---------\r\n\r\n## What changes were proposed in this pull request?\r\n\r\nA followup of https://github.com/apache/spark/pull/23043\r\n\r\nThere are 4 places we need to deal with NaN and -0.0:\r\n1. comparison expressions. `-0.0` and `0.0` should be treated as same. Different NaNs should be treated as same.\r\n2. Join keys. `-0.0` and `0.0` should be treated as same. Different NaNs should be treated as same.\r\n3. grouping keys. `-0.0` and `0.0` should be assigned to the same group. Different NaNs should be assigned to the same group.\r\n4. window partition keys. `-0.0` and `0.0` should be treated as same. Different NaNs should be treated as same.\r\n\r\nThe case 1 is OK. Our comparison already handles NaN and -0.0, and for struct/array/map, we will recursively compare the fields/elements.\r\n\r\nCase 2, 3 and 4 are problematic, as they compare `UnsafeRow` binary directly, and different NaNs have different binary representation, and the same thing happens for -0.0 and 0.0.\r\n\r\nTo fix it, a simple solution is: normalize float/double when building unsafe data (`UnsafeRow`, `UnsafeArrayData`, `UnsafeMapData`). Then we don't need to worry about it anymore.\r\n\r\nFollowing this direction, this PR moves the handling of NaN and -0.0 from `Platform` to `UnsafeWriter`, so that places like `UnsafeRow.setFloat` will not handle them, which reduces the perf overhead. It's also easier to add comments explaining why we do it in `UnsafeWriter`.\r\n\r\n## How was this patch tested?\r\n\r\nexisting tests"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/445518627","html_url":"https://github.com/apache/spark/pull/23265#issuecomment-445518627","issue_url":"https://api.github.com/repos/apache/spark/issues/23265","id":445518627,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTUxODYyNw==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-12-09T08:05:20Z","updated_at":"2018-12-09T08:05:20Z","author_association":"NONE","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99884/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}