{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15428","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15428/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15428/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15428/events","html_url":"https://github.com/apache/spark/pull/15428","id":182194472,"number":15428,"title":"[SPARK-17219][ML] enhanced NaN value handling in Bucketizer","user":{"login":"VinceShieh","id":2673819,"avatar_url":"https://avatars.githubusercontent.com/u/2673819?v=3","gravatar_id":"","url":"https://api.github.com/users/VinceShieh","html_url":"https://github.com/VinceShieh","followers_url":"https://api.github.com/users/VinceShieh/followers","following_url":"https://api.github.com/users/VinceShieh/following{/other_user}","gists_url":"https://api.github.com/users/VinceShieh/gists{/gist_id}","starred_url":"https://api.github.com/users/VinceShieh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VinceShieh/subscriptions","organizations_url":"https://api.github.com/users/VinceShieh/orgs","repos_url":"https://api.github.com/users/VinceShieh/repos","events_url":"https://api.github.com/users/VinceShieh/events{/privacy}","received_events_url":"https://api.github.com/users/VinceShieh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-10-11T07:42:55Z","updated_at":"2016-10-24T06:36:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15428","html_url":"https://github.com/apache/spark/pull/15428","diff_url":"https://github.com/apache/spark/pull/15428.diff","patch_url":"https://github.com/apache/spark/pull/15428.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR is an enhancement of PR with commit ID:57dc326bd00cf0a49da971e9c573c48ae28acaa2.\r\nNaN is a special type of value which is commonly seen as invalid. But We find that there are certain cases where NaN are also valuable, thus need special handling. We provided user when dealing NaN values with 3 options, to either reserve an extra bucket for NaN values, or remove the NaN values, or report an error, by setting handleNaN \"keep\", \"skip\", or \"error\"(default) respectively.\r\n\r\n'''Before:\r\nval bucketizer: Bucketizer = new Bucketizer()\r\n          .setInputCol(\"feature\")\r\n          .setOutputCol(\"result\")\r\n          .setSplits(splits)\r\n'''After:\r\nval bucketizer: Bucketizer = new Bucketizer()\r\n          .setInputCol(\"feature\")\r\n          .setOutputCol(\"result\")\r\n          .setSplits(splits)\r\n          .setHandleNaN(\"keep\")\r\n\r\n## How was this patch tested?\r\nTests added in QuantileDiscretizerSuite, BucketizerSuite and DataFrameStatSuite\r\n\r\nSigned-off-by: VinceShieh <vincent.xie@intel.com>"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/255659132","html_url":"https://github.com/apache/spark/pull/15428#issuecomment-255659132","issue_url":"https://api.github.com/repos/apache/spark/issues/15428","id":255659132,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-10-24T06:36:24Z","updated_at":"2016-10-24T06:36:24Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67430/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}