{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15212","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15212/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15212/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15212/events","html_url":"https://github.com/apache/spark/pull/15212","id":178808665,"number":15212,"title":"[SPARK-17645][MLLIB][ML][WIP]add feature selector method based on: False Discovery Rate (FDR) and Family wise error rate (FWE)","user":{"login":"mpjlu","id":13826327,"avatar_url":"https://avatars.githubusercontent.com/u/13826327?v=3","gravatar_id":"","url":"https://api.github.com/users/mpjlu","html_url":"https://github.com/mpjlu","followers_url":"https://api.github.com/users/mpjlu/followers","following_url":"https://api.github.com/users/mpjlu/following{/other_user}","gists_url":"https://api.github.com/users/mpjlu/gists{/gist_id}","starred_url":"https://api.github.com/users/mpjlu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpjlu/subscriptions","organizations_url":"https://api.github.com/users/mpjlu/orgs","repos_url":"https://api.github.com/users/mpjlu/repos","events_url":"https://api.github.com/users/mpjlu/events{/privacy}","received_events_url":"https://api.github.com/users/mpjlu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-09-23T07:53:40Z","updated_at":"2016-09-27T12:59:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15212","html_url":"https://github.com/apache/spark/pull/15212","diff_url":"https://github.com/apache/spark/pull/15212.diff","patch_url":"https://github.com/apache/spark/pull/15212.patch"},"body":"## What changes were proposed in this pull request?\r\nUnivariate feature selection works by selecting the best features based on univariate statistical tests. \r\nFDR and FWE are a popular univariate statistical test for feature selection.  \r\nIn 2005, the Benjamini and Hochberg paper on FDR was identified as one of the 25 most-cited statistical papers. The FDR uses the Benjamini-Hochberg procedure in this PR. https://en.wikipedia.org/wiki/False_discovery_rate. \r\nIn statistics, FWE is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses tests.\r\nhttps://en.wikipedia.org/wiki/Family-wise_error_rate\r\n\r\nWe add  FDR and FWE methods for ChiSqSelector in this PR, like it is implemented in scikit-learn. \r\nhttp://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection\r\n\r\n## How was this patch tested?\r\n\r\nut will be added soon\r\n\r\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\r\n\r\n\r\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/249857167","html_url":"https://github.com/apache/spark/pull/15212#issuecomment-249857167","issue_url":"https://api.github.com/repos/apache/spark/issues/15212","id":249857167,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-09-27T12:59:26Z","updated_at":"2016-09-27T12:59:26Z","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}