{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12730","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12730/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12730/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12730/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/12730","id":388246185,"node_id":"MDU6SXNzdWUzODgyNDYxODU=","number":12730,"title":"Add paired test for cross-validation","user":{"login":"amueller","id":449558,"node_id":"MDQ6VXNlcjQ0OTU1OA==","avatar_url":"https://avatars3.githubusercontent.com/u/449558?v=4","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-12-06T14:47:51Z","updated_at":"2018-12-07T14:30:38Z","closed_at":null,"author_association":"MEMBER","body":"It would be nice to add error bars for grid-search results. Right now I think we're showing std over folds, which is not the right thing (there's a nice talk & package by Max Kuhn about it, also mentioned in todays NIPS invited talk).\r\nWe should implement a paired test to have a better measure of uncertainty between parameter settings."},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/445249537","html_url":"https://github.com/scikit-learn/scikit-learn/issues/12730#issuecomment-445249537","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/12730","id":445249537,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTI0OTUzNw==","user":{"login":"MiladShahidi","id":36179054,"node_id":"MDQ6VXNlcjM2MTc5MDU0","avatar_url":"https://avatars0.githubusercontent.com/u/36179054?v=4","gravatar_id":"","url":"https://api.github.com/users/MiladShahidi","html_url":"https://github.com/MiladShahidi","followers_url":"https://api.github.com/users/MiladShahidi/followers","following_url":"https://api.github.com/users/MiladShahidi/following{/other_user}","gists_url":"https://api.github.com/users/MiladShahidi/gists{/gist_id}","starred_url":"https://api.github.com/users/MiladShahidi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MiladShahidi/subscriptions","organizations_url":"https://api.github.com/users/MiladShahidi/orgs","repos_url":"https://api.github.com/users/MiladShahidi/repos","events_url":"https://api.github.com/users/MiladShahidi/events{/privacy}","received_events_url":"https://api.github.com/users/MiladShahidi/received_events","type":"User","site_admin":false},"created_at":"2018-12-07T14:30:38Z","updated_at":"2018-12-07T14:30:38Z","author_association":"NONE","body":"I found this [talk](https://www.youtube.com/watch?v=MJdsnJCWIuA) in which he mentions [Tidyposterior](https://cran.r-project.org/web/packages/tidyposterior/tidyposterior.pdf). And Tidyposterior cites that [paper](http://jmlr.org/papers/volume18/16-305/16-305.pdf).\r\n\r\nYou're right @adrinjalali . The paper beats the hell out of frequentist t-tests and then proposes a Bayesian approach. Maybe we could implement both? Even after all that criticism, t-test does not seem totally useless to me, and it is what most people want to do, at least for now."}}
{"actor":{"display_login":"MiladShahidi"}}