{"action":"created","issue":{"url":"https://api.github.com/repos/mlr-org/mlr/issues/1034","repository_url":"https://api.github.com/repos/mlr-org/mlr","labels_url":"https://api.github.com/repos/mlr-org/mlr/issues/1034/labels{/name}","comments_url":"https://api.github.com/repos/mlr-org/mlr/issues/1034/comments","events_url":"https://api.github.com/repos/mlr-org/mlr/issues/1034/events","html_url":"https://github.com/mlr-org/mlr/issues/1034","id":167021582,"number":1034,"title":"Add unit tests to check for wrong default values","user":{"login":"giuseppec","id":4238748,"avatar_url":"https://avatars.githubusercontent.com/u/4238748?v=3","gravatar_id":"","url":"https://api.github.com/users/giuseppec","html_url":"https://github.com/giuseppec","followers_url":"https://api.github.com/users/giuseppec/followers","following_url":"https://api.github.com/users/giuseppec/following{/other_user}","gists_url":"https://api.github.com/users/giuseppec/gists{/gist_id}","starred_url":"https://api.github.com/users/giuseppec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/giuseppec/subscriptions","organizations_url":"https://api.github.com/users/giuseppec/orgs","repos_url":"https://api.github.com/users/giuseppec/repos","events_url":"https://api.github.com/users/giuseppec/events{/privacy}","received_events_url":"https://api.github.com/users/giuseppec/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mlr-org/mlr/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/mlr-org/mlr/labels/hiwi","name":"hiwi","color":"c2e0c6"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-07-22T10:59:41Z","updated_at":"2016-07-22T11:12:22Z","closed_at":null,"body":"In mlr we maybe have wrong default values, for example in the `?mboost::boost_control` documentation it says that `center = TRUE`  is the default. In our mlr param set, the default is `FALSE`, see:\r\n\r\n```r\r\ngetParamSet(lrn)$pars[[\"center\"]]\r\n#     Type len   Def Constr Req Tunable Trafo\r\n# 1 logical   - FALSE      -   -    TRUE     -\r\n# Used: train.\r\n```\r\n\r\nThis is the issue that is caused:\r\n\r\n```r\r\nlrn = makeLearner(\"regr.glmboost\")\r\nset.seed(1);(bench <- benchmark(lrn, bh.task))\r\n#                task.id    learner.id mse.test.mean\r\n# 1 BostonHousing-example regr.glmboost      25.95779\r\n\r\nlrn2 = setHyperPars2(lrn, getDefaults(lrn$par.set))\r\nset.seed(1);(bench2 <- benchmark(lrn2, bh.task))\r\n#                 task.id    learner.id mse.test.mean\r\n# 1 BostonHousing-example regr.glmboost      42.50329\r\n```\r\n\r\nMaybe we should add a test for all learners such as:\r\n\r\n```\r\ntestthat::expect_equal(as.data.frame(bench), as.data.frame(bench2))\r\n```"},"comment":{"url":"https://api.github.com/repos/mlr-org/mlr/issues/comments/234518255","html_url":"https://github.com/mlr-org/mlr/issues/1034#issuecomment-234518255","issue_url":"https://api.github.com/repos/mlr-org/mlr/issues/1034","id":234518255,"user":{"login":"giuseppec","id":4238748,"avatar_url":"https://avatars.githubusercontent.com/u/4238748?v=3","gravatar_id":"","url":"https://api.github.com/users/giuseppec","html_url":"https://github.com/giuseppec","followers_url":"https://api.github.com/users/giuseppec/followers","following_url":"https://api.github.com/users/giuseppec/following{/other_user}","gists_url":"https://api.github.com/users/giuseppec/gists{/gist_id}","starred_url":"https://api.github.com/users/giuseppec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/giuseppec/subscriptions","organizations_url":"https://api.github.com/users/giuseppec/orgs","repos_url":"https://api.github.com/users/giuseppec/repos","events_url":"https://api.github.com/users/giuseppec/events{/privacy}","received_events_url":"https://api.github.com/users/giuseppec/received_events","type":"User","site_admin":false},"created_at":"2016-07-22T11:12:22Z","updated_at":"2016-07-22T11:12:22Z","body":"In general, making sure that our defaults are always correct is very important for reproducibility in openml because we decided not to upload default hyperparameter values to openml. Applying a learner `lrn = makeLearner(\"regr.glmboost\")` and `lrn2 = setHyperPars2(lrn, getDefaults(lrn$par.set))` on the same task with the same seed should therefore always return the same result, shouldn't it?"}}
{"actor":{"display_login":"giuseppec"}}