{"action":"created","issue":{"url":"https://api.github.com/repos/mlr-org/mlr/issues/2463","repository_url":"https://api.github.com/repos/mlr-org/mlr","labels_url":"https://api.github.com/repos/mlr-org/mlr/issues/2463/labels{/name}","comments_url":"https://api.github.com/repos/mlr-org/mlr/issues/2463/comments","events_url":"https://api.github.com/repos/mlr-org/mlr/issues/2463/events","html_url":"https://github.com/mlr-org/mlr/pull/2463","id":373098662,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI1MTEzMDE5","number":2463,"title":"Enable caching of filter values during tuning","user":{"login":"pat-s","id":8430564,"node_id":"MDQ6VXNlcjg0MzA1NjQ=","avatar_url":"https://avatars3.githubusercontent.com/u/8430564?v=4","gravatar_id":"","url":"https://api.github.com/users/pat-s","html_url":"https://github.com/pat-s","followers_url":"https://api.github.com/users/pat-s/followers","following_url":"https://api.github.com/users/pat-s/following{/other_user}","gists_url":"https://api.github.com/users/pat-s/gists{/gist_id}","starred_url":"https://api.github.com/users/pat-s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pat-s/subscriptions","organizations_url":"https://api.github.com/users/pat-s/orgs","repos_url":"https://api.github.com/users/pat-s/repos","events_url":"https://api.github.com/users/pat-s/events{/privacy}","received_events_url":"https://api.github.com/users/pat-s/received_events","type":"User","site_admin":false},"labels":[{"id":421273968,"node_id":"MDU6TGFiZWw0MjEyNzM5Njg=","url":"https://api.github.com/repos/mlr-org/mlr/labels/pr-work%20in%20progress%20-%20not%20done","name":"pr-work in progress - not done","color":"d4c5f9","default":false},{"id":556929614,"node_id":"MDU6TGFiZWw1NTY5Mjk2MTQ=","url":"https://api.github.com/repos/mlr-org/mlr/labels/project%20-%20base","name":"project - base","color":"bfd4f2","default":false},{"id":54191380,"node_id":"MDU6TGFiZWw1NDE5MTM4MA==","url":"https://api.github.com/repos/mlr-org/mlr/labels/type-enhancement","name":"type-enhancement","color":"84b6eb","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-23T16:51:21Z","updated_at":"2018-10-23T17:17:55Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/mlr-org/mlr/pulls/2463","html_url":"https://github.com/mlr-org/mlr/pull/2463","diff_url":"https://github.com/mlr-org/mlr/pull/2463.diff","patch_url":"https://github.com/mlr-org/mlr/pull/2463.patch"},"body":"fixes #1995 \t\r\n\r\n# Implementation\r\n\r\n- caching via `memoise::memoise()`\r\n- caching is done on the filesystem because memory caching is not supported in parallel processes (https://github.com/r-lib/memoise/issues/77)\r\n- argument `cache = T/F` can be passed to `tuneParams()` or `resample()` to prevent caching for nondeterministic filters\r\n- removal of argument `nselect` in `generateFilterValuesData()` and all filters because its not used\r\n- making `generateFilterValuesData()` internal because `filterFeatures()` can do everything of `generateFilterValuesData()`\r\n- setting `perc = 1` as default in `filterFeatures()`\r\n- filter values are attached to the task as task$fval (data.frame)\r\n\r\n\r\n# Benchmark example\r\n\r\n#### WITH Cache\r\n\r\n``` r\r\nlibrary(mlr)\r\n#> Loading required package: ParamHelpers\r\nlibrary(microbenchmark)\r\n\r\nlrn = makeFilterWrapper(learner = \"regr.ksvm\", fw.method = \"chi.squared\")\r\nps = makeParamSet(makeNumericParam(\"fw.perc\", lower = 0, upper = 1),\r\n                  makeNumericParam(\"C\", lower = -10, upper = 10, \r\n                                   trafo = function(x) 2^x),\r\n                  makeNumericParam(\"sigma\", lower = -10, upper = 10,\r\n                                   trafo = function(x) 2^x)\r\n)\r\nrdesc = makeResampleDesc(\"CV\", iters = 3)\r\n\r\ny <- rnorm(100)\r\nx <- matrix(rnorm(100 * 2000), 100, 2000)\r\ndat <- data.frame(data.frame(y, as.data.frame(x)))\r\ntask = makeRegrTask(target = \"y\", data = dat)\r\n\r\nset.seed(123)\r\nmicrobenchmark(tuneParams(lrn, task = task, resampling = rdesc, par.set = ps,\r\n                 control = makeTuneControlRandom(maxit = 10)),\r\n               times = 1)\r\n#> [Tune] Started tuning learner regr.ksvm.filtered for parameter set:\r\n#>            Type len Def    Constr Req Tunable Trafo\r\n#> fw.perc numeric   -   -    0 to 1   -    TRUE     -\r\n#> C       numeric   -   - -10 to 10   -    TRUE     Y\r\n#> sigma   numeric   -   - -10 to 10   -    TRUE     Y\r\n#> With control class: TuneControlRandom\r\n#> Imputation value: Inf\r\n#> [Tune-x] 1: fw.perc=0.962; C=4.08; sigma=1.23\r\n#> [Tune-y] 1: mse.test.mean=0.8765976; time: 0.3 min\r\n#> [Tune-x] 2: fw.perc=0.403; C=195; sigma=0.152\r\n#> [Tune-y] 2: mse.test.mean=0.8765976; time: 0.0 min\r\n#> [Tune-x] 3: fw.perc=0.288; C=0.0104; sigma=0.0106\r\n#> [Tune-y] 3: mse.test.mean=0.8866969; time: 0.0 min\r\n#> [Tune-x] 4: fw.perc=0.482; C=0.0326; sigma=0.0196\r\n#> [Tune-y] 4: mse.test.mean=0.8861229; time: 0.0 min\r\n#> [Tune-x] 5: fw.perc=0.674; C=0.00189; sigma=16.2\r\n#> [Tune-y] 5: mse.test.mean=0.8871299; time: 0.0 min\r\n#> [Tune-x] 6: fw.perc=0.352; C=0.283; sigma=85.6\r\n#> [Tune-y] 6: mse.test.mean=0.8809662; time: 0.0 min\r\n#> [Tune-x] 7: fw.perc=0.919; C=0.0491; sigma=597\r\n#> [Tune-y] 7: mse.test.mean=0.8852991; time: 0.0 min\r\n#> [Tune-x] 8: fw.perc=0.728; C=13.2; sigma=0.00203\r\n#> [Tune-y] 8: mse.test.mean=0.8767769; time: 0.0 min\r\n#> [Tune-x] 9: fw.perc=0.395; C=0.736; sigma=2.31\r\n#> [Tune-y] 9: mse.test.mean=0.8785797; time: 0.0 min\r\n#> [Tune-x] 10: fw.perc=0.698; C=318; sigma=5.16\r\n#> [Tune-y] 10: mse.test.mean=0.8765976; time: 0.0 min\r\n#> [Tune] Result: fw.perc=0.403; C=195; sigma=0.152 : mse.test.mean=0.8765976\r\n#> Unit: seconds\r\n#>                                                                                                              expr\r\n#>  tuneParams(lrn, task = task, resampling = rdesc, par.set = ps,      control = makeTuneControlRandom(maxit = 10))\r\n#>       min       lq     mean   median       uq      max neval\r\n#>  29.17415 29.17415 29.17415 29.17415 29.17415 29.17415     1\r\n```\r\n\r\n<sup>Created on 2018-10-23 by the [reprex package](https://reprex.tidyverse.org) (v0.2.1)</sup>\r\n\r\n#### WITHOUT caching\r\n\r\n``` r\r\nlibrary(mlr)\r\n#> Loading required package: ParamHelpers\r\nlibrary(microbenchmark)\r\n\r\nlrn = makeFilterWrapper(learner = \"regr.ksvm\", fw.method = \"chi.squared\")\r\nps = makeParamSet(makeNumericParam(\"fw.perc\", lower = 0, upper = 1),\r\n                  makeNumericParam(\"C\", lower = -10, upper = 10, \r\n                                   trafo = function(x) 2^x),\r\n                  makeNumericParam(\"sigma\", lower = -10, upper = 10,\r\n                                   trafo = function(x) 2^x)\r\n)\r\nrdesc = makeResampleDesc(\"CV\", iters = 3)\r\n\r\ny <- rnorm(100)\r\nx <- matrix(rnorm(100 * 2000), 100, 2000)\r\ndat <- data.frame(data.frame(y, as.data.frame(x)))\r\ntask = makeRegrTask(target = \"y\", data = dat)\r\n\r\nset.seed(123)\r\nmicrobenchmark(tuneParams(lrn, task = task, resampling = rdesc, par.set = ps,\r\n                 control = makeTuneControlRandom(maxit = 10)),\r\n               times = 1)\r\n#> [Tune] Started tuning learner regr.ksvm.filtered for parameter set:\r\n#>            Type len Def    Constr Req Tunable Trafo\r\n#> fw.perc numeric   -   -    0 to 1   -    TRUE     -\r\n#> C       numeric   -   - -10 to 10   -    TRUE     Y\r\n#> sigma   numeric   -   - -10 to 10   -    TRUE     Y\r\n#> With control class: TuneControlRandom\r\n#> Imputation value: Inf\r\n#> [Tune-x] 1: fw.perc=0.962; C=4.08; sigma=1.23\r\n#> [Tune-y] 1: mse.test.mean=1.0014789; time: 0.3 min\r\n#> [Tune-x] 2: fw.perc=0.403; C=195; sigma=0.152\r\n#> [Tune-y] 2: mse.test.mean=1.0014789; time: 0.3 min\r\n#> [Tune-x] 3: fw.perc=0.288; C=0.0104; sigma=0.0106\r\n#> [Tune-y] 3: mse.test.mean=1.0035937; time: 0.2 min\r\n#> [Tune-x] 4: fw.perc=0.482; C=0.0326; sigma=0.0196\r\n#> [Tune-y] 4: mse.test.mean=1.0033925; time: 0.3 min\r\n#> [Tune-x] 5: fw.perc=0.674; C=0.00189; sigma=16.2\r\n#> [Tune-y] 5: mse.test.mean=1.0038892; time: 0.2 min\r\n#> [Tune-x] 6: fw.perc=0.352; C=0.283; sigma=85.6\r\n#> [Tune-y] 6: mse.test.mean=1.0010886; time: 0.2 min\r\n#> [Tune-x] 7: fw.perc=0.919; C=0.0491; sigma=597\r\n#> [Tune-y] 7: mse.test.mean=1.0033925; time: 0.3 min\r\n#> [Tune-x] 8: fw.perc=0.728; C=13.2; sigma=0.00203\r\n#> [Tune-y] 8: mse.test.mean=1.0001677; time: 0.3 min\r\n#> [Tune-x] 9: fw.perc=0.395; C=0.736; sigma=2.31\r\n#> [Tune-y] 9: mse.test.mean=1.0022391; time: 0.2 min\r\n#> [Tune-x] 10: fw.perc=0.698; C=318; sigma=5.16\r\n#> [Tune-y] 10: mse.test.mean=1.0014789; time: 0.2 min\r\n#> [Tune] Result: fw.perc=0.728; C=13.2; sigma=0.00203 : mse.test.mean=1.0001677\r\n#> Unit: seconds\r\n#>                                                                                                              expr\r\n#>  tuneParams(lrn, task = task, resampling = rdesc, par.set = ps,      control = makeTuneControlRandom(maxit = 10))\r\n#>       min       lq     mean   median       uq      max neval\r\n#>  154.3293 154.3293 154.3293 154.3293 154.3293 154.3293     1\r\n```\r\n\r\n<sup>Created on 2018-10-23 by the [reprex package](https://reprex.tidyverse.org) (v0.2.1)</sup>"},"comment":{"url":"https://api.github.com/repos/mlr-org/mlr/issues/comments/432338713","html_url":"https://github.com/mlr-org/mlr/pull/2463#issuecomment-432338713","issue_url":"https://api.github.com/repos/mlr-org/mlr/issues/2463","id":432338713,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMjMzODcxMw==","user":{"login":"larskotthoff","id":579233,"node_id":"MDQ6VXNlcjU3OTIzMw==","avatar_url":"https://avatars2.githubusercontent.com/u/579233?v=4","gravatar_id":"","url":"https://api.github.com/users/larskotthoff","html_url":"https://github.com/larskotthoff","followers_url":"https://api.github.com/users/larskotthoff/followers","following_url":"https://api.github.com/users/larskotthoff/following{/other_user}","gists_url":"https://api.github.com/users/larskotthoff/gists{/gist_id}","starred_url":"https://api.github.com/users/larskotthoff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larskotthoff/subscriptions","organizations_url":"https://api.github.com/users/larskotthoff/orgs","repos_url":"https://api.github.com/users/larskotthoff/repos","events_url":"https://api.github.com/users/larskotthoff/events{/privacy}","received_events_url":"https://api.github.com/users/larskotthoff/received_events","type":"User","site_admin":false},"created_at":"2018-10-23T17:17:55Z","updated_at":"2018-10-23T17:17:55Z","author_association":"CONTRIBUTOR","body":"Could you add a test that checks that results are the same with and without caching please?"}}
{"actor":{"display_login":"larskotthoff"}}