{"action":"closed","issue":{"url":"https://api.github.com/repos/topepo/caret/issues/401","repository_url":"https://api.github.com/repos/topepo/caret","labels_url":"https://api.github.com/repos/topepo/caret/issues/401/labels{/name}","comments_url":"https://api.github.com/repos/topepo/caret/issues/401/comments","events_url":"https://api.github.com/repos/topepo/caret/issues/401/events","html_url":"https://github.com/topepo/caret/issues/401","id":146086538,"number":401,"title":"Seed values for K-fold CV","user":{"login":"crossxwill","id":4138594,"avatar_url":"https://avatars.githubusercontent.com/u/4138594?v=3","gravatar_id":"","url":"https://api.github.com/users/crossxwill","html_url":"https://github.com/crossxwill","followers_url":"https://api.github.com/users/crossxwill/followers","following_url":"https://api.github.com/users/crossxwill/following{/other_user}","gists_url":"https://api.github.com/users/crossxwill/gists{/gist_id}","starred_url":"https://api.github.com/users/crossxwill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crossxwill/subscriptions","organizations_url":"https://api.github.com/users/crossxwill/orgs","repos_url":"https://api.github.com/users/crossxwill/repos","events_url":"https://api.github.com/users/crossxwill/events{/privacy}","received_events_url":"https://api.github.com/users/crossxwill/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-04-05T19:25:28Z","updated_at":"2016-04-06T00:08:57Z","closed_at":"2016-04-06T00:08:56Z","body":"I created seed values to ensure that the observations in a single fold are identical between models. For example, if observations 1, 2, 4, 5, 10 are in fold 1, I expect the same observations to be in fold 1 across all models.\r\n\r\nHowever, this doesn't appear to be the case. The stepwise algorithm below actually selects all the covariates in every single fold. This is identical to fitting a lm model with all the covariates. However, the RMSE are never the same for a single fold.\r\n\r\n### Minimal example:\r\n```{R}\r\nlibrary(urca)\r\n\r\ndata(denmark)\r\n\r\nlibrary(caret)\r\n\r\n### set seed values\r\nset.seed(123)\r\nseeds <- vector(mode = \"list\", length = 11)         ### number of folds + 1\r\nfor(i in 1:10) seeds[[i]] <- sample.int(1000, 3)    ### number of fits per model (for lm there is only 1 fit per model)\r\nseeds[[11]] <- 1983                                 ### seed value for 10-fold CV\r\n\r\n### Use 10-fold cross-validation\r\nctrl <- trainControl(method = \"cv\",\r\n                     number = 10,\r\n                     seeds = seeds)\r\n\r\ncs.lm.mod <- train(form=LRY ~ . - ENTRY, data=denmark, trControl=ctrl, metric=\"RMSE\", maximize=FALSE, method=\"lm\")\r\ncs.stepwise <- train(form=LRY  ~ . - ENTRY, data=denmark, trControl=ctrl, metric=\"RMSE\", maximize=FALSE, method=\"lmStepAIC\", trace=1)\r\n\r\ncs.out.of.sample <- resamples(list(OLS=cs.lm.mod, OLS.Stepwise=cs.stepwise))\r\n\r\ncs.lm.mod$resample\r\n\r\ncs.stepwise$resample\r\n```\r\n"}}