{"action":"created","issue":{"url":"https://api.github.com/repos/topepo/caret/issues/351","labels_url":"https://api.github.com/repos/topepo/caret/issues/351/labels{/name}","comments_url":"https://api.github.com/repos/topepo/caret/issues/351/comments","events_url":"https://api.github.com/repos/topepo/caret/issues/351/events","html_url":"https://github.com/topepo/caret/issues/351","id":126412755,"number":351,"title":"Wrong OOB estimates with cforest method","user":{"login":"asardaes","id":7768461,"avatar_url":"https://avatars.githubusercontent.com/u/7768461?v=3","gravatar_id":"","url":"https://api.github.com/users/asardaes","html_url":"https://github.com/asardaes","followers_url":"https://api.github.com/users/asardaes/followers","following_url":"https://api.github.com/users/asardaes/following{/other_user}","gists_url":"https://api.github.com/users/asardaes/gists{/gist_id}","starred_url":"https://api.github.com/users/asardaes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asardaes/subscriptions","organizations_url":"https://api.github.com/users/asardaes/orgs","repos_url":"https://api.github.com/users/asardaes/repos","events_url":"https://api.github.com/users/asardaes/events{/privacy}","received_events_url":"https://api.github.com/users/asardaes/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-01-13T12:56:53Z","updated_at":"2016-01-18T12:35:29Z","closed_at":"2016-01-13T14:44:14Z","body":"This is a tricky one, and it took me a while to figure it out.\r\n\r\nWhen using `\"oob\"` as training method, the default function to obtain Accuracy and Kappa is the following:\r\n\r\n```r\r\nobs <- x@data@get(\"response\")[,1]\r\npred <- predict(x,  x@data@get(\"input\"), OOB = TRUE)\r\npostResample(pred, obs)\r\n```\r\n\r\nThe second argument to `predict` is the `newdata` parameter. This should be ok, since `x@data@get(\"input\")` just loads the input data. However, looking at the source code from the `party` package, I realized that the *call* to the actual prediction function (which is a C function) does the boolean operation `OOB && is.null(newdata)` to determine which output to give. Therefore, the only way to get the true OOB estimate is by calling the `predict` generic with `newdata = NULL`, which is the default when calling the function as `predict(x, OOB = TRUE)`...\r\n\r\nI realized I was getting OOB Accuracy estimates that were much larger than the one for `finalModel`, and this is the reason.\r\n\r\n------\r\n\r\nProof:\r\n```r\r\nlibrary(party)\r\n\r\ndata(mtcars)\r\n\r\ncforest_fit <- cforest(mpg ~ ., data = mtcars, controls = cforest_unbiased(mtry = 0))\r\n\r\n# True OOB prediction\r\npred1 <- predict(cforest_fit, OOB = TRUE)\r\n\r\n# This should be equal to 'pred1'...\r\npred2 <- predict(cforest_fit, newdata = cforest_fit@data@get(\"input\"), OOB = TRUE)\r\n\r\n# Prediction without using OOB\r\npred3 <- predict(cforest_fit, OOB = FALSE)\r\n\r\n# This is FALSE\r\nidentical(pred1, pred2)\r\n\r\n# This is actually TRUE, i.e. 'pred2' sort of ignores OOB = TRUE\r\nidentical(pred2, pred3)\r\n```"},"comment":{"url":"https://api.github.com/repos/topepo/caret/issues/comments/172517162","html_url":"https://github.com/topepo/caret/issues/351#issuecomment-172517162","issue_url":"https://api.github.com/repos/topepo/caret/issues/351","id":172517162,"user":{"login":"asardaes","id":7768461,"avatar_url":"https://avatars.githubusercontent.com/u/7768461?v=3","gravatar_id":"","url":"https://api.github.com/users/asardaes","html_url":"https://github.com/asardaes","followers_url":"https://api.github.com/users/asardaes/followers","following_url":"https://api.github.com/users/asardaes/following{/other_user}","gists_url":"https://api.github.com/users/asardaes/gists{/gist_id}","starred_url":"https://api.github.com/users/asardaes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asardaes/subscriptions","organizations_url":"https://api.github.com/users/asardaes/orgs","repos_url":"https://api.github.com/users/asardaes/repos","events_url":"https://api.github.com/users/asardaes/events{/privacy}","received_events_url":"https://api.github.com/users/asardaes/received_events","type":"User","site_admin":false},"created_at":"2016-01-18T12:35:29Z","updated_at":"2016-01-18T12:35:29Z","body":"I should also point out that the `predict` generic for `randomForest` has a similar behavior. \r\n\r\nI see you've taken that into account during training, but be aware that if you use `train` with `rf`, `predict(train.rf) != predict(train.rf$finalModel)`, which may or may not be desired."}}