{"action":"opened","issue":{"url":"https://api.github.com/repos/hadley/modelr/issues/37","repository_url":"https://api.github.com/repos/hadley/modelr","labels_url":"https://api.github.com/repos/hadley/modelr/issues/37/labels{/name}","comments_url":"https://api.github.com/repos/hadley/modelr/issues/37/comments","events_url":"https://api.github.com/repos/hadley/modelr/issues/37/events","html_url":"https://github.com/hadley/modelr/issues/37","id":197281198,"number":37,"title":"rsquare not robust","user":{"login":"drsimonj","id":11235783,"avatar_url":"https://avatars.githubusercontent.com/u/11235783?v=3","gravatar_id":"","url":"https://api.github.com/users/drsimonj","html_url":"https://github.com/drsimonj","followers_url":"https://api.github.com/users/drsimonj/followers","following_url":"https://api.github.com/users/drsimonj/following{/other_user}","gists_url":"https://api.github.com/users/drsimonj/gists{/gist_id}","starred_url":"https://api.github.com/users/drsimonj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drsimonj/subscriptions","organizations_url":"https://api.github.com/users/drsimonj/orgs","repos_url":"https://api.github.com/users/drsimonj/repos","events_url":"https://api.github.com/users/drsimonj/events{/privacy}","received_events_url":"https://api.github.com/users/drsimonj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-22T22:38:25Z","updated_at":"2016-12-22T22:38:25Z","closed_at":null,"body":"Looks like [rsquare calculation](https://github.com/hadley/modelr/blob/master/R/quality.R#L36-L38) is \"SSR / SST\". For a more robust solution (explanation below), I suggest using \"1 - SSE / SST\" instead. For this, the function could be rewritten as:\r\n\r\n```r\r\nrsquare <- function(model, data) {\r\n  1 - stats::var(residuals(model, data)) / stats::var(response(model, data), na.rm = TRUE)\r\n}\r\n```\r\n\r\nHappy to submit a PR later, but I've got one live now, so don't want to tangle them up.\r\n\r\n## Explanation\r\n\r\n\"SSR / SST\" has few caveats. Very relevant (I think), it doesn't apply when a model fitted on one data set (training data) is used to predict new data (test set). Here's a worked example where we should get a negative R-Squared (predicted values are worse than the mean), but we get `Inf`.\r\n\r\n```r\r\nlibrary(modelr)\r\n\r\nset.seed(12)\r\n# Training data\r\ntrain <- data.frame(\r\n  x = 1:10,\r\n  y = 1:10 + rnorm(10, sd = .1)\r\n)\r\n# Test data with constant `y`\r\ntest <- data.frame(\r\n  x = 1:10,\r\n  y = 5\r\n)\r\n\r\nmod <- lm(y ~ x, train)\r\nrsquare(mod, train)\r\n#> [1] 0.9989631\r\n\r\nrsquare(mod, test)\r\n#> [1] Inf\r\n\r\n# Safer to calculate R-squared using residuals (easy to see in plot)\r\nplot(test, ylim = c(1, 10))\r\nabline(coefficients(mod))\r\n```\r\n\r\n![rplot](https://cloud.githubusercontent.com/assets/11235783/21442607/00b198fc-c8f3-11e6-990b-e84828e68862.png)\r\n\r\n\r\n"}}
{"actor":{"display_login":"drsimonj"}}