{"action":"opened","issue":{"url":"https://api.github.com/repos/WinVector/vtreat/issues/19","repository_url":"https://api.github.com/repos/WinVector/vtreat","labels_url":"https://api.github.com/repos/WinVector/vtreat/issues/19/labels{/name}","comments_url":"https://api.github.com/repos/WinVector/vtreat/issues/19/comments","events_url":"https://api.github.com/repos/WinVector/vtreat/issues/19/events","html_url":"https://github.com/WinVector/vtreat/issues/19","id":337989093,"node_id":"MDU6SXNzdWUzMzc5ODkwOTM=","number":19,"title":"Unable to create \"rare\" level for categorical variable with two levels","user":{"login":"peterzsohar","id":6389807,"node_id":"MDQ6VXNlcjYzODk4MDc=","avatar_url":"https://avatars3.githubusercontent.com/u/6389807?v=4","gravatar_id":"","url":"https://api.github.com/users/peterzsohar","html_url":"https://github.com/peterzsohar","followers_url":"https://api.github.com/users/peterzsohar/followers","following_url":"https://api.github.com/users/peterzsohar/following{/other_user}","gists_url":"https://api.github.com/users/peterzsohar/gists{/gist_id}","starred_url":"https://api.github.com/users/peterzsohar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterzsohar/subscriptions","organizations_url":"https://api.github.com/users/peterzsohar/orgs","repos_url":"https://api.github.com/users/peterzsohar/repos","events_url":"https://api.github.com/users/peterzsohar/events{/privacy}","received_events_url":"https://api.github.com/users/peterzsohar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-03T17:22:39Z","updated_at":"2018-07-03T17:22:39Z","closed_at":null,"author_association":"NONE","body":"I am trying to create a \"rare\" level dummy from a two-level factor variable that has a rare level with low frequency.\r\nIn the test/holdout data I expect to see a potential third level (also rare) that I would like to bundle with the rare level in the train data after applying the treatment.\r\n\r\nExample below:\r\n```\r\nlibrary(vtreat)\r\n\r\nset.seed(1133)\r\nN = 10000\r\np0_y = 0.5\r\np0_x = 0.999\r\n\r\ndf = data.frame(y = sample(c(0, 1), size = N, prob = c(p0_y, 1 - p0_y), replace = TRUE),\r\n                x = as.factor(sample(c(0, 1), size = N, prob = c(p0_x, 1 - p0_x), replace = TRUE)))\r\n\r\nprint(summary(df))\r\n\r\n#       y          x       \r\n# Min.   :0.0000   0:9987  \r\n# 1st Qu.:0.0000   1:  13  \r\n# Median :1.0000           \r\n# Mean   :0.5034           \r\n# 3rd Qu.:1.0000           \r\n# Max.   :1.0000 \r\n\r\ntreatmentsC = designTreatmentsC(dframe = df,\r\n                                varlist = c(\"x\"),\r\n                                outcomename = \"y\",\r\n                                outcometarget = 1,\r\n                                minFraction = 0.0,\r\n                                rareCount = 50)\r\n\r\n[1] \"vtreat 1.2.1 inspecting inputs Tue Jul 03 19:02:23 2018\"\r\n[1] \"designing treatments Tue Jul 03 19:02:23 2018\"\r\n[1] \" have initial level statistics Tue Jul 03 19:02:23 2018\"\r\nError in .designTreatmentsXS(dframe, varlist, outcomename, zoY, zC, zTarget,  : \r\n  no usable vars\r\n```\r\nI would expect the get back variables `x_lev_x_0` and `x_lev_rare`.\r\nThen if the test/holdout data has factor level `2` it could be bundled with `x_lev_rare`.\r\n\r\nHowever, if I set `rareCount = 0` (or anything less than the frequency of the rare level) I get `x_lev_x_0` and `x_lev_x_1` as expected but then I can't bundle the novel level from the test/holdout without manually inspecting `novel_level_summary()`."}}
{"actor":{"display_login":"peterzsohar"}}