{"action":"opened","issue":{"url":"https://api.github.com/repos/hadley/forcats/issues/12","repository_url":"https://api.github.com/repos/hadley/forcats","labels_url":"https://api.github.com/repos/hadley/forcats/issues/12/labels{/name}","comments_url":"https://api.github.com/repos/hadley/forcats/issues/12/comments","events_url":"https://api.github.com/repos/hadley/forcats/issues/12/events","html_url":"https://github.com/hadley/forcats/issues/12","id":170680841,"number":12,"title":"Feature request: Lump biggest possible minority into `Other` category","user":{"login":"huftis","id":100251,"avatar_url":"https://avatars.githubusercontent.com/u/100251?v=3","gravatar_id":"","url":"https://api.github.com/users/huftis","html_url":"https://github.com/huftis","followers_url":"https://api.github.com/users/huftis/followers","following_url":"https://api.github.com/users/huftis/following{/other_user}","gists_url":"https://api.github.com/users/huftis/gists{/gist_id}","starred_url":"https://api.github.com/users/huftis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/huftis/subscriptions","organizations_url":"https://api.github.com/users/huftis/orgs","repos_url":"https://api.github.com/users/huftis/repos","events_url":"https://api.github.com/users/huftis/events{/privacy}","received_events_url":"https://api.github.com/users/huftis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-11T15:54:39Z","updated_at":"2016-08-11T15:54:39Z","closed_at":null,"body":"Here’s a use case I sometimes encounter. I have a factor with a large number of levels (e.g. medical diagnoses), but most of them occur very infrequently, and a few occur frequently (e.g. heart disease and cancer). Showing all of them (e.g. on a bar chart) takes up too much space, so I want to lump the infrequent ones into an `Other` category.\r\n\r\nHowever, the *number* of high-frequent levels vary, and so does the proportion (sometimes it’s 1%, sometimes 5%), so I can’t use `fct_collapse()`. Here’s a method that works well for me: Lump all the low-count levels into an `Other` level so that the count of the `Other` level is less than (or perhaps less than or equal to) the *lowest* count of the rest of the levels. In other words, Lump the biggest possible minority into `Other` category. That’s a sensible definition of ‘Other’, and the bar of the `Other` category in the bar chart will always be shorter than (or equal to) the rest of the bars, which looks good.\r\n\r\nExample:\r\n\r\n```\r\n> x = factor(rep(LETTERS[1:9], times=c(2, 82, 1, 51, 5, 62, 11, 1,90)))\r\n> count = sort(table(x))\r\n> count\r\nx\r\n C  H  A  E  G  D  F  B  I \r\n 1  1  2  5 11 51 62 82 90\r\n```\r\n\r\nHere we would lump CHAEG into `Other`, because their total count is 20, which is less than the count of D, but if we add D, the count will increase to 20 + 51 = 71, which is greater than the count of the next level, F (62).\r\n\r\nHere’s a simple proof-of-concept code to find the levels that needs to be merged into `Other`:\r\n\r\n```\r\n> less = cumsum(head(count, -1)) <= count[-1] # Or <\r\n> names(count)[which(less)]                   # Lump these into ‘Other’ category\r\n[1] \"C\" \"H\" \"A\" \"E\" \"G\"\r\n```\r\n\r\n(It needs to be refined to work with some corner cases, e.g. when the the input or output has length 1, or when there are ties in the counts, but it shows the idea.)"}}
{"actor":{"display_login":"huftis"}}