{"action":"created","issue":{"url":"https://api.github.com/repos/kbenoit/quanteda/issues/296","repository_url":"https://api.github.com/repos/kbenoit/quanteda","labels_url":"https://api.github.com/repos/kbenoit/quanteda/issues/296/labels{/name}","comments_url":"https://api.github.com/repos/kbenoit/quanteda/issues/296/comments","events_url":"https://api.github.com/repos/kbenoit/quanteda/issues/296/events","html_url":"https://github.com/kbenoit/quanteda/issues/296","id":188804954,"number":296,"title":"applyDictionary() for tokens will not record keys not matched","user":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"labels":[{"id":11645692,"url":"https://api.github.com/repos/kbenoit/quanteda/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":476279735,"url":"https://api.github.com/repos/kbenoit/quanteda/labels/tokenizing","name":"tokenizing","color":"e99695","default":false}],"state":"open","locked":false,"assignee":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"assignees":[{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},{"login":"koheiw","id":6572963,"avatar_url":"https://avatars.githubusercontent.com/u/6572963?v=3","gravatar_id":"","url":"https://api.github.com/users/koheiw","html_url":"https://github.com/koheiw","followers_url":"https://api.github.com/users/koheiw/followers","following_url":"https://api.github.com/users/koheiw/following{/other_user}","gists_url":"https://api.github.com/users/koheiw/gists{/gist_id}","starred_url":"https://api.github.com/users/koheiw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/koheiw/subscriptions","organizations_url":"https://api.github.com/users/koheiw/orgs","repos_url":"https://api.github.com/users/koheiw/repos","events_url":"https://api.github.com/users/koheiw/events{/privacy}","received_events_url":"https://api.github.com/users/koheiw/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2016-11-11T17:02:41Z","updated_at":"2016-11-11T17:08:47Z","closed_at":null,"body":"### Consequence\r\n\r\nKeys not matched will not be recorded as zero-occurring keys in a dfm constructed from the tokens to which the dictionary has been applied.\r\n\r\n### Illustration\r\n\r\n```r\r\ntxt <- c(d1 = \"The United States is bordered by the Atlantic Ocean and the Pacific Ocean.\",\r\n         d2 = \"The Supreme Court of the United States is seldom in a united state.\")\r\ntoks <- tokens(txt)\r\n\r\ndict <- dictionary(list(Countries = c(\"States\"),\r\n                        oceans = c(\"Atlantic\", \"Pacific\"),\r\n                        gameconsoles = c(\"Xbox\", \"Nintendo\")))\r\n\r\n# gamesconsoles is gone\r\ndfm(applyDictionary(toks, dict, valuetype = \"fixed\"), verbose = FALSE)\r\n## Document-feature matrix of: 2 documents, 2 features (25% sparse).\r\n## 2 x 2 sparse Matrix of class \"dfmSparse\"\r\n##     features\r\n## docs Countries oceans\r\n##   d1         1      2\r\n##   d2         1      0\r\n\r\n# gamesconsoles is back\r\ndfm(toks, dictionary = dict, valuetype = \"fixed\", verbose = FALSE)\r\n## Document-feature matrix of: 2 documents, 3 features (0% sparse).\r\n## 2 x 3 sparse Matrix of class \"dfmSparse\"\r\n##     features\r\n## docs Countries oceans gameconsoles\r\n##   d1         1      2            0\r\n##   d2         1      0            0\r\n```\r\n\r\n### Suggested solution\r\n\r\nMake the `dfm() constructor use the dictionary attribute that is added to the `tokens` object by `applyDictionary.tokens()`.  This has the keys, and can make sure that even empty keys not found in the `types` attribute of the `tokens` object also become (zero-frequency) features.\r\n\r\n```r\r\n> str(attr(applyDictionary(toks, dict, valuetype = \"fixed\"), \"dictionary\"))\r\nFormal class 'dictionary' [package \"quanteda\"] with 4 slots\r\n  ..@ .Data       :List of 3\r\n  .. ..$ : chr \"states\"\r\n  .. ..$ : chr [1:2] \"atlantic\" \"pacific\"\r\n  .. ..$ : chr [1:2] \"xbox\" \"nintendo\"\r\n  ..@ concatenator: chr \" \"\r\n  ..@ format      : NULL\r\n  ..@ file        : NULL\r\n```"},"comment":{"url":"https://api.github.com/repos/kbenoit/quanteda/issues/comments/260005166","html_url":"https://github.com/kbenoit/quanteda/issues/296#issuecomment-260005166","issue_url":"https://api.github.com/repos/kbenoit/quanteda/issues/296","id":260005166,"user":{"login":"kbenoit","id":2182246,"avatar_url":"https://avatars.githubusercontent.com/u/2182246?v=3","gravatar_id":"","url":"https://api.github.com/users/kbenoit","html_url":"https://github.com/kbenoit","followers_url":"https://api.github.com/users/kbenoit/followers","following_url":"https://api.github.com/users/kbenoit/following{/other_user}","gists_url":"https://api.github.com/users/kbenoit/gists{/gist_id}","starred_url":"https://api.github.com/users/kbenoit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbenoit/subscriptions","organizations_url":"https://api.github.com/users/kbenoit/orgs","repos_url":"https://api.github.com/users/kbenoit/repos","events_url":"https://api.github.com/users/kbenoit/events{/privacy}","received_events_url":"https://api.github.com/users/kbenoit/received_events","type":"User","site_admin":false},"created_at":"2016-11-11T17:08:47Z","updated_at":"2016-11-11T17:08:47Z","body":"### Other possible solution\r\n\r\nIf we add a value to the `types` vector for the empty dictionary keys, we could construct the dfm so that it always includes a feature for every type, even if there are no values for that type in the tokens list. We would need to be careful then about how or when we apply `tokens_hashed_recompile()` (in `tokens.R`) since this always reindexes the `types` table to sequential integers for just those types that actually occur in the tokens list."}}
{"actor":{"display_login":"kbenoit"}}