{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2639","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2639/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2639/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2639/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/2639","id":23838705,"number":2639,"title":"Use succinct tries for vocabulary storage","user":{"login":"kmike","id":107893,"avatar_url":"https://avatars.githubusercontent.com/u/107893?v=3","gravatar_id":"","url":"https://api.github.com/users/kmike","html_url":"https://github.com/kmike","followers_url":"https://api.github.com/users/kmike/followers","following_url":"https://api.github.com/users/kmike/following{/other_user}","gists_url":"https://api.github.com/users/kmike/gists{/gist_id}","starred_url":"https://api.github.com/users/kmike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmike/subscriptions","organizations_url":"https://api.github.com/users/kmike/orgs","repos_url":"https://api.github.com/users/kmike/repos","events_url":"https://api.github.com/users/kmike/events{/privacy}","received_events_url":"https://api.github.com/users/kmike/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/labels/New%20Feature","name":"New Feature","color":"02e10c"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":37,"created_at":"2013-12-06T05:29:34Z","updated_at":"2016-10-05T19:26:46Z","closed_at":null,"body":"Hey,\r\n\r\nPython dictionary is quite wasteful for string data, so `CountVectorizer`, `TfidfVectorizer` and `DictVectorizer` all could take much less memory with an another data structure for `vocabulary_`. \r\n\r\nHere is a quick prototype that persists `vocabulary_` of `CountVectorizer` and `TfidfVectorizer` in [MARISA-Trie](https://code.google.com/p/marisa-trie/) via Python [wrapper](https://github.com/kmike/marisa-trie): https://gist.github.com/kmike/7814472.\r\n\r\nI used [this](https://gist.github.com/kmike/7813450) script to measure memory and speed of `fit` and `dump`, and [this](https://gist.github.com/kmike/7815149) script to measure parameters of `load` and `transform`.\r\n\r\nThe results are quite cool: for '20 newsgroups' data (fit on training subset, transform on test subset) memory consumption of different vectorizers after loading is the following:\r\n\r\n*  `CountVectorizer()`: 94MB;\r\n*  `CountVectorizer(ngram_range=(1,2)`: 666MB;\r\n*  `MarisaCountVectorizer()`: 1.2MB;\r\n*  `MarisaCountVectorizer(ngram_range=(1,2))`: 13.3MB;\r\n\r\nSo using a good succinct Trie implementation gives us 50x-80x reduction of memory usage; it also makes serializing and deserializing almost instant. And such vectorizers don't have HashingVectorizer limitations.\r\n\r\nThe downside is that `fit` method is 2x-3x slower and requires slightly more memory (but I think it could be changed to require less memory than CountVectorizer.fit), and `transform` is about 2x slower. Full output: https://gist.github.com/kmike/7815156\r\n\r\nWhat do you think about adding something similar to scikit-learn? \r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/251773861","html_url":"https://github.com/scikit-learn/scikit-learn/issues/2639#issuecomment-251773861","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2639","id":251773861,"user":{"login":"amueller","id":449558,"avatar_url":"https://avatars.githubusercontent.com/u/449558?v=3","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"created_at":"2016-10-05T19:26:46Z","updated_at":"2016-10-05T19:26:46Z","body":"Should we summarize and close and make a new issue? This one got pretty carried away lol"}}
{"actor":{"display_login":"amueller"}}