{"action":"created","comment":{"url":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/comments/195911837","pull_request_review_id":129372620,"id":195911837,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5NTkxMTgzNw==","diff_hunk":"@@ -459,25 +469,34 @@ def __str__(self):\n             self.__class__.__name__, len(self.wv.index2word), self.vector_size, self.alpha\n         )\n \n-    def build_vocab(self, sentences, update=False, progress_per=10000, keep_raw_vocab=False, trim_rule=None, **kwargs):\n+    def build_vocab(self, sentences, multistream=False, workers=None, update=False, progress_per=10000,\n+                    keep_raw_vocab=False, trim_rule=None, **kwargs):\n         \"\"\"Build vocabulary from a sequence of sentences (can be a once-only generator stream).\n         Each sentence is a iterable of iterables (can simply be a list of unicode strings too).\n \n         Parameters\n         ----------\n-        sentences : iterable of iterables\n+        sentences : {iterable of iterables, list or tuple of iterable of iterables}","path":"gensim/models/base_any2vec.py","position":94,"original_position":94,"commit_id":"74e7b022f0ae621916f92fabd8171e16cdb4c5aa","original_commit_id":"74e7b022f0ae621916f92fabd8171e16cdb4c5aa","user":{"login":"piskvorky","id":610412,"node_id":"MDQ6VXNlcjYxMDQxMg==","avatar_url":"https://avatars1.githubusercontent.com/u/610412?v=4","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"body":"Well, we'd have to put the \"peeked\" items back of course. We cannot just discard them :)\r\n\r\nWe're already doing something similar in `utils.is_corpus()`.\r\n\r\nQuestion is, is it possible to do something similar here as well? What is the risk of unintended consequences, such as accidentally materializing some huge corpus?\r\n\r\nBecause it is a bit hackish, so the tradeoff (pros and cons) between having an extra explicit parameter vs being simpler for user is not clearcut to me.","created_at":"2018-06-16T20:52:19Z","updated_at":"2018-06-16T20:52:19Z","html_url":"https://github.com/RaRe-Technologies/gensim/pull/2078#discussion_r195911837","pull_request_url":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/2078","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/comments/195911837"},"html":{"href":"https://github.com/RaRe-Technologies/gensim/pull/2078#discussion_r195911837"},"pull_request":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/2078"}},"in_reply_to_id":195749836},"pull_request":{"url":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/2078","id":192426069,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkyNDI2MDY5","html_url":"https://github.com/RaRe-Technologies/gensim/pull/2078","diff_url":"https://github.com/RaRe-Technologies/gensim/pull/2078.diff","patch_url":"https://github.com/RaRe-Technologies/gensim/pull/2078.patch","issue_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2078","number":2078,"state":"open","locked":false,"title":"[GSoC 2018] Multistream API for vocabulary building in *2vec","user":{"login":"persiyanov","id":3997997,"node_id":"MDQ6VXNlcjM5OTc5OTc=","avatar_url":"https://avatars1.githubusercontent.com/u/3997997?v=4","gravatar_id":"","url":"https://api.github.com/users/persiyanov","html_url":"https://github.com/persiyanov","followers_url":"https://api.github.com/users/persiyanov/followers","following_url":"https://api.github.com/users/persiyanov/following{/other_user}","gists_url":"https://api.github.com/users/persiyanov/gists{/gist_id}","starred_url":"https://api.github.com/users/persiyanov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/persiyanov/subscriptions","organizations_url":"https://api.github.com/users/persiyanov/orgs","repos_url":"https://api.github.com/users/persiyanov/repos","events_url":"https://api.github.com/users/persiyanov/events{/privacy}","received_events_url":"https://api.github.com/users/persiyanov/received_events","type":"User","site_admin":false},"body":"This PR introduces __multistream API__ for building vocabularies in __*2Vec models__. \r\n\r\n# Main idea\r\nTraining any2vec model involves two main steps:\r\n1. Passing over your data in order to build vocabulary __(x)__\r\n2. Training the model\r\n\r\nThis pull request optimizes the __first step__.\r\n\r\nThe main idea is to parallelize `scan_vocab` method, by allowing users to pass _several streams of data_ (i.e. several `LineSentence` iterators for different files). Under the hood, the implementation uses `multiprocessing` python module creating `multiprocessing.Pool(processes=min(workers, len(input_streams)))` and __reading all input streams in parallel__.\r\n\r\n# How to use it\r\n\r\nIt's really simple to start using multistream mode. Only you need is to have multiple files you want to train on and passing additional flag `multistream=True` in your code:\r\n\r\n### Current singlestream approaches to train, i.e. word2vec model\r\n\r\n1. Train by construction\r\n\r\n```python\r\nsentences = LineSentence('myfile.txt')\r\nmodel = Word2Vec(sentences, workers=10, ...other params...)\r\n```\r\n\r\n2. Construct without training, then build vocabulary and train\r\n\r\n```python\r\nsentences = LineSentence('myfile.txt')\r\nmodel = Word2Vec(workers=10, ...other params...)\r\nmodel.build_vocab(sentences)\r\nmodel.train(sentences, ...train params...)\r\n```\r\n\r\n### New multistream mode\r\n\r\n1. Train by construction (`build_vocab` is done in a parallel way)\r\n\r\n```python\r\ninput_streams = [LineSentence('myfile1.txt'), LineSentence('myfile2.txt'), ...]\r\nmodel = Word2Vec(input_streams, multistream=True, workers=10, ...other params...)\r\n```\r\n\r\n2. Construct without training, then build vocabulary (in a parallel way) and train\r\n\r\n```python\r\ninput_streams = [LineSentence('myfile1.txt'), LineSentence('myfile2.txt'), ...]\r\nmodel = Word2Vec(workers=10, ...other params...)\r\nmodel.build_vocab(input_streams, multistream=True)  # Optionally, you can pass `workers` param here which differs from one passed in constructor.\r\nmodel.train(input_streams, multistream=True, ...train params...)\r\n```\r\n\r\nMostly the same changes are required for `Doc2Vec` and `FastText` models.\r\n\r\n# Why to use it?\r\n\r\nThis feature could substantially reduce vocabulary building time, especially if you have complex data iterators (more CPU-bound, i.e. reading a line from a file, then do complex preprocessing, removing stop words, lemmatizing, stemming etc).\r\n\r\nOn benchmarks with simplest IO-iterators __multistream mode__ produces up to __3x boost__ on building vocabulary step. In its turn, building vocabulary could take up to several hours (try to build vocabulary from smth like this https://dumps.wikimedia.org/enwiki/20180320/ :) )\r\n\r\nLikely, the more CPU-complex data iterators you use, the more speedup you will get.","created_at":"2018-06-04T14:26:39Z","updated_at":"2018-06-16T20:52:19Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b20f388c3c2d5ec3f75e98fbfea58e6897b81d94","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/2078/commits","review_comments_url":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/2078/comments","review_comment_url":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/comments{/number}","comments_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2078/comments","statuses_url":"https://api.github.com/repos/RaRe-Technologies/gensim/statuses/74e7b022f0ae621916f92fabd8171e16cdb4c5aa","head":{"label":"persiyanov:feature/gsoc-multistream-vocab","ref":"feature/gsoc-multistream-vocab","sha":"74e7b022f0ae621916f92fabd8171e16cdb4c5aa","user":{"login":"persiyanov","id":3997997,"node_id":"MDQ6VXNlcjM5OTc5OTc=","avatar_url":"https://avatars1.githubusercontent.com/u/3997997?v=4","gravatar_id":"","url":"https://api.github.com/users/persiyanov","html_url":"https://github.com/persiyanov","followers_url":"https://api.github.com/users/persiyanov/followers","following_url":"https://api.github.com/users/persiyanov/following{/other_user}","gists_url":"https://api.github.com/users/persiyanov/gists{/gist_id}","starred_url":"https://api.github.com/users/persiyanov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/persiyanov/subscriptions","organizations_url":"https://api.github.com/users/persiyanov/orgs","repos_url":"https://api.github.com/users/persiyanov/repos","events_url":"https://api.github.com/users/persiyanov/events{/privacy}","received_events_url":"https://api.github.com/users/persiyanov/received_events","type":"User","site_admin":false},"repo":{"id":123994588,"node_id":"MDEwOlJlcG9zaXRvcnkxMjM5OTQ1ODg=","name":"gensim","full_name":"persiyanov/gensim","owner":{"login":"persiyanov","id":3997997,"node_id":"MDQ6VXNlcjM5OTc5OTc=","avatar_url":"https://avatars1.githubusercontent.com/u/3997997?v=4","gravatar_id":"","url":"https://api.github.com/users/persiyanov","html_url":"https://github.com/persiyanov","followers_url":"https://api.github.com/users/persiyanov/followers","following_url":"https://api.github.com/users/persiyanov/following{/other_user}","gists_url":"https://api.github.com/users/persiyanov/gists{/gist_id}","starred_url":"https://api.github.com/users/persiyanov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/persiyanov/subscriptions","organizations_url":"https://api.github.com/users/persiyanov/orgs","repos_url":"https://api.github.com/users/persiyanov/repos","events_url":"https://api.github.com/users/persiyanov/events{/privacy}","received_events_url":"https://api.github.com/users/persiyanov/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/persiyanov/gensim","description":"Topic Modelling for Humans","fork":true,"url":"https://api.github.com/repos/persiyanov/gensim","forks_url":"https://api.github.com/repos/persiyanov/gensim/forks","keys_url":"https://api.github.com/repos/persiyanov/gensim/keys{/key_id}","collaborators_url":"https://api.github.com/repos/persiyanov/gensim/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/persiyanov/gensim/teams","hooks_url":"https://api.github.com/repos/persiyanov/gensim/hooks","issue_events_url":"https://api.github.com/repos/persiyanov/gensim/issues/events{/number}","events_url":"https://api.github.com/repos/persiyanov/gensim/events","assignees_url":"https://api.github.com/repos/persiyanov/gensim/assignees{/user}","branches_url":"https://api.github.com/repos/persiyanov/gensim/branches{/branch}","tags_url":"https://api.github.com/repos/persiyanov/gensim/tags","blobs_url":"https://api.github.com/repos/persiyanov/gensim/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/persiyanov/gensim/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/persiyanov/gensim/git/refs{/sha}","trees_url":"https://api.github.com/repos/persiyanov/gensim/git/trees{/sha}","statuses_url":"https://api.github.com/repos/persiyanov/gensim/statuses/{sha}","languages_url":"https://api.github.com/repos/persiyanov/gensim/languages","stargazers_url":"https://api.github.com/repos/persiyanov/gensim/stargazers","contributors_url":"https://api.github.com/repos/persiyanov/gensim/contributors","subscribers_url":"https://api.github.com/repos/persiyanov/gensim/subscribers","subscription_url":"https://api.github.com/repos/persiyanov/gensim/subscription","commits_url":"https://api.github.com/repos/persiyanov/gensim/commits{/sha}","git_commits_url":"https://api.github.com/repos/persiyanov/gensim/git/commits{/sha}","comments_url":"https://api.github.com/repos/persiyanov/gensim/comments{/number}","issue_comment_url":"https://api.github.com/repos/persiyanov/gensim/issues/comments{/number}","contents_url":"https://api.github.com/repos/persiyanov/gensim/contents/{+path}","compare_url":"https://api.github.com/repos/persiyanov/gensim/compare/{base}...{head}","merges_url":"https://api.github.com/repos/persiyanov/gensim/merges","archive_url":"https://api.github.com/repos/persiyanov/gensim/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/persiyanov/gensim/downloads","issues_url":"https://api.github.com/repos/persiyanov/gensim/issues{/number}","pulls_url":"https://api.github.com/repos/persiyanov/gensim/pulls{/number}","milestones_url":"https://api.github.com/repos/persiyanov/gensim/milestones{/number}","notifications_url":"https://api.github.com/repos/persiyanov/gensim/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/persiyanov/gensim/labels{/name}","releases_url":"https://api.github.com/repos/persiyanov/gensim/releases{/id}","deployments_url":"https://api.github.com/repos/persiyanov/gensim/deployments","created_at":"2018-03-05T23:59:24Z","updated_at":"2018-06-07T10:54:13Z","pushed_at":"2018-06-15T14:41:46Z","git_url":"git://github.com/persiyanov/gensim.git","ssh_url":"git@github.com:persiyanov/gensim.git","clone_url":"https://github.com/persiyanov/gensim.git","svn_url":"https://github.com/persiyanov/gensim","homepage":"","size":63164,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"lgpl-2.1","name":"GNU Lesser General Public License v2.1","spdx_id":"LGPL-2.1","url":"https://api.github.com/licenses/lgpl-2.1","node_id":"MDc6TGljZW5zZTEx"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"RaRe-Technologies:develop","ref":"develop","sha":"700a1be6eec406335d99b3c6fdb3f750e5cf2371","user":{"login":"RaRe-Technologies","id":12515886,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEyNTE1ODg2","avatar_url":"https://avatars3.githubusercontent.com/u/12515886?v=4","gravatar_id":"","url":"https://api.github.com/users/RaRe-Technologies","html_url":"https://github.com/RaRe-Technologies","followers_url":"https://api.github.com/users/RaRe-Technologies/followers","following_url":"https://api.github.com/users/RaRe-Technologies/following{/other_user}","gists_url":"https://api.github.com/users/RaRe-Technologies/gists{/gist_id}","starred_url":"https://api.github.com/users/RaRe-Technologies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RaRe-Technologies/subscriptions","organizations_url":"https://api.github.com/users/RaRe-Technologies/orgs","repos_url":"https://api.github.com/users/RaRe-Technologies/repos","events_url":"https://api.github.com/users/RaRe-Technologies/events{/privacy}","received_events_url":"https://api.github.com/users/RaRe-Technologies/received_events","type":"Organization","site_admin":false},"repo":{"id":1349775,"node_id":"MDEwOlJlcG9zaXRvcnkxMzQ5Nzc1","name":"gensim","full_name":"RaRe-Technologies/gensim","owner":{"login":"RaRe-Technologies","id":12515886,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEyNTE1ODg2","avatar_url":"https://avatars3.githubusercontent.com/u/12515886?v=4","gravatar_id":"","url":"https://api.github.com/users/RaRe-Technologies","html_url":"https://github.com/RaRe-Technologies","followers_url":"https://api.github.com/users/RaRe-Technologies/followers","following_url":"https://api.github.com/users/RaRe-Technologies/following{/other_user}","gists_url":"https://api.github.com/users/RaRe-Technologies/gists{/gist_id}","starred_url":"https://api.github.com/users/RaRe-Technologies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RaRe-Technologies/subscriptions","organizations_url":"https://api.github.com/users/RaRe-Technologies/orgs","repos_url":"https://api.github.com/users/RaRe-Technologies/repos","events_url":"https://api.github.com/users/RaRe-Technologies/events{/privacy}","received_events_url":"https://api.github.com/users/RaRe-Technologies/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/RaRe-Technologies/gensim","description":"Topic Modelling for Humans","fork":false,"url":"https://api.github.com/repos/RaRe-Technologies/gensim","forks_url":"https://api.github.com/repos/RaRe-Technologies/gensim/forks","keys_url":"https://api.github.com/repos/RaRe-Technologies/gensim/keys{/key_id}","collaborators_url":"https://api.github.com/repos/RaRe-Technologies/gensim/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/RaRe-Technologies/gensim/teams","hooks_url":"https://api.github.com/repos/RaRe-Technologies/gensim/hooks","issue_events_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/events{/number}","events_url":"https://api.github.com/repos/RaRe-Technologies/gensim/events","assignees_url":"https://api.github.com/repos/RaRe-Technologies/gensim/assignees{/user}","branches_url":"https://api.github.com/repos/RaRe-Technologies/gensim/branches{/branch}","tags_url":"https://api.github.com/repos/RaRe-Technologies/gensim/tags","blobs_url":"https://api.github.com/repos/RaRe-Technologies/gensim/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/RaRe-Technologies/gensim/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/RaRe-Technologies/gensim/git/refs{/sha}","trees_url":"https://api.github.com/repos/RaRe-Technologies/gensim/git/trees{/sha}","statuses_url":"https://api.github.com/repos/RaRe-Technologies/gensim/statuses/{sha}","languages_url":"https://api.github.com/repos/RaRe-Technologies/gensim/languages","stargazers_url":"https://api.github.com/repos/RaRe-Technologies/gensim/stargazers","contributors_url":"https://api.github.com/repos/RaRe-Technologies/gensim/contributors","subscribers_url":"https://api.github.com/repos/RaRe-Technologies/gensim/subscribers","subscription_url":"https://api.github.com/repos/RaRe-Technologies/gensim/subscription","commits_url":"https://api.github.com/repos/RaRe-Technologies/gensim/commits{/sha}","git_commits_url":"https://api.github.com/repos/RaRe-Technologies/gensim/git/commits{/sha}","comments_url":"https://api.github.com/repos/RaRe-Technologies/gensim/comments{/number}","issue_comment_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/comments{/number}","contents_url":"https://api.github.com/repos/RaRe-Technologies/gensim/contents/{+path}","compare_url":"https://api.github.com/repos/RaRe-Technologies/gensim/compare/{base}...{head}","merges_url":"https://api.github.com/repos/RaRe-Technologies/gensim/merges","archive_url":"https://api.github.com/repos/RaRe-Technologies/gensim/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/RaRe-Technologies/gensim/downloads","issues_url":"https://api.github.com/repos/RaRe-Technologies/gensim/issues{/number}","pulls_url":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls{/number}","milestones_url":"https://api.github.com/repos/RaRe-Technologies/gensim/milestones{/number}","notifications_url":"https://api.github.com/repos/RaRe-Technologies/gensim/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/RaRe-Technologies/gensim/labels{/name}","releases_url":"https://api.github.com/repos/RaRe-Technologies/gensim/releases{/id}","deployments_url":"https://api.github.com/repos/RaRe-Technologies/gensim/deployments","created_at":"2011-02-10T07:43:04Z","updated_at":"2018-06-16T13:35:17Z","pushed_at":"2018-06-16T20:36:05Z","git_url":"git://github.com/RaRe-Technologies/gensim.git","ssh_url":"git@github.com:RaRe-Technologies/gensim.git","clone_url":"https://github.com/RaRe-Technologies/gensim.git","svn_url":"https://github.com/RaRe-Technologies/gensim","homepage":"","size":63271,"stargazers_count":7176,"watchers_count":7176,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2755,"mirror_url":null,"archived":false,"open_issues_count":203,"license":{"key":"lgpl-2.1","name":"GNU Lesser General Public License v2.1","spdx_id":"LGPL-2.1","url":"https://api.github.com/licenses/lgpl-2.1","node_id":"MDc6TGljZW5zZTEx"},"forks":2755,"open_issues":203,"watchers":7176,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/2078"},"html":{"href":"https://github.com/RaRe-Technologies/gensim/pull/2078"},"issue":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2078"},"comments":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/issues/2078/comments"},"review_comments":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/2078/comments"},"review_comment":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/pulls/2078/commits"},"statuses":{"href":"https://api.github.com/repos/RaRe-Technologies/gensim/statuses/74e7b022f0ae621916f92fabd8171e16cdb4c5aa"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"piskvorky"}}