{"action":"opened","issue":{"url":"https://api.github.com/repos/bmabey/pyLDAvis/issues/70","repository_url":"https://api.github.com/repos/bmabey/pyLDAvis","labels_url":"https://api.github.com/repos/bmabey/pyLDAvis/issues/70/labels{/name}","comments_url":"https://api.github.com/repos/bmabey/pyLDAvis/issues/70/comments","events_url":"https://api.github.com/repos/bmabey/pyLDAvis/issues/70/events","html_url":"https://github.com/bmabey/pyLDAvis/issues/70","id":173405453,"number":70,"title":"Visualizing topics without corpus","user":{"login":"nguyenhoan1988","id":2057220,"avatar_url":"https://avatars.githubusercontent.com/u/2057220?v=3","gravatar_id":"","url":"https://api.github.com/users/nguyenhoan1988","html_url":"https://github.com/nguyenhoan1988","followers_url":"https://api.github.com/users/nguyenhoan1988/followers","following_url":"https://api.github.com/users/nguyenhoan1988/following{/other_user}","gists_url":"https://api.github.com/users/nguyenhoan1988/gists{/gist_id}","starred_url":"https://api.github.com/users/nguyenhoan1988/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nguyenhoan1988/subscriptions","organizations_url":"https://api.github.com/users/nguyenhoan1988/orgs","repos_url":"https://api.github.com/users/nguyenhoan1988/repos","events_url":"https://api.github.com/users/nguyenhoan1988/events{/privacy}","received_events_url":"https://api.github.com/users/nguyenhoan1988/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-26T09:07:47Z","updated_at":"2016-08-26T09:07:47Z","closed_at":null,"body":"I train a `gensim` LDA model with a very large corpus (almost 100M documents), and I train it in the **streaming fashion**. Therefore, I cannot reproduce the corpus that is needed for the `prepare` method of `pyLDAvis`.\r\n\r\nBut the `corpus` is needed in this method. I try to fake a small corpus with a sample of the training data but it does not work, and give me the following exception:\r\n```\r\nanaconda/lib/python2.7/site-packages/pyLDAvis/gensim.pyc in _extract_data(topic_model, corpus, dictionary, doc_topic_dists)\r\n     26    beta = 0.01\r\n     27    fnames_argsort = np.asarray(list(dictionary.token2id.values()), dtype=np.int_)\r\n---> 28    term_freqs = corpus_csc.sum(axis=1).A.ravel()[fnames_argsort]\r\n     29    term_freqs[term_freqs == 0] = beta\r\n     30    doc_lengths = corpus_csc.sum(axis=0).A.ravel()\r\n\r\nIndexError: index 66999 is out of bounds for axis 1 with size 66988\r\n```\r\n\r\nIs there any methods to work around this problem?"}}
{"actor":{"display_login":"nguyenhoan1988"}}