{"action":"created","issue":{"url":"https://api.github.com/repos/piskvorky/gensim/issues/646","repository_url":"https://api.github.com/repos/piskvorky/gensim","labels_url":"https://api.github.com/repos/piskvorky/gensim/issues/646/labels{/name}","comments_url":"https://api.github.com/repos/piskvorky/gensim/issues/646/comments","events_url":"https://api.github.com/repos/piskvorky/gensim/issues/646/events","html_url":"https://github.com/piskvorky/gensim/issues/646","id":144269308,"number":646,"title":"SystemError when using ldamulticore on big corpus","user":{"login":"hejunqing","id":5152663,"avatar_url":"https://avatars.githubusercontent.com/u/5152663?v=3","gravatar_id":"","url":"https://api.github.com/users/hejunqing","html_url":"https://github.com/hejunqing","followers_url":"https://api.github.com/users/hejunqing/followers","following_url":"https://api.github.com/users/hejunqing/following{/other_user}","gists_url":"https://api.github.com/users/hejunqing/gists{/gist_id}","starred_url":"https://api.github.com/users/hejunqing/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hejunqing/subscriptions","organizations_url":"https://api.github.com/users/hejunqing/orgs","repos_url":"https://api.github.com/users/hejunqing/repos","events_url":"https://api.github.com/users/hejunqing/events{/privacy}","received_events_url":"https://api.github.com/users/hejunqing/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/piskvorky/gensim/labels/bug","name":"bug","color":"e10c02"}],"state":"closed","locked":false,"assignee":{"login":"tmylk","id":1261528,"avatar_url":"https://avatars.githubusercontent.com/u/1261528?v=3","gravatar_id":"","url":"https://api.github.com/users/tmylk","html_url":"https://github.com/tmylk","followers_url":"https://api.github.com/users/tmylk/followers","following_url":"https://api.github.com/users/tmylk/following{/other_user}","gists_url":"https://api.github.com/users/tmylk/gists{/gist_id}","starred_url":"https://api.github.com/users/tmylk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmylk/subscriptions","organizations_url":"https://api.github.com/users/tmylk/orgs","repos_url":"https://api.github.com/users/tmylk/repos","events_url":"https://api.github.com/users/tmylk/events{/privacy}","received_events_url":"https://api.github.com/users/tmylk/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2016-03-29T13:44:38Z","updated_at":"2016-03-31T03:37:03Z","closed_at":"2016-03-31T03:37:03Z","body":"I was running ldamulticore with default settings on a big corpus,and come up with following error:\r\nTraceback (most recent call last):\r\n  File \"/home/nlip/anaconda2/lib/python2.7/multiprocessing/queues.py\", line 268, in _feed\r\n    send(obj)\r\nSystemError: NULL result without error in PyObject_Call\r\n\r\nThe corpus was 533M,containing 5837527 documents,1037945 features, 77704086 non-zero entries.I have tried serializing it to MmCorpus and Bzip it,but doesn't work. When I take only 1000000 document, ldamulticore runs fine.\r\n\r\nIf it is a bug ,please fix it as soon as possible and inform me .If not,please tell me how to fix it."},"comment":{"url":"https://api.github.com/repos/piskvorky/gensim/issues/comments/203737284","html_url":"https://github.com/piskvorky/gensim/issues/646#issuecomment-203737284","issue_url":"https://api.github.com/repos/piskvorky/gensim/issues/646","id":203737284,"user":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"created_at":"2016-03-31T03:37:03Z","updated_at":"2016-03-31T03:37:03Z","body":"I don't think you need the ShardedCorpus. Instead, try trimming your vocabulary by calling `filter_extremes()`: http://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary.filter_extremes\r\n\r\n1.1 million features sounds excessive."}}