{"action":"created","issue":{"url":"https://api.github.com/repos/piskvorky/gensim/issues/571","labels_url":"https://api.github.com/repos/piskvorky/gensim/issues/571/labels{/name}","comments_url":"https://api.github.com/repos/piskvorky/gensim/issues/571/comments","events_url":"https://api.github.com/repos/piskvorky/gensim/issues/571/events","html_url":"https://github.com/piskvorky/gensim/issues/571","id":124475772,"number":571,"title":"Race Condition in Muli-threaded word2vec","user":{"login":"elmar-haussmann","id":14123414,"avatar_url":"https://avatars.githubusercontent.com/u/14123414?v=3","gravatar_id":"","url":"https://api.github.com/users/elmar-haussmann","html_url":"https://github.com/elmar-haussmann","followers_url":"https://api.github.com/users/elmar-haussmann/followers","following_url":"https://api.github.com/users/elmar-haussmann/following{/other_user}","gists_url":"https://api.github.com/users/elmar-haussmann/gists{/gist_id}","starred_url":"https://api.github.com/users/elmar-haussmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elmar-haussmann/subscriptions","organizations_url":"https://api.github.com/users/elmar-haussmann/orgs","repos_url":"https://api.github.com/users/elmar-haussmann/repos","events_url":"https://api.github.com/users/elmar-haussmann/events{/privacy}","received_events_url":"https://api.github.com/users/elmar-haussmann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-12-31T14:28:12Z","updated_at":"2016-01-07T11:14:01Z","closed_at":null,"body":"The current development branch uses batch-based multi-threading. Some parts of the code are not synchronized properly and susceptible to race conditions. In particular, workers execute `self.jobs_left -= 1` and the job producer: `self.jobs_left += 1`. This is not an atomic operation. Depending on timing and release of the GIL this leads to a wrong number of jobs left. Since the termination condition for training checks whether jobs are left, training often never finishes properly.\r\n\r\nI verified this by outputting the number of jobs left at the end of training. The longer the training before, the higher the chances of a corrupted value and training not terminating.\r\n\r\nAccess should be properly synchronized (via locking) or another thread-safe data structure could be used instead."},"comment":{"url":"https://api.github.com/repos/piskvorky/gensim/issues/comments/169633145","html_url":"https://github.com/piskvorky/gensim/issues/571#issuecomment-169633145","issue_url":"https://api.github.com/repos/piskvorky/gensim/issues/571","id":169633145,"user":{"login":"piskvorky","id":610412,"avatar_url":"https://avatars.githubusercontent.com/u/610412?v=3","gravatar_id":"","url":"https://api.github.com/users/piskvorky","html_url":"https://github.com/piskvorky","followers_url":"https://api.github.com/users/piskvorky/followers","following_url":"https://api.github.com/users/piskvorky/following{/other_user}","gists_url":"https://api.github.com/users/piskvorky/gists{/gist_id}","starred_url":"https://api.github.com/users/piskvorky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piskvorky/subscriptions","organizations_url":"https://api.github.com/users/piskvorky/orgs","repos_url":"https://api.github.com/users/piskvorky/repos","events_url":"https://api.github.com/users/piskvorky/events{/privacy}","received_events_url":"https://api.github.com/users/piskvorky/received_events","type":"User","site_admin":false},"created_at":"2016-01-07T11:14:01Z","updated_at":"2016-01-07T11:14:01Z","body":"Yes, either that, or using a C counter (like in the page I linked to), making the operation atomic.\r\n\r\nIf using a lock, can you time the difference before/after the change? Say on the text9 corpus. Just as a sanity check, to make sure nothing unexpectedly horrible happened to performance.\r\n\r\nCheers!"}}