{"action":"created","issue":{"url":"https://api.github.com/repos/timvieira/lazygrad/issues/1","repository_url":"https://api.github.com/repos/timvieira/lazygrad","labels_url":"https://api.github.com/repos/timvieira/lazygrad/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/timvieira/lazygrad/issues/1/comments","events_url":"https://api.github.com/repos/timvieira/lazygrad/issues/1/events","html_url":"https://github.com/timvieira/lazygrad/issues/1","id":185765455,"number":1,"title":"Tip - how to parallelise without using locking","user":{"login":"jkkummerfeld","id":1298052,"avatar_url":"https://avatars.githubusercontent.com/u/1298052?v=3","gravatar_id":"","url":"https://api.github.com/users/jkkummerfeld","html_url":"https://github.com/jkkummerfeld","followers_url":"https://api.github.com/users/jkkummerfeld/followers","following_url":"https://api.github.com/users/jkkummerfeld/following{/other_user}","gists_url":"https://api.github.com/users/jkkummerfeld/gists{/gist_id}","starred_url":"https://api.github.com/users/jkkummerfeld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkkummerfeld/subscriptions","organizations_url":"https://api.github.com/users/jkkummerfeld/orgs","repos_url":"https://api.github.com/users/jkkummerfeld/repos","events_url":"https://api.github.com/users/jkkummerfeld/events{/privacy}","received_events_url":"https://api.github.com/users/jkkummerfeld/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-27T19:58:58Z","updated_at":"2016-10-27T21:08:14Z","closed_at":null,"body":"Hi,\r\nI came across this after reading your EMNLP paper and looking for the related code (vocrf seems to be private?). Thought you might be interested in a slight tweak that I added when I wrote about this in my thesis (https://www2.eecs.berkeley.edu/Pubs/TechRpts/2016/EECS-2016-138.pdf page 87 / 101 counting front matter). Essentially, I modified the update to be able to do multi-threaded lockless processing of batches, at the cost of potentially running part of inference with stale weights (very rarely though, so shouldn't be a big deal). The idea is very similar to Hogwild.\r\n\r\nThinking about this again, even that risk could be avoided by reading the variables in the correct order... hmmm, I can't amend my thesis, but I might write this up and put a note on my website :)\r\n\r\nJonathan"},"comment":{"url":"https://api.github.com/repos/timvieira/lazygrad/issues/comments/256770258","html_url":"https://github.com/timvieira/lazygrad/issues/1#issuecomment-256770258","issue_url":"https://api.github.com/repos/timvieira/lazygrad/issues/1","id":256770258,"user":{"login":"timvieira","id":219898,"avatar_url":"https://avatars.githubusercontent.com/u/219898?v=3","gravatar_id":"","url":"https://api.github.com/users/timvieira","html_url":"https://github.com/timvieira","followers_url":"https://api.github.com/users/timvieira/followers","following_url":"https://api.github.com/users/timvieira/following{/other_user}","gists_url":"https://api.github.com/users/timvieira/gists{/gist_id}","starred_url":"https://api.github.com/users/timvieira/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timvieira/subscriptions","organizations_url":"https://api.github.com/users/timvieira/orgs","repos_url":"https://api.github.com/users/timvieira/repos","events_url":"https://api.github.com/users/timvieira/events{/privacy}","received_events_url":"https://api.github.com/users/timvieira/received_events","type":"User","site_admin":false},"created_at":"2016-10-27T21:08:14Z","updated_at":"2016-10-27T21:08:14Z","body":"The VoCRF code isn't public yet - sorry about the stale link. I'll have it up these week. \r\n\r\nI've run into the stale weight problem before. Actually, you run into it *without* parallelism! In my case, I was avoiding allocating a sparse gradient vector (for efficiency) by directly updating the lazy weights as I run the backward pass to compute gradients, but this results slightly wrong updates.\r\n\r\nThanks for the pointer! I'd gladly accept a patch, BTW :-)\r\n\r\nI'm not too concerned with parallelism since people rarely do parallel stuff in Python thanks to the GIL. The stale weight problem that I mentioned above, does concern me, however. A similar trick to the one you can probably fix that one too.\r\n\r\nWill you be at EMNLP?"}}
{"actor":{"display_login":"timvieira"}}