{"action":"created","issue":{"url":"https://api.github.com/repos/twitter/torch-distlearn/issues/7","repository_url":"https://api.github.com/repos/twitter/torch-distlearn","labels_url":"https://api.github.com/repos/twitter/torch-distlearn/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/twitter/torch-distlearn/issues/7/comments","events_url":"https://api.github.com/repos/twitter/torch-distlearn/issues/7/events","html_url":"https://github.com/twitter/torch-distlearn/issues/7","id":138311463,"number":7,"title":"Example using optim?","user":{"login":"willfrey","id":13784361,"avatar_url":"https://avatars.githubusercontent.com/u/13784361?v=3","gravatar_id":"","url":"https://api.github.com/users/willfrey","html_url":"https://github.com/willfrey","followers_url":"https://api.github.com/users/willfrey/followers","following_url":"https://api.github.com/users/willfrey/following{/other_user}","gists_url":"https://api.github.com/users/willfrey/gists{/gist_id}","starred_url":"https://api.github.com/users/willfrey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/willfrey/subscriptions","organizations_url":"https://api.github.com/users/willfrey/orgs","repos_url":"https://api.github.com/users/willfrey/repos","events_url":"https://api.github.com/users/willfrey/events{/privacy}","received_events_url":"https://api.github.com/users/willfrey/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-03T22:07:14Z","updated_at":"2016-03-04T15:05:39Z","closed_at":null,"body":"Can you provide an example of how to use AllReduceSGD with the optim package?\r\n\r\nI'm confused about whether or not I should normalize gradParameters like they do in [this example.]https://github.com/torch/tutorials/blob/master/2_supervised/4_train.lua) The behavior of AllReduceSGD.sumAndNormalizeGradients isn't clear to me, specifically whether (1) it sums and normalizes the accumulated gradient across all nodes or (2) if it takes the batch size given to the torch-dataset package into account.\r\n\r\nI'm curious about the last part because you normalize the batch size, per node in your cifar example.\r\n\r\nThanks."},"comment":{"url":"https://api.github.com/repos/twitter/torch-distlearn/issues/comments/192314237","html_url":"https://github.com/twitter/torch-distlearn/issues/7#issuecomment-192314237","issue_url":"https://api.github.com/repos/twitter/torch-distlearn/issues/7","id":192314237,"user":{"login":"zakattacktwitter","id":4247897,"avatar_url":"https://avatars.githubusercontent.com/u/4247897?v=3","gravatar_id":"","url":"https://api.github.com/users/zakattacktwitter","html_url":"https://github.com/zakattacktwitter","followers_url":"https://api.github.com/users/zakattacktwitter/followers","following_url":"https://api.github.com/users/zakattacktwitter/following{/other_user}","gists_url":"https://api.github.com/users/zakattacktwitter/gists{/gist_id}","starred_url":"https://api.github.com/users/zakattacktwitter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zakattacktwitter/subscriptions","organizations_url":"https://api.github.com/users/zakattacktwitter/orgs","repos_url":"https://api.github.com/users/zakattacktwitter/repos","events_url":"https://api.github.com/users/zakattacktwitter/events{/privacy}","received_events_url":"https://api.github.com/users/zakattacktwitter/received_events","type":"User","site_admin":false},"created_at":"2016-03-04T15:05:38Z","updated_at":"2016-03-04T15:05:38Z","body":"Hi,\r\n\r\nWe just wrote our own quick version of SGD for the cifar10 example. This block of code is equivalent to calling optim.SGD (https://github.com/twitter/torch-distlearn/blob/master/examples/cifar10.lua#L159). The most basic version of optim.SGD is just a multiply and add.\r\n\r\nThe batch normalization occurs in AllReduceSGD.sumAndNormalizeGradients as you suspected. This is equivalent to what you see in the torch example here (https://github.com/torch/tutorials/blob/master/2_supervised/4_train.lua#L173).\r\n\r\nThe reason we do this inside the AllReduceSGD code is because not all nodes in the training set will have the same number of steps every time. For example if you have 11 items to train on, your batch size is 4 and you train on 2 nodes. You would see 3 iterations thru the loop with batch sizes of 4, 4 and 3. The last batch of 3 would have 2 items from the first node and 1 from the second node.\r\n\r\nWe hide this corner case and handle it for you. Hope this helps!\r\n\r\n\r\n"}}