{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/1827","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/1827/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/1827/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/1827/events","html_url":"https://github.com/dmlc/mxnet/issues/1827","id":147599268,"number":1827,"title":"A question about update_params function in multi gpu mode","user":{"login":"ascust","id":12552820,"avatar_url":"https://avatars.githubusercontent.com/u/12552820?v=3","gravatar_id":"","url":"https://api.github.com/users/ascust","html_url":"https://github.com/ascust","followers_url":"https://api.github.com/users/ascust/followers","following_url":"https://api.github.com/users/ascust/following{/other_user}","gists_url":"https://api.github.com/users/ascust/gists{/gist_id}","starred_url":"https://api.github.com/users/ascust/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ascust/subscriptions","organizations_url":"https://api.github.com/users/ascust/orgs","repos_url":"https://api.github.com/users/ascust/repos","events_url":"https://api.github.com/users/ascust/events{/privacy}","received_events_url":"https://api.github.com/users/ascust/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-04-12T00:46:32Z","updated_at":"2016-04-12T08:28:46Z","closed_at":null,"body":"I saw in model.py, the _update_params function is like this:\r\n```\r\nfor k, p in enumerate(zip(arg_list, grad_list)):\r\n            # faked an index here, to make optimizer create diff\r\n            # state for the same index but on diff devs, TODO(mli)\r\n            # use a better solution latter\r\n            w, g = p\r\n            updater(index*num_device+k, g, w)\r\n```\r\nFor example if I have two gpus, and I want to update a weight based on gradients on two devices. In this way, do it mean it will run the update twice? including updating moment and other states. What if I sum the gradients up first and update the weight once using the result. Be specific, I copy the gradient from the other device and do the summation on one device, which applies to more gpus. Is this better? @tqchen @piiswrong "},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/208781049","html_url":"https://github.com/dmlc/mxnet/issues/1827#issuecomment-208781049","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/1827","id":208781049,"user":{"login":"Godricly","id":17567935,"avatar_url":"https://avatars.githubusercontent.com/u/17567935?v=3","gravatar_id":"","url":"https://api.github.com/users/Godricly","html_url":"https://github.com/Godricly","followers_url":"https://api.github.com/users/Godricly/followers","following_url":"https://api.github.com/users/Godricly/following{/other_user}","gists_url":"https://api.github.com/users/Godricly/gists{/gist_id}","starred_url":"https://api.github.com/users/Godricly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Godricly/subscriptions","organizations_url":"https://api.github.com/users/Godricly/orgs","repos_url":"https://api.github.com/users/Godricly/repos","events_url":"https://api.github.com/users/Godricly/events{/privacy}","received_events_url":"https://api.github.com/users/Godricly/received_events","type":"User","site_admin":false},"created_at":"2016-04-12T08:28:46Z","updated_at":"2016-04-12T08:28:46Z","body":"I think this function is only called when you have 1gpu on 1 machine. Try printing sth in your experiment should be helpful. "}}