{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/7365","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/7365/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/7365/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/7365/events","html_url":"https://github.com/pytorch/pytorch/issues/7365","id":321032193,"number":7365,"title":"nn.DataParallel fills None grads with 0","user":{"login":"karandwivedi42","id":9624554,"avatar_url":"https://avatars1.githubusercontent.com/u/9624554?v=4","gravatar_id":"","url":"https://api.github.com/users/karandwivedi42","html_url":"https://github.com/karandwivedi42","followers_url":"https://api.github.com/users/karandwivedi42/followers","following_url":"https://api.github.com/users/karandwivedi42/following{/other_user}","gists_url":"https://api.github.com/users/karandwivedi42/gists{/gist_id}","starred_url":"https://api.github.com/users/karandwivedi42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karandwivedi42/subscriptions","organizations_url":"https://api.github.com/users/karandwivedi42/orgs","repos_url":"https://api.github.com/users/karandwivedi42/repos","events_url":"https://api.github.com/users/karandwivedi42/events{/privacy}","received_events_url":"https://api.github.com/users/karandwivedi42/received_events","type":"User","site_admin":false},"labels":[{"id":443484050,"url":"https://api.github.com/repos/pytorch/pytorch/labels/medium%20priority","name":"medium priority","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-05-08T03:42:16Z","updated_at":"2018-05-24T08:29:05Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"On pytorch 0.4.0, using nn.DataParallel leads to grads of parameters not used being filled with 0. This is a problem when the model has many parameters but only small part of it is used in forward.\r\n\r\nTest:\r\n```python\r\nimport torch\r\nfrom torch import nn\r\nclass Test(nn.Module):\r\n    def __init__(self):\r\n        super(Test, self).__init__()\r\n        self.l1 = nn.Linear(10,10)\r\n        self.l2 = nn.Linear(10,10)\r\n\r\n    def forward(self, input):\r\n        return self.l1(input)\r\n\r\nprint('CPU:')\r\nnet = Test()\r\ninputs = torch.randn(1,10)\r\noutputs = net(inputs)\r\nloss = torch.norm(outputs)\r\nloss.backward()\r\nfor j,p in net.named_parameters():\r\n    if p.grad is not None:\r\n        print(j, p.shape, torch.norm(p.grad))\r\n    else:\r\n        print(j)\r\nprint('GPU:')\r\nnet = Test().to('cuda')\r\ninputs = torch.randn(1,10).to('cuda')\r\noutputs = net(inputs)\r\nloss = torch.norm(outputs)\r\nloss.backward()\r\nfor j,p in net.named_parameters():\r\n    if p.grad is not None:\r\n        print(j, p.shape, torch.norm(p.grad))\r\n    else:\r\n        print(j)\r\nprint('MultiGPU:')\r\nnet = nn.DataParallel(Test().to('cuda'))\r\ninputs = torch.randn(1,10).to('cuda')\r\noutputs = net(inputs)\r\nloss = torch.norm(outputs)\r\nloss.backward()\r\nfor j,p in net.module.named_parameters():\r\n    if p.grad is not None:\r\n        print(j, p.shape, torch.norm(p.grad))\r\n    else:\r\n        print(j)\r\n```\r\nFor multi GPU we see all grads are filled with zeros. For CPU and GPU the grads are None."},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/391632996","html_url":"https://github.com/pytorch/pytorch/issues/7365#issuecomment-391632996","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/7365","id":391632996,"user":{"login":"apaszke","id":4583066,"avatar_url":"https://avatars3.githubusercontent.com/u/4583066?v=4","gravatar_id":"","url":"https://api.github.com/users/apaszke","html_url":"https://github.com/apaszke","followers_url":"https://api.github.com/users/apaszke/followers","following_url":"https://api.github.com/users/apaszke/following{/other_user}","gists_url":"https://api.github.com/users/apaszke/gists{/gist_id}","starred_url":"https://api.github.com/users/apaszke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apaszke/subscriptions","organizations_url":"https://api.github.com/users/apaszke/orgs","repos_url":"https://api.github.com/users/apaszke/repos","events_url":"https://api.github.com/users/apaszke/events{/privacy}","received_events_url":"https://api.github.com/users/apaszke/received_events","type":"User","site_admin":false},"created_at":"2018-05-24T08:29:05Z","updated_at":"2018-05-24T08:29:05Z","author_association":"MEMBER","body":"I don't think we can resolve this without knowing the set of parameters you'll use in `forward` upfront. The problem is in `broadcast_coalesced`, which will compact them into blocks quite arbitrarily, and as long as a single parameter gets grad, all other will get zeros."}}
{"actor":{"display_login":"apaszke"}}