{"action":"created","issue":{"url":"https://api.github.com/repos/gpleiss/efficient_densenet_pytorch/issues/36","repository_url":"https://api.github.com/repos/gpleiss/efficient_densenet_pytorch","labels_url":"https://api.github.com/repos/gpleiss/efficient_densenet_pytorch/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/gpleiss/efficient_densenet_pytorch/issues/36/comments","events_url":"https://api.github.com/repos/gpleiss/efficient_densenet_pytorch/issues/36/events","html_url":"https://github.com/gpleiss/efficient_densenet_pytorch/issues/36","id":318963426,"number":36,"title":"`nn.DataPallel` fails for checkpoint feature","user":{"login":"wandering007","id":6906411,"avatar_url":"https://avatars3.githubusercontent.com/u/6906411?v=4","gravatar_id":"","url":"https://api.github.com/users/wandering007","html_url":"https://github.com/wandering007","followers_url":"https://api.github.com/users/wandering007/followers","following_url":"https://api.github.com/users/wandering007/following{/other_user}","gists_url":"https://api.github.com/users/wandering007/gists{/gist_id}","starred_url":"https://api.github.com/users/wandering007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wandering007/subscriptions","organizations_url":"https://api.github.com/users/wandering007/orgs","repos_url":"https://api.github.com/users/wandering007/repos","events_url":"https://api.github.com/users/wandering007/events{/privacy}","received_events_url":"https://api.github.com/users/wandering007/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-04-30T17:05:23Z","updated_at":"2018-05-14T04:29:59Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I just want to benchmark the new implementation of efficient densenet with the code [here](https://gist.github.com/wandering007/d7f3db11285f382fefa39a9cda99ad0d). However, it seems that the used checkpointed modules are not broadcast to multiple GPUs as I got the following errors:\r\n\r\n```\r\n  File \"/home/changmao/efficient_densenet_pytorch/models/densenet.py\", line 16, in bn_function\r\n    bottleneck_output = conv(relu(norm(concated_features)))\r\n  File \"/home/changmao/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 491, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/changmao/anaconda3/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py\", line 49, in forward\r\n    self.training or not self.track_running_stats, self.momentum, self.eps)\r\n  File \"/home/changmao/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py\", line 1194, in batch_norm\r\n    training, momentum, eps, torch.backends.cudnn.enabled\r\nRuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_batch_norm)\r\n```\r\nI think that the checkpoint feature provides weak support for `nn.DataParallel`."},"comment":{"url":"https://api.github.com/repos/gpleiss/efficient_densenet_pytorch/issues/comments/388694121","html_url":"https://github.com/gpleiss/efficient_densenet_pytorch/issues/36#issuecomment-388694121","issue_url":"https://api.github.com/repos/gpleiss/efficient_densenet_pytorch/issues/36","id":388694121,"user":{"login":"wandering007","id":6906411,"avatar_url":"https://avatars3.githubusercontent.com/u/6906411?v=4","gravatar_id":"","url":"https://api.github.com/users/wandering007","html_url":"https://github.com/wandering007","followers_url":"https://api.github.com/users/wandering007/followers","following_url":"https://api.github.com/users/wandering007/following{/other_user}","gists_url":"https://api.github.com/users/wandering007/gists{/gist_id}","starred_url":"https://api.github.com/users/wandering007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wandering007/subscriptions","organizations_url":"https://api.github.com/users/wandering007/orgs","repos_url":"https://api.github.com/users/wandering007/repos","events_url":"https://api.github.com/users/wandering007/events{/privacy}","received_events_url":"https://api.github.com/users/wandering007/received_events","type":"User","site_admin":false},"created_at":"2018-05-14T04:29:59Z","updated_at":"2018-05-14T04:29:59Z","author_association":"CONTRIBUTOR","body":"@gpleiss Yes, it works fine. However, there is one thing that I've noticed before and have to mention, though out of the scope of this issue. With checkpointing feature, the whole autograd computation graph is broken into pieces. And the current `nn.DataParallel` backward process is roughly *backward in each GPU asynchronously* and *inter-GPU communication for collecting weight gradients (aka, reduce)* for **each** autograd computation graph. That is, if one checkpoint contains weight for updating, there would be an inter-gpu synchronization process for accumulating the gradients in it, which is time-consuming. Considering that the current efficient DenseNet contains so many checkpointed `nn.BatchNorm2d` modules, a lot of time will be spent on the inter-GPU communications and gradient accumulation. From my test, the backward of efficent DenseNet for multi-GPUs is at least 100x slower that the normal version..."}}
{"actor":{"display_login":"wandering007"}}