{"action":"created","issue":{"url":"https://api.github.com/repos/soumith/imagenet-multiGPU.torch/issues/49","repository_url":"https://api.github.com/repos/soumith/imagenet-multiGPU.torch","labels_url":"https://api.github.com/repos/soumith/imagenet-multiGPU.torch/issues/49/labels{/name}","comments_url":"https://api.github.com/repos/soumith/imagenet-multiGPU.torch/issues/49/comments","events_url":"https://api.github.com/repos/soumith/imagenet-multiGPU.torch/issues/49/events","html_url":"https://github.com/soumith/imagenet-multiGPU.torch/issues/49","id":133295852,"number":49,"title":"nGPU=4 is slower than nGPU=1","user":{"login":"chienlinhuang1116","id":16109566,"avatar_url":"https://avatars.githubusercontent.com/u/16109566?v=3","gravatar_id":"","url":"https://api.github.com/users/chienlinhuang1116","html_url":"https://github.com/chienlinhuang1116","followers_url":"https://api.github.com/users/chienlinhuang1116/followers","following_url":"https://api.github.com/users/chienlinhuang1116/following{/other_user}","gists_url":"https://api.github.com/users/chienlinhuang1116/gists{/gist_id}","starred_url":"https://api.github.com/users/chienlinhuang1116/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chienlinhuang1116/subscriptions","organizations_url":"https://api.github.com/users/chienlinhuang1116/orgs","repos_url":"https://api.github.com/users/chienlinhuang1116/repos","events_url":"https://api.github.com/users/chienlinhuang1116/events{/privacy}","received_events_url":"https://api.github.com/users/chienlinhuang1116/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-02-12T17:16:31Z","updated_at":"2016-10-27T06:16:24Z","closed_at":"2016-02-15T03:35:45Z","body":"Hi,\r\n\r\nDo you know why the results showed that “nGPU=4” is still slower than “nGPU=1”?\r\n\r\n-nGPU 1 -batchSize 128\r\nEpoch: [1][2/10000]     Time 0.835 Err 6.9080 Top1-%: 0.00 LR 1e-02 DataLoadingTime 0.005\r\nEpoch: [1][3/10000]     Time 0.834 Err 6.9073 Top1-%: 0.00 LR 1e-02 DataLoadingTime 3.233\r\nEpoch: [1][4/10000]     Time 0.834 Err 6.9104 Top1-%: 0.00 LR 1e-02 DataLoadingTime 0.003\r\nEpoch: [1][5/10000]     Time 0.834 Err 6.9075 Top1-%: 0.00 LR 1e-02 DataLoadingTime 2.871\r\nEpoch: [1][6/10000]     Time 0.836 Err 6.9064 Top1-%: 0.00 LR 1e-02 DataLoadingTime 0.003\r\nEpoch: [1][7/10000]     Time 0.833 Err 6.9077 Top1-%: 0.00 LR 1e-02 DataLoadingTime 2.776\r\n\r\n-nGPU 4 -batchSize 512\r\nEpoch: [1][2/10000]     Time 3.915 Err 6.9070 Top1-%: 0.00 LR 1e-02 DataLoadingTime 0.003\r\nEpoch: [1][3/10000]     Time 4.449 Err 6.9081 Top1-%: 0.00 LR 1e-02 DataLoadingTime 11.843\r\nEpoch: [1][4/10000]     Time 3.906 Err 6.9079 Top1-%: 0.20 LR 1e-02 DataLoadingTime 0.005\r\nEpoch: [1][5/10000]     Time 3.898 Err 6.9078 Top1-%: 0.20 LR 1e-02 DataLoadingTime 7.108\r\nEpoch: [1][6/10000]     Time 3.902 Err 6.9079 Top1-%: 0.20 LR 1e-02 DataLoadingTime 0.005\r\n\r\n-nGPU 8 -batchSize 1024\r\nEpoch: [1][2/10000]     Time 7.186 Err 6.9080 Top1-%: 0.20 LR 1e-02 DataLoadingTime 0.006\r\nEpoch: [1][3/10000]     Time 6.947 Err 6.9079 Top1-%: 0.10 LR 1e-02 DataLoadingTime 24.149\r\nEpoch: [1][4/10000]     Time 6.724 Err 6.9080 Top1-%: 0.00 LR 1e-02 DataLoadingTime 0.007\r\nEpoch: [1][5/10000]     Time 7.773 Err 6.9080 Top1-%: 0.10 LR 1e-02 DataLoadingTime 16.892\r\nEpoch: [1][6/10000]     Time 6.731 Err 6.9081 Top1-%: 0.00 LR 1e-02 DataLoadingTime 0.007\r\n\r\nThank you.\r\n\r\nChien-Lin\r\n"},"comment":{"url":"https://api.github.com/repos/soumith/imagenet-multiGPU.torch/issues/comments/256555863","html_url":"https://github.com/soumith/imagenet-multiGPU.torch/issues/49#issuecomment-256555863","issue_url":"https://api.github.com/repos/soumith/imagenet-multiGPU.torch/issues/49","id":256555863,"user":{"login":"buttomnutstoast","id":4762052,"avatar_url":"https://avatars.githubusercontent.com/u/4762052?v=3","gravatar_id":"","url":"https://api.github.com/users/buttomnutstoast","html_url":"https://github.com/buttomnutstoast","followers_url":"https://api.github.com/users/buttomnutstoast/followers","following_url":"https://api.github.com/users/buttomnutstoast/following{/other_user}","gists_url":"https://api.github.com/users/buttomnutstoast/gists{/gist_id}","starred_url":"https://api.github.com/users/buttomnutstoast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/buttomnutstoast/subscriptions","organizations_url":"https://api.github.com/users/buttomnutstoast/orgs","repos_url":"https://api.github.com/users/buttomnutstoast/repos","events_url":"https://api.github.com/users/buttomnutstoast/events{/privacy}","received_events_url":"https://api.github.com/users/buttomnutstoast/received_events","type":"User","site_admin":false},"created_at":"2016-10-27T06:16:24Z","updated_at":"2016-10-27T06:16:24Z","body":"In my experience, you can try:\r\n1) use multi-threads in DataParallelTable\r\n2) call model:getParameter() before model:forward()\r\n3) see if nccl is well installed\r\n4) maybe hardware issues..."}}
{"actor":{"display_login":"buttomnutstoast"}}