{"action":"created","issue":{"url":"https://api.github.com/repos/akanimax/pro_gan_pytorch/issues/4","repository_url":"https://api.github.com/repos/akanimax/pro_gan_pytorch","labels_url":"https://api.github.com/repos/akanimax/pro_gan_pytorch/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/akanimax/pro_gan_pytorch/issues/4/comments","events_url":"https://api.github.com/repos/akanimax/pro_gan_pytorch/issues/4/events","html_url":"https://github.com/akanimax/pro_gan_pytorch/issues/4","id":372626107,"node_id":"MDU6SXNzdWUzNzI2MjYxMDc=","number":4,"title":"Multi-GPU version runs much slower than on single card","user":{"login":"deartonym","id":12375764,"node_id":"MDQ6VXNlcjEyMzc1NzY0","avatar_url":"https://avatars3.githubusercontent.com/u/12375764?v=4","gravatar_id":"","url":"https://api.github.com/users/deartonym","html_url":"https://github.com/deartonym","followers_url":"https://api.github.com/users/deartonym/followers","following_url":"https://api.github.com/users/deartonym/following{/other_user}","gists_url":"https://api.github.com/users/deartonym/gists{/gist_id}","starred_url":"https://api.github.com/users/deartonym/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deartonym/subscriptions","organizations_url":"https://api.github.com/users/deartonym/orgs","repos_url":"https://api.github.com/users/deartonym/repos","events_url":"https://api.github.com/users/deartonym/events{/privacy}","received_events_url":"https://api.github.com/users/deartonym/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-10-22T17:51:27Z","updated_at":"2018-10-22T19:28:45Z","closed_at":null,"author_association":"NONE","body":"I found this problem at last weekend. The GPU volatile utilizing is only 11% on each card (I am using 4 titan xp). However when I use single card, the usage would be 45% (not that optimal, maybe >= 80% would be better). I guess it has something to do with the EMA thing. I will try to debug it. Just an info.\r\n\r\nBest,\r\nShu"},"comment":{"url":"https://api.github.com/repos/akanimax/pro_gan_pytorch/issues/comments/431949267","html_url":"https://github.com/akanimax/pro_gan_pytorch/issues/4#issuecomment-431949267","issue_url":"https://api.github.com/repos/akanimax/pro_gan_pytorch/issues/4","id":431949267,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTk0OTI2Nw==","user":{"login":"deartonym","id":12375764,"node_id":"MDQ6VXNlcjEyMzc1NzY0","avatar_url":"https://avatars3.githubusercontent.com/u/12375764?v=4","gravatar_id":"","url":"https://api.github.com/users/deartonym","html_url":"https://github.com/deartonym","followers_url":"https://api.github.com/users/deartonym/followers","following_url":"https://api.github.com/users/deartonym/following{/other_user}","gists_url":"https://api.github.com/users/deartonym/gists{/gist_id}","starred_url":"https://api.github.com/users/deartonym/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deartonym/subscriptions","organizations_url":"https://api.github.com/users/deartonym/orgs","repos_url":"https://api.github.com/users/deartonym/repos","events_url":"https://api.github.com/users/deartonym/events{/privacy}","received_events_url":"https://api.github.com/users/deartonym/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T19:28:45Z","updated_at":"2018-10-22T19:28:45Z","author_association":"NONE","body":"I have tested several parameters.\r\n\r\n**Single GPU**:\r\n\r\nbatch size 32:\r\nepoch[1/30] 1417.54 examples/sec, 0.59 min/epoch. Rest runtime is 0.28 hour\r\n\r\nbatch_size 64:\r\nepoch[1/30] 2640.38 examples/sec, 0.32 min/epoch. Rest runtime is 0.15 hour \r\n\r\n**4 GPUs**:\r\nbatch size 128 (32 on each):\r\nepoch[1/30] 845.78 examples/sec, 0.99 min/epoch. Rest runtime is 0.48 hour\r\n\r\nbatch size 256 (64 on each):\r\nepoch[1/30] 1402.12 examples/sec, 0.60 min/epoch. Rest runtime is 0.29 hour\r\n\r\nAs shows, by running the generator and discriminator on 4 GPU will even make it worse.\r\nI have no idea whether we can put EMA as DataParallel on multiple GPU as well. A simple:\r\n`ema = torch.nn.DataParallel(ema)`\r\nwill not work. Since PyTorch will try to devide the input for ema, which are the current parameters of network, as 4 different batches into different 4GPUs to update the moving parameters. However parameters are not databatches, I am not sure whether we can find the correct part of the parameter for ema to update distributedly.\r\n\r\nBest,\r\nShu"}}
{"actor":{"display_login":"deartonym"}}