{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/3675","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/3675/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/3675/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/3675/events","html_url":"https://github.com/dmlc/mxnet/issues/3675","id":186401404,"number":3675,"title":"How to choose batch size when using multi-GPU","user":{"login":"hfutxrg","id":3160803,"avatar_url":"https://avatars.githubusercontent.com/u/3160803?v=3","gravatar_id":"","url":"https://api.github.com/users/hfutxrg","html_url":"https://github.com/hfutxrg","followers_url":"https://api.github.com/users/hfutxrg/followers","following_url":"https://api.github.com/users/hfutxrg/following{/other_user}","gists_url":"https://api.github.com/users/hfutxrg/gists{/gist_id}","starred_url":"https://api.github.com/users/hfutxrg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hfutxrg/subscriptions","organizations_url":"https://api.github.com/users/hfutxrg/orgs","repos_url":"https://api.github.com/users/hfutxrg/repos","events_url":"https://api.github.com/users/hfutxrg/events{/privacy}","received_events_url":"https://api.github.com/users/hfutxrg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-10-31T21:23:59Z","updated_at":"2016-11-01T03:05:01Z","closed_at":null,"body":"Hi All,\r\n\r\nI used MXNet to run the GoogleNet and Inception-BN networks in examples/image_classification. I got good speedup in both networks, but the batch size choice is different in these two models. \r\n\r\nIn GoogleNet, the batch size is constant 144 for 1 GPU, 2 GPUs and 4 GPUs.\r\nBut in Inception-BN, the batch size is 64 for 1 GPU, 128 for 2 GPUs and 256 for 4 GPUs.\r\n\r\nBased on the description in https://github.com/dmlc/mxnet/blob/master/docs/how_to/multi_devices.md, the batch size should be constant for 1 GPU and multiple GPUs. This is because when multiple GPUs are used, the samples in one batch is distributed to all GPUs. But why for Inception-BN, we need to change the batch size? Another post https://github.com/dmlc/mxnet/issues/2283 also changed the batch size. \r\n\r\nRegards,\r\nRengan\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/257479019","html_url":"https://github.com/dmlc/mxnet/issues/3675#issuecomment-257479019","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/3675","id":257479019,"user":{"login":"hfutxrg","id":3160803,"avatar_url":"https://avatars.githubusercontent.com/u/3160803?v=3","gravatar_id":"","url":"https://api.github.com/users/hfutxrg","html_url":"https://github.com/hfutxrg","followers_url":"https://api.github.com/users/hfutxrg/followers","following_url":"https://api.github.com/users/hfutxrg/following{/other_user}","gists_url":"https://api.github.com/users/hfutxrg/gists{/gist_id}","starred_url":"https://api.github.com/users/hfutxrg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hfutxrg/subscriptions","organizations_url":"https://api.github.com/users/hfutxrg/orgs","repos_url":"https://api.github.com/users/hfutxrg/repos","events_url":"https://api.github.com/users/hfutxrg/events{/privacy}","received_events_url":"https://api.github.com/users/hfutxrg/received_events","type":"User","site_admin":false},"created_at":"2016-11-01T03:05:01Z","updated_at":"2016-11-01T03:05:01Z","body":"I did not mean the absolute value of batch size. I mean what the batch size should be when using multiple GPUs compared to the batch size when using only 1 GPU. For example, when using 1 GPU, we set the batch size as M. If the goal is just to see the performance speedup when using multiple GPUs, then for the same model using 2 GPUs, the batch size should also be M instead of 2M, right? Because if we use 2M, then the learning will be different from M. \r\n\r\nIn GoogleNet, I did use M (M=64) for 1 GPU and multiple GPUs and got good speedup. But for Inception-BN, I had to use M for 1 GPU and 2M for 2 GPUs. That's the part I do not understand. "}}
{"actor":{"display_login":"hfutxrg"}}