{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/CNTK/issues/440","repository_url":"https://api.github.com/repos/Microsoft/CNTK","labels_url":"https://api.github.com/repos/Microsoft/CNTK/issues/440/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/CNTK/issues/440/comments","events_url":"https://api.github.com/repos/Microsoft/CNTK/issues/440/events","html_url":"https://github.com/Microsoft/CNTK/issues/440","id":153193429,"number":440,"title":"Why is the training speed on multiple machine with multiple-GPUs lower than on single machine with multiple-GPUs ?","user":{"login":"NewEricWang","id":18108440,"avatar_url":"https://avatars.githubusercontent.com/u/18108440?v=3","gravatar_id":"","url":"https://api.github.com/users/NewEricWang","html_url":"https://github.com/NewEricWang","followers_url":"https://api.github.com/users/NewEricWang/followers","following_url":"https://api.github.com/users/NewEricWang/following{/other_user}","gists_url":"https://api.github.com/users/NewEricWang/gists{/gist_id}","starred_url":"https://api.github.com/users/NewEricWang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NewEricWang/subscriptions","organizations_url":"https://api.github.com/users/NewEricWang/orgs","repos_url":"https://api.github.com/users/NewEricWang/repos","events_url":"https://api.github.com/users/NewEricWang/events{/privacy}","received_events_url":"https://api.github.com/users/NewEricWang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-05T08:56:57Z","updated_at":"2016-05-06T03:06:00Z","closed_at":null,"body":"I ran two DNN parallel training experiments, one delivered on single machine with 2 k40 GPUs, another delivered on two machines where there are 3 k40 GPUs in total. I found the training on multiple machine is 2 times slower than on single machine. These GPUs is same type.  I guess the reason is the communication speed between two machine is limited by the network bandwidth of gigabit Ethernet. Must 10-gigabit Ethernet or InfiniBand be used if I want to speed up by multiple-machine parallel?"},"comment":{"url":"https://api.github.com/repos/Microsoft/CNTK/issues/comments/217337990","html_url":"https://github.com/Microsoft/CNTK/issues/440#issuecomment-217337990","issue_url":"https://api.github.com/repos/Microsoft/CNTK/issues/440","id":217337990,"user":{"login":"Alexey-Kamenev","id":8118464,"avatar_url":"https://avatars.githubusercontent.com/u/8118464?v=3","gravatar_id":"","url":"https://api.github.com/users/Alexey-Kamenev","html_url":"https://github.com/Alexey-Kamenev","followers_url":"https://api.github.com/users/Alexey-Kamenev/followers","following_url":"https://api.github.com/users/Alexey-Kamenev/following{/other_user}","gists_url":"https://api.github.com/users/Alexey-Kamenev/gists{/gist_id}","starred_url":"https://api.github.com/users/Alexey-Kamenev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Alexey-Kamenev/subscriptions","organizations_url":"https://api.github.com/users/Alexey-Kamenev/orgs","repos_url":"https://api.github.com/users/Alexey-Kamenev/repos","events_url":"https://api.github.com/users/Alexey-Kamenev/events{/privacy}","received_events_url":"https://api.github.com/users/Alexey-Kamenev/received_events","type":"User","site_admin":false},"created_at":"2016-05-06T03:06:00Z","updated_at":"2016-05-06T03:06:00Z","body":"What is your DNN architecture? And just to make sure, you do set parallelTrain to true, right?"}}