{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/3228","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/3228/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/3228/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/3228/events","html_url":"https://github.com/dmlc/mxnet/issues/3228","id":175154390,"number":3228,"title":"BUG: The performance of multi-GPU training is decreased dramatically at current revision.","user":{"login":"miaow1988","id":3820816,"avatar_url":"https://avatars.githubusercontent.com/u/3820816?v=3","gravatar_id":"","url":"https://api.github.com/users/miaow1988","html_url":"https://github.com/miaow1988","followers_url":"https://api.github.com/users/miaow1988/followers","following_url":"https://api.github.com/users/miaow1988/following{/other_user}","gists_url":"https://api.github.com/users/miaow1988/gists{/gist_id}","starred_url":"https://api.github.com/users/miaow1988/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/miaow1988/subscriptions","organizations_url":"https://api.github.com/users/miaow1988/orgs","repos_url":"https://api.github.com/users/miaow1988/repos","events_url":"https://api.github.com/users/miaow1988/events{/privacy}","received_events_url":"https://api.github.com/users/miaow1988/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-09-06T03:43:02Z","updated_at":"2016-09-08T17:20:53Z","closed_at":null,"body":"The performance of multi-GPU training is decreased dramatically after I update the code from 5f278803303d5019b76c5aaecfd8c8f27758558e to the latest revision.\r\nIn particular, using four GPUs and 5f278803303d5019b76c5aaecfd8c8f27758558e, I got 242.26 samples/sec for Inception-BN network at training. However, using four GPUs and the latest version. I got only about 90 samples/sec for Inception-BN network at training."},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/245671968","html_url":"https://github.com/dmlc/mxnet/issues/3228#issuecomment-245671968","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/3228","id":245671968,"user":{"login":"Trangle","id":3235116,"avatar_url":"https://avatars.githubusercontent.com/u/3235116?v=3","gravatar_id":"","url":"https://api.github.com/users/Trangle","html_url":"https://github.com/Trangle","followers_url":"https://api.github.com/users/Trangle/followers","following_url":"https://api.github.com/users/Trangle/following{/other_user}","gists_url":"https://api.github.com/users/Trangle/gists{/gist_id}","starred_url":"https://api.github.com/users/Trangle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Trangle/subscriptions","organizations_url":"https://api.github.com/users/Trangle/orgs","repos_url":"https://api.github.com/users/Trangle/repos","events_url":"https://api.github.com/users/Trangle/events{/privacy}","received_events_url":"https://api.github.com/users/Trangle/received_events","type":"User","site_admin":false},"created_at":"2016-09-08T17:20:53Z","updated_at":"2016-09-08T17:20:53Z","body":"The same problem but with update #3075, rollback early will be OK.\r\nAnother problem is the speed will be slower ans slower during training, like:\r\n2016-09-08 23:35:02,492 Node[0] Start training with [gpu(1), gpu(0)]\r\n2016-09-08 23:35:29,963 Node[0] Epoch[0] Batch [50]\tSpeed: 875.63 samples/sec\tTrain-accuracy=0.225000\r\n2016-09-08 23:35:29,963 Node[0] Epoch[0] Batch [50]\tSpeed: 875.63 samples/sec\tTrain-top_k_accuracy_2=0.377031\r\n2016-09-08 23:35:44,855 Node[0] Epoch[0] Batch [100]\tSpeed: 859.58 samples/sec\tTrain-accuracy=0.313594\r\n2016-09-08 23:35:44,855 Node[0] Epoch[0] Batch [100]\tSpeed: 859.58 samples/sec\tTrain-top_k_accuracy_2=0.484297\r\n2016-09-08 23:36:00,513 Node[0] Epoch[0] Batch [150]\tSpeed: 817.53 samples/sec\tTrain-accuracy=0.346250\r\n2016-09-08 23:36:00,513 Node[0] Epoch[0] Batch [150]\tSpeed: 817.53 samples/sec\tTrain-top_k_accuracy_2=0.522891\r\n2016-09-08 23:36:16,523 Node[0] Epoch[0] Batch [200]\tSpeed: 799.50 samples/sec\tTrain-accuracy=0.395313\r\n2016-09-08 23:36:16,523 Node[0] Epoch[0] Batch [200]\tSpeed: 799.50 samples/sec\tTrain-top_k_accuracy_2=0.567734\r\n2016-09-08 23:36:32,545 Node[0] Epoch[0] Batch [250]\tSpeed: 798.95 samples/sec\tTrain-accuracy=0.415938\r\n2016-09-08 23:36:32,546 Node[0] Epoch[0] Batch [250]\tSpeed: 798.95 samples/sec\tTrain-top_k_accuracy_2=0.596172\r\n2016-09-08 23:36:48,555 Node[0] Epoch[0] Batch [300]\tSpeed: 799.50 samples/sec\tTrain-accuracy=0.456094\r\n2016-09-08 23:36:48,555 Node[0] Epoch[0] Batch [300]\tSpeed: 799.50 samples/sec\tTrain-top_k_accuracy_2=0.635547\r\n2016-09-08 23:37:04,805 Node[0] Epoch[0] Batch [350]\tSpeed: 787.74 samples/sec\tTrain-accuracy=0.495469\r\n2016-09-08 23:37:04,805 Node[0] Epoch[0] Batch [350]\tSpeed: 787.74 samples/sec\tTrain-top_k_accuracy_2=0.664687\r\n2016-09-08 23:37:21,542 Node[0] Epoch[0] Batch [400]\tSpeed: 764.77 samples/sec\tTrain-accuracy=0.522109\r\n2016-09-08 23:37:21,542 Node[0] Epoch[0] Batch [400]\tSpeed: 764.77 samples/sec\tTrain-top_k_accuracy_2=0.687969\r\n2016-09-08 23:37:39,036 Node[0] Epoch[0] Batch [450]\tSpeed: 731.72 samples/sec\tTrain-accuracy=0.561953\r\n2016-09-08 23:37:39,036 Node[0] Epoch[0] Batch [450]\tSpeed: 731.72 samples/sec\tTrain-top_k_accuracy_2=0.722031\r\n2016-09-08 23:37:41,203 Node[0] Epoch[0] Resetting Data Iterator\r\n2016-09-08 23:37:41,203 Node[0] Epoch[0] Time cost=147.089\r\n2016-09-08 23:37:41,400 Node[0] Saved checkpoint to \"H:/data/bot_animal/models/net_cnnv2-0-0001.params\"\r\n2016-09-08 23:37:49,036 Node[0] Epoch[0] Validation-accuracy=0.545343\r\n2016-09-08 23:37:49,036 Node[0] Epoch[0] Validation-top_k_accuracy_2=0.709329\r\n2016-09-08 23:38:06,970 Node[0] Epoch[1] Batch [50]\tSpeed: 719.42 samples/sec\tTrain-accuracy=0.586562\r\n2016-09-08 23:38:06,970 Node[0] Epoch[1] Batch [50]\tSpeed: 719.42 samples/sec\tTrain-top_k_accuracy_2=0.742891\r\n2016-09-08 23:38:25,776 Node[0] Epoch[1] Batch [100]\tSpeed: 680.63 samples/sec\tTrain-accuracy=0.612344\r\n2016-09-08 23:38:25,776 Node[0] Epoch[1] Batch [100]\tSpeed: 680.63 samples/sec\tTrain-top_k_accuracy_2=0.760625\r\n2016-09-08 23:38:45,683 Node[0] Epoch[1] Batch [150]\tSpeed: 642.99 samples/sec\tTrain-accuracy=0.630234\r\n2016-09-08 23:38:45,683 Node[0] Epoch[1] Batch [150]\tSpeed: 642.99 samples/sec\tTrain-top_k_accuracy_2=0.776016\r\n2016-09-08 23:39:06,322 Node[0] Epoch[1] Batch [200]\tSpeed: 620.19 samples/sec\tTrain-accuracy=0.646797\r\n2016-09-08 23:39:06,322 Node[0] Epoch[1] Batch [200]\tSpeed: 620.19 samples/sec\tTrain-top_k_accuracy_2=0.796094\r\n2016-09-08 23:39:27,276 Node[0] Epoch[1] Batch [250]\tSpeed: 610.89 samples/sec\tTrain-accuracy=0.660000\r\n2016-09-08 23:39:27,276 Node[0] Epoch[1] Batch [250]\tSpeed: 610.89 samples/sec\tTrain-top_k_accuracy_2=0.800312\r\n2016-09-08 23:39:48,714 Node[0] Epoch[1] Batch [300]\tSpeed: 597.04 samples/sec\tTrain-accuracy=0.680234\r\n2016-09-08 23:39:48,719 Node[0] Epoch[1] Batch [300]\tSpeed: 597.04 samples/sec\tTrain-top_k_accuracy_2=0.818359\r\n2016-09-08 23:40:10,693 Node[0] Epoch[1] Batch [350]\tSpeed: 582.51 samples/sec\tTrain-accuracy=0.692500\r\n2016-09-08 23:40:10,693 Node[0] Epoch[1] Batch [350]\tSpeed: 582.51 samples/sec\tTrain-top_k_accuracy_2=0.826562\r\n2016-09-08 23:40:32,963 Node[0] Epoch[1] Batch [400]\tSpeed: 574.76 samples/sec\tTrain-accuracy=0.705469\r\n2016-09-08 23:40:32,963 Node[0] Epoch[1] Batch [400]\tSpeed: 574.76 samples/sec\tTrain-top_k_accuracy_2=0.831562\r\n2016-09-08 23:40:55,618 Node[0] Epoch[1] Batch [450]\tSpeed: 565.02 samples/sec\tTrain-accuracy=0.723437\r\n2016-09-08 23:40:55,618 Node[0] Epoch[1] Batch [450]\tSpeed: 565.02 samples/sec\tTrain-top_k_accuracy_2=0.845625\r\n2016-09-08 23:40:57,861 Node[0] Epoch[1] Resetting Data Iterator\r\n2016-09-08 23:40:57,861 Node[0] Epoch[1] Time cost=188.824\r\n2016-09-08 23:40:58,158 Node[0] Saved checkpoint to \"H:/data/bot_animal/models/net_cnnv2-0-0002.params\"\r\n2016-09-08 23:41:05,997 Node[0] Epoch[1] Validation-accuracy=0.651961\r\n2016-09-08 23:41:05,997 Node[0] Epoch[1] Validation-top_k_accuracy_2=0.797564\r\n2016-09-08 23:41:28,134 Node[0] Epoch[2] Batch [50]\tSpeed: 582.00 samples/sec\tTrain-accuracy=0.728672\r\n2016-09-08 23:41:28,134 Node[0] Epoch[2] Batch [50]\tSpeed: 582.00 samples/sec\tTrain-top_k_accuracy_2=0.848437\r\n2016-09-08 23:41:51,263 Node[0] Epoch[2] Batch [100]\tSpeed: 553.39 samples/sec\tTrain-accuracy=0.733984\r\n2016-09-08 23:41:51,263 Node[0] Epoch[2] Batch [100]\tSpeed: 553.39 samples/sec\tTrain-top_k_accuracy_2=0.854609\r\n2016-09-08 23:42:14,671 Node[0] Epoch[2] Batch [150]\tSpeed: 546.82 samples/sec\tTrain-accuracy=0.740234\r\n2016-09-08 23:42:14,671 Node[0] Epoch[2] Batch [150]\tSpeed: 546.82 samples/sec\tTrain-top_k_accuracy_2=0.860781\r\n2016-09-08 23:42:38,443 Node[0] Epoch[2] Batch [200]\tSpeed: 538.47 samples/sec\tTrain-accuracy=0.746641\r\n2016-09-08 23:42:38,443 Node[0] Epoch[2] Batch [200]\tSpeed: 538.47 samples/sec\tTrain-top_k_accuracy_2=0.864844\r\n2016-09-08 23:43:02,299 Node[0] Epoch[2] Batch [250]\tSpeed: 536.55 samples/sec\tTrain-accuracy=0.749297\r\n2016-09-08 23:43:02,299 Node[0] Epoch[2] Batch [250]\tSpeed: 536.55 samples/sec\tTrain-top_k_accuracy_2=0.869141\r\n2016-09-08 23:43:26,358 Node[0] Epoch[2] Batch [300]\tSpeed: 532.03 samples/sec\tTrain-accuracy=0.763047\r\n2016-09-08 23:43:26,359 Node[0] Epoch[2] Batch [300]\tSpeed: 532.03 samples/sec\tTrain-top_k_accuracy_2=0.878281\r\n2016-09-08 23:43:50,516 Node[0] Epoch[2] Batch [350]\tSpeed: 529.89 samples/sec\tTrain-accuracy=0.772734\r\n2016-09-08 23:43:50,516 Node[0] Epoch[2] Batch [350]\tSpeed: 529.89 samples/sec\tTrain-top_k_accuracy_2=0.881094\r\n2016-09-08 23:44:14,898 Node[0] Epoch[2] Batch [400]\tSpeed: 524.98 samples/sec\tTrain-accuracy=0.769531\r\n2016-09-08 23:44:14,898 Node[0] Epoch[2] Batch [400]\tSpeed: 524.98 samples/sec\tTrain-top_k_accuracy_2=0.876563\r\n2016-09-08 23:44:39,318 Node[0] Epoch[2] Batch [450]\tSpeed: 524.16 samples/sec\tTrain-accuracy=0.783047\r\n2016-09-08 23:44:39,318 Node[0] Epoch[2] Batch [450]\tSpeed: 524.16 samples/sec\tTrain-top_k_accuracy_2=0.883047\r\n2016-09-08 23:44:41,750 Node[0] Epoch[2] Resetting Data Iterator\r\n2016-09-08 23:44:41,750 Node[0] Epoch[2] Time cost=215.751\r\n2016-09-08 23:44:42,033 Node[0] Saved checkpoint to \"H:/data/bot_animal/models/net_cnnv2-0-0003.params\"\r\n2016-09-08 23:44:49,696 Node[0] Epoch[2] Validation-accuracy=0.683287\r\n2016-09-08 23:44:49,697 Node[0] Epoch[2] Validation-top_k_accuracy_2=0.823912\r\n2016-09-08 23:45:14,043 Node[0] Epoch[3] Batch [50]\tSpeed: 530.42 samples/sec\tTrain-accuracy=0.782031\r\n2016-09-08 23:45:14,045 Node[0] Epoch[3] Batch [50]\tSpeed: 530.42 samples/sec\tTrain-top_k_accuracy_2=0.885547\r\n2016-09-08 23:45:38,563 Node[0] Epoch[3] Batch [100]\tSpeed: 522.04 samples/sec\tTrain-accuracy=0.787734\r\n2016-09-08 23:45:38,565 Node[0] Epoch[3] Batch [100]\tSpeed: 522.04 samples/sec\tTrain-top_k_accuracy_2=0.886094\r\n2016-09-08 23:46:03,214 Node[0] Epoch[3] Batch [150]\tSpeed: 519.27 samples/sec\tTrain-accuracy=0.789141\r\n2016-09-08 23:46:03,214 Node[0] Epoch[3] Batch [150]\tSpeed: 519.27 samples/sec\tTrain-top_k_accuracy_2=0.889687\r\n2016-09-08 23:46:27,908 Node[0] Epoch[3] Batch [200]\tSpeed: 518.34 samples/sec\tTrain-accuracy=0.793359\r\n2016-09-08 23:46:27,908 Node[0] Epoch[3] Batch [200]\tSpeed: 518.34 samples/sec\tTrain-top_k_accuracy_2=0.894766\r\n2016-09-08 23:46:52,717 Node[0] Epoch[3] Batch [250]\tSpeed: 515.96 samples/sec\tTrain-accuracy=0.797422\r\n2016-09-08 23:46:52,717 Node[0] Epoch[3] Batch [250]\tSpeed: 515.96 samples/sec\tTrain-top_k_accuracy_2=0.894453\r\n2016-09-08 23:47:17,509 Node[0] Epoch[3] Batch [300]\tSpeed: 516.27 samples/sec\tTrain-accuracy=0.800234\r\n2016-09-08 23:47:17,509 Node[0] Epoch[3] Batch [300]\tSpeed: 516.27 samples/sec\tTrain-top_k_accuracy_2=0.894375\r\n2016-09-08 23:47:42,410 Node[0] Epoch[3] Batch [350]\tSpeed: 514.08 samples/sec\tTrain-accuracy=0.807109\r\n2016-09-08 23:47:42,410 Node[0] Epoch[3] Batch [350]\tSpeed: 514.08 samples/sec\tTrain-top_k_accuracy_2=0.904922\r\n2016-09-08 23:48:07,289 Node[0] Epoch[3] Batch [400]\tSpeed: 514.49 samples/sec\tTrain-accuracy=0.809453\r\n2016-09-08 23:48:07,289 Node[0] Epoch[3] Batch [400]\tSpeed: 514.49 samples/sec\tTrain-top_k_accuracy_2=0.903516\r\n2016-09-08 23:48:32,085 Node[0] Epoch[3] Batch [450]\tSpeed: 516.19 samples/sec\tTrain-accuracy=0.813438\r\n2016-09-08 23:48:32,085 Node[0] Epoch[3] Batch [450]\tSpeed: 516.19 samples/sec\tTrain-top_k_accuracy_2=0.907969\r\n2016-09-08 23:48:34,585 Node[0] Epoch[3] Resetting Data Iterator\r\n2016-09-08 23:48:34,586 Node[0] Epoch[3] Time cost=224.890\r\n2016-09-08 23:48:34,877 Node[0] Saved checkpoint to \"H:/data/bot_animal/models/net_cnnv2-0-0004.params\"\r\n2016-09-08 23:48:42,569 Node[0] Epoch[3] Validation-accuracy=0.741958\r\n2016-09-08 23:48:42,569 Node[0] Epoch[3] Validation-top_k_accuracy_2=0.854090\r\n2016-09-08 23:49:07,190 Node[0] Epoch[4] Batch [50]\tSpeed: 525.99 samples/sec\tTrain-accuracy=0.813516\r\n2016-09-08 23:49:07,190 Node[0] Epoch[4] Batch [50]\tSpeed: 525.99 samples/sec\tTrain-top_k_accuracy_2=0.906484\r\n2016-09-08 23:49:32,032 Node[0] Epoch[4] Batch [100]\tSpeed: 515.26 samples/sec\tTrain-accuracy=0.817031\r\n2016-09-08 23:49:32,032 Node[0] Epoch[4] Batch [100]\tSpeed: 515.26 samples/sec\tTrain-top_k_accuracy_2=0.905937\r\n2016-09-08 23:49:57,131 Node[0] Epoch[4] Batch [150]\tSpeed: 509.96 samples/sec\tTrain-accuracy=0.821094\r\n2016-09-08 23:49:57,131 Node[0] Epoch[4] Batch [150]\tSpeed: 509.96 samples/sec\tTrain-top_k_accuracy_2=0.909141\r\n2016-09-08 23:50:22,255 Node[0] Epoch[4] Batch [200]\tSpeed: 509.49 samples/sec\tTrain-accuracy=0.820391\r\n2016-09-08 23:50:22,255 Node[0] Epoch[4] Batch [200]\tSpeed: 509.49 samples/sec\tTrain-top_k_accuracy_2=0.912891\r\n2016-09-08 23:50:47,184 Node[0] Epoch[4] Batch [250]\tSpeed: 513.44 samples/sec\tTrain-accuracy=0.823438\r\n2016-09-08 23:50:47,184 Node[0] Epoch[4] Batch [250]\tSpeed: 513.44 samples/sec\tTrain-top_k_accuracy_2=0.911563\r\n2016-09-08 23:51:12,216 Node[0] Epoch[4] Batch [300]\tSpeed: 511.39 samples/sec\tTrain-accuracy=0.825937\r\n2016-09-08 23:51:12,216 Node[0] Epoch[4] Batch [300]\tSpeed: 511.39 samples/sec\tTrain-top_k_accuracy_2=0.913125\r\n2016-09-08 23:51:37,359 Node[0] Epoch[4] Batch [350]\tSpeed: 509.07 samples/sec\tTrain-accuracy=0.832187\r\n2016-09-08 23:51:37,359 Node[0] Epoch[4] Batch [350]\tSpeed: 509.07 samples/sec\tTrain-top_k_accuracy_2=0.919297\r\n2016-09-08 23:52:02,466 Node[0] Epoch[4] Batch [400]\tSpeed: 509.84 samples/sec\tTrain-accuracy=0.830547\r\n2016-09-08 23:52:02,466 Node[0] Epoch[4] Batch [400]\tSpeed: 509.84 samples/sec\tTrain-top_k_accuracy_2=0.916719\r\n2016-09-08 23:52:27,539 Node[0] Epoch[4] Batch [450]\tSpeed: 510.51 samples/sec\tTrain-accuracy=0.836484\r\n2016-09-08 23:52:27,539 Node[0] Epoch[4] Batch [450]\tSpeed: 510.51 samples/sec\tTrain-top_k_accuracy_2=0.919375"}}
{"actor":{"display_login":"Trangle"}}