{"action":"opened","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/2166","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/2166/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/2166/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/2166/events","html_url":"https://github.com/dmlc/mxnet/issues/2166","id":155188767,"number":2166,"title":"multi machine train seem efficient but low accuracy","user":{"login":"GekTo","id":5905329,"avatar_url":"https://avatars.githubusercontent.com/u/5905329?v=3","gravatar_id":"","url":"https://api.github.com/users/GekTo","html_url":"https://github.com/GekTo","followers_url":"https://api.github.com/users/GekTo/followers","following_url":"https://api.github.com/users/GekTo/following{/other_user}","gists_url":"https://api.github.com/users/GekTo/gists{/gist_id}","starred_url":"https://api.github.com/users/GekTo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GekTo/subscriptions","organizations_url":"https://api.github.com/users/GekTo/orgs","repos_url":"https://api.github.com/users/GekTo/repos","events_url":"https://api.github.com/users/GekTo/events{/privacy}","received_events_url":"https://api.github.com/users/GekTo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-17T06:54:13Z","updated_at":"2016-05-17T06:54:13Z","closed_at":null,"body":"Hi all,\r\nI am testing the distributed train over multi-machine mxnet. I used the train_mnist.py and test with lenet. And I set one machine as dispatcher， and a half of the rest machine as workers and the rest as server.\r\nHowever, even though the time used for train one epoch is linear decrease as the machine I used, the train accuracy and validation accuracy isn't increase with the train epoch any more.\r\nI have checked that the ps-lite is working fine, cause when set PS_VERBOSE=1, connection info is right, set PS_VERBOSE=2, lots of communication log is print.\r\nSo, How to explain those low accuracy and no increase of it?\r\n\r\nthe following is some log info parsed from tools/parselog.py\r\n1 node:\r\n| epoch | train-accuracy | valid-accuracy | time |\r\n| --- | --- | --- | --- |\r\n|  1 | 0.766714 | 0.904172 | 238.0 |\r\n|  2 | 0.853887 | 0.889674 | 229.8 |\r\n|  3 | 0.921867 | 0.887795 | 228.5 |\r\n|  4 | 0.935588 | 0.952324 | 230.6 |\r\n|  5 | 0.934608 | 0.932267 | 229.3 |\r\n|  6 | 0.938147 | 0.933068 | 230.7 |\r\n|  7 | 0.938343 | 0.940856 | 228.8 |\r\n|  8 | 0.937893 | 0.945137 | 226.7 |\r\n|  9 | 0.933089 | 0.941256 | 230.7 |\r\n| 10 | 0.879812 | 0.903922 | 227.7 |\r\n5 nodes:\r\n| epoch | train-accuracy | valid-accuracy | time |\r\n| --- | --- | --- | --- |\r\n|  1 | 0.650182 | 0.648287 | 119.8 |\r\n|  2 | 0.649210 | 0.651492 | 120.1 |\r\n|  3 | 0.650011 | 0.648387 | 121.1 |\r\n|  4 | 0.653091 | 0.651267 | 121.8 |\r\n|  5 | 0.695041 | 0.783979 | 121.3 |\r\n|  6 | 0.694958 | 0.649164 | 121.6 |\r\n|  7 | 0.666900 | 0.650115 | 121.2 |\r\n|  8 | 0.649944 | 0.649714 | 121.2 |\r\n|  9 | 0.649923 | 0.652294 | 120.7 |\r\n| 10 | 0.655019 | 0.652794 | 120.3 |\r\n8 nodes:\r\n| epoch | train-accuracy | valid-accuracy | time |\r\n| --- | --- | --- | --- |\r\n|  1 | 0.647549 | 0.647949 | 63.5 |\r\n|  2 | 0.649656 | 0.648951 | 62.5 |\r\n|  3 | 0.649381 | 0.655633 | 62.2 |\r\n|  4 | 0.650186 | 0.646125 | 62.5 |\r\n|  5 | 0.651363 | 0.654194 | 62.6 |\r\n|  6 | 0.676048 | 0.695826 | 62.3 |\r\n|  7 | 0.653946 | 0.646716 | 62.1 |\r\n|  8 | 0.659827 | 0.651650 | 62.7 |\r\n|  9 | 0.667860 | 0.668586 | 62.9 |\r\n| 10 | 0.708588 | 0.683080 | 62.9 |\r\n10 nodes:\r\n| epoch | train-accuracy | valid-accuracy | time |\r\n| --- | --- | --- | --- |\r\n|  1 | 0.648937 | 0.647786 | 51.3 |\r\n|  2 | 0.649609 | 0.648516 | 50.6 |\r\n|  3 | 0.649013 | 0.647604 | 50.0 |\r\n|  4 | 0.648921 | 0.646510 | 49.2 |\r\n|  5 | 0.649273 | 0.652031 | 49.5 |\r\n|  6 | 0.651932 | 0.652240 | 49.2 |\r\n|  7 | 0.650055 | 0.644635 | 49.6 |\r\n|  8 | 0.651932 | 0.651458 | 49.4 |\r\n|  9 | 0.681527 | 0.651875 | 49.9 |\r\n| 10 | 0.666961 | 0.682656 | 50.0 |\r\n20 nodes:\r\n| epoch | train-accuracy | valid-accuracy | time |\r\n| --- | --- | --- | --- |\r\n|  1 | 0.648939 | 0.645368 | 25.5 |\r\n|  2 | 0.649168 | 0.645703 | 25.1 |\r\n|  3 | 0.649907 | 0.646708 | 25.1 |\r\n|  4 | 0.649588 | 0.648884 | 24.9 |\r\n|  5 | 0.650000 | 0.648493 | 25.0 |\r\n|  6 | 0.650255 | 0.645089 | 24.6 |\r\n|  7 | 0.648964 | 0.646233 | 24.8 |\r\n|  8 | 0.649011 | 0.644029 | 24.4 |\r\n|  9 | 0.649231 | 0.645954 | 25.0 |\r\n| 10 | 0.650972 | 0.645368 | 24.8 |\r\n50 nodes:\r\n| epoch | train-accuracy | valid-accuracy | time |\r\n| --- | --- | --- | --- |\r\n|  1 | 0.648767 | 0.648776 | 10.3 |\r\n|  2 | 0.648980 | 0.647396 | 9.9 |\r\n|  3 | 0.649384 | 0.649922 | 9.9 |\r\n|  4 | 0.649948 | 0.644609 | 9.8 |\r\n|  5 | 0.649427 | 0.646901 | 9.9 |\r\n|  6 | 0.648490 | 0.646667 | 10.0 |\r\n|  7 | 0.647743 | 0.646901 | 9.9 |\r\n|  8 | 0.648655 | 0.648333 | 9.9 |\r\n|  9 | 0.648329 | 0.648047 | 9.8 |\r\n| 10 | 0.649002 | 0.649479 | 9.9 |\r\n100 nodes:\r\n| epoch | train-accuracy | valid-accuracy | time |\r\n| --- | --- | --- | --- |\r\n|  1 | 0.647895 | 0.647227 | 5.6 |\r\n|  2 | 0.647717 | 0.655469 | 5.0 |\r\n|  3 | 0.650195 | 0.648984 | 4.9 |\r\n|  4 | 0.649154 | 0.649805 | 4.9 |\r\n|  5 | 0.649839 | 0.644766 | 5.0 |\r\n|  6 | 0.648190 | 0.647461 | 4.9 |\r\n|  7 | 0.648954 | 0.648906 | 4.9 |\r\n|  8 | 0.648411 | 0.653633 | 5.0 |\r\n|  9 | 0.649926 | 0.645547 | 4.9 |\r\n| 10 | 0.649115 | 0.649414 | 4.9 |\r\n\r\nBTW,  the cpu-only version is quite slow, anyway to speed up?\r\n"}}