{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/1410","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/1410/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/1410/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/1410/events","html_url":"https://github.com/dmlc/mxnet/issues/1410","id":131179663,"number":1410,"title":"Distributed training -- where is the final model? Why are validation scores different?","user":{"login":"svohara","id":4420784,"avatar_url":"https://avatars.githubusercontent.com/u/4420784?v=3","gravatar_id":"","url":"https://api.github.com/users/svohara","html_url":"https://github.com/svohara","followers_url":"https://api.github.com/users/svohara/followers","following_url":"https://api.github.com/users/svohara/following{/other_user}","gists_url":"https://api.github.com/users/svohara/gists{/gist_id}","starred_url":"https://api.github.com/users/svohara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/svohara/subscriptions","organizations_url":"https://api.github.com/users/svohara/orgs","repos_url":"https://api.github.com/users/svohara/repos","events_url":"https://api.github.com/users/svohara/events{/privacy}","received_events_url":"https://api.github.com/users/svohara/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"mli","id":421857,"avatar_url":"https://avatars.githubusercontent.com/u/421857?v=3","gravatar_id":"","url":"https://api.github.com/users/mli","html_url":"https://github.com/mli","followers_url":"https://api.github.com/users/mli/followers","following_url":"https://api.github.com/users/mli/following{/other_user}","gists_url":"https://api.github.com/users/mli/gists{/gist_id}","starred_url":"https://api.github.com/users/mli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mli/subscriptions","organizations_url":"https://api.github.com/users/mli/orgs","repos_url":"https://api.github.com/users/mli/repos","events_url":"https://api.github.com/users/mli/events{/privacy}","received_events_url":"https://api.github.com/users/mli/received_events","type":"User","site_admin":false},"milestone":null,"comments":7,"created_at":"2016-02-03T22:49:10Z","updated_at":"2016-02-04T18:37:54Z","closed_at":null,"body":"First of all, I would like to express that I appreciate the significant development effort required to produce mxnet, and I am very excited to use this tool and hopefully to contribute features as time goes on. However, there are areas that could use more polish. My reason for using mxnet is for distributed training (cluster/AWS). Since few of the most popular packages do a good job in this regard, it is the single most important differentiating factor, in my opinion, for using mxnet. \r\n\r\nAs such, I seem to be having a disconnect between what I expect the distributed training process should produce (and how validation results should be reported per epoch) and what I observe actually happens. I'm using ssh based distributed training on AWS. Training cifar10 on 3 nodes, for example, generates checkpoints for each of the three nodes. Training seems to be successful, yet there is no obvious \"final model\" that is produced and the three node-specific models that are saved by the process at the last epoch yield different validation scores.\r\n\r\n**Issue 1**\r\nThe user expects that there will be an obvious final model produced as a result of the training process, and there is not with how the examples/tutorials are structured.\r\n\r\n**Issue 2**\r\nThe user expects that after parameter synchronization, the models on each of the distributed nodes are, well, synchronized (meaning identical parameters). They appear not to be. If they were, then after each synchronization event, the validation scores should be the same.\r\n\r\n**Expectations**\r\nThese two expectations may be the result of a misunderstanding on how distributed training works and what a parameter server actually synchronizes. I outline my understanding below, and if this is way off, then I would ask for some user-friendly high-level documentation with a training flow chart of some sort that makes it clear what each artifact represents and what to do at the end of the process to generate the single \"final\" model that I desire to use at run-time for predictions on new data.\r\n\r\nMy basic understanding is that, when using \"dist-sync\", at regular points in time (e.g., each epoch, but maybe more frequently? each mini-batch?), the parameter server collects the parameters from each of the nodes, performs some sort of aggregation to generate a consolidated/updated parameter set, and then distributes this update to each of the nodes. At this point in time, after synchronization but before continuing to train, the models at each of the nodes should be identical. After the synchronization, each node continues the training process independently until the next synchronization point. The process continues until the training completes, and then a final synchronization should yield a single consolidated model with parameters aggregated from the nodes.\r\n\r\n**Observations**\r\nMy observations on where what happens clashes with my expectations:\r\n\r\n1. When done, I am left with K different models (param files) and symbols (json), where K is the number of nodes. I would expect that a distributed training job should end up producing one final consolidated model, just as if I had only used single-node training, but of course, faster. The other artifacts produced by the separate nodes are useful, but when all is done, it should be obvious which is my final model that I should use for prediction.\r\n\r\n2. When I load the K different models that result, and test their performance on the validation data, I get different results! I expect that after syncing using the parameter server, that the final three models would be the same. If not, at what point can/should I checkpoint in order to get the model after the parameter synchronization has completed?\r\n\r\n3. When one observes the training output of the K nodes, at each epoch when the validation accuracy is reported, each of the K models has a different value. Is this because the validation data has also been partitioned and thus the K models are evaluating on different data? (If so, I think this should at least be optional -- partition training but not the presumably smaller validation data).\r\n\r\n4. If the answer to 3 is NOT that the validation data was partitioned, and in fact all K models are evaluating on the same validation set, then this becomes a similar question to 2 above. Why don't we, during the dist-sync process, end up with K identical models after the synchronization? This is what I intuitively expect, but it would seem the validation results are produced prior to synchronization. Why is this good/desired? Doesn't the user want to know, at the end of each epoch, how the model (in general, after updates from the K nodes) is performing?\r\n\r\n**Bottom Line**\r\nIn summary, the mechanism by which we train the model (local, distributed, sych/asynch, multi-gpu, cpus, etc.) is irrelevant at the highest-level of reporting performance at each epoch and at the end of the process. The name of the final model params and symbol files produced should be independent of the training mechanism.\r\n\r\nAnyone writing a wrapper or automation tool around using mxnet to train a deep net would want the outputs to be the same format regardless of mechanism. There should not be extra undocumented work that the user must perform in order to take the results of distributed training and produce the desired final model.\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/179990906","html_url":"https://github.com/dmlc/mxnet/issues/1410#issuecomment-179990906","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/1410","id":179990906,"user":{"login":"mli","id":421857,"avatar_url":"https://avatars.githubusercontent.com/u/421857?v=3","gravatar_id":"","url":"https://api.github.com/users/mli","html_url":"https://github.com/mli","followers_url":"https://api.github.com/users/mli/followers","following_url":"https://api.github.com/users/mli/following{/other_user}","gists_url":"https://api.github.com/users/mli/gists{/gist_id}","starred_url":"https://api.github.com/users/mli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mli/subscriptions","organizations_url":"https://api.github.com/users/mli/orgs","repos_url":"https://api.github.com/users/mli/repos","events_url":"https://api.github.com/users/mli/events{/privacy}","received_events_url":"https://api.github.com/users/mli/received_events","type":"User","site_admin":false},"created_at":"2016-02-04T18:37:54Z","updated_at":"2016-02-04T18:37:54Z","body":"the reason you see different validation acc during training is because each node only runs validation on a part of the validation data. assume there are n nodes, we partition validation data into n parts, node i only report the results on part i."}}