{"action":"opened","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/3787","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/3787/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/3787/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/3787/events","html_url":"https://github.com/dmlc/mxnet/issues/3787","id":188431900,"number":3787,"title":"Multiple-node training never ends...","user":{"login":"FreemanX","id":9710644,"avatar_url":"https://avatars.githubusercontent.com/u/9710644?v=3","gravatar_id":"","url":"https://api.github.com/users/FreemanX","html_url":"https://github.com/FreemanX","followers_url":"https://api.github.com/users/FreemanX/followers","following_url":"https://api.github.com/users/FreemanX/following{/other_user}","gists_url":"https://api.github.com/users/FreemanX/gists{/gist_id}","starred_url":"https://api.github.com/users/FreemanX/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FreemanX/subscriptions","organizations_url":"https://api.github.com/users/FreemanX/orgs","repos_url":"https://api.github.com/users/FreemanX/repos","events_url":"https://api.github.com/users/FreemanX/events{/privacy}","received_events_url":"https://api.github.com/users/FreemanX/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-10T06:40:33Z","updated_at":"2016-11-10T06:40:33Z","closed_at":null,"body":"![image](https://cloud.githubusercontent.com/assets/9710644/20166611/f394c888-a750-11e6-9925-3602ff23f83d.png)\r\nAs shown above, the process of training ends but the program doesn't. I set PS_VERBOSE=2 and these are the outputs after training. The lower Terminal shows the current network traffic status.  I have no idea what's going on here...\r\nI notice that there are '? => id' in it, does it lose track of certain processes? \r\nMy running script:\r\nDMLC_INTERFACE=em3 PS_VERBOSE=2 python ~/mxnet/tools/launch.py --launcher ssh -n 2 -s 1 -H hosts2 python train_mnist.py --lr 0.05 --kv-store dist_sync --batch-size 4096 --num-epochs 4 --gpus 0,1,2,3"}}
{"actor":{"display_login":"FreemanX"}}