{"action":"opened","issue":{"url":"https://api.github.com/repos/Microsoft/CNTK/issues/3419","repository_url":"https://api.github.com/repos/Microsoft/CNTK","labels_url":"https://api.github.com/repos/Microsoft/CNTK/issues/3419/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/CNTK/issues/3419/comments","events_url":"https://api.github.com/repos/Microsoft/CNTK/issues/3419/events","html_url":"https://github.com/Microsoft/CNTK/issues/3419","id":364587684,"node_id":"MDU6SXNzdWUzNjQ1ODc2ODQ=","number":3419,"title":"Distributed Training Error","user":{"login":"chung1204","id":23748374,"node_id":"MDQ6VXNlcjIzNzQ4Mzc0","avatar_url":"https://avatars3.githubusercontent.com/u/23748374?v=4","gravatar_id":"","url":"https://api.github.com/users/chung1204","html_url":"https://github.com/chung1204","followers_url":"https://api.github.com/users/chung1204/followers","following_url":"https://api.github.com/users/chung1204/following{/other_user}","gists_url":"https://api.github.com/users/chung1204/gists{/gist_id}","starred_url":"https://api.github.com/users/chung1204/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chung1204/subscriptions","organizations_url":"https://api.github.com/users/chung1204/orgs","repos_url":"https://api.github.com/users/chung1204/repos","events_url":"https://api.github.com/users/chung1204/events{/privacy}","received_events_url":"https://api.github.com/users/chung1204/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-27T17:53:40Z","updated_at":"2018-09-27T17:53:40Z","closed_at":null,"author_association":"NONE","body":"I want to train VGG16_ImageNet_Distributed.py at multiple node using mpiexec (two gpu on one node)\r\nso, I followed instructions in https://docs.microsoft.com/en-us/cognitive-toolkit/Multiple-GPUs-and-machines.\r\nWhen I was training on one node, it worked well. However, it did not work when I trained on multiple node.\r\nAn error occurred that does not import module numpy or cntk at other node.\r\nSince I train through anaconda, i changed default python path to anaconda. But it did not solve the problem..\r\nIs there a solution to this problem??\r\n\r\n\r\nthe errors are as follows :\r\n**Traceback (most recent call last):\r\n  File \"/home/cslee/cntk/Examples/Image/Classification/VGG/Python/VGG16_modify_2.py\", line 11, in <module>\r\n    import numpy as np\r\nImportError: No module named numpy\r\nTraceback (most recent call last):\r\n  File \"/home/cslee/cntk/Examples/Image/Classification/VGG/Python/VGG16_modify_2.py\", line 11, in <module>\r\n    import numpy as np\r\nImportError: No module named numpy\r\n-------------------------------------------------------\r\nPrimary job  terminated normally, but 1 process returned\r\na non-zero exit code.. Per user-direction, the job has been aborted.\r\n-------------------------------------------------------\r\n--------------------------------------------------------------------------\r\nmpiexec detected that one or more processes exited with non-zero status, thus causing\r\nthe job to be terminated. The first process to do so was:\r\n\r\n  Process name: [[62073,1],2]\r\n  Exit code:    1**\r\n"}}
{"actor":{"display_login":"chung1204"}}