{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/669","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/669/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/669/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/669/events","html_url":"https://github.com/uber/horovod/issues/669","id":386118873,"node_id":"MDU6SXNzdWUzODYxMTg4NzM=","number":669,"title":"Horovod hangs on multi-machine keras program test","user":{"login":"Yang-HangWA","id":38546249,"node_id":"MDQ6VXNlcjM4NTQ2MjQ5","avatar_url":"https://avatars0.githubusercontent.com/u/38546249?v=4","gravatar_id":"","url":"https://api.github.com/users/Yang-HangWA","html_url":"https://github.com/Yang-HangWA","followers_url":"https://api.github.com/users/Yang-HangWA/followers","following_url":"https://api.github.com/users/Yang-HangWA/following{/other_user}","gists_url":"https://api.github.com/users/Yang-HangWA/gists{/gist_id}","starred_url":"https://api.github.com/users/Yang-HangWA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yang-HangWA/subscriptions","organizations_url":"https://api.github.com/users/Yang-HangWA/orgs","repos_url":"https://api.github.com/users/Yang-HangWA/repos","events_url":"https://api.github.com/users/Yang-HangWA/events{/privacy}","received_events_url":"https://api.github.com/users/Yang-HangWA/received_events","type":"User","site_admin":false},"labels":[{"id":665452437,"node_id":"MDU6TGFiZWw2NjU0NTI0Mzc=","url":"https://api.github.com/repos/uber/horovod/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-11-30T10:06:15Z","updated_at":"2018-12-03T11:47:07Z","closed_at":null,"author_association":"NONE","body":"machine A:user1(not root)\r\nmachine B:user2 (not root)\r\nMy data and program on machine A,then I can ssh to machine B without password.\r\nwhen I run:\r\n`mpirun -np 4 -H localhost:2,server_two:2 -bind-to none -map-by slot -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -x NCCL_IB_DISABLE=1  -mca pml ob1 python src/train.py \r\n`\r\nNothing happened.\r\nIt works fine if I only run the program on machine A. Do I need to copy the data and program to machine B？ Or there is something I have missed?"},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/443683046","html_url":"https://github.com/uber/horovod/issues/669#issuecomment-443683046","issue_url":"https://api.github.com/repos/uber/horovod/issues/669","id":443683046,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MzY4MzA0Ng==","user":{"login":"Yang-HangWA","id":38546249,"node_id":"MDQ6VXNlcjM4NTQ2MjQ5","avatar_url":"https://avatars0.githubusercontent.com/u/38546249?v=4","gravatar_id":"","url":"https://api.github.com/users/Yang-HangWA","html_url":"https://github.com/Yang-HangWA","followers_url":"https://api.github.com/users/Yang-HangWA/followers","following_url":"https://api.github.com/users/Yang-HangWA/following{/other_user}","gists_url":"https://api.github.com/users/Yang-HangWA/gists{/gist_id}","starred_url":"https://api.github.com/users/Yang-HangWA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yang-HangWA/subscriptions","organizations_url":"https://api.github.com/users/Yang-HangWA/orgs","repos_url":"https://api.github.com/users/Yang-HangWA/repos","events_url":"https://api.github.com/users/Yang-HangWA/events{/privacy}","received_events_url":"https://api.github.com/users/Yang-HangWA/received_events","type":"User","site_admin":false},"created_at":"2018-12-03T11:47:07Z","updated_at":"2018-12-03T11:47:07Z","author_association":"NONE","body":"I use another machine to test this.The command as follow:\r\n`mpirun -np 4 -H machine_A:2,machine_C:2 -bind-to none -map-by slot -x NCCL_DEBUG=ERROR -x NCCL_IB_DISABLE=1 -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib python /home/yanghang/horovod/examples/keras_mnist_advanced.py`\r\nIt seems something else wrong.\r\n`2018-12-03 19:26:34.980094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10405 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:04:00.0, compute capability: 6.1)\r\nx_train shape: (60000, 28, 28, 1)\r\n60000 train samples\r\n10000 test samples\r\nx_train shape: (60000, 28, 28, 1)\r\n60000 train samples\r\n10000 test samples\r\nEpoch 1/24\r\nEpoch 1/24\r\nEpoch 1/24\r\nEpoch 1/24\r\nUsing TensorFlow backend.\r\nUsing TensorFlow backend.\r\nTraceback (most recent call last):\r\n  File \"/home/yanghang/horovod/examples/keras_mnist_advanced.py\", line 122, in <module>\r\n    validation_steps=3 * test_batches // hvd.size())\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/legacy/interfaces.py\", line 91, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/engine/training.py\", line 1418, in fit_generator\r\n    initial_epoch=initial_epoch)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/engine/training_generator.py\", line 217, in fit_generator\r\n    class_weight=class_weight)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/engine/training.py\", line 1217, in train_on_batch\r\n    outputs = self.train_function(ins)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py\", line 2715, in __call__\r\n    return self._call(inputs)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py\", line 2675, in _call\r\n    fetched = self._callable_fn(*array_vals)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1439, in __call__\r\n    run_metadata_ptr)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py\", line 528, in __exit__\r\n    c_api.TF_GetCode(self.status.status))\r\ntensorflow.python.framework.errors_impl.FailedPreconditionError: Mismatched ALLREDUCE CPU/GPU device selection: One rank specified device GPU, but another rank specified device CPU.\r\n\t [[{{node training/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_dense_2_MatMul_grad_MatMul_1_0}} = HorovodAllreduce[T=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](training/Adadelta/gradients/dense_2/MatMul_grad/MatMul_1/_229)]]\r\nTraceback (most recent call last):\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1334, in _do_call\r\n    return fn(*args)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1319, in _run_fn\r\n    options, feed_dict, fetch_list, target_list, run_metadata)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1407, in _call_tf_sessionrun\r\n    run_metadata)\r\ntensorflow.python.framework.errors_impl.FailedPreconditionError: Mismatched ALLREDUCE CPU/GPU device selection: One rank specified device GPU, but another rank specified device CPU.\r\n\t [[{{node training/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_dense_2_MatMul_grad_MatMul_1_0}} = HorovodAllreduce[T=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](training/Adadelta/gradients/dense_2/MatMul_grad/MatMul_1)]]\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/yanghang/horovod/examples/keras_mnist_advanced.py\", line 122, in <module>\r\n    validation_steps=3 * test_batches // hvd.size())\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/legacy/interfaces.py\", line 87, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/models.py\", line 1223, in fit_generator\r\n    initial_epoch=initial_epoch)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/legacy/interfaces.py\", line 87, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/engine/training.py\", line 2096, in fit_generator\r\n    class_weight=class_weight)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/engine/training.py\", line 1814, in train_on_batch\r\n    outputs = self.train_function(ins)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py\", line 2352, in __call__\r\n    **self.session_kwargs)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 929, in run\r\n    run_metadata_ptr)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1152, in _run\r\n    feed_dict_tensor, options, run_metadata)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1328, in _do_run\r\n    run_metadata)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py\", line 1348, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors_impl.FailedPreconditionError: Mismatched ALLREDUCE CPU/GPU device selection: One rank specified device GPU, but another rank specified device CPU.\r\n\t [[node training/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_dense_2_MatMul_grad_MatMul_1_0 (defined at <string>:51)  = HorovodAllreduce[T=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](training/Adadelta/gradients/dense_2/MatMul_grad/MatMul_1)]]\r\n\r\nCaused by op 'training/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_dense_2_MatMul_grad_MatMul_1_0', defined at:\r\n  File \"/home/yanghang/horovod/examples/keras_mnist_advanced.py\", line 122, in <module>\r\n    validation_steps=3 * test_batches // hvd.size())\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/legacy/interfaces.py\", line 87, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/models.py\", line 1223, in fit_generator\r\n    initial_epoch=initial_epoch)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/legacy/interfaces.py\", line 87, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/engine/training.py\", line 1978, in fit_generator\r\n    self._make_train_function()\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/engine/training.py\", line 990, in _make_train_function\r\n    loss=self.total_loss)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/legacy/interfaces.py\", line 87, in wrapper\r\n    return func(*args, **kwargs)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/keras/optimizers.py\", line 344, in get_updates\r\n    grads = self.get_gradients(loss, params)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/horovod/_keras/__init__.py\", line 55, in get_gradients\r\n    compression=self._compression)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/horovod/tensorflow/__init__.py\", line 83, in allreduce\r\n    summed_tensor_compressed = _allreduce(tensor_compressed)\r\n  File \"/home/yanghang/anaconda3/lib/python3.5/site-packages/horovod/tensorflow/mpi_ops.py\", line 90, in _allreduce\r\n    return MPI_LIB.horovod_allreduce(tensor, name=name)\r\n  File \"<string>\", line 51, in horovod_allreduce\r\n .....\r\n` \r\nWhen I run my program to train,I met another problem.The command as follow:\r\n `mpirun -np 4 -H machine_A:2,machine_C:2 -bind-to none -map-by slot -x NCCL_DEBUG=ERROR -x NCCL_IB_DISABLE=1 -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib python /home/yanghang/temp_code/train.py`\r\nThe message not the same,the program hangs with info:\r\n`Train on 1884 data, test on 209 data\r\nTrain on 1884 data, test on 209 data\r\nTrain on 1884 data, test on 209 data\r\nTrain on 1884 data, test on 209 data\r\nEpoch 1/9\r\nEpoch 1/9\r\nEpoch 1/9\r\nEpoch 1/9\r\nWARNING: One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock. \r\nStalled ops:\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllgather_training_Adadelta_gradients_embedding_1_embedding_lookup_grad_Reshape_0 [missing ranks: 0, 1, 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllgather_training_Adadelta_gradients_embedding_1_embedding_lookup_grad_Reshape_1_0 [missing ranks: 0, 1]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllgather_training_Adadelta_gradients_embedding_1_GatherV2_grad_Reshape_0 [missing ranks: 2, 3]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_1_convolution_ExpandDims_1_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_2_convolution_ExpandDims_1_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_2_Reshape_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_1_Reshape_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_3_convolution_ExpandDims_1_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllgather_training_Adadelta_gradients_embedding_1_GatherV2_grad_Reshape_1_0 [missing ranks: 2, 3]\r\nWARNING: One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock. \r\nStalled ops:\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllgather_training_Adadelta_gradients_embedding_1_embedding_lookup_grad_Reshape_0 [missing ranks: 0, 1, 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllgather_training_Adadelta_gradients_embedding_1_embedding_lookup_grad_Reshape_1_0 [missing ranks: 0, 1]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllgather_training_Adadelta_gradients_embedding_1_GatherV2_grad_Reshape_0 [missing ranks: 2, 3]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_1_convolution_ExpandDims_1_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_2_convolution_ExpandDims_1_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_2_Reshape_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_1_Reshape_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllreduce_training_Adadelta_gradients_conv1d_3_convolution_ExpandDims_1_grad_Reshape_0 [missing ranks: 2]\r\ntraining/Adadelta/DistributedAdadelta_Allreduce/HorovodAllgather_training_Adadelta_gradients_embedding_1_GatherV2_grad_Reshape_1_0 [missing ranks: 2, 3]\r\nWARNING: One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock. \r\n`\r\n"}}
{"actor":{"display_login":"Yang-HangWA"}}