{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/1623","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/1623/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/1623/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/1623/events","html_url":"https://github.com/dmlc/mxnet/issues/1623","id":140142621,"number":1623,"title":"Distributed training over Infiniband","user":{"login":"vcodreanu","id":7404928,"avatar_url":"https://avatars.githubusercontent.com/u/7404928?v=3","gravatar_id":"","url":"https://api.github.com/users/vcodreanu","html_url":"https://github.com/vcodreanu","followers_url":"https://api.github.com/users/vcodreanu/followers","following_url":"https://api.github.com/users/vcodreanu/following{/other_user}","gists_url":"https://api.github.com/users/vcodreanu/gists{/gist_id}","starred_url":"https://api.github.com/users/vcodreanu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vcodreanu/subscriptions","organizations_url":"https://api.github.com/users/vcodreanu/orgs","repos_url":"https://api.github.com/users/vcodreanu/repos","events_url":"https://api.github.com/users/vcodreanu/events{/privacy}","received_events_url":"https://api.github.com/users/vcodreanu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-03-11T09:57:56Z","updated_at":"2016-03-22T23:45:02Z","closed_at":null,"body":"I am experiencing some issues when using distributed training in combination with DMLC_INTERFACE=\"ib0\". I have successfully trained many models using the default DMLC_INTERFACE (eth0), but I've hit some bandwidth limits on some large models and thus tried the Infiniband option. \r\n\r\nI am using the dmlc_mpi launcher and the output is as follows:\r\n\r\nWith DMLC_INTERFACE not set mxnet starts successfully:\r\n\r\n```\r\nCurrently Loaded Modulefiles:\r\n 1) cudnn/7.0-v4-prod            4) cuda/7.5.18                 \r\n 2) python/2.7.9                 5) opencv/gnu/2.4.10           \r\n 3) mxnet/2016.02.23             6) mpi/mvapich2-gdr/2.1-cuda75 \r\nmpirun -n 4  -env DMLC_ROLE server -env DMLC_PS_ROOT_PORT 9894 -env DMLC_PS_ROOT_URI 10.3.200.83 -env DMLC_NUM_SERVER 4 -env DMLC_NUM_WORKER 4 --hostfile /home/valeriuc/work/mxnet/example/image-classification/hosts python /home/valeriuc/work/mxnet/example/image-classification/train_imagenet_full.py --batch-size 96 --lr 0.05 --lr-factor .94 --gpus 0,1 --kv-store dist_sync --data-dir /projects/2/managed_datasets/imagenet-full --network inception-v3-full --model-prefix model/ilsvrc21k-8n\r\nmpirun -n 4  -env DMLC_ROLE worker -env DMLC_PS_ROOT_PORT 9894 -env DMLC_PS_ROOT_URI 10.3.200.83 -env DMLC_NUM_SERVER 4 -env DMLC_NUM_WORKER 4 --hostfile /home/valeriuc/work/mxnet/example/image-classification/hosts python /home/valeriuc/work/mxnet/example/image-classification/train_imagenet_full.py --batch-size 96 --lr 0.05 --lr-factor .94 --gpus 0,1 --kv-store dist_sync --data-dir /projects/2/managed_datasets/imagenet-full --network inception-v3-full --model-prefix model/ilsvrc21k-8n\r\n2016-03-11 10:48:16,220 Node[1] start with arguments Namespace(batch_size=96, clip_gradient=5.0, data_dir='/projects/2/managed_datasets/imagenet-full', data_shape=299, gpus='0,1', kv_store='dist_sync', load_epoch=None, log_dir='/tmp/', log_file=None, lr=0.05, lr_factor=0.94, lr_factor_epoch=1, model_prefix='model/ilsvrc21k-8n', network='inception-v3-full', num_classes=21841, num_epochs=20, num_examples=14192019, train_dataset='train.rec', val_dataset='val.rec')\r\n2016-03-11 10:48:16,221 Node[3] start with arguments Namespace(batch_size=96, clip_gradient=5.0, data_dir='/projects/2/managed_datasets/imagenet-full', data_shape=299, gpus='0,1', kv_store='dist_sync', load_epoch=None, log_dir='/tmp/', log_file=None, lr=0.05, lr_factor=0.94, lr_factor_epoch=1, model_prefix='model/ilsvrc21k-8n', network='inception-v3-full', num_classes=21841, num_epochs=20, num_examples=14192019, train_dataset='train.rec', val_dataset='val.rec')\r\n2016-03-11 10:48:16,220 Node[2] start with arguments Namespace(batch_size=96, clip_gradient=5.0, data_dir='/projects/2/managed_datasets/imagenet-full', data_shape=299, gpus='0,1', kv_store='dist_sync', load_epoch=None, log_dir='/tmp/', log_file=None, lr=0.05, lr_factor=0.94, lr_factor_epoch=1, model_prefix='model/ilsvrc21k-8n', network='inception-v3-full', num_classes=21841, num_epochs=20, num_examples=14192019, train_dataset='train.rec', val_dataset='val.rec')\r\n2016-03-11 10:48:16,221 Node[0] start with arguments Namespace(batch_size=96, clip_gradient=5.0, data_dir='/projects/2/managed_datasets/imagenet-full', data_shape=299, gpus='0,1', kv_store='dist_sync', load_epoch=None, log_dir='/tmp/', log_file=None, lr=0.05, lr_factor=0.94, lr_factor_epoch=1, model_prefix='model/ilsvrc21k-8n', network='inception-v3-full', num_classes=21841, num_epochs=20, num_examples=14192019, train_dataset='train.rec', val_dataset='val.rec')\r\n[10:48:16] src/io/iter_image_recordio.cc[10:48:16] src/io/iter_image_recordio.cc:[10:48:16] src/io/iter_image_recordio.cc::212: ImageRecordIOParser: /projects/2/managed_datasets/imagenet-full/train.rec, use 1 threads for decoding..\r\n212: ImageRecordIOParser: /projects/2/managed_datasets/imagenet-full/train.rec, use 1 threads for decoding..\r\n212: ImageRecordIOParser: /projects/2/managed_datasets/imagenet-full/train.rec, use 1 threads for decoding..\r\n[10:48:16] src/io/iter_image_recordio.cc:212: ImageRecordIOParser: /projects/2/managed_datasets/imagenet-full/train.rec, use 1 threads for decoding..\r\n[10:48:17] src/io/iter_image_recordio.cc:212: ImageRecordIOParser: /projects/2/managed_datasets/imagenet-full/val.rec, use 1 threads for decoding..\r\n[10:48:17] src/io/iter_image_recordio.cc:212: ImageRecordIOParser: /projects/2/managed_datasets/imagenet-full/val.rec, use 1 threads for decoding..\r\n[10:48:17] src/io/iter_image_recordio.cc:212: ImageRecordIOParser: /projects/2/managed_datasets/imagenet-full/val.rec, use 1 threads for decoding..\r\n[10:48:17] src/io/iter_image_recordio.cc:212: ImageRecordIOParser: /projects/2/managed_datasets/imagenet-full/val.rec, use 1 threads for decoding..\r\n2016-03-11 10:48:18,321 Node[1] Start training with [gpu(0), gpu(1)]\r\n2016-03-11 10:48:18,351 Node[2] Start training with [gpu(0), gpu(1)]\r\n2016-03-11 10:48:18,359 Node[3] Start training with [gpu(0), gpu(1)]\r\n2016-03-11 10:48:18,396 Node[0] Start training with [gpu(0), gpu(1)]\r\n```\r\n\r\n\r\nWith DMLC_INTERFACE=\"ib0\" mxnet freezes:\r\n\r\n```\r\nCurrently Loaded Modulefiles:\r\n 1) cudnn/7.0-v4-prod            4) cuda/7.5.18                 \r\n 2) python/2.7.9                 5) opencv/gnu/2.4.10           \r\n 3) mxnet/2016.02.23             6) mpi/mvapich2-gdr/2.1-cuda75 \r\nmpirun -n 4  -env DMLC_ROLE server -env DMLC_PS_ROOT_PORT 9986 -env DMLC_PS_ROOT_URI 10.3.200.83 -env DMLC_NUM_SERVER 4 -env DMLC_NUM_WORKER 4 --hostfile /home/valeriuc/work/mxnet/example/image-classification/hosts python /home/valeriuc/work/mxnet/example/image-classification/train_imagenet_full.py --batch-size 96 --lr 0.05 --lr-factor .94 --gpus 0,1 --kv-store dist_sync --data-dir /projects/2/managed_datasets/imagenet-full --network inception-v3-full --model-prefix model/ilsvrc21k-8n\r\nmpirun -n 4  -env DMLC_ROLE worker -env DMLC_PS_ROOT_PORT 9986 -env DMLC_PS_ROOT_URI 10.3.200.83 -env DMLC_NUM_SERVER 4 -env DMLC_NUM_WORKER 4 --hostfile /home/valeriuc/work/mxnet/example/image-classification/hosts python /home/valeriuc/work/mxnet/example/image-classification/train_imagenet_full.py --batch-size 96 --lr 0.05 --lr-factor .94 --gpus 0,1 --kv-store dist_sync --data-dir /projects/2/managed_datasets/imagenet-full --network inception-v3-full --model-prefix model/ilsvrc21k-8n\r\n```\r\n\r\nAnd on the computing nodes I see processes like:\r\n```\r\n/hpc/sw/mvapich2-gdr-2.1-cuda-7.5-intel/bin/hydra_pmi_proxy --control-port gcn2:51087 --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0\r\nvaleriuc  68642  68543  1 10:45 ?        00:00:00           python /home/valeriuc/work/mxnet/example/image-classification/train_imagenet_full.py --batch-size 96 --lr 0.05 --lr-factor .94 --gpus 0,1 --kv-store dist_sync --data-dir /projects/2/managed_datasets/imagenet-full --network inception-v3-full --model-prefix model/ilsvrc21k-8n\r\n```\r\n\r\nWhen I Ctrl-C the process I get:\r\n```\r\nPress Ctrl-C again to force abort\r\nTraceback (most recent call last):\r\n  File \"dmlc_slurm.py\", line 92, in <module>\r\n    pscmd=(' '.join(args.command) + ' ' + ' '.join(unknown)))\r\n  File \"/hpc/sw/mxnet-2016.02.23/tracker/tracker.py\", line 424, in submit\r\n    pserver.join()\r\n  File \"/hpc/sw/mxnet-2016.02.23/tracker/tracker.py\", line 358, in join\r\n    self.thread.join(100)\r\n  File \"/hpc/sw/python-2.7.9/lib/python2.7/threading.py\", line 960, in join\r\n    self.__block.wait(delay)\r\n  File \"/hpc/sw/python-2.7.9/lib/python2.7/threading.py\", line 359, in wait\r\n    _sleep(delay)\r\nKeyboardInterrupt\r\n```\r\n\r\nOur system has multiple NICs (ib0 and ib1) and I get the same behavior when setting any of them. Also, it happens in the case of both kv-stores: dist_sync and dist_async.\r\n\r\nCould you advise me on what to try? Or what can I do to add get more verbose debug information from mxnet?"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/200081731","html_url":"https://github.com/dmlc/mxnet/issues/1623#issuecomment-200081731","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/1623","id":200081731,"user":{"login":"mli","id":421857,"avatar_url":"https://avatars.githubusercontent.com/u/421857?v=3","gravatar_id":"","url":"https://api.github.com/users/mli","html_url":"https://github.com/mli","followers_url":"https://api.github.com/users/mli/followers","following_url":"https://api.github.com/users/mli/following{/other_user}","gists_url":"https://api.github.com/users/mli/gists{/gist_id}","starred_url":"https://api.github.com/users/mli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mli/subscriptions","organizations_url":"https://api.github.com/users/mli/orgs","repos_url":"https://api.github.com/users/mli/repos","events_url":"https://api.github.com/users/mli/events{/privacy}","received_events_url":"https://api.github.com/users/mli/received_events","type":"User","site_admin":false},"created_at":"2016-03-22T23:45:02Z","updated_at":"2016-03-22T23:45:02Z","body":"the problem seems that the scheduler, which uses eth0, is failed to send\ndata to another machine's ib0 interface.\n\ncan you try the `--host-ip` way? namely let the scheduler also use the ib0\ninterface.\n\nOn Tue, Mar 22, 2016 at 7:33 PM, vcodreanu <notifications@github.com> wrote:\n\n> yes, it seems that this is the case. I started the job directly from a\n> compute node (before I was starting it from a different \"scheduler\" node)\n> and it goes a bit forward ahead. But it goes only on the server/worker\n> placed on the node that launches the job (10.3.200.92 in this case).\n>\n> [00:27:41] src/van.cc:76: Bind to role=schedulerid=1, ip=10.3.200.92,\n> port=9876\n> [00:27:41] src/van.cc:76: Bind to role=server, ip=10.202.203.92, port=40666\n> [00:27:41] src/van.cc:226: S => 1: Meta: request=0, push=0, simple_app=0,\n> control={ cmd=ADD_NODE, node={ role=server, ip=10.202.203.92, port=40666 } }\n> [00:27:41] src/van.cc:291: H[1] <= 2147483647: Meta: request=0, push=0,\n> simple_app=0, control={ cmd=ADD_NODE, node={ role=server, ip=10.202.203.92,\n> port=40666 } }\n> [00:27:41] src/van.cc:76: Bind to role=worker, ip=10.202.203.92, port=38127\n> [00:27:41] src/van.cc:226: W => 1: Meta: request=0, push=0, simple_app=0,\n> control={ cmd=ADD_NODE, node={ role=worker, ip=10.202.203.92, port=38127 } }\n> [00:27:41] src/van.cc:291: H[1] <= 2147483647: Meta: request=0, push=0,\n> simple_app=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.202.203.92,\n> port=38127 } }\n> [00:27:42] src/van.cc:76: Bind to role=server, ip=10.202.203.93, port=49706\n> [00:27:42] src/van.cc:226: S => 1: Meta: request=0, push=0, simple_app=0,\n> control={ cmd=ADD_NODE, node={ role=server, ip=10.202.203.93, port=49706 } }\n> [00:27:42] src/van.cc:291: H[1] <= 2147483647: Meta: request=0, push=0,\n> simple_app=0, control={ cmd=ADD_NODE, node={ role=server, ip=10.202.203.93,\n> port=49706 } }\n> [00:27:42] src/van.cc:76: Bind to role=worker, ip=10.202.203.93, port=58320\n> [00:27:42] src/van.cc:226: W => 1: Meta: request=0, push=0, simple_app=0,\n> control={ cmd=ADD_NODE, node={ role=worker, ip=10.202.203.93, port=58320 } }\n> [00:27:42] src/van.cc:291: H[1] <= 2147483647: Meta: request=0, push=0,\n> simple_app=0, control={ cmd=ADD_NODE, node={ role=worker, ip=10.202.203.93,\n> port=58320 } }\n> [00:27:42] src/van.cc:348: assign rank=8 to node role=server,\n> ip=10.202.203.93, port=49706\n> [00:27:42] src/van.cc:348: assign rank=9 to node role=worker,\n> ip=10.202.203.93, port=58320\n> [00:27:42] src/van.cc:348: assign rank=11 to node role=worker,\n> ip=10.202.203.92, port=38127\n> [00:27:42] src/van.cc:348: assign rank=10 to node role=server,\n> ip=10.202.203.92, port=40666\n> [00:27:42] src/van.cc:226: H[1] => 9: Meta: request=0, push=0,\n> simple_app=0, control={ cmd=ADD_NODE, node={ role=serverid=8,\n> ip=10.202.203.93, port=49706 role=workerid=9, ip=10.202.203.93, port=58320\n> role=workerid=11, ip=10.202.203.92, port=38127 role=serverid=10,\n> ip=10.202.203.92, port=40666 role=schedulerid=1, ip=10.3.200.92, port=9876\n> } }\n> [00:27:42] src/van.cc:226: H[1] => 11: Meta: request=0, push=0,\n> simple_app=0, control={ cmd=ADD_NODE, node={ role=serverid=8,\n> ip=10.202.203.93, port=49706 role=workerid=9, ip=10.202.203.93, port=58320\n> role=workerid=11, ip=10.202.203.92, port=38127 role=serverid=10,\n> ip=10.202.203.92, port=40666 role=schedulerid=1, ip=10.3.200.92, port=9876\n> } }\n> [00:27:42] src/van.cc:226: H[1] => 8: Meta: request=0, push=0,\n> simple_app=0, control={ cmd=ADD_NODE, node={ role=serverid=8,\n> ip=10.202.203.93, port=49706 role=workerid=9, ip=10.202.203.93, port=58320\n> role=workerid=11, ip=10.202.203.92, port=38127 role=serverid=10,\n> ip=10.202.203.92, port=40666 role=schedulerid=1, ip=10.3.200.92, port=9876\n> } }\n> [00:27:42] src/van.cc:226: H[1] => 10: Meta: request=0, push=0,\n> simple_app=0, control={ cmd=ADD_NODE, node={ role=serverid=8,\n> ip=10.202.203.93, port=49706 role=workerid=9, ip=10.202.203.93, port=58320\n> role=workerid=11, ip=10.202.203.92, port=38127 role=serverid=10,\n> ip=10.202.203.92, port=40666 role=schedulerid=1, ip=10.3.200.92, port=9876\n> } }\n> [00:27:42] src/van.cc:359: the scheduler is connected to 2 workers and 2\n> servers\n> [00:27:42] src/van.cc:291: W <= 1: Meta: request=0, push=0, simple_app=0,\n> control={ cmd=ADD_NODE, node={ role=serverid=8, ip=10.202.203.93,\n> port=49706 role=workerid=9, ip=10.202.203.93, port=58320 role=workerid=11,\n> ip=10.202.203.92, port=38127 role=serverid=10, ip=10.202.203.92, port=40666\n> role=schedulerid=1, ip=10.3.200.92, port=9876 } }\n> [00:27:42] src/van.cc:291: S <= 1: Meta: request=0, push=0, simple_app=0,\n> control={ cmd=ADD_NODE, node={ role=serverid=8, ip=10.202.203.93,\n> port=49706 role=workerid=9, ip=10.202.203.93, port=58320 role=workerid=11,\n> ip=10.202.203.92, port=38127 role=serverid=10, ip=10.202.203.92, port=40666\n> role=schedulerid=1, ip=10.3.200.92, port=9876 } }\n> [00:27:42] src/van.cc:366: W[11] is connected to others\n> [00:27:42] src/van.cc:366: S[10] is connected to others\n> [00:27:42] src/van.cc:226: S[10] => 1: Meta: request=1, push=42,\n> simple_app=0, control={ cmd=BARRIER, barrier_group=7 }\n> [00:27:42] src/van.cc:226: W[11] => 1: Meta: request=1, push=0,\n> simple_app=0, control={ cmd=BARRIER, barrier_group=7 }\n> [00:27:42] src/van.cc:226: H[1] => 1: Meta: request=1, push=42,\n> simple_app=0, control={ cmd=BARRIER, barrier_group=7 }\n> [00:27:42] src/van.cc:291: H[1] <= 1: Meta: request=1, push=0,\n> simple_app=0, control={ cmd=BARRIER, barrier_group=7 }\n> [00:27:42] src/van.cc:291: H[1] <= 10: Meta: request=1, push=0,\n> simple_app=0, control={ cmd=BARRIER, barrier_group=7 }\n> [00:27:42] src/van.cc:291: H[1] <= 11: Meta: request=1, push=0,\n> simple_app=0, control={ cmd=BARRIER, barrier_group=7 }\n>\n> but it seems strange...we have many mpi programs running over ib0. any\n> ideas?\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly or view it on GitHub\n> <https://github.com/dmlc/mxnet/issues/1623#issuecomment-200079309>\n>\n"}}