{"action":"created","issue":{"url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/58","repository_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark","labels_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/58/labels{/name}","comments_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/58/comments","events_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/58/events","html_url":"https://github.com/facebookresearch/maskrcnn-benchmark/issues/58","id":375109867,"node_id":"MDU6SXNzdWUzNzUxMDk4Njc=","number":58,"title":"Problem with multi-GPU training","user":{"login":"robpal1990","id":24491417,"node_id":"MDQ6VXNlcjI0NDkxNDE3","avatar_url":"https://avatars0.githubusercontent.com/u/24491417?v=4","gravatar_id":"","url":"https://api.github.com/users/robpal1990","html_url":"https://github.com/robpal1990","followers_url":"https://api.github.com/users/robpal1990/followers","following_url":"https://api.github.com/users/robpal1990/following{/other_user}","gists_url":"https://api.github.com/users/robpal1990/gists{/gist_id}","starred_url":"https://api.github.com/users/robpal1990/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robpal1990/subscriptions","organizations_url":"https://api.github.com/users/robpal1990/orgs","repos_url":"https://api.github.com/users/robpal1990/repos","events_url":"https://api.github.com/users/robpal1990/events{/privacy}","received_events_url":"https://api.github.com/users/robpal1990/received_events","type":"User","site_admin":false},"labels":[{"id":1108323488,"node_id":"MDU6TGFiZWwxMTA4MzIzNDg4","url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/labels/dependency%20bug","name":"dependency bug","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2018-10-29T16:40:14Z","updated_at":"2018-11-22T10:49:59Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\n\r\nI have successfully built maskrcnn_benchmark on Ubuntu 16.04. My workstation has 4x1080Ti (CUDA 9.2, cuDNN 7, Nvidia drivers 410.48) and I tried to train on COCO dataset on multiple GPUs. I used the script provided in \"Perform training on COCO dataset\" section.\r\n\r\nOne GPU worked fine with: \r\n```\r\npython tools/train_net.py --config-file \"configs/e2e_mask_rcnn_R_50_FPN_1x.yaml\" SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 720000 SOLVER.STEPS \"(480000, 640000)\" TEST.IMS_PER_BATCH 1\r\n```\r\n\r\nThen I used \r\n\r\n```\r\nexport NGPUS=2\r\npython -m torch.distributed.launch --nproc_per_node=$NGPUS tools/train_net.py --config-file \"configs/e2e_mask_rcnn_R_50_FPN_1x.yaml\" SOLVER.IMS_PER_BATCH 4 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 720000 SOLVER.STEPS \"(480000, 640000)\" TEST.IMS_PER_BATCH 1\r\n```\r\n(the same train_net.py file and the same config, changed images per batch to 4) and everything worked fine.\r\n\r\nNext I tried the same thing for 3 GPUs (changed NGPUS=3, images per batch to 6) and the training gets stuck during the first interation. I have the following logging information and it does not change:\r\n\r\n```\r\n2018-10-29 17:20:55,722 maskrcnn_benchmark.trainer INFO: Start training\r\n2018-10-29 17:20:58,453 maskrcnn_benchmark.trainer INFO: eta: 22 days, 18:04:18  iter: 0  loss: 6.7175 (6.7175)  loss_classifier: 4.4688 (4.4688)  loss_box_reg: 0.0044 (0.0044)  loss_mask: 1.4084 (1.4084)  loss_objectness: 0.7262 (0.7262)  loss_rpn_box_reg: 0.1097 (0.1097)  time: 2.7304 (2.7304)  data: 2.4296 (2.4296)  lr: 0.000833  max mem: 1749\r\n```\r\nThe GPU memory is used, the temperature goes up, but nothing is happening (I tried multiple times and then gave up).\r\n\r\nAny ideas? I'd be grateful for help."},"comment":{"url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/comments/440991278","html_url":"https://github.com/facebookresearch/maskrcnn-benchmark/issues/58#issuecomment-440991278","issue_url":"https://api.github.com/repos/facebookresearch/maskrcnn-benchmark/issues/58","id":440991278,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MDk5MTI3OA==","user":{"login":"zimenglan-sysu-512","id":13795888,"node_id":"MDQ6VXNlcjEzNzk1ODg4","avatar_url":"https://avatars3.githubusercontent.com/u/13795888?v=4","gravatar_id":"","url":"https://api.github.com/users/zimenglan-sysu-512","html_url":"https://github.com/zimenglan-sysu-512","followers_url":"https://api.github.com/users/zimenglan-sysu-512/followers","following_url":"https://api.github.com/users/zimenglan-sysu-512/following{/other_user}","gists_url":"https://api.github.com/users/zimenglan-sysu-512/gists{/gist_id}","starred_url":"https://api.github.com/users/zimenglan-sysu-512/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zimenglan-sysu-512/subscriptions","organizations_url":"https://api.github.com/users/zimenglan-sysu-512/orgs","repos_url":"https://api.github.com/users/zimenglan-sysu-512/repos","events_url":"https://api.github.com/users/zimenglan-sysu-512/events{/privacy}","received_events_url":"https://api.github.com/users/zimenglan-sysu-512/received_events","type":"User","site_admin":false},"created_at":"2018-11-22T10:49:59Z","updated_at":"2018-11-22T10:49:59Z","author_association":"NONE","body":"hi @chengyangfu  @fmassa \r\ni find that after installing the new version of nccl, the hang doesn't occur. it seems to be solved."}}
{"actor":{"display_login":"zimenglan-sysu-512"}}