{"action":"created","issue":{"url":"https://api.github.com/repos/espnet/espnet/issues/279","repository_url":"https://api.github.com/repos/espnet/espnet","labels_url":"https://api.github.com/repos/espnet/espnet/issues/279/labels{/name}","comments_url":"https://api.github.com/repos/espnet/espnet/issues/279/comments","events_url":"https://api.github.com/repos/espnet/espnet/issues/279/events","html_url":"https://github.com/espnet/espnet/issues/279","id":338843233,"node_id":"MDU6SXNzdWUzMzg4NDMyMzM=","number":279,"title":"Multi Gpus with pytorch backend problem.","user":{"login":"ZhuoranLyu","id":8801452,"node_id":"MDQ6VXNlcjg4MDE0NTI=","avatar_url":"https://avatars0.githubusercontent.com/u/8801452?v=4","gravatar_id":"","url":"https://api.github.com/users/ZhuoranLyu","html_url":"https://github.com/ZhuoranLyu","followers_url":"https://api.github.com/users/ZhuoranLyu/followers","following_url":"https://api.github.com/users/ZhuoranLyu/following{/other_user}","gists_url":"https://api.github.com/users/ZhuoranLyu/gists{/gist_id}","starred_url":"https://api.github.com/users/ZhuoranLyu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ZhuoranLyu/subscriptions","organizations_url":"https://api.github.com/users/ZhuoranLyu/orgs","repos_url":"https://api.github.com/users/ZhuoranLyu/repos","events_url":"https://api.github.com/users/ZhuoranLyu/events{/privacy}","received_events_url":"https://api.github.com/users/ZhuoranLyu/received_events","type":"User","site_admin":false},"labels":[{"id":777173273,"node_id":"MDU6TGFiZWw3NzcxNzMyNzM=","url":"https://api.github.com/repos/espnet/espnet/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-07-06T08:02:54Z","updated_at":"2018-07-12T17:01:54Z","closed_at":null,"author_association":"NONE","body":"I am using Pytorch 0.4 with 4 GTX 1080Ti. When I run using pytorch backend and multi gpus, it gives me this error.\r\n\r\n`# asr_train.py --ngpu 4 --backend pytorch --outdir exp/tr_en_vggblstmp_e4_subsample1_2_2_1_1_unit320_proj320_d1_unit300_location_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs30_mli800_mlo150/results --debugmode 1 --dict data/lang_1char/tr_en_units.txt --debugdir exp/tr_en_vggblstmp_e4_subsample1_2_2_1_1_unit320_proj320_d1_unit300_location_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs30_mli800_mlo150 --minibatches 0 --verbose 0 --resume --train-json dump/tr_en/deltafalse/data.json --valid-json dump/dt_en/deltafalse/data.json --etype vggblstmp --elayers 4 --eunits 320 --eprojs 320 --subsample 1_2_2_1_1 --dlayers 1 --dunits 300 --atype location --aconv-chans 10 --aconv-filts 100 --mtlalpha 0.5 --batch-size 30 --maxlen-in 800 --maxlen-out 150 --opt adadelta --epochs 100 \r\n# Started at Fri Jul  6 10:05:31 CST 2018\r\n#\r\n2018-07-06 10:05:31,582 (asr_train:146) WARNING: Skip DEBUG/INFO messages\r\n2018-07-06 10:05:31,587 (asr_train:186) WARNING: CUDA_VISIBLE_DEVICES is not set.\r\n2018-07-06 10:05:35,803 (e2e_asr_attctc_th:198) WARNING: Subsampling is not performed for vgg*. It is performed in max pooling layers at CNN.\r\nException in main training loop: torch/csrc/autograd/variable.cpp:115: get_grad_fn: Assertion `output_nr == 0` failed.\r\nTraceback (most recent call last):\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/chainer/training/trainer.py\", line 306, in run\r\n    update()\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/chainer/training/updaters/standard_updater.py\", line 149, in update\r\n    self.update_core()\r\n  File \"/home/lvzhuoran/code/espnet-master/src/asr/asr_pytorch.py\", line 122, in update_core\r\n    loss = 1. / self.num_gpu * self.model(x)\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py\", line 491, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/lvzhuoran/code/espnet-master/src/asr/asr_pytorch.py\", line 168, in forward\r\n    outputs = self.parallel_apply(replicas, inputs, kwargs)\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py\", line 124, in parallel_apply\r\n    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py\", line 65, in parallel_apply\r\n    raise output\r\nWill finalize trainer extensions and updater before reraising the exception.\r\n\u001B[JTraceback (most recent call last):\r\n  File \"/home/lvzhuoran/code/espnet-master/egs/voxforge/asr1/../../../src/bin/asr_train.py\", line 224, in <module>\r\n    main()\r\n  File \"/home/lvzhuoran/code/espnet-master/egs/voxforge/asr1/../../../src/bin/asr_train.py\", line 218, in main\r\n    train(args)\r\n  File \"/home/lvzhuoran/code/espnet-master/src/asr/asr_pytorch.py\", line 377, in train\r\n    trainer.run()\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/chainer/training/trainer.py\", line 320, in run\r\n    six.reraise(*sys.exc_info())\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/chainer/training/trainer.py\", line 306, in run\r\n    update()\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/chainer/training/updaters/standard_updater.py\", line 149, in update\r\n    self.update_core()\r\n  File \"/home/lvzhuoran/code/espnet-master/src/asr/asr_pytorch.py\", line 122, in update_core\r\n    loss = 1. / self.num_gpu * self.model(x)\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py\", line 491, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/home/lvzhuoran/code/espnet-master/src/asr/asr_pytorch.py\", line 168, in forward\r\n    outputs = self.parallel_apply(replicas, inputs, kwargs)\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py\", line 124, in parallel_apply\r\n    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])\r\n  File \"/home/lvzhuoran/code/espnet-master/tools/venv/local/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py\", line 65, in parallel_apply\r\n    raise output\r\nRuntimeError: torch/csrc/autograd/variable.cpp:115: get_grad_fn: Assertion `output_nr == 0` failed.`\r\n\r\n\r\nI am using Pytorch 0.4. I googled it and found [this](https://github.com/pytorch/pytorch/issues/7092) link to be useful. \r\n\r\nThanks,\r\nGeorge.\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/espnet/espnet/issues/comments/404581183","html_url":"https://github.com/espnet/espnet/issues/279#issuecomment-404581183","issue_url":"https://api.github.com/repos/espnet/espnet/issues/279","id":404581183,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDU4MTE4Mw==","user":{"login":"sw005320","id":11741550,"node_id":"MDQ6VXNlcjExNzQxNTUw","avatar_url":"https://avatars0.githubusercontent.com/u/11741550?v=4","gravatar_id":"","url":"https://api.github.com/users/sw005320","html_url":"https://github.com/sw005320","followers_url":"https://api.github.com/users/sw005320/followers","following_url":"https://api.github.com/users/sw005320/following{/other_user}","gists_url":"https://api.github.com/users/sw005320/gists{/gist_id}","starred_url":"https://api.github.com/users/sw005320/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sw005320/subscriptions","organizations_url":"https://api.github.com/users/sw005320/orgs","repos_url":"https://api.github.com/users/sw005320/repos","events_url":"https://api.github.com/users/sw005320/events{/privacy}","received_events_url":"https://api.github.com/users/sw005320/received_events","type":"User","site_admin":false},"created_at":"2018-07-12T17:01:54Z","updated_at":"2018-07-12T17:01:54Z","author_association":"CONTRIBUTOR","body":"@kan-bayashi @bobchennan, do you have any ideas?\r\n(I don't have two GPU slots now to test it....)"}}
{"actor":{"display_login":"sw005320"}}