{"action":"created","issue":{"url":"https://api.github.com/repos/espnet/espnet/issues/419","repository_url":"https://api.github.com/repos/espnet/espnet","labels_url":"https://api.github.com/repos/espnet/espnet/issues/419/labels{/name}","comments_url":"https://api.github.com/repos/espnet/espnet/issues/419/comments","events_url":"https://api.github.com/repos/espnet/espnet/issues/419/events","html_url":"https://github.com/espnet/espnet/issues/419","id":362917892,"node_id":"MDU6SXNzdWUzNjI5MTc4OTI=","number":419,"title":"error when using ngpu=8","user":{"login":"chenzhehuai","id":12264312,"node_id":"MDQ6VXNlcjEyMjY0MzEy","avatar_url":"https://avatars3.githubusercontent.com/u/12264312?v=4","gravatar_id":"","url":"https://api.github.com/users/chenzhehuai","html_url":"https://github.com/chenzhehuai","followers_url":"https://api.github.com/users/chenzhehuai/followers","following_url":"https://api.github.com/users/chenzhehuai/following{/other_user}","gists_url":"https://api.github.com/users/chenzhehuai/gists{/gist_id}","starred_url":"https://api.github.com/users/chenzhehuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenzhehuai/subscriptions","organizations_url":"https://api.github.com/users/chenzhehuai/orgs","repos_url":"https://api.github.com/users/chenzhehuai/repos","events_url":"https://api.github.com/users/chenzhehuai/events{/privacy}","received_events_url":"https://api.github.com/users/chenzhehuai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-09-23T06:28:02Z","updated_at":"2018-09-24T06:50:07Z","closed_at":null,"author_association":"NONE","body":"version:\r\n84f72a4950a23ee92e3a554ccc4960e565e5939e\r\n\r\ncommand (in librispeech):\r\nasr_train.py --ngpu 8 --backend pytorch --outdir exp/train_100_blstmp_e8_subsample1_2_2_1_1_unit320_proj320_d1_unit300_location_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs20_mli800_mlo150.2c/results --debugmode 1 --dict data/lang_1char/train_100_units.txt --debugdir exp/train_100_blstmp_e8_subsample1_2_2_1_1_unit320_proj320_d1_unit300_location_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs20_mli800_mlo150.2c --minibatches 0 --verbose 5 --resume --train-json dump/train_100/deltafalse/data.json --valid-json dump/dev/deltafalse/data.json --etype blstmp --elayers 8 --eunits 320 --eprojs 320 --subsample 1_2_2_1_1 --dlayers 1 --dunits 300 --atype location --aconv-chans 10 --aconv-filts 100 --mtlalpha 0.5 --batch-size 5 --maxlen-in 800 --maxlen-out 150 --opt adadelta --epochs 15\r\n\r\nerror log:\r\nException in main training loop: invalid gradient at index 0 - expected shape [7] but got [8]\r\nTraceback (most recent call last):\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/chainer/training/trainer.py\", line 306, in run\r\n    update()\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/chainer/training/updaters/standard_updater.py\", line 149, in update\r\n    self.update_core()\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/src/asr/asr_pytorch.py\", line 123, in update_core\r\n    loss.backward(loss.new_ones(self.ngpu))  # Backprop\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/torch/tensor.py\", line 93, in backward\r\n    torch.autograd.backward(self, gradient, retain_graph, create_graph)\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/torch/autograd/__init__.py\", line 90, in backward\r\n    allow_unreachable=True)  # allow_unreachable flag\r\nWill finalize trainer extensions and updater before reraising the exception.\r\nTraceback (most recent call last):\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/egs/partlibrispeech/asr1/../../../codes/espnet-work//src/bin/asr_train.py\", line 193, in <module>\r\n    main()\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/egs/partlibrispeech/asr1/../../../codes/espnet-work//src/bin/asr_train.py\", line 187, in main\r\n    train(args)\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/src/asr/asr_pytorch.py\", line 353, in train\r\n    trainer.run()\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/chainer/training/trainer.py\", line 320, in run\r\n    six.reraise(*sys.exc_info())\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/six.py\", line 693, in reraise\r\n    raise value\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/chainer/training/trainer.py\", line 306, in run\r\n    update()\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/chainer/training/updaters/standard_updater.py\", line 149, in update\r\n    self.update_core()\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/src/asr/asr_pytorch.py\", line 123, in update_core\r\n    loss.backward(loss.new_ones(self.ngpu))  # Backprop\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/torch/tensor.py\", line 93, in backward\r\n    torch.autograd.backward(self, gradient, retain_graph, create_graph)\r\n  File \"/mnt/homedir/chenzhehuai/works/e2e/codes/espnet-git/tools/venv/lib/python3.7/site-packages/torch/autograd/__init__.py\", line 90, in backward\r\n    allow_unreachable=True)  # allow_unreachable flag\r\nRuntimeError: invalid gradient at index 0 - expected shape [7] but got [8]\r\n\r\nnote:\r\nit always exists. However, using different sizes of batch will change the number of iterations before coming into this error"},"comment":{"url":"https://api.github.com/repos/espnet/espnet/issues/comments/423888479","html_url":"https://github.com/espnet/espnet/issues/419#issuecomment-423888479","issue_url":"https://api.github.com/repos/espnet/espnet/issues/419","id":423888479,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzg4ODQ3OQ==","user":{"login":"chenzhehuai","id":12264312,"node_id":"MDQ6VXNlcjEyMjY0MzEy","avatar_url":"https://avatars3.githubusercontent.com/u/12264312?v=4","gravatar_id":"","url":"https://api.github.com/users/chenzhehuai","html_url":"https://github.com/chenzhehuai","followers_url":"https://api.github.com/users/chenzhehuai/followers","following_url":"https://api.github.com/users/chenzhehuai/following{/other_user}","gists_url":"https://api.github.com/users/chenzhehuai/gists{/gist_id}","starred_url":"https://api.github.com/users/chenzhehuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenzhehuai/subscriptions","organizations_url":"https://api.github.com/users/chenzhehuai/orgs","repos_url":"https://api.github.com/users/chenzhehuai/repos","events_url":"https://api.github.com/users/chenzhehuai/events{/privacy}","received_events_url":"https://api.github.com/users/chenzhehuai/received_events","type":"User","site_admin":false},"created_at":"2018-09-24T06:50:07Z","updated_at":"2018-09-24T06:50:07Z","author_association":"NONE","body":"I made some modification like the following. But still obtain this error\r\n```-def make_batchset(data, batch_size, max_length_in, max_length_out, num_batches=0):\r\n+def make_batchset(data, batch_size, max_length_in, max_length_out, num_batches=0, min_batch_size=1):\r\n     # sort it by input lengths (long to short)\r\n     sorted_data = sorted(data.items(), key=lambda data: int(\r\n         data[1]['input'][0]['shape'][0]), reverse=True)\r\n@@ -37,7 +37,7 @@ def make_batchset(data, batch_size, max_length_in, max_length_out, num_batches=0\r\n         # if ilen = 1000 and max_length_in = 800\r\n         # then b = batchsize / 2\r\n         # and max(1, .) avoids batchsize = 0\r\n-        b = max(1, int(batch_size / (1 + factor)))\r\n+        b = max(min_batch_size, int(batch_size / (1 + factor)))\r\n         end = min(len(sorted_data), start + b)\r\n         minibatch.append(sorted_data[start:end])\r\n         if end == len(sorted_data):```"}}
{"actor":{"display_login":"chenzhehuai"}}