{"action":"opened","issue":{"url":"https://api.github.com/repos/harvardnlp/seq2seq-attn/issues/65","repository_url":"https://api.github.com/repos/harvardnlp/seq2seq-attn","labels_url":"https://api.github.com/repos/harvardnlp/seq2seq-attn/issues/65/labels{/name}","comments_url":"https://api.github.com/repos/harvardnlp/seq2seq-attn/issues/65/comments","events_url":"https://api.github.com/repos/harvardnlp/seq2seq-attn/issues/65/events","html_url":"https://github.com/harvardnlp/seq2seq-attn/issues/65","id":185565384,"number":65,"title":"out of memory during the training","user":{"login":"jdxyw","id":119963,"avatar_url":"https://avatars.githubusercontent.com/u/119963?v=3","gravatar_id":"","url":"https://api.github.com/users/jdxyw","html_url":"https://github.com/jdxyw","followers_url":"https://api.github.com/users/jdxyw/followers","following_url":"https://api.github.com/users/jdxyw/following{/other_user}","gists_url":"https://api.github.com/users/jdxyw/gists{/gist_id}","starred_url":"https://api.github.com/users/jdxyw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdxyw/subscriptions","organizations_url":"https://api.github.com/users/jdxyw/orgs","repos_url":"https://api.github.com/users/jdxyw/repos","events_url":"https://api.github.com/users/jdxyw/events{/privacy}","received_events_url":"https://api.github.com/users/jdxyw/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-27T03:32:16Z","updated_at":"2016-10-27T03:32:16Z","closed_at":null,"body":"Hi\r\n\r\nI am trying to train a seq2seq model like it using my own dataset\r\n\r\n> th train.lua -data_file ../../data/demo-train.hdf5 -val_data_file ../../data/demo-val.hdf5 -savefile model-demo -brnn 1 -attn 1 -multi_attn 3 -gpuid 3 -max_batch_l 64 -lr_decay 0.8 -num_layers 4 -rnn_size 1024 -dropout 0.3 -curriculum 1\r\n\r\nAt the beginning, it works well. However, after 4800 batch, it threw an error about out of memory.\r\n\r\nMy machine has 196G memory with Tesla K40m. Since the training has begun, why it would fail during the training.\r\n\r\n`Epoch: 1, Batch: 4800/15072, Batch size: 64, LR: 1.0000, PPL: 246.53, |Param|: 690.31, |GParam|: 5.21, Training: 1817/1308/509 total/source/target tokens/sec\r\nEpoch: 1, Batch: 4850/15072, Batch size: 64, LR: 1.0000, PPL: 244.21, |Param|: 691.18, |GParam|: 5.53, Training: 1816/1311/505 total/source/target tokens/sec\r\nEpoch: 1, Batch: 4900/15072, Batch size: 64, LR: 1.0000, PPL: 241.79, |Param|: 692.06, |GParam|: 5.30, Training: 1815/1314/501 total/source/target tokens/sec\r\nEpoch: 1, Batch: 4950/15072, Batch size: 64, LR: 1.0000, PPL: 239.34, |Param|: 692.97, |GParam|: 5.76, Training: 1814/1317/496 total/source/target tokens/sec\r\nEpoch: 1, Batch: 5000/15072, Batch size: 64, LR: 1.0000, PPL: 236.97, |Param|: 693.87, |GParam|: 6.32, Training: 1813/1321/492 total/source/target tokens/sec\r\nEpoch: 1, Batch: 5050/15072, Batch size: 64, LR: 1.0000, PPL: 234.39, |Param|: 694.76, |GParam|: 5.80, Training: 1812/1324/488 total/source/target tokens/sec\r\nTHCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-2002/cutorch/lib/THC/generic/THCStorage.cu line=40 error=2 : out of memory\r\n/home/work/dev/torch/distro/install/bin/luajit: ...v/torch/distro/install/share/lua/5.1/nngraph/nesting.lua:34: cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-2002/cutorch/lib/THC/generic/THCStorage.cu:40\r\nstack traceback:\r\n        [C]: in function 'resizeAs'\r\n        ...v/torch/distro/install/share/lua/5.1/nngraph/nesting.lua:34: in function 'resizeNestedAs'\r\n        ...v/torch/distro/install/share/lua/5.1/nngraph/gmodule.lua:37: in function 'getTotalGradOutput'\r\n        ...v/torch/distro/install/share/lua/5.1/nngraph/gmodule.lua:404: in function 'neteval'\r\n        ...v/torch/distro/install/share/lua/5.1/nngraph/gmodule.lua:454: in function 'updateGradInput'\r\n        ...v/torch/distro/install/share/lua/5.1/nngraph/gmodule.lua:420: in function 'neteval'\r\n        ...v/torch/distro/install/share/lua/5.1/nngraph/gmodule.lua:454: in function 'updateGradInput'\r\n        ...ork/dev/torch/distro/install/share/lua/5.1/nn/Module.lua:31: in function 'backward'\r\n        train.lua:535: in function 'train_batch'\r\n        train.lua:745: in function 'train'\r\n        train.lua:1071: in function 'main'\r\n        train.lua:1074: in main chunk\r\n        [C]: in function 'dofile'\r\n        ...rch/distro/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk\r\n        [C]: at 0x00405810`\r\n\r\n"}}
{"actor":{"display_login":"jdxyw"}}