{"action":"created","issue":{"url":"https://api.github.com/repos/SeanNaren/deepspeech.torch/issues/60","repository_url":"https://api.github.com/repos/SeanNaren/deepspeech.torch","labels_url":"https://api.github.com/repos/SeanNaren/deepspeech.torch/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/SeanNaren/deepspeech.torch/issues/60/comments","events_url":"https://api.github.com/repos/SeanNaren/deepspeech.torch/issues/60/events","html_url":"https://github.com/SeanNaren/deepspeech.torch/issues/60","id":184118075,"number":60,"title":"Cuda out of memory error while training on LibriSpeech dataset","user":{"login":"suhaspillai","id":12215110,"avatar_url":"https://avatars.githubusercontent.com/u/12215110?v=3","gravatar_id":"","url":"https://api.github.com/users/suhaspillai","html_url":"https://github.com/suhaspillai","followers_url":"https://api.github.com/users/suhaspillai/followers","following_url":"https://api.github.com/users/suhaspillai/following{/other_user}","gists_url":"https://api.github.com/users/suhaspillai/gists{/gist_id}","starred_url":"https://api.github.com/users/suhaspillai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suhaspillai/subscriptions","organizations_url":"https://api.github.com/users/suhaspillai/orgs","repos_url":"https://api.github.com/users/suhaspillai/repos","events_url":"https://api.github.com/users/suhaspillai/events{/privacy}","received_events_url":"https://api.github.com/users/suhaspillai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-10-20T02:07:23Z","updated_at":"2016-11-19T22:19:16Z","closed_at":null,"body":"I am trying to train the model on libri speech dev-clean dataset, where my train split  = 2503 and val split = 200. I reduced my val split thinking this might be the issue . Based on the memory consumption (which I checked using nvidia-smi), I think all the training data is loaded at once and so is the validation right? . Did anyone face this issue?\nFollowing is the stack trace\n\n```\nTHCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-6130/cutorch/lib/THC/generic/THCStorage.cu line=40 error=2 : out of memory\n/home/sbp3624/torch/install/bin/luajit: /home/sbp3624/torch/install/share/lua/5.1/nn/Container.lua:67: \nIn 2 module of nn.Sequential:\nIn 1 module of nn.Sequential:\nIn 5 module of cudnn.BatchBRNNReLU:\n/home/sbp3624/torch/install/share/lua/5.1/cudnn/RNN.lua:308: cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-6130/cutorch/lib/THC/generic/THCStorage.cu:40\nstack traceback:\n    [C]: in function 'resize'\n    /home/sbp3624/torch/install/share/lua/5.1/cudnn/RNN.lua:308: in function </home/sbp3624/torch/install/share/lua/5.1/cudnn/RNN.lua:262>\n    [C]: in function 'xpcall'\n    /home/sbp3624/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'\n    /home/sbp3624/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function </home/sbp3624/torch/install/share/lua/5.1/nn/Sequential.lua:41>\n    [C]: in function 'xpcall'\n    /home/sbp3624/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'\n    /home/sbp3624/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function </home/sbp3624/torch/install/share/lua/5.1/nn/Sequential.lua:41>\n    [C]: in function 'xpcall'\n    /home/sbp3624/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'\n    /home/sbp3624/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'\n    ./ModelEvaluator.lua:70: in function 'runEvaluation'\n    ./Network.lua:78: in function 'testNetwork'\n    ./Network.lua:170: in function 'trainNetwork'\n    Train.lua:42: in main chunk\n    [C]: in function 'dofile'\n    ...3624/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk\n    [C]: at 0x00406670\n\nWARNING: If you see a stack trace below, it doesn't point to the place where this error occurred. Please use only the one above.\nstack traceback:\n    [C]: in function 'error'\n    /home/sbp3624/torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors'\n    /home/sbp3624/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'\n    ./ModelEvaluator.lua:70: in function 'runEvaluation'\n    ./Network.lua:78: in function 'testNetwork'\n    ./Network.lua:170: in function 'trainNetwork'\n    Train.lua:42: in main chunk\n    [C]: in function 'dofile'\n    ...3624/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk\n    [C]: at 0x00406670\n```\n"},"comment":{"url":"https://api.github.com/repos/SeanNaren/deepspeech.torch/issues/comments/261743477","html_url":"https://github.com/SeanNaren/deepspeech.torch/issues/60#issuecomment-261743477","issue_url":"https://api.github.com/repos/SeanNaren/deepspeech.torch/issues/60","id":261743477,"user":{"login":"SeanNaren","id":6707363,"avatar_url":"https://avatars.githubusercontent.com/u/6707363?v=3","gravatar_id":"","url":"https://api.github.com/users/SeanNaren","html_url":"https://github.com/SeanNaren","followers_url":"https://api.github.com/users/SeanNaren/followers","following_url":"https://api.github.com/users/SeanNaren/following{/other_user}","gists_url":"https://api.github.com/users/SeanNaren/gists{/gist_id}","starred_url":"https://api.github.com/users/SeanNaren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SeanNaren/subscriptions","organizations_url":"https://api.github.com/users/SeanNaren/orgs","repos_url":"https://api.github.com/users/SeanNaren/repos","events_url":"https://api.github.com/users/SeanNaren/events{/privacy}","received_events_url":"https://api.github.com/users/SeanNaren/received_events","type":"User","site_admin":false},"created_at":"2016-11-19T22:19:16Z","updated_at":"2016-11-19T22:19:16Z","body":"The GPU memory will always increase as we move through the epoch as the batches get larger and larger (and those RNNs take a lot of memory!). Largest batch per GPU which has 12gb of vram is usually around 15 from what I've seen because of this. \r\n\r\nI know Baidu were able to get much larger batches but they had their own internal software and were super memory efficient when it came to how to using memory. Sadly with Torch it's a little bit more difficult, but should still be trainable with smaller batch sizes! Hopefully this help!"}}
{"actor":{"display_login":"SeanNaren"}}