{"action":"opened","issue":{"url":"https://api.github.com/repos/DmitryUlyanov/online-neural-doodle/issues/1","repository_url":"https://api.github.com/repos/DmitryUlyanov/online-neural-doodle","labels_url":"https://api.github.com/repos/DmitryUlyanov/online-neural-doodle/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/DmitryUlyanov/online-neural-doodle/issues/1/comments","events_url":"https://api.github.com/repos/DmitryUlyanov/online-neural-doodle/issues/1/events","html_url":"https://github.com/DmitryUlyanov/online-neural-doodle/issues/1","id":158804196,"number":1,"title":"Out of memory error","user":{"login":"BrianWestlake","id":2127792,"avatar_url":"https://avatars.githubusercontent.com/u/2127792?v=3","gravatar_id":"","url":"https://api.github.com/users/BrianWestlake","html_url":"https://github.com/BrianWestlake","followers_url":"https://api.github.com/users/BrianWestlake/followers","following_url":"https://api.github.com/users/BrianWestlake/following{/other_user}","gists_url":"https://api.github.com/users/BrianWestlake/gists{/gist_id}","starred_url":"https://api.github.com/users/BrianWestlake/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BrianWestlake/subscriptions","organizations_url":"https://api.github.com/users/BrianWestlake/orgs","repos_url":"https://api.github.com/users/BrianWestlake/repos","events_url":"https://api.github.com/users/BrianWestlake/events{/privacy}","received_events_url":"https://api.github.com/users/BrianWestlake/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-06-07T00:22:22Z","updated_at":"2016-06-07T00:22:22Z","closed_at":null,"body":" When I execute \r\n\r\nCUDA_VISIBLE_DEVICES=0 th feedforward_neural_doodle.lua -model_name skip_noise_4 -masks_hdf5 data/starry/gen_doodles.hdf5 -batch_size 4 -num_mask_noise_times 0 -num_noise_channels 0 -learning_rate 1e-1 -half false\r\n\r\nI get the followingh result:\r\n\r\n[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.\r\n[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 574671192\r\nSuccessfully loaded data/pretrained/VGG_ILSVRC_19_layers.caffemodel\r\nconv1_1: 64 3 3 3\r\nconv1_2: 64 64 3 3\r\nconv2_1: 128 64 3 3\r\nconv2_2: 128 128 3 3\r\nconv3_1: 256 128 3 3\r\nconv3_2: 256 256 3 3\r\nconv3_3: 256 256 3 3\r\nconv3_4: 256 256 3 3\r\nconv4_1: 512 256 3 3\r\nconv4_2: 512 512 3 3\r\nconv4_3: 512 512 3 3\r\nconv4_4: 512 512 3 3\r\nconv5_1: 512 512 3 3\r\nconv5_2: 512 512 3 3\r\nconv5_3: 512 512 3 3\r\nconv5_4: 512 512 3 3\r\nfc6: 1 1 25088 4096\r\nfc7: 1 1 4096 4096\r\nfc8: 1 1 4096 1000\r\nSetting up style layer  \t2\t:\trelu1_1\t\r\nReplacing max pooling at layer 5 with average pooling\t\r\nSetting up style layer  \t7\t:\trelu2_1\t\r\nReplacing max pooling at layer 10 with average pooling\t\r\nSetting up style layer  \t12\t:\trelu3_1\t\r\nReplacing max pooling at layer 19 with average pooling\t\r\nSetting up style layer  \t21\t:\trelu4_1\t\r\nReplacing max pooling at layer 28 with average pooling\t\r\nSetting up style layer  \t30\t:\trelu5_1\t\r\n        Optimize        \t\r\nTHCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-7288/cutorch/lib/THC/generic/THCStorage.cu line=41 error=2 : out of memory\r\n/home/andrew/torch/install/bin/luajit: /home/andrew/torch/install/share/lua/5.1/nn/Container.lua:67: \r\nIn 3 module of nn.Sequential:\r\nIn 1 module of nn.Sequential:\r\n/home/andrew/torch/install/share/lua/5.1/nn/THNN.lua:109: cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-7288/cutorch/lib/THC/generic/THCStorage.cu:41\r\nstack traceback:\r\n\t[C]: in function 'v'\r\n\t/home/andrew/torch/install/share/lua/5.1/nn/THNN.lua:109: in function 'SpatialReplicationPadding_updateGradInput'\r\n\t...h/install/share/lua/5.1/nn/SpatialReplicationPadding.lua:41: in function 'updateGradInput'\r\n\t/home/andrew/torch/install/share/lua/5.1/nn/Module.lua:31: in function </home/andrew/torch/install/share/lua/5.1/nn/Module.lua:29>\r\n\t[C]: in function 'xpcall'\r\n\t/home/andrew/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'\r\n\t/home/andrew/torch/install/share/lua/5.1/nn/Sequential.lua:88: in function </home/andrew/torch/install/share/lua/5.1/nn/Sequential.lua:78>\r\n\t[C]: in function 'xpcall'\r\n\t/home/andrew/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'\r\n\t/home/andrew/torch/install/share/lua/5.1/nn/Sequential.lua:84: in function 'backward'\r\n\tfeedforward_neural_doodle.lua:167: in function 'opfunc'\r\n\t/home/andrew/torch/install/share/lua/5.1/optim/adam.lua:33: in function 'optim_method'\r\n\tfeedforward_neural_doodle.lua:199: in main chunk\r\n\t[C]: in function 'dofile'\r\n\t...drew/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk\r\n\t[C]: at 0x00406670\r\n\r\nI'm running with multiple GTX 980s, so GPU m,emory should not be an issue.\r\n\r\nI have tried running with both -backend cudnn and -backen nn with no difference to the outcome.\r\n\r\nI have been able to run the fast-neural-doodle project with no problems on my machine, so prerequisites such as python, torch and cuda appear to have been set up correctly.\r\n\r\nAny idea of the cause of this problem?\r\n\r\n\r\n"}}
{"actor":{"display_login":"BrianWestlake"}}