{"action":"closed","issue":{"url":"https://api.github.com/repos/OpenNMT/OpenNMT-py/issues/1089","repository_url":"https://api.github.com/repos/OpenNMT/OpenNMT-py","labels_url":"https://api.github.com/repos/OpenNMT/OpenNMT-py/issues/1089/labels{/name}","comments_url":"https://api.github.com/repos/OpenNMT/OpenNMT-py/issues/1089/comments","events_url":"https://api.github.com/repos/OpenNMT/OpenNMT-py/issues/1089/events","html_url":"https://github.com/OpenNMT/OpenNMT-py/issues/1089","id":386876750,"node_id":"MDU6SXNzdWUzODY4NzY3NTA=","number":1089,"title":"Conv2conv out of memory at validation time","user":{"login":"ZeweiChu","id":2027005,"node_id":"MDQ6VXNlcjIwMjcwMDU=","avatar_url":"https://avatars1.githubusercontent.com/u/2027005?v=4","gravatar_id":"","url":"https://api.github.com/users/ZeweiChu","html_url":"https://github.com/ZeweiChu","followers_url":"https://api.github.com/users/ZeweiChu/followers","following_url":"https://api.github.com/users/ZeweiChu/following{/other_user}","gists_url":"https://api.github.com/users/ZeweiChu/gists{/gist_id}","starred_url":"https://api.github.com/users/ZeweiChu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ZeweiChu/subscriptions","organizations_url":"https://api.github.com/users/ZeweiChu/orgs","repos_url":"https://api.github.com/users/ZeweiChu/repos","events_url":"https://api.github.com/users/ZeweiChu/events{/privacy}","received_events_url":"https://api.github.com/users/ZeweiChu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-12-03T15:42:24Z","updated_at":"2018-12-04T18:01:59Z","closed_at":"2018-12-04T18:01:59Z","author_association":"NONE","body":"Here is the script I used. \r\n\r\nFor preprocessing: \r\n```python ../OpenNMT-py/preprocess.py -train_src \"$INPUT_FILENAME\".train.src.txt -train_tgt \"$INPUT_FILENAME\".train.tgt.txt -valid_src \"$INPUT_FILENAME\".valid.src.txt -valid_tgt \"$INPUT_FILENAME\".valid.tgt.txt -save_data data.share -share_vocab -src_seq_length_trunc 500 -src_seq_length 50000 -tgt_seq_length 50000```\r\n\r\nFor training:\r\n```python ../OpenNMT-py/train.py -data data.share -save_model $MODEL_NAME/model -world_size 1 -gpu_ranks 0 -global_attention general -batch_size 8 -encoder_type cnn -decoder_type cnn -enc_layers 7 -dec_layers 3 -cnn_kernel_width 5 -optim adam -learning_rate 0.001 -start_decay_steps 10000 -decay_steps 5000 -share_embeddings -save_checkpoint_steps 10000```\r\n\r\nAt 10000 steps:\r\n\r\n> [2018-12-03 09:28:41,180 INFO] Loading valid dataset from data.share.valid.0.pt, number of examples: 5200\r\n> Traceback (most recent call last):\r\n>   File \"../OpenNMT-py/train.py\", line 120, in <module>\r\n>     main(opt)\r\n>   File \"../OpenNMT-py/train.py\", line 51, in main\r\n>     single_main(opt, 0)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/train_single.py\", line 154, in main\r\n>     opt.valid_steps)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/trainer.py\", line 190, in train\r\n>     valid_stats = self.validate(valid_iter)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/trainer.py\", line 236, in validate\r\n>     outputs, attns = self.model(src, tgt, src_lengths)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/models/model.py\", line 43, in forward\r\n>     enc_state, memory_bank, lengths = self.encoder(src, lengths)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/encoders/cnn_encoder.py\", line 40, in forward\r\n>     out = self.cnn(emb_remap)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/utils/cnn_factory.py\", line 52, in forward\r\n>     x = x + conv(x)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/utils/cnn_factory.py\", line 31, in forward\r\n>     x_var = self.conv(x_var)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/modules/weight_norm.py\", line 168, in forward\r\n>     self.padding, self.dilation, self.groups)\r\n> RuntimeError: CUDA error: out of memory\r\n\r\nIf I change batch size to 32, I get out of memory immediately:\r\n\r\n> [2018-12-03 09:06:10,089 INFO] Loading train dataset from data.share.train.0.pt, number of examples: 220615\r\n> [2018-12-03 09:06:10,149 INFO]  * vocabulary size. source = 50004; target = 50004\r\n> [2018-12-03 09:06:10,149 INFO] Building model...\r\n> [2018-12-03 09:06:12,557 INFO] NMTModel(\r\n>   (encoder): CNNEncoder(\r\n>     (embeddings): Embeddings(\r\n>       (make_embedding): Sequential(\r\n>         (emb_luts): Elementwise(\r\n>           (0): Embedding(50004, 500, padding_idx=1)\r\n>         )\r\n>       )\r\n>     )\r\n>     (linear): Linear(in_features=500, out_features=500, bias=True)\r\n>     (cnn): StackedCNN(\r\n>       (layers): ModuleList(\r\n>         (0): GatedConv(\r\n>           (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0))\r\n>           (dropout): Dropout(p=0.3)\r\n>         )\r\n>         (1): GatedConv(\r\n>           (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0))\r\n>           (dropout): Dropout(p=0.3)\r\n>         )\r\n>         (2): GatedConv(\r\n>           (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0))\r\n>           (dropout): Dropout(p=0.3)\r\n>         )\r\n>         (3): GatedConv(\r\n>           (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0))\r\n>           (dropout): Dropout(p=0.3)\r\n>         )\r\n>         (4): GatedConv(\r\n>           (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0))\r\n>           (dropout): Dropout(p=0.3)\r\n>         )\r\n>         (5): GatedConv(\r\n>           (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0))\r\n>           (dropout): Dropout(p=0.3)\r\n>         )\r\n>         (6): GatedConv(\r\n>           (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0))\r\n>           (dropout): Dropout(p=0.3)\r\n>         )\r\n>       )\r\n>     )\r\n>   )\r\n>   (decoder): CNNDecoder(\r\n>     (embeddings): Embeddings(\r\n>       (make_embedding): Sequential(\r\n>         (emb_luts): Elementwise(\r\n>           (0): Embedding(50004, 500, padding_idx=1)\r\n>         )\r\n>       )\r\n>     )\r\n>     (linear): Linear(in_features=500, out_features=500, bias=True)\r\n>     (conv_layers): ModuleList(\r\n>       (0): GatedConv(\r\n>         (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1))\r\n>         (dropout): Dropout(p=0.3)\r\n>       )\r\n>       (1): GatedConv(\r\n>         (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1))\r\n>         (dropout): Dropout(p=0.3)\r\n>       )\r\n>       (2): GatedConv(\r\n>         (conv): WeightNormConv2d(500, 1000, kernel_size=(5, 1), stride=(1, 1))\r\n>         (dropout): Dropout(p=0.3)\r\n>       )\r\n>     )\r\n>     (attn_layers): ModuleList(\r\n>       (0): ConvMultiStepAttention(\r\n>         (linear_in): Linear(in_features=500, out_features=500, bias=True)\r\n>       )\r\n>       (1): ConvMultiStepAttention(\r\n>         (linear_in): Linear(in_features=500, out_features=500, bias=True)\r\n>       )\r\n>       (2): ConvMultiStepAttention(\r\n>         (linear_in): Linear(in_features=500, out_features=500, bias=True)\r\n>       )\r\n>     )\r\n>   )\r\n>   (generator): Sequential(\r\n>     (0): Linear(in_features=500, out_features=50004, bias=True)\r\n>     (1): LogSoftmax()\r\n>   )\r\n> )\r\n> [2018-12-03 09:06:12,557 INFO] encoder: 42766500\r\n> [2018-12-03 09:06:12,557 INFO] decoder: 33560004\r\n> [2018-12-03 09:06:12,557 INFO] * number of parameters: 76326504\r\n> [2018-12-03 09:06:12,558 INFO] Starting training on GPU: [0]\r\n> [2018-12-03 09:06:12,558 INFO] Start training...\r\n> [2018-12-03 09:06:15,761 INFO] Loading train dataset from data.share.train.0.pt, number of examples: 220615\r\n> Traceback (most recent call last):\r\n>   File \"../OpenNMT-py/train.py\", line 120, in <module>\r\n>     main(opt)\r\n>   File \"../OpenNMT-py/train.py\", line 51, in main\r\n>     single_main(opt, 0)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/train_single.py\", line 154, in main\r\n>     opt.valid_steps)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/trainer.py\", line 175, in train\r\n>     report_stats)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/trainer.py\", line 283, in _gradient_accumulation\r\n>     self.model(src, tgt, src_lengths)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/models/model.py\", line 43, in forward\r\n>     enc_state, memory_bank, lengths = self.encoder(src, lengths)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/encoders/cnn_encoder.py\", line 40, in forward\r\n>     out = self.cnn(emb_remap)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/utils/cnn_factory.py\", line 52, in forward\r\n>     x = x + conv(x)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/utils/cnn_factory.py\", line 31, in forward\r\n>     x_var = self.conv(x_var)\r\n>   File \"/home/yijingxiao/dev/anaconda3/envs/py37-torch-0.4.1/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 477, in __call__\r\n>     result = self.forward(*input, **kwargs)\r\n>   File \"/home/yijingxiao/research/post-modifier/OpenNMT-py/onmt/modules/weight_norm.py\", line 168, in forward\r\n>     self.padding, self.dilation, self.groups)\r\n> RuntimeError: CUDA error: out of memory"}}
{"actor":{"display_login":"ZeweiChu"}}