{"action":"created","issue":{"url":"https://api.github.com/repos/Element-Research/rnn/issues/129","repository_url":"https://api.github.com/repos/Element-Research/rnn","labels_url":"https://api.github.com/repos/Element-Research/rnn/issues/129/labels{/name}","comments_url":"https://api.github.com/repos/Element-Research/rnn/issues/129/comments","events_url":"https://api.github.com/repos/Element-Research/rnn/issues/129/events","html_url":"https://github.com/Element-Research/rnn/issues/129","id":131246573,"number":129,"title":"Problem with using Sequencer with non-recurrent modules","user":{"login":"kmul00","id":6665222,"avatar_url":"https://avatars.githubusercontent.com/u/6665222?v=3","gravatar_id":"","url":"https://api.github.com/users/kmul00","html_url":"https://github.com/kmul00","followers_url":"https://api.github.com/users/kmul00/followers","following_url":"https://api.github.com/users/kmul00/following{/other_user}","gists_url":"https://api.github.com/users/kmul00/gists{/gist_id}","starred_url":"https://api.github.com/users/kmul00/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmul00/subscriptions","organizations_url":"https://api.github.com/users/kmul00/orgs","repos_url":"https://api.github.com/users/kmul00/repos","events_url":"https://api.github.com/users/kmul00/events{/privacy}","received_events_url":"https://api.github.com/users/kmul00/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-02-04T05:22:27Z","updated_at":"2016-02-04T20:37:31Z","closed_at":null,"body":"Hi,\r\n\r\nI have a model where I have a `CNN` (as an *encoder*) followed by an `LSTM`, acting on a *sequential data*. It can be thought of like, *features* are learnt from my *input* using a `CNN` and these *features* are then fed into an `LSTM` for final *classification*. For ease, I wrapped my modules in a `sequencer` to allow using *sequential* data.\r\n\r\nMy **input** is as follows :\r\n```lua\r\ninput = torch.Tensor(time_step X batch_size X channels X image_width X image_height)\r\n```\r\nMy **model** is as follows:\r\n```lua\r\n\r\n-- a normal three layer CNN encoder\r\n-- conv -> batch_norm -> activation -> pooling -> dropout\r\n\r\ncnn_model = nn.Sequential()\r\ncnn_model:add(SpatialConvolution):add(BatchNormalization):add(ReLU):add(MaxPooling):add(Dropout)\r\ncnn_model:add(SpatialConvolution):add(BatchNormalization):add(ReLU):add(MaxPooling):add(Dropout)\r\ncnn_model:add(SpatialConvolution):add(BatchNormalization):add(ReLU):add(MaxPooling):add(Dropout)\r\ncnn_model:Reshape()  -- to flatten out the features while feeding into LSTM\r\n\r\n-- LSTM layer\r\n\r\nlstm_model = nn.LSTM(input_size, hidden_size)\r\nlstm_model:add(Dropout)\r\n\r\n-- putting everything together\r\n\r\nmodel = nn.Sequential()\r\nmodel:add(nn.SplitTable(1))  -- sequencer expects table of inputs\r\nmodel:add(nn.Sequencer(cnn_model))  -- putting the CNN into sequencer\r\nmodel:add(nn.Sequencer(lstm_model))  -- followed by LSTM which acts on output of the CNN\r\nmodel:add(nn.SelectTable(-1)) -- many-to-one LSTM\r\nmodel:add(nn.Linear()):add(nn.LogSoftMax())  -- apply criterion on top of this\r\n\r\ncriterion = nn.ClassNLLCriterion()\r\n```\r\n\r\nNow **training** is okay, error decreasing as expected. Problem arises while **testing**. For verification purpose I am using the same *train set* as my *test set*. \r\n\r\nWhile **testing** when I set \r\n```lua\r\nmodel:evaluate()\r\n```\r\nthings are not working out. The accuracy keeps oscillating at a very low value (~5-10%), even though my *training accuracy* is much higher and I am using the same data for *testing* also.\r\n\r\nBut instead if I use \r\n```lua\r\ncnn_model:evaluate()\r\nlstm_model:evaluate()\r\n```\r\nthings are going perfectly, as expected !\r\n\r\nI have previously used `LSTM` without any problem. So my guess is, I am doing something wrong while putting the `CNN` in the `sequencer`.\r\n\r\nCan someone point out the reason for this and where exactly is my mistake ? \r\n\r\nThanks !\r\n\r\n**NB**: I have omitted details such as `kernel_size`, `hidden_unit_size`, `linear_unit_size`, *etc*, for ease of readability. There is no problem like dimension mismatch or something of that sort. "},"comment":{"url":"https://api.github.com/repos/Element-Research/rnn/issues/comments/180039503","html_url":"https://github.com/Element-Research/rnn/issues/129#issuecomment-180039503","issue_url":"https://api.github.com/repos/Element-Research/rnn/issues/129","id":180039503,"user":{"login":"soumith","id":1310570,"avatar_url":"https://avatars.githubusercontent.com/u/1310570?v=3","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"created_at":"2016-02-04T20:37:31Z","updated_at":"2016-02-04T20:37:31Z","body":"@kmul00 @nicholas-leonard output2 wont be equal to output, because the running_mean and running_var are exponential averages over some momentum. "}}