{"action":"opened","issue":{"url":"https://api.github.com/repos/Element-Research/rnn/issues/180","repository_url":"https://api.github.com/repos/Element-Research/rnn","labels_url":"https://api.github.com/repos/Element-Research/rnn/issues/180/labels{/name}","comments_url":"https://api.github.com/repos/Element-Research/rnn/issues/180/comments","events_url":"https://api.github.com/repos/Element-Research/rnn/issues/180/events","html_url":"https://github.com/Element-Research/rnn/issues/180","id":143880190,"number":180,"title":"Implementation of DRAM model","user":{"login":"vyouman","id":9965927,"avatar_url":"https://avatars.githubusercontent.com/u/9965927?v=3","gravatar_id":"","url":"https://api.github.com/users/vyouman","html_url":"https://github.com/vyouman","followers_url":"https://api.github.com/users/vyouman/followers","following_url":"https://api.github.com/users/vyouman/following{/other_user}","gists_url":"https://api.github.com/users/vyouman/gists{/gist_id}","starred_url":"https://api.github.com/users/vyouman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vyouman/subscriptions","organizations_url":"https://api.github.com/users/vyouman/orgs","repos_url":"https://api.github.com/users/vyouman/repos","events_url":"https://api.github.com/users/vyouman/events{/privacy}","received_events_url":"https://api.github.com/users/vyouman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-28T03:00:44Z","updated_at":"2016-03-28T03:00:44Z","closed_at":null,"body":"Hi, I'm trying to implement the Deep Recurrent Attention Model described in the paper http://arxiv.org/pdf/1412.7755v2.pdf to apply to image caption generation instead of image classification. I will probably be able to use most of the modules in the RAM model implemented in the rnn package. For the case, I don't need to modify the Reinforce.lua interface and ReinforceNormal.lua since it's able to deal with table of rewards at every time step per batch now. All I should do here is to write a new Criterion and I've written one. And I think I should modify the RecurrentAttention module.\r\n\r\n There is a context network, which is a 3-layer convolution network or some other CNN model presented in the paper to get the feature of the low-resolution images to feed to the second recurrent layer to get the first location. I come up with two approaches:\r\n1. Maybe I should assemble the context network, the second recurrent layer and the location network to be the locator expected in the RecurrentAttention.\r\n2. Or I should use the context network to deal with the low-resolution image indepently and feed the feature to the 2nd recurrent layer in the first time step directly.\r\nIt seems that the second approach is more natural and easy to implement than the first one, because It seems I only need to modify the feature fed in the first time step here. \r\nhttps://github.com/Element-Research/rnn/blob/master/RecurrentAttention.lua#L44-L48\r\n\r\nFor the second question, I'll need to add some other input to the first recurrent layer, in my case, it's word vector of every time step. Finally, instead of predict classes of multi-objects, I expect to predict the caption describing the image. And there's also some logic to deal with the captions of variable lengths so I'll probably have to encapsulate a LSTM layer to do this stuff, we can think of it as a language model without the last LogSoftmax layer here, call it **lm** for now. And then wrap it with a Recursor with the glimpse network as a rnn like https://github.com/Element-Research/rnn/blob/master/examples/recurrent-visual-attention.lua#L130-L131, we use\r\n\r\n> rnn = nn.Recurrent(opt.hiddenSize, glimpse, lm, nn\\[opt.transfer\\](), 99999)\r\n\r\nat last wrap the rnn and the locator as https://github.com/Element-Research/rnn/blob/master/examples/recurrent-visual-attention.lua#L145. So here comes the question, how can I pass the additional input like word vector which is not the direct input of the rnn expected by the RecurrentAttention model? The rnn here is compsed of the glimpse network and a recurrent layer, where the glimpse network expect an input of {image, location}, and in this case the recurrent layer not only expect the gt vector gained from the glimpse network but also the word vector. Should I modify the RecurrentAttention module to get more inputs? Or I don't need to modify the input of the RecurrentAttention module and the word vector can directly go from the **lm** module I'm going to implement above. Do you think it's feasible to do so or you have some more elegant way to implement it?\r\n\r\nI'm a new babie of the rnn package and torch7. I'll appreciate your suggestion.:p\r\n\r\n\r\n\r\n\r\n\r\n\r\n"}}