{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/2033","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/2033/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/2033/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/2033/events","html_url":"https://github.com/BVLC/caffe/pull/2033","id":59849829,"number":2033,"title":"Unrolled recurrent layers (RNN, LSTM)","user":{"login":"jeffdonahue","id":965190,"avatar_url":"https://avatars.githubusercontent.com/u/965190?v=3","gravatar_id":"","url":"https://api.github.com/users/jeffdonahue","html_url":"https://github.com/jeffdonahue","followers_url":"https://api.github.com/users/jeffdonahue/followers","following_url":"https://api.github.com/users/jeffdonahue/following{/other_user}","gists_url":"https://api.github.com/users/jeffdonahue/gists{/gist_id}","starred_url":"https://api.github.com/users/jeffdonahue/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeffdonahue/subscriptions","organizations_url":"https://api.github.com/users/jeffdonahue/orgs","repos_url":"https://api.github.com/users/jeffdonahue/repos","events_url":"https://api.github.com/users/jeffdonahue/events{/privacy}","received_events_url":"https://api.github.com/users/jeffdonahue/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BVLC/caffe/labels/ES","name":"ES","color":"bfdadc"},{"url":"https://api.github.com/repos/BVLC/caffe/labels/JL","name":"JL","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":56,"created_at":"2015-03-04T19:38:35Z","updated_at":"2016-02-01T17:36:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/BVLC/caffe/pulls/2033","html_url":"https://github.com/BVLC/caffe/pull/2033","diff_url":"https://github.com/BVLC/caffe/pull/2033.diff","patch_url":"https://github.com/BVLC/caffe/pull/2033.patch"},"body":"(Replaces #1873)\r\n\r\nBased on #2032 (adds EmbedLayer -- not needed for, but often used with RNNs in practice, and is needed for my examples), which in turn is based on #1977.\r\n\r\nThis adds an abstract class `RecurrentLayer` intended to support recurrent architectures (RNNs, LSTMs, etc.) using an internal network unrolled in time.  `RecurrentLayer` implementations (here, just `RNNLayer` and `LSTMLayer`) specify the recurrent architecture by filling in a NetParameter with appropriate layers.\r\n\r\n`RecurrentLayer` requires 2 input (bottom) Blobs.  The first -- the input data itself -- has shape `T x N x ...` and the second -- the \"sequence continuation indicators\" `delta` -- has shape `T x N`, each holding `T` timesteps of `N` independent \"streams\".  `delta_{t,n}` should be a binary indicator (i.e., value in {0, 1}), where a value of 0 means that timestep t of stream n is the beginning of a *new* sequence, and a value of 1 means that timestep t of stream n is *continuing* the sequence from timestep t-1 of stream n.  Under the hood, the previous timestep's hidden state is multiplied by these delta values.  The fact that these indicators are specified on a per-timestep and per-stream basis allows for streams of arbitrary different lengths without any padding or truncation.  At the beginning of the forward pass, the final hidden state from the previous forward pass (`h_T`) is copied into the initial hidden state for the new forward pass (`h_0`), allowing for exact inference across arbitrarily long sequences, even if `T == 1`.  However, if any sequences cross batch boundaries, backpropagation through time is approximate -- it is truncated along the batch boundaries.\r\n\r\nNote that the `T x N` arrangement in memory, used for computational efficiency, is somewhat counterintuitive, as it requires one to \"interleave\" the data streams.\r\n\r\nThere is also an *optional* third input whose dimensions are simply `N x ...` (i.e. the first axis must have dimension `N` and the others can be anything) which is a \"static\" input to the LSTM.  It's equivalent to (but more efficient than) copying the input across the `T` timesteps and concatenating it with the \"dynamic\" first input (I was using my `TileLayer` -- #2083 -- for this purpose at some point before adding the static input).  It's used in my captioning experiments to input the image features as they don't change over time.  For most problems there will be no such \"static\" input and you should simply ignore it and just specify the first two input blobs.\r\n\r\nI've added scripts to download COCO2014 (and splits), and prototxts for training a language model and LRCN captioning model on the data. From the Caffe root directory, you should be able to download and parse the data by doing:\r\n\r\n```\r\ncd data/coco\r\n./get_coco_aux.sh # download train/val/test splits\r\n./download_tools.sh # download official COCO tool\r\ncd tools\r\npython setup.py install # follow instructions to install tools and download COCO data if needed\r\ncd ../../.. # back to caffe root\r\n./examples/coco_caption/coco_to_hdf5_data.py\r\n```\r\n\r\nThen, you can train a language model using `./examples/coco_caption/train_language_model.sh`, or train LRCN for captioning using `./examples/coco_caption/train_lrcn.sh` (assuming you have downloaded `models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel`).\r\n\r\nStill on the TODO list: upload a pretrained model to the zoo; add a tool to preview generated image captions and compute retrieval & generation scores."},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/178088443","html_url":"https://github.com/BVLC/caffe/pull/2033#issuecomment-178088443","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/2033","id":178088443,"user":{"login":"shaibagon","id":1818351,"avatar_url":"https://avatars.githubusercontent.com/u/1818351?v=3","gravatar_id":"","url":"https://api.github.com/users/shaibagon","html_url":"https://github.com/shaibagon","followers_url":"https://api.github.com/users/shaibagon/followers","following_url":"https://api.github.com/users/shaibagon/following{/other_user}","gists_url":"https://api.github.com/users/shaibagon/gists{/gist_id}","starred_url":"https://api.github.com/users/shaibagon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shaibagon/subscriptions","organizations_url":"https://api.github.com/users/shaibagon/orgs","repos_url":"https://api.github.com/users/shaibagon/repos","events_url":"https://api.github.com/users/shaibagon/events{/privacy}","received_events_url":"https://api.github.com/users/shaibagon/received_events","type":"User","site_admin":false},"created_at":"2016-02-01T17:36:23Z","updated_at":"2016-02-01T17:36:23Z","body":"@elliotchencv I believe the static feature should be of dimension `N x D1 x D2` - you should have the same static feature for all time steps T, but different for each stream/batch."}}