{"action":"closed","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/2612","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/2612/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/2612/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/2612/events","html_url":"https://github.com/fchollet/keras/issues/2612","id":153063776,"number":2612,"title":"Implementing simple neural attention model (for padded inputs)","user":{"login":"anayebi","id":12835914,"avatar_url":"https://avatars.githubusercontent.com/u/12835914?v=3","gravatar_id":"","url":"https://api.github.com/users/anayebi","html_url":"https://github.com/anayebi","followers_url":"https://api.github.com/users/anayebi/followers","following_url":"https://api.github.com/users/anayebi/following{/other_user}","gists_url":"https://api.github.com/users/anayebi/gists{/gist_id}","starred_url":"https://api.github.com/users/anayebi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anayebi/subscriptions","organizations_url":"https://api.github.com/users/anayebi/orgs","repos_url":"https://api.github.com/users/anayebi/repos","events_url":"https://api.github.com/users/anayebi/events{/privacy}","received_events_url":"https://api.github.com/users/anayebi/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-05-04T16:55:43Z","updated_at":"2016-05-05T16:45:45Z","closed_at":"2016-05-05T16:45:45Z","body":"I have an input sequence (because it is variable length, used the Keras `pad_sequences` function to zero pad it), and I wanted to basically average the outputs of my RNN and concatenate it with its prediction at the last timestep, which would then be the model's prediction.\r\n\r\nHere is my code to better visualize what I mean (I'm using the Keras 1.0 Model class):\r\n\r\n>   input_sentences = Input(shape=(max_sen_length,), dtype='int32')\r\n     x = Embedding(input_dim=vocab_size, output_dim=vocab_dim, input_length=max_sen_length, mask_zero=True)(input_sentences)\r\n    lstm_out = LSTM(150, return_sequences=True)(x)\r\n    attention = TimeDistributedDense(32, activation='softmax')(lstm_out)\r\n    context = Lambda(avg_timesteps, output_shape=avg_timesteps_output_shape)(attention)\r\n    last_lstm_out = Lambda(last_timestep, output_shape=last_timestep_output_shape)(lstm_out)\r\n    concat = Merge([last_lstm_out, context], mode='concat')\r\n\r\nNote that `avg_timesteps` and `last_timestep` were custom functions I wrote to work with the `Lambda` layers (I won't include them here for brevity).\r\n\r\nThe issue I am running into is that `Lambda` layers do **not** support masking, so I was not able to get the above code to work. Is there a way to essentially implement my context layer (which is the average of the outputs of the `TimeDistributedDense` layer)? \r\n\r\nI am aware of the various Keras threads about implementing neural attention, but this is a relatively simple version of that (since my model's overall prediction is based on the last timestep, so I'm just averaging the predictions up to then and concatenating them with the RNN's last output, `last_lstm_out`), based on Section 3.1 of [this paper](http://arxiv.org/pdf/1508.04025v5.pdf). I figure this is probably doable, so I'd be happy to hear anyone's thoughts on this. Thanks!"}}