{"action":"created","issue":{"url":"https://api.github.com/repos/keras-team/keras/issues/8296","repository_url":"https://api.github.com/repos/keras-team/keras","labels_url":"https://api.github.com/repos/keras-team/keras/issues/8296/labels{/name}","comments_url":"https://api.github.com/repos/keras-team/keras/issues/8296/comments","events_url":"https://api.github.com/repos/keras-team/keras/issues/8296/events","html_url":"https://github.com/keras-team/keras/pull/8296","id":269448036,"number":8296,"title":"Recurrent Attention API: Cell wrapper base class [work in progress]","user":{"login":"andhus","id":5502349,"avatar_url":"https://avatars2.githubusercontent.com/u/5502349?v=4","gravatar_id":"","url":"https://api.github.com/users/andhus","html_url":"https://github.com/andhus","followers_url":"https://api.github.com/users/andhus/followers","following_url":"https://api.github.com/users/andhus/following{/other_user}","gists_url":"https://api.github.com/users/andhus/gists{/gist_id}","starred_url":"https://api.github.com/users/andhus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andhus/subscriptions","organizations_url":"https://api.github.com/users/andhus/orgs","repos_url":"https://api.github.com/users/andhus/repos","events_url":"https://api.github.com/users/andhus/events{/privacy}","received_events_url":"https://api.github.com/users/andhus/received_events","type":"User","site_admin":false},"labels":[{"id":808058525,"url":"https://api.github.com/repos/keras-team/keras/labels/Reviewers%20wanted","name":"Reviewers wanted","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2017-10-29T23:31:10Z","updated_at":"2018-04-11T12:53:59Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/keras-team/keras/pulls/8296","html_url":"https://github.com/keras-team/keras/pull/8296","diff_url":"https://github.com/keras-team/keras/pull/8296.diff","patch_url":"https://github.com/keras-team/keras/pull/8296.patch"},"body":"## Intro\r\nThis PR implements step 3 (and example of step 4) as laid out in the [API doc](https://docs.google.com/document/d/1psFXnmMlSTg5JapgZKz26ag-zBu3ERrxkKoEzNpzl4w/edit?usp=sharing). See [PR for step 1](https://github.com/fchollet/keras/pull/7980) (step 2 is put on hold and) and the [API issue](https://github.com/fchollet/keras/issues/7633) for further background.\r\n\r\nThe main addition is an abstract base class for implementing _attention cell wrappers_: [`RecurrentAttentionCellWrapperABC`](https://github.com/fchollet/keras/pull/8296/files#diff-9c4188ddc4dd173d80f64feed5b89412R19). This class is pretty solid and I've tried to make the docs really detailed to clarify the important concepts.\r\n\r\nThere is also an implementation (of the base class) for Mixture of Gaussian 1D (i.e. sequence) attention with an example included - but here there are some more points to discuss:\r\n\r\n### 1) Should we add a distribution module?\r\nI will try to make a case for this: firstly, it is generally relevant to make it easy to implement \"mixture of density\" networks, and secondly, \"predicting distributions\" is an important part of attention mechanisms and the same logic could be reused. The main idea is to keep all parameters of any distribution in a \"flat output vector\" and gather the _activation function_ for parameters and the corresponding _loss_ (typicall -log(pdf)) in a class for each distribution. I'm happy make a separate API suggestion for this.\r\n\r\n### 2) Should we have (make it easy to write) layers with _several_ wrapped layers (and/or its own weights) internally?\r\nSo far there are only a few cases in Keras where a _layer is injected into another_; `TimeDistributed`, `Bidirectional` and most recently RNN `cell` -> `RNN`. In all of these cases there are _no additional parameters_ (weights) in the receiving layers. For such a case it is enough to extend the `Wrapper` class to get most of the required API (like `get/set_weights` etc.) \"for free\".\r\n\r\nWith recurrent attention it is different: we should wrap a core RNN cell with additional learnable transforms with their own weights. Moreover, an attention mechanism can be quite complex and consist of \"several layers\".\r\n\r\nThe pain-point is this: normally when we want to define complex transforms we should use the _functional API_ and compose it based on the atomic layers. But this is not possible (or at least not straight fwd) when defining a RNN cell transformation. This is why I suggested step 2, the [`FunctionalRNNCell`](https://github.com/andhus/keras/blob/recurrent_attention_api_constants_support_before_removing_func_cell/keras/layers/recurrent.py#L198).\r\n\r\nIf we instead take the standard approach of using low level `add_weight` for all internal transformations of a layer we end up with a lot of boiler-plate and configuration code. This is e.g. the case with the `MixtureOfGaussian1DAttention` in this PR: The `__init__` method takes 12 parameters (all the regularisers, initializers etc...) when all I really want to do is to _inject_ a `Dense` layer which already implements this and adequately groups the parameters. I think one could motivate something like a `MultiLayerWrapper` class that takes care of the boilerplate for these cases (`get/set_weights` etc.). Something like https://github.com/andhus/keras/blob/205d057b7078bfe885b967f39ab6324271c764fa/keras/layers/attention.py#L17 (@fchollet if you recall, something similar was mentioned in my original attention API suggestion...)\r\n\r\n### 3) Should we add support for returning state _sequences_ from RNN (not only final states)?\r\nThe most important thing that is not supported in this PR is that one can not get out the \"attention encoding\" or the \"attention location parameters\" (where is it looking) for the full sequence processed. The first is a problem because typically one want to stack multiple recurrent layers, let the first \"drive/interact with\" the attention mechanism but then feed the attention encoding to all later layers (\"cascading\").  The second is problem because it is very useful to be able to analyse/visualise what part of the input is attended to a any point in time. There are uggly/hacky ways of solving this (by concatenating this info to the wrapped cell output) but it causes a lot of additional complexity - and if we could just get out the full state sequences the information would basically already be there. Like this:\r\n```\r\ny, [h, c] = LSTM(32, return_sequences=True, return_state_sequences=True)(x)\r\n# h.shape[1] == c.shape[1] == y.shape[1] == x.shape[1]\r\n```\r\n\r\n### Other TODOs\r\n- Tests for `RecurrentAttentionCellWrapperABC`.\r\n- Improve recurrent attention example to include benchmark with encoder-decoder approach, see: https://github.com/andhus/extkeras/blob/master/examples/attention/README.md\r\n- Add machine translation (with attention!) example."},"comment":{"url":"https://api.github.com/repos/keras-team/keras/issues/comments/380441140","html_url":"https://github.com/keras-team/keras/pull/8296#issuecomment-380441140","issue_url":"https://api.github.com/repos/keras-team/keras/issues/8296","id":380441140,"user":{"login":"andreyzharkov","id":14358106,"avatar_url":"https://avatars0.githubusercontent.com/u/14358106?v=4","gravatar_id":"","url":"https://api.github.com/users/andreyzharkov","html_url":"https://github.com/andreyzharkov","followers_url":"https://api.github.com/users/andreyzharkov/followers","following_url":"https://api.github.com/users/andreyzharkov/following{/other_user}","gists_url":"https://api.github.com/users/andreyzharkov/gists{/gist_id}","starred_url":"https://api.github.com/users/andreyzharkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreyzharkov/subscriptions","organizations_url":"https://api.github.com/users/andreyzharkov/orgs","repos_url":"https://api.github.com/users/andreyzharkov/repos","events_url":"https://api.github.com/users/andreyzharkov/events{/privacy}","received_events_url":"https://api.github.com/users/andreyzharkov/received_events","type":"User","site_admin":false},"created_at":"2018-04-11T12:53:59Z","updated_at":"2018-04-11T12:53:59Z","author_association":"NONE","body":"Hi, @andhus, is there any example of how should your model be used at prediction time? As far as I understand the only possible solution now is to predict each symbol with the new run of entire model feeding it previous output label and state, am I right? Also I think it would be beneficial to add that prediction part to the example if you don't mind"}}
{"actor":{"display_login":"andreyzharkov"}}