title: Adding Attention on top of simple LSTM layer in Tensorflow 2.0
body: <p>I have a simple network of one LSTM and two Dense layers as such:</p>
  
  <pre><code>model = tf.keras.Sequential()
  model.add(layers.LSTM(20, input_shape=(train_X.shape[1], train_X.shape[2])))
  model.add(layers.Dense(20, activation='sigmoid'))
  model.add(layers.Dense(1, activation='sigmoid'))
  model.compile(loss='mean_squared_error')
  </code></pre>
  
  <p>It is training on data with 3 inputs (normalized 0 to 1.0) and 1 output (binary) for the purpose of classification. The data is time series data where there is a relation between time steps.</p>
  
  <pre><code>    var1(t)   var2(t)   var3(t)  var4(t)
  0  0.448850  0.503847  0.498571      0.0
  1  0.450992  0.503480  0.501215      0.0
  2  0.451011  0.506655  0.503049      0.0
  </code></pre>
  
  <p>The model is trained as such:</p>
  
  <pre><code>history = model.fit(train_X, train_y, epochs=2800, batch_size=40, validation_data=(test_X, test_y), verbose=2, shuffle=False)
  model.summary()
  </code></pre>
  
  <p>Giving the model summary:</p>
  
  <pre><code>Model: "sequential"
  _________________________________________________________________
  Layer (type)                 Output Shape              Param #   
  =================================================================
  lstm (LSTM)                  (None, 20)                1920      
  _________________________________________________________________
  dense (Dense)                (None, 20)                420       
  _________________________________________________________________
  dense_1 (Dense)              (None, 1)                 21        
  =================================================================
  Total params: 2,361
  Trainable params: 2,361
  Non-trainable params: 0
  </code></pre>
  
  <p>The model works reasonably well. Now I am trying to replace the Dense(20) layer with an Attention layer. All the examples, tutorials, etc. online (including the TF docs) are for seq2seq models with an embedding layer at the input layer. I understand the seq2seq implementations in TF v1.x but I cannot find any documentation for what I am trying to do. I believe in the new API (v2.0) I need to do something like this:</p>
  
  <pre><code>lstm = layers.LSTM(20, input_shape=(train_X.shape[1], train_X.shape[2]), return_sequences=True)
  lstm = tf.keras.layers.Bidirectional(lstm)
  attention = layers.Attention() # this does not work
  
  model = tf.keras.Sequential()
  model.add(lstm)
  model.add(attention)
  model.add(layers.Dense(1, activation='sigmoid'))
  model.compile(loss='mean_squared_error')
  </code></pre>
  
  <p>And of course I get the error "<em>Attention layer must be called on a list of inputs, namely [query, value] or [query, value, key]</em>"</p>
  
  <p>I do not understand the solution to this in version (2.0) and for this case (time series data with fixed length input). Any ideas on adding attention to this type of problem is welcome.</p>