title: Implementing Attention in Keras
body: <p>I am trying to implement attention in keras over a simple lstm:</p>
  
  <pre><code>model_2_input = Input(shape=(500,))
  #model_2 = Conv1D(100, 10, activation='relu')(model_2_input)
  model_2 = Dense(64, activation='sigmoid')(model_2_input)
  model_2 = Dense(64, activation='sigmoid')(model_2)
  
  model_1_input = Input(shape=(None, 2048))
  model_1 = LSTM(64, dropout_U = 0.2, dropout_W = 0.2, return_sequences=True)(model_1_input)
  model_1, state_h, state_c = LSTM(16, dropout_U = 0.2, dropout_W = 0.2, return_sequences=True, return_state=True)(model_1) # dropout_U = 0.2, dropout_W = 0.2,
  
  
  #print(state_c.shape)
  match = dot([model_1, state_h], axes=(0, 0))
  match = Activation('softmax')(match)
  match = dot([match, state_h], axes=(0, 0))
  print(match.shape)
  
  merged = concatenate([model_2, match], axis=1)
  print(merged.shape)
  merged = Dense(4, activation='softmax')(merged)
  print(merged.shape)
  model = Model(inputs=[model_2_input , model_1_input], outputs=merged)
  adam = Adam()
  model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
  </code></pre>
  
  <p>I am getting the error in line:</p>
  
  <pre><code>merged = concatenate([model_2, match], axis=1)
  </code></pre>
  
  <blockquote>
    <p>'Got inputs shapes: %s' % (input_shape)) ValueError: A <code>Concatenate</code>
    layer requires inputs with matching shapes except for the concat axis.
    Got inputs shapes: [(None, 64), (16, 1)]</p>
  </blockquote>
  
  <p>The implementation is very simple, just take dot product of lstm output and with the hidden states and use it as weighing function to compute the hidden state itself.</p>
  
  <p>How to resolve the error? Especially how to get the attention concept working?</p>