<p>In the original Bahdanau's paper, the decoder has only a single LSTM layer. There are various approaches how to deal with multiple layers. The quite usual thing to do is to do the attention between the layers (which you obviously did not do, see e.g., <a href="https://arxiv.org/abs/1609.08144" rel="nofollow noreferrer">a paper by Google</a>). If you use multiple decoder layers like this, you can use only the last layer (i.e., do <code>h_decoder[1]</code>), alternatively, you can concatenate the layers (i.e., in torch call <a href="https://pytorch.org/docs/stable/generated/torch.cat.html#torch-cat" rel="nofollow noreferrer"><code>torch.cat</code></a> or <a href="https://www.tensorflow.org/api_docs/python/tf/concat" rel="nofollow noreferrer"><code>tf.concat</code></a> in the 0-th dimension).</p>
<p>The matrices <em>W</em><sub>decoder</sub> and <em>W</em><sub>encoder</sub> ensure that both the encoder and decoder states get projected to the same dimension (regardless if you what you did with the decoder layers), so you can do the summation.</p>
<p>The only remaining issue is that the encoder states have the max-length dimension. The trick here is that you need to add a dimension to the projected decoder state, so the summation gets broadcasted and the projected decoder state get summed with all the encoder states. In PyTorch, just call <a href="https://pytorch.org/docs/stable/generated/torch.unsqueeze.html" rel="nofollow noreferrer"><code>unsqueeze</code></a>, in TensorFlow <a href="https://www.tensorflow.org/api_docs/python/tf/expand_dims" rel="nofollow noreferrer"><code>expand_dims</code></a> in the 0-th dimension on the projected decoder state.</p>
State dimensions in Bahdanau Attention
<p>I'm currently trying to compute this function to get Bahdanau's attention
<a href="https://i.stack.imgur.com/hpnlf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hpnlf.png" alt="enter image description here" /></a></p>
<p>My question is with the H for the decoder and the encoder.</p>
<p>In one implementation, I see an h encoder with the dimensions: [max source Len, batch size, hidden size]</p>
<p>and a h decoder  with the following dimensions: [#lstm layers, batch size, hidden dim]</p>
<p>How can I compute the addition if the dimensions for the W matrices have to be the same according to:
<a href="https://blog.floydhub.com/attention-mechanism/#bahdanau-att-step1" rel="nofollow noreferrer">https://blog.floydhub.com/attention-mechanism/#bahdanau-att-step1</a></p>
<p>Thanks for the help</p>