<p>I went through this <a href="https://arxiv.org/pdf/1508.04025.pdf" rel="noreferrer">Effective Approaches to Attention-based Neural Machine Translation</a>. In the section <em>3.1</em> They have mentioned the difference between two attentions as follows,</p>
<ol>
<li><p><strong>Luong attention</strong> used top hidden layer states in both of encoder and decoder. <em>But <strong>Bahdanau attention</strong> take concatenation of forward and backward source hidden state (Top Hidden Layer)</em>.</p>
</li>
<li><p>In <strong>Luong attention</strong> they get the decoder hidden state at time <em><strong>t</strong></em>. Then calculate attention scores and from that get the context vector which will be concatenated with hidden state of the decoder and then predict.</p>
<p>But in the <strong>Bahdanau</strong> at time <em><strong>t</strong></em> we consider about <em><strong>t-1</strong></em> hidden state of the decoder. Then we calculate alignment , context vectors as above. But then we concatenate this context with hidden state of the decoder at <em><strong>t-1</strong></em>. So before the softmax this concatenated vector goes inside a GRU.</p>
</li>
<li><p>Luong has diffferent types of alignments. <strong>Bahdanau</strong> has only concat score alignment model.</p>
</li>
</ol>
<p><a href="https://i.stack.imgur.com/tiQkz.png" rel="noreferrer"><img src="https://i.stack.imgur.com/tiQkz.png" alt="Alignment methdods" /></a></p>
What is the difference between Luong attention and Bahdanau attention?
<p>These two attentions are used in <strong>seq2seq</strong> modules. The two different attentions are introduced as multiplicative and additive attentions in <a href="https://www.tensorflow.org/versions/master/api_guides/python/contrib.seq2seq" rel="noreferrer">this</a> TensorFlow documentation. What is the difference?</p>