<p>Attention mechanism in the context of encoder-decoder means that decoder at each time step "attends" to the "useful" parts of the encoder. This is implemented as, for example, averaging encoder's outputs and feeding that value (called context) into a decoder at a given time step.</p>

<p><code>dynamic_rnn</code> computes outputs of LSTM cells across all time steps and gives you the final value. So, there is no way to tell the model that the cell state at time step <code>t</code> should depend not only on the output of the previous cell and input, but also on additional information such as <strong>context</strong>. You can control computation at each time step of encoder or decoder LSTM using <code>raw_rnn</code>.</p>

<p>If I understand correctly, in this tutorial the author feeds ground truth input as input to the decoder at each time step. However, this is not the usual way it is done. Usually, you want to feed the output of decoder at time <code>t</code> as input to decoder at time <code>t+1</code>. In short, the input to the decoder at each time step is variable, whereas in <code>dynamic_rnn</code> it is predefined.</p>

<p>Refer to for more technical details: <a href="https://arxiv.org/abs/1409.0473" rel="nofollow noreferrer">https://arxiv.org/abs/1409.0473</a></p>
what is the difference between tf.nn.dynamic_rnn and tf.nn.raw_rnn in tensorflow?
<p>I went through <a href="https://github.com/ematvey/tensorflow-seq2seq-tutorials/blob/master/1-seq2seq.ipynb" rel="nofollow noreferrer">this tutorial</a> . In the last block it says that dynamic_rnn function cannot apply to calculate attention. But what I don't understand is all we need is the hidden state of the decoder in order to find the attention which will be work out with encoder symbols. </p>