<p>I asked myself the same thing today and found this question. I have never implemented an attention mechanism myself, but from <a href="https://arxiv.org/abs/1409.0473" rel="noreferrer">this paper</a> it seems a little bit more than just a straight softmax. For each output y<sub>i</sub> of the decoder network, a context vector <strong>c</strong><sub>i</sub> is computed as a weighted sum of the encoder hidden states <strong>h</strong><sub>1</sub>, ..., <strong>h</strong><sub>T</sub>:</p>

<p><strong>c</strong><sub>i</sub> = α<sub>i1</sub><strong>h</strong><sub>1</sub>+...+α<sub>iT</sub><strong>h</strong><sub>T</sub></p>

<p>The number of time steps T may be different for each sample because the coefficients α<sub>ij</sub> are not vector of fixed size. In fact, they are computed by softmax(e<sub>i1</sub>, ..., e<sub>iT</sub>), where each e<sub>ij</sub> is the output of a neural network whose input is the encoder hidden state <strong>h</strong><sub>j</sub> and the decoder hidden state <strong>s</strong><sub>i-1</sub>:</p>

<p>e<sub>ij</sub> = f(<strong>s</strong><sub>i-1</sub>, <strong>h</strong><sub>j</sub>)</p>

<p>Thus, before y<sub>i</sub> is computed, this neural network must be evaluated T times, producing T weights α<sub>i1</sub>,...,α<sub>iT</sub>.  Also, <a href="https://github.com/ilivans/tf-rnn-attention/blob/master/attention.py" rel="noreferrer">this tensorflow impementation</a> might be useful.</p>
How can LSTM attention have variable length input
<p>The attention mechanism of LSTM is a straight softmax feed forward network that takes in the hidden states of each time step of the encoder and the decoder's current state.</p>

<p>These 2 steps seems to contradict and can't wrap my head around:
1) The number of inputs to a feed forward network needs to be predefined
2) the number of hidden states of the encoder is variable (depends on number of time steps during encoding).</p>

<p>Am I misunderstanding something? Also would training be the same as if I were to train a regular encoder/decoder network or would I have to train the attention mechanism separately?</p>

<p>Thanks in Advance</p>