<p>Using only the last hidden state without attention has insufficient representation power, especially when the hidden size is small. A few systems prior the invention of attention are </p>

<p><a href="https://arxiv.org/abs/1409.3215" rel="nofollow noreferrer">https://arxiv.org/abs/1409.3215</a></p>

<p><a href="https://arxiv.org/abs/1506.05869" rel="nofollow noreferrer">https://arxiv.org/abs/1506.05869</a></p>
Does simple deep seq2seq without peeking or attention converge?
<p>Are there any successful application of deep seq2seq model where the decoder read ONLY the encoder's output state (final step of encoder's internal state) at its first step, and carry out multiple steps decoding? </p>

<p>I.e. no peeking, no attention etc. At each step the decoder's input is only the previous step's output and state.</p>

<p>I could see a few seq2seq autoencoder implementation, wonder if they really converge after a long time of training, especially when the internal state is small.</p>