 Before the introduction of Transformers, most state-of-the-art natural language processing systems relied on gated recurrent neural networks (RNNs), such as LSTMs, with added attention mechanisms.The Transformer built upon these attention technologies without using an RNN structure, highlighting the fact that the attention mechanisms alone, without recurrent sequential processing, are powerful enough to achieve the performance of RNNs with attention.Gated RNNs process tokens sequentially, maintaining an internal state vector that contains a latent representation of the data seen after every token.