 This problem was addressed by the introduction of attention mechanisms.Attention mechanisms let a model directly look at, and draw from, the latent state at any earlier point in the sentence.The attention layer can access all previous latent states and weighs them according to some learned measure of relevancy to the current token, providing sharper information about far-away relevant tokens.