University of Oxford; DeepMind; University of Oxford, DeepMind
Bobby He, James Martens, Guodong Zhang, Aleksandar Botev, Andrew Brock, Samuel L Smith, Yee Whye Teh
signal propagation;neural networks and kernels;deep transformers;self-attention;residual connections;layer normalisation;rank collapse;positional encoding
Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation