Dept of Computer Science, ETH Zürich; MPI for Intelligent Systems, Tübingen; Department of Mathematics and Computer Science, University of Basel; Dept of Computer Science, ETH Zürich; Dept of Computer Science, ETH Zürich; Robotics & ML, CSEM SA, Alpnach, Switzerland
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, Aurelien Lucchi
Theory of Transformers;Transformers;Rank Collapse;Gradient Vanishing
Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse