Department of Computer Science, Stanford University; Department of Computer Science and Engineering, University at Buffalo, SUNY; Adobe Research
Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, Christopher Ré
Sparsity;Low-rank;Locality Sensitive Hashing;Kernel Approximation;Efficient Transformers
Scatterbrain: Unifying Sparse and Low-rank Attention