Georgia Institute of Technology; Google Research
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby
mixture of experts;sparse;vision;language;deep learning;superglue;imagenet
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints