Meta AI; University of Washington, Seattle
Mixture-of-Experts;Efficiency;Transformer;Architecture;Pretraining;LLM
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model