Peking University; Beijing Institute of Technology; Microsoft Corporation
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, XIA SONG, Xian-Ling Mao, Heyan Huang, Furu Wei
On the Representation Collapse of Sparse Mixture of Experts