The University of North Carolina at Chapel Hill; The University of Texas at Austin; The Chinese University of Hong Kong; MIT, Harvard University, The University of North Carolina at Chapel Hill
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, Tianlong Chen
Sparse Mixture-of-Experts;Efficiency;Merging;Compression
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy