VITA Group, University of Texas at Austin
Tianlong Chen, Zhenyu Zhang, AJAY JAISWAL, Shiwei Liu, Zhangyang Wang
Sparse Mixture-of-Experts;Random Routing;Transformer Training;Dropout
Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers