University of Illinois at Urbana-Champaign; Microsoft
Liliang Ren, Yang Liu, Shuohang Wang, Yichong Xu, Chenguang Zhu, Cheng Xiang Zhai
Sequence Modeling;Modularity;Sparsity;Attention Mechanism;State Space Model;Mixture of Experts;Neural Network;Transformer
Sparse Modular Activation for Efficient Sequence Modeling