Meta AI; Johns Hopkins University
Stratified Mixture-of-Experts;Parameter-efficiency;Dynamic capacity
Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity