1. The paper needs a comprehensive analysis of sparse MoE, including the communication overhead (all to all). Currently, it's not clear where the performance gain comes from, basically, different number of experts incurs different communication overhead.
2. The evaluation needs experiments on distributed deployment and a larger model.
3. For the arguments that the existing approach has two key limitations, the authors should present key experiment results for demonstration.
2. The evaluation needs experiments on distributed deployment and a larger model.