1. The paper studies the case where the student distills knowledge to the teacher, which improves the teacher's performance. However, the improvements could potentially be due to regularization effects rather than distillation as claimed, since all fine-tuning is performed for 10 epochs and without early-stopping. The fine-tuning on GLUE without validation early-stopping usually has very high variances, proper ablation studies are needed to verify.
2. The performance gains (especially, the Community KD) over standard one-way distillation seem quite marginal based on the experiments when compared to other BERT distillation techniques like BERT-PKD, TinyBERT, MobileBERT, or BERT-of-Theseus. This is not a good signal given the training process is more complex with co-training and co-distillation compared to other distillation techniques.
3. Evaluations are limited to BERT models only. Testing on other PLMs would be more convincing.
1. The paper studies the case where the student distills knowledge to the teacher, which improves the teacher's performance. However, the improvements could potentially be due to regularization effects rather than distillation as claimed, since all fine-tuning is performed for 10 epochs and without early-stopping. The fine-tuning on GLUE without validation early-stopping usually has very high variances, proper ablation studies are needed to verify.