- The claim is grounded in empirical findings and does not provide a solid mathematical foundation.
- Although I acknowledge that KD and LS are not identical, I believe KD can be viewed as a special form of LS. This is particularly true when the teacher network is uniformly distributed and the temperature is set at 1, then LS and KD are equivalent.
- The authors only compared one of the existing works in this area and did not sufficiently address related works.
Here are some related works for LS and KD:
Lee, Dongkyu, Ka Chun Cheung, and Nevin Zhang. "Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
Zhang, Zhilu, and Mert Sabuncu. "Self-distillation as instance-specific label smoothing." Advances in Neural Information Processing Systems 33 (2020): 2184-2195.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. "Revisit knowledge distillation: a teacher-free framework." arXiv preprint arXiv:1909.11723, 2019.
Yun, Sukmin, et al. "Regularizing class-wise predictions via self-knowledge distillation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
- Although I acknowledge that KD and LS are not identical, I believe KD can be viewed as a special form of LS. This is particularly true when the teacher network is uniformly distributed and the temperature is set at 1, then LS and KD are equivalent.