KAIST; National University of Singapore; KAIST, AITRICS
Seanie Lee, Minki Kang, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi
self-distillation;adaptation of pre-trained models;regularization
Self-Distillation for Further Pre-training of Transformers