ByteDance; The University of Hong Kong; University of Oxford
Ruifei He; Shuyang Sun; Jihan Yang; Song Bai; Xiaojuan Qi
Knowledge Distillation As Efficient Pre-Training: Faster Convergence, Higher Data-Efficiency, and Better Transferability