to the student model (shallow model). Hinton et al. [@hinton2015distilling] proposed a method to apply teacher prediction as a “soft-label” and distill teacher classifier’s knowledge to the student. Moreover, they proposed a temperature cross entropy instead of $L2$ distance as the loss function. Romero et al. [@romero2014fitnets] proposed a so-called “hint” procedure to guide the training of the student model. There are also other approaches to distill knowledge between different domains such as from RGB to depth images [@gupta2016cross; @su2016adapting]. Knowledge distillation has been also applied to the object detection task. Chen et al. [@chen2017learning] proposed a method which adopts all of the soft labeling (labels generated by the teacher), the hard labeling (the ground truth) and the hint procedure to transfer knowledge from the teacher with deep feature extractor to the student with a shallow feature extractor. They adopt a two-stage method (FasterRCNN [@ren2015faster]) in their system. Mehta et al. [@mehta2018object] applied the same procedure to one stage method (Tiny-Yolo v2).

Mullapudi et al. [@mullapudi2018online] proposed an online model distillation for efficient segmentation. They adopt a light CNN model as a student and a heavy model as a teacher. At the inference time, the student model is trained periodically 