input image and the weights of student’s TKD decoders get updates at specific frames from the oracle model’s result. Finally, we design a decision procedure using an LSTM model, to generate the signals that indicate the right timing to use the Oracle knowledge.

Specifically, we train Tiny-Yolo with a general decoder over the COCO dataset [@lin2014microsoft]. The design of Tiny-Yolo has two general decoders to improve the accuracy of different object sizes. We first make a copy of the general decoders bounded together as TKD decoder. The TKD decoder is updated during the inference stage. We only update the last three layers of Tiny-Yolo and treat it as the decoder, since it yields enough performance in practice. We keep the general decoder from Tiny-Yolo together with the TKD decoder to make the final detection. TKD decoder and general decoder are executed in two parallel threads which do not increase the latency. This will preserve the chance of detecting viable objects addressing the challenge (3) in Sec. \[distillation\].

Distillation Loss
-----------------

Before describing our distillation loss, we provide a brief overview of the other distillation loss functions. First, Chen et al. [@chen2017learning] proposed a combination of hint procedure and weighted loss 