 Knowledge distillation consists of training a smaller network, called "distilled model", on a dataset called transfer set (different than the dataset used to train the large model) using as loss function the cross entropy between the output of the distilled model formula_5 and the output formula_6 produced by the large model on the same record (or the average of the individual outputs, if the large model is an ensemble), using a high value of softmax temperature formula_3 for both models
In this context, a high temperature increases the entropy of the output, and therefore provides more information to learn for the distilled model compared to hard targets, at the same time reducing the variance of the gradient between different records and therefore allowing higher learning rates.If ground truth is available for the transfer set, the process can be strengthened by adding to the loss the cross entropy between the output of the distilled model (computed with formula_9) and the known label formula_10
where the component of the loss with respect to the large model is weighted by a factor of formula_12 since, as the temperature increases, the gradient of the loss with respect to the model weights scales by a factor of formula_13.Under the assumption that the logits have zero mean, it is possible to show that model compression is a special case of knowledge distillation.