 If ground truth is available for the transfer set, the process can be strengthened by adding to the loss the cross entropy between the output of the distilled model (computed with formula_9) and the known label formula_10
where the component of the loss with respect to the large model is weighted by a factor of formula_12 since, as the temperature increases, the gradient of the loss with respect to the model weights scales by a factor of formula_13.Under the assumption that the logits have zero mean, it is possible to show that model compression is a special case of knowledge distillation.The gradient of the knowledge distillation loss formula_14 with respect to the logit of the distilled model formula_15 is given by
where formula_17 are the logits of the large model.