 The gradient of the knowledge distillation loss formula_14 with respect to the logit of the distilled model formula_15 is given by
where formula_17 are the logits of the large model.For large values of formula_3 this can be approximated as
and under the zero-mean hypothesis formula_20 it becomes formula_21, which is the derivative of formula_22, i.e.the loss is equivalent to matching the logits of the two models, as done in model compression.