 For large values of formula_3 this can be approximated as
and under the zero-mean hypothesis formula_20 it becomes formula_21, which is the derivative of formula_22, i.e.the loss is equivalent to matching the logits of the two models, as done in model compression.Doug Wozniak

Douglas Chester Wozniak is a Republican member of the Michigan House of Representatives.