 Given a large model as a function of the vector variable formula_1, trained for a specific classification task, typically the final layer of the network is a softmax in the form
where formula_3 is a parameter called "temperature", that for a standard softmax is normally set to 1.The softmax operator converts the logit values formula_4 to pseudo-probabilities, and higher values of temperature have the effect of generating a softer distribution of pseudo-probabilities among the output classes.Knowledge distillation consists of training a smaller network, called "distilled model", on a dataset called transfer set (different than the dataset used to train the large model) using as loss function the cross entropy between the output of the distilled model formula_5 and the output formula_6 produced by the large model on the same record (or the average of the individual outputs, if the large model is an ensemble), using a high value of softmax temperature formula_3 for both models
In this context, a high temperature increases the entropy of the output, and therefore provides more information to learn for the distilled model compared to hard targets, at the same time reducing the variance of the gradient between different records and therefore allowing higher learning rates.