 Compression was achieved by training a smaller model on large amounts of pseudo-data labelled by a higher-performing ensemble, optimising to match the logit of the compressed model to the logit of the ensemble.Knowledge distillation is a generalisation of such approach, introduced by Geoffrey Hinton et al.in 2015, in a preprint that formulated the concept and showed some results achieved in the task of image classification.