 Model compression, a methodology to compress the knowledge of multiple models into a single neural network, was introduced in 2006.Compression was achieved by training a smaller model on large amounts of pseudo-data labelled by a higher-performing ensemble, optimising to match the logit of the compressed model to the logit of the ensemble.Knowledge distillation is a generalisation of such approach, introduced by Geoffrey Hinton et al.