 Knowledge distillation

In machine learning, knowledge distillation is the process of transferring knowledge from a large model to a smaller one.While large models (such as very deep neural networks or ensembles of many models) are able to better learn how to generalise from the data compared to smaller ones, they have the obvious drawback of being computationally expensive to evaluate, therefore limiting their applications to sufficiently powerful hardware.Knowledge distillation allows to transfer the knowledge learnt by a large model to a smaller one, that would not be able to easily learn it directly from the data itself, producing a new model that is faster to evaluate, and therefore deployable on less powerful hardware (such as a mobile device), while at the same time experiencing only a small loss of classification performance compared to the original, large model.