 Knowledge distillation

In machine learning, knowledge distillation is the process of transferring knowledge from a large model to a smaller one.While large models (such as very deep neural networks or ensembles of many models) are able to better learn how to generalise from the data compared to smaller ones, they have the obvious drawback of being computationally expensive to evaluate, therefore limiting their applications to sufficiently powerful hardware.