 Knowledge distillation has been successfully used in several applications of machine learning such as object detection, acoustic models, and natural language processing.What it is desired from a good model is good generalization capability, therefore transferring the knowledge from a large to a small model needs to somehow teach to the latter how to generalise in the same way as the former.If both models are trained on the same data, the small model will be worse at generalisation because of its inferior expressive power.