Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a distilled machine learning model. One of the methods includes training a cumbersome machine learning model, wherein the cumbersome machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes; and training a distilled machine learning model on a plurality of training inputs, wherein the distilled machine learning model is also configured to receive inputs and generate scores for the plurality of classes, comprising: processing each training input using the cumbersome machine learning model to generate a cumbersome target soft output for the training input; and training the distilled machine learning model to, for each of the training inputs, generate a soft output that matches the cumbersome target soft output for the training input.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method performed by one or more computers, the method comprising: training a cumbersome machine learning model, wherein the cumbersome machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes; and training a distilled machine learning model on a plurality of training inputs, wherein the distilled machine learning model is also configured to receive inputs and generate scores for the plurality of classes, comprising: processing each training input using the cumbersome machine learning model to generate a cumbersome target soft output for the training input; and training the distilled machine learning model to, for each of the training inputs, generate a soft output that matches the cumbersome target soft output for the training input. 2. The method of claim 1, wherein a soft output of a machine learning model comprises a respective soft score for each of the plurality of classes generated by a last layer of the machine learning model, and wherein each soft score satisfies: q i = exp  ( z i / T ) Σ j  exp  ( z j / T ) , wherein qi is the score for a class i, zi is a weighted combination of outputs of a previous layer of the machine learning model received by the last layer, j ranges from 1 to a total number of classes in the plurality of classes, T is a temperature constant, and wherein to generate the soft scores, T is set to a higher value than used to generate scores for the classes after the machine learning model has been trained. 3. The method of claim 2, wherein the value used for T to generate scores for the classes after the machine learning model has been trained is 1. 4. The method of claim 3, further comprising setting the value of T to 1 for the distilled machine learning model after the distilled machine learning model has been trained. 5. The method of claim 1, wherein a soft output of a machine learning model comprises a respective soft score for each of the plurality of classes generated by a last layer of the machine learning model, and wherein the soft scores define a softer score distribution over the plurality of classes than scores generated by the machine learning model after the machine learning model has been trained. 6. The method of claim 1, wherein the cumbersome machine learning model is an ensemble model that is a composition of a plurality of machine learning models, and wherein the respective score generated by the cumbersome machine learning model for each class is a combination of scores generated for the class by the machine learning models. 7. The method of claim 6, wherein the plurality of ensemble models are trained separately. 8. The method of claim 6, wherein processing each training input using the cumbersome machine learning model comprises: generating a plurality of ensemble scores for the training input using the ensemble models; and combining the ensemble scores to generate the cumbersome target soft output for the training input. 9. The method of claim 6, wherein the plurality of machine learning models comprises one or more full machine learning models and one or more specialist machine learning models, wherein each full machine learning model is configured to receive an input and generate a respective full score for each of the plurality of classes, and wherein each specialist machine learning model is configured to receive an input and generate a respective specialist score for a respective subset of the plurality of classes. 10. The method of claim 9, wherein each specialist machine learning model is further configured to generate a specialist score for a dustbin class that is an aggregate of all of the classes that are not included in the subset for the specialist machine learning model. 11. The method of claim 9, wherein combining the ensemble scores to generate the cumbersome target soft output for the training input comprises: combining full scores generated for the training input by each of the full machine learning models to generate initial scores for the training input; and for each class for which the initial score is high enough, combining the full scores for the class with specialist scores generated for the class by the specialist models for which the class is included in the subset of classes for the specialist model. 12. The method of claim 9, further comprising: generating one or more subsets of the plurality of classes by clustering the classes that are frequently predicted together by the full models into the same subset; and assigning each subset to a respective specialist model. 13. The method of claim 1, wherein training the distilled machine learning model on the plurality of training inputs further comprises: training the distilled machine learning model to generate unsoftened outputs that match a known output for each of the training inputs. 14. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: training a cumbersome machine learning model, wherein the cumbersome machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes; and training a distilled machine learning model on a plurality of training inputs, wherein the distilled machine learning model is also configured to receive inputs and generate scores for the plurality of classes, comprising: processing each training input using the cumbersome machine learning model to generate a cumbersome target soft output for the training input; and training the distilled machine learning model to, for each of the training inputs, generate a soft output that matches the cumbersome target soft output for the training input. 15. The system of claim 14, wherein a soft output of a machine learning model comprises a respective soft score for each of the plurality of classes generated by a last layer of the machine learning model, and wherein each soft score satisfies: q i = exp  ( z i / T ) Σ j  exp  ( z j / T ) , wherein qi is the score for a class i, zi is a weighted combination of outputs of a previous layer of the machine learning model received by the last layer, j ranges from 1 to a total number of classes in the plurality of classes, T is a temperature constant, and wherein to generate the soft scores, T is set to a higher value than used to generate scores for the classes after the machine learning model has been trained. 16. The system of claim 15, wherein the value used for T to generate scores for the classes after the machine learning model has been trained is 1. 17. The system of claim 16, further comprising setting the value of T to 1 for the distilled machine learning model after the distilled machine learning model has been trained. 18. The system of claim 14, wherein a soft output of a machine learning model comprises a respective soft score for each of the plurality of classes generated by a last layer of the machine learning model, and wherein the soft scores define a softer score distribution over the plurality of classes than scores generated by the machine learning model after the machine learning model has been trained. 19. The system of claim 14, wherein the cumbersome machine learning model is an ensemble model that is a composition of a plurality of machine learning models, and wherein the respective score generated by the cumbersome machine learning model for each class is a combination of scores generated for the class by the machine learning models. 20. A computer program product encoded on one or more non-transitory computer storage media, the computer program product storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: training a cumbersome machine learning model, wherein the cumbersome machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes; and training a distilled machine learning model on a plurality of training inputs, wherein the distilled machine learning model is also configured to receive inputs and generate scores for the plurality of classes, comprising: processing each training input using the cumbersome machine learning model to generate a cumbersome target soft output for the training input; and training the distilled machine learning model to, for each of the training inputs, generate a soft output that matches the cumbersome target soft output for the training input.