While modern  generative  models are able to synthesize high-fidelity, visually appealing images, successfully generating examples that are useful for recognition tasks remains an elusive goal. To this end, our key insight is that the examples should be synthesized to recover classifier decision boundaries that would be learned from a large amount of real examples. More concretely, we treat a classifier trained on synthetic examples as ''student'' and a classifier trained on real examples as ''teacher''. By introducing knowledge distillation into a meta-learning framework, we encourage the generative model to produce examples in a way that enables the student classifier to mimic the behavior of the teacher. To mitigate the potential gap between student and teacher classifiers, we further propose to distill the knowledge in a progressive manner, either by gradually strengthening the teacher or weakening the student. We demonstrate the use of our model-agnostic distillation approach to deal with data scarcity, significantly improving few-shot learning performance on miniImageNet and ImageNet1K benchmarks. Over the past decade, generative image modeling has progressed remarkably with the emergence of deep learning techniques. Modern generative models, such as the variants of generative adversarial networks (GANs) (Goodfellow et al., 2014; Karras et al., 2018; Brock et al., 2019) and variational auto-encoders (VAEs) (Kingma & Welling, 2014; Razavi et al., 2019) , are able to synthesize high-fidelity, visually appealing images, with successful applications ranging from superresolution (Ledig et al., 2017) to artistic manipulation (Zhu et al., 2017) . However, when it comes to their use in discriminative visual recognition tasks, these images are still far from satisfactory. The performance of the classifiers trained on synthetic images is substantially inferior to that of the classifiers trained on real images (Dai et al., 2017; Shmelkov et al., 2018) . In this paper, we make a step towards building generative models that are recognition task oriented, thus enabling synthesizing examples in a way that helps the classification algorithm learn better classifiers. This is of great promise to deal with data scarcity in real-world scenarios, such as addressing few-shot learning which aims to recognize novel categories from one, or only a few, annotated examples (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017) . Instead of matching training data distribution or aiming for realism, our key insight is that the examples should be synthesized to recover or stabilize classifier decision boundaries that would be learned from real samples. More precisely, let us consider a 2-way 2-shot classification problem in Figure 1 . We aim to learn a good classifier (the purple boundary in Figure 1a ) that distinguishes the two classes based on 2 training examples per class. Ideally, the hope is that the boundary in Figure 1a should be as close as possible to the classifier (the red boundary in Figure 1b ) that would be learned from a large set of real samples (on the order of hundreds or thousands of). To this end, we synthesize additional examples for each class based on its available 2 examples, so that the resulting classifier (the red solid boundary in Figure 1c ) produced by the synthesized examples together with the few real examples remains unchanged from the desired classifier (the red dashed boundary in Figure 1b or Figure 1c ). To minimize the discrepancy between the classifier trained on synthetic examples and the classifier trained on real examples, we leverage the idea of knowledge distillation proposed by Hinton et al. (2015) . While knowledge distillation was developed for model compression, in which a lightweight "student" model is trained to mimic the behavior of a larger, high-capacity "teacher" model, here Figure 1 : Knowledge distillation of generative models for few-shot learning. We aim to recognize two novel classes from 2 examples per class (Figure 1a) . The desired classifier is the one that would be learned from abundant real examples (Figure 1b) . To this end, we distill the knowledge of the desired large-sample (dashed) classifier into a generative model, and thus enable it to produce additional examples from the few real examples in a way that minimizes the discrepancy between the (solid) classifier trained on synthesized examples together with the few real examples and the large-sample (dashed) classifier (Figure 1c) . Real examples are shown as squares, synthetic examples as triangles, and classifier decision boundaries as solid or dashed lines. we focus on models of the same capacity but trained on different types of data. Specifically, we treat the classifier trained on synthetic examples along with few real examples as the student, and treat the classifier trained on a large amount of real examples as the teacher. Using the distillation loss function (Hinton et al., 2015) , our generative model is encouraged to produce such kind of examples that enable the student classifier to output the distribution of class probabilities predicted by the teacher. To make the generative model applicable to a broad range of categories, we further incorporate the distillation process into a meta-learning framework as in . Through meta-learning, we construct a variety of few-shot learning tasks from base categories with abundant labeled examples, thus being able to learn a generic, category-shared generative model. For a novel few-shot recognition task on unseen categories, we use the learned generative model to synthesize additional examples and produce an augmented training set for learning classifiers. While we show that the basic framework of meta-learning with distillation already performs well, directly distilling the knowledge into the generative model might be still challenging. This is because the decision boundaries of the student and teacher classifiers could be far away from each other at the beginning of the training, if the teacher is produced by a large amount of real examples while the student has access to only few real examples. To mitigate this issue, we propose to distill the knowledge in a progressive manner and explore two different avenues of dual directions -(1) we start with a teacher and a student trained on a small number of real examples, and we gradually strengthen the teacher by re-training it with increasing number of real examples; (2) we start with a teacher and a student trained on a large number of real examples, and we gradually weaken the student by removing its real examples. During both of the processes, the generative model is trained progressively as well by producing more synthetic examples. Finally, we introduce ensemble of distillation and train independently several distillation processes on different student-teacher pairs, thus leading to a diverse collection of generative models and effectively reducing the variance of few-shot classifiers. We demonstrate that our progressive distillation facilitates learning generative models to be directly useful for discriminative recognition tasks, significantly improving few-shot learning performance on both the widely benchmarked miniImageNet and much larger-scale ImageNet1K datasets. In particular, our approach is general and model-agnostic, which can synthesize in different feature spaces and can be combined with different meta-learning models to improve their performance. In this paper, we introduced a general framework of meta-learning with knowledge distillation to guide the learning of generative models to be directly useful for discriminative recognition tasks. We apply our approach to few-shot learning, where the amount of available data is very limited and therefore this kind of generation is in particular helpful. By progressively distilling the knowledge and benefiting from diverse teachers, our approach achieves state-of-the-art results on heavily benchmarked miniImageNet and ImageNet1K few-shot classification datasets.
This paper introduces progressive knowledge distillation for learning generative models that are recognition task oriented This paper demonstrates easy-to-hard curriculum learning to train a generative model to improve few-shot classification.