Knowledge Distillation (KD) is a widely used technique in recent deep learning research to obtain small and simple models whose performance is on a par with their large and complex counterparts. Standard Knowledge Distillation tends to be time-consuming because of the training time spent to obtain a teacher model that would then provide guidance for the student model. It might be possible to cut short the time by training a teacher model on the fly, but it is not trivial to have such a high-capacity teacher that gives quality guidance to student models this way. To improve this, we present a novel framework of Knowledge Distillation exploiting dark knowledge from the whole training set. In this framework, we propose a simple and effective implementation named Distillation by Utilizing Peer Samples (DUPS) in one generation. We verify our algorithm on numerous experiments. Compared with standard training on modern architectures, DUPS achieves an average improvement of 1%-2% on various tasks with nearly zero extra cost. Considering some typical Knowledge Distillation methods which are much more time-consuming, we also get comparable or even better performance using DUPS. Recent years have witnessed continuous development of deep neural network models. A general trend is that improvements in model performance are usually coupled with more complex architecture designs and higher cost of computation. In order to obtain more compact models with higher quality, the idea of Knowledge Distillation (KD) first emerged in the form of knowledge transfer between models (Buciluǎ et al., 2006) . KD takes advantage of the "dark knowledge" by transferring it from teacher models to student models so as to facilitate the latter's training process (Hinton et al., 2015) . Student models, with the availability of softened output vectors from teacher models in KD, have access to richer information in comparison to directly learning from hard labels provided by training set. KD significantly improves smaller models' performance, and thus it further allows model compression. Although great progress has been made in this area, much more training cost is incurred due to involved time-consuming mid-output (e.g. feature maps) alignment when training student models, on top of extra training of a huge teacher model. It is ad meaningful objective of finding more efficient KD methods. Recent works by (Furlanello et al., 2018) and (Lan et al., 2018b) show that a stronger teacher model is not the necessary condition for improving the student model. Their research shows that it is possible that the student model's performance can be significantly improved by an identically structured teacher model. Although the techniques remain inefficient due to the cost of multi-generation (at least one extra) training of teacher models, these works give important hints that cheaper teachers with considerable effectiveness may exist. Recently (Yang et al., 2018) extend these works, trying to obtain continuously improved teachers by introducing the cyclic learning rate technique in one-generation training. They propose Snapshot Distillation (SD), which uses models obtained from earlier checkpoints as teachers and skips the process of separately training a teacher model. Inspired by recent interesting ideas of dataset distillation for objectives on other research areas, we propose a novel approach for KD in this paper. Instead of relying on the assitance of a separate teacher model or checkpoint, we exploit hidden knowledge in the dataset to generate a surrogate teacher. Specifically, we first define a more general framework of knowledge distillation utilizing the whole dataset to generate extra supervision signals, rather than using a single sample alone. Then we propose a very simple yet effective implementation of one-generation KD, called Distillation by Utilizing Peer Samples (DUPS). In DUPS, each sample borrows continuously boosted secondary information from a random subset of peer samples belonging to the same category on the fly. We perform extensive experiments on CIFAR100 dataset, with various modern architectures such as PreActResNet, WideResNet, and ResNeXt, demonstrating that our proposed DUPS gains significant improvement compared to standard SGD training with nearly zero extra computation cost. DUPS also outperforms recent one-generation KD method SnapShot Distillation (Yang et al., 2018) on most architectures. Moreover, we validate our algorithm on more practical tasks, include ImageNet classification, transfer learning, and language model. Experiments show that DUPS is generally effective across different tasks. In summary, our main contributions include: 1) To the best of our knowledge, we are the first to propose an extension framework of Knowledge Distillation utilizing the whole training set other than a single sample. 2) Under this framework we implement a general on-the-fly algorithm DUPS which achieves significant improvement at almost no extra cost. The rest of the paper is organized as follows. Section 2 presents prior works related to this paper. Section 3 introduces our methodology of the general knowledge distillation framework. Section 4 demonstrates our experimental results and provides some discussions. Section 5 concludes the paper. Here we give a short discussion about how and why DUPS brings benefits. We first demonstrate some empirical characteristics of DUPS observed in our experiments. We plot the learning curve of the whole training procedure of PreActResNet18 as Fig. 1 . For better demonstration purpose, we divide the training process into only 4 stages for DUPS training, with each stage consisting of 40 epochs. We observe that SGD and DUPS display almost the same standard of performance in the first stage as expected. While at the 41th epoch, both training and test accuracy of DUPS get a sharp rise due to involving teacher signal generated in the 40th epoch. Then both training and test accuracy drop slightly for a few epochs, and then return to the trend of slowly rising for the remaining epochs until next stage. A similar phenomenon also appears at the beginning of next stage, although the magnitude of accuracy improvement becomes much more smaller. As the model reaches the beginning of final stage, we no longer see increases in accuracy since training is nearly saturated. We notice that since the first sharp rising, DUPS continuously outperforms SGD by a stable gap for the following training epochs until convergence. We also investigate the influence of different choices of hyper-parameters specific to DUPS. The most important two are the number of stages and number of random peer samples. We run a grid search method to validate different combinations of these two variables. We use update intervals, or equivalently number of epochs per stage, instead of number of stages for clarity in this experiments. In Fig. 2 we see that performance of DUPS does not seem to be very sensitive to most combinations of the hyperparameters. When the number of peer samples increases to 5 or more, model accuracy tends to be over 77.3%. Even when the number of peer samples is low, a good choice of the value of update interval can boost the model performance significantly. For example, when number of peer samples is 1 and update interval is set to be between 20 to 40, DUPS still delivers satisfying results which is comparable to its best performance. Low accuracy of the model only happens consistently when the value of update interval is large. If update interval is set to 80, the model, teacher-student knowledge transfer only takes place once during the whole training process. Consequently, the opportunity to distill the knowledge obtained from the dataset is too rare for the model to benefit from DUPS. In this paper, we have introduced a general framework for one-generation KD: We incorporate the information contained within the dataset into teacher-student optimization. We have also proposed an effective implementation of this general framework named DUPS. With extensive experiments, this simple yet effective algorithm is verified to be effective in improving model performance in tasks like image classification, transfer learning and language modeling with almost no additional cost in training resources. The demonstrated success of DUPS imply that utilizing dataset information during training potentially allow us to gain even more benefits.
We present a novel framework of Knowledge Distillation utilizing peer samples as the teacher Proposes a method for improving the effectiveness of knowledge distillation by softening the labels used and employing a dataset instead of a single sample. This paper proposes to address the extra computational cost of training with knowledge distillation, building on the recently proposed Snapshot Distillation technique.