Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly,  the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available. Self-supervised learning on a general-domain text corpus followed by end-task learning is the twostaged training approach that enabled deep-and-wide Transformer-based networks (Vaswani et al., 2017) to advance language understanding (Devlin et al., 2018; Yang et al., 2019b; Sun et al., 2019b; . However, state-of-the-art models have hundreds of millions of parameters, incurring a high computational cost. Our goal is to realize their gains under a restricted memory and latency budget. We seek a training method that is well-performing, general and simple and can leverage additional resources such as unlabeled task data. Before considering compression techniques, we start with the following research question: Could we directly train small models using the same two-staged approach? In other words, we explore the idea of applying language model (LM) pre-training and task fine-tuning to compact architectures directly. This simple baseline has so far been overlooked by the NLP community, potentially based on an underlying assumption that the limited capacity of compact models is capitalized better when focusing on the end task rather than a general language model objective. Concurrent work to ours proposes variations of the standard pre-training+fine-tuning procedure, but with limited generality (Sun et al., 2019a; Sanh, 2019) . We make the surprising finding that pre-training+fine-tuning in its original formulation is a competitive method for building compact models. For further gains, we additionally leverage knowledge distillation (Hinton et al., 2015) , the standard technique for model compression. A compact student is trained to recover the predictions of a highly accurate teacher. In addition to the posited regularization effect of these soft labels (Hinton et al., 2015) , distillation provides a means of producing pseudo-labels for unlabeled data. By regarding LM pre-training of compact models as a student initialization strategy, we can take advantage of both methods. The resulting algorithm is a sequence of three standard training operations: masked LM (MLM) pre-training (Devlin et al., 2018) , task-specific distillation, and optional fine-tuning. From here on, we will refer to it as Pre-trained Distillation (PD) ( Figure 1 ). As we will show in Get loss L ← − y P Ω (y|x) log P θ (y|x) In a controlled study following data and model architecture settings in concurrent work (Section 4), we show that Pre-trained Distillation outperforms or is competitive with more elaborate approaches which use either more sophisticated distillation of task knowledge (Sun et al., 2019a) or more sophisticated pre-training from unlabeled text (Sanh, 2019) . The former distill task knowledge from intermediate teacher activations, starting with a heuristically initialized student. The latter fine-tune a compact model that is pre-trained on unlabeled text with the help of a larger LM teacher. One of the most noteworthy contributions of our paper are the extensive experiments that examine how Pre-trained Distillation and its baselines perform under various conditions. We investigate two axes that have been under-studied in previous work: model size and amount/quality of unlabeled data. While experimenting with 24 models of various sizes (4m to 110m parameters) and depth/width trade-offs, we observe that pre-trained students can leverage depth much better than width; in contrast, this property is not visible for randomly-initialized models. For the second axis, we vary the amount of unlabeled data, as well as its similarity to the labeled set. Interestingly, Pretrained Distillation is more robust to these variations in the transfer set than standard distillation. Finally, in order to gain insight into the interaction between LM pre-training and task-specific distillation, we sequentially apply these operations on the same dataset. In this experiment, chaining the two operations performs better than any one of them applied in isolation, despite the fact that a single dataset was used for both steps. This compounding effect is surprising, indicating that pre-training and distillation are learning complementary aspects of the data. Given the effectiveness of LM pre-training on compact architectures, we will make our 24 pretrained miniature BERT models publicly available in order to accelerate future research.
Studies how self-supervised learning and knowledge distillation interact in the context of building compact models. Investigates training compact pre-trained language models via distillation and shows that using a teacher for distilling a compact student model performs better than directly pre-training the model. This submission shows that pre-training a student directly on masked language modeling is better than distillation, and the best is to combine both and distill from that pre-trained student model.