In this work, we study how the large-scale pretrain-finetune framework changes the behavior of a neural language generator. We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task. We find that after standard fine-tuning, the model forgets important language generation skills acquired during large-scale pre-training. We demonstrate the forgetting phenomenon through a detailed behavior analysis from the perspectives of context sensitivity and knowledge transfer. Adopting the concept of data mixing, we propose an intuitive fine-tuning strategy named "mix-review''. We find that mix-review effectively regularize the fine-tuning process, and the forgetting problem is largely alleviated. Finally, we discuss interesting behavior of the resulting dialogue model and its implications.
 Large-scale unsupervised pre-training (Peters et al., 2018; Devlin et al., 2018; Song et al., 2019) has recently been shown to greatly boost the performance of natural language processing (NLP) models, and has attracted much research interest. Despite its huge success, there is a fundamental question remaining to be answered: Is there some crucial weakness in the standard NLP pretrain-finetune framework? In this work, we take the viewpoint of language generation and show that the answer is, to some extent, yes. In particular, we find that the key to answer this question is a concept we denote as data separation. Although various unsupervised pre-training strategies have been proposed for better utilization of large-scale text data, on a high level the pretrain-finetune framework can be viewed as a simple two-stage procedure: (1) use large-scale text data to pre-train the model, and (2) use target task data to fine-tune the model. Data separation refers to (almost) zero-overlapping data usage of the two stages. In this work we study the pretrain-finetune framework from the viewpoint of neural language generation (NLG). In particular, we focus on the open-domain dialogue response task, for the following reasons: (1) There is high similarity between the target dialogue response task (conditional NLG) and the pre-training language modeling (LM) objective, so we expect that language generation skills learnt during pre-training can be well transferred to the down-stream target task. (2) The sequenceto-sequence (seq2seq) nature of the model allows us to characterize the model's generation behavior in various ways (e.g. context sensitivity). We briefly summarize our contributions as follows. To study how pretrain-finetuning changes the model's behavior, we conduct a behavior analysis from the perspectives of context sensitivity and knowledge transfer. Our main finding is that in the fine-tuning stage, data separation causes the model to forget important language generation skills acquired during pre-training. Motivated by this analysis, we adopt the concept of data mixing and propose a mix-review fine-tuning strategy, where we combine the pre-training and fine-tuning objective. We find that mix-review effectively regularize the fine-tuning process, and the forgetting problem is largely alleviated. Finally, we demonstrate and discuss interesting behavior of the resulting dialogue model and its implications. In this work, we analyze forgetting problem for the standard NLP pretrain-finetune framework in the viewpoint of language generation. We adopt the concept of "data mixing" and propose the mix-review fine-tuning strategy. We demonstrate that mix-review can effectively help the model remember important generation skills learned during pre-training. Through a detailed behavior analysis, we find that under the surface of the performance boost for standard metrics, large-scale pre-training changes the model's generative behavior in various profound ways (e.g. context sensitivity). More importantly, the behavior change is influenced by the nature of data itself. For example, we demonstrate that we can discuss news with the resulting dialogue model, even when the fine-tuning data is not about news (Dailydialogue). This opens the exciting possibility of a completely data-driven way to customize a language generator.
We identify the forgetting problem in fine-tuning of pre-trained NLG models, and propose the mix-review strategy to address it. This paper analyzes the forgetting problem in the pretraining-finetuning framework from the perspective of context sensitivity and knowledge transfer, and proposes a fine-tuning strategy which outperforms the weight decay method. Study of the forgetting problem in the pretrain-finetune framework, specifically in dialogue response generation tasks, and proposal of a mix-review strategy to alleviate the forgetting issue.