Title: Topic Aware Transformer: Domain Shift for Unconditional Text Generation Model. Abstract: Our goal is to adapt pre-trained language models (PLMs) to support unconditional text generation tasks.
Because Transformer-based models are pre-trained on more massive and heterogeneous corpora than specific target corpus,
the gap between these corpora and the target corpus raises the question of whether these PLMs will actually benefit this task even after fine-tuning.
As the domain adaptation of PLMs needs to bridge this gap,
we propose a framework, Topic Aware Transformer (TAT), that adapts PLMs for target-aware text generation while alleviating catastrophic forgetting.
The motivation of TAT to distill the target-specific knowledge as topics,
and steer PLMs toward these topics.
This requirement and motivation lead us to introduce a topic steering layer (TSL) as an additional layer,
and Topic Distribution Modeling (TDM) as a training task.
Experiments show that these components resolve the gap as the domain shift,
and can tailor PLMs to generate text to better reflect a given small fine-tuning corpus.