empirically show that our approach significantly reduces the training time of ELMo, while maintaining competitive performance on most of the end tasks. (Section \[Experiment\])

We conduct a thorough analysis of our approach, discussing different modeling choices and computational efficiency. We also analyze the subword language model, a strong baseline used in GPT and BERT to circumvent the problem of large vocabulary. (Section \[Analysis\])

Related Work {#sec:relatedwork}
============

#### Contextual Representation

We review recently proposed contextual representation models from two aspects: how they are trained and how these pre-trained models are used in downstream tasks.

CoVe [@cove] trained a machine translation model, and used the source language encoder as a contextual representation model for other downstream tasks. As large in-domain parallel corpus is hard to obtain, the potential of CoVe is limited. In contrast, a few recent approaches learn contextual encoder on unannotated corpus with language model objectives. ELMo [@elmo] concatenated a forward and a backward LSTM-based language model while GPT-1 [@GPT] and GPT-2 [@GPT2] used unidirectional transformer-based language models. BERT [@bert] introduced masked language models and provided deep bidirectional representation.

There are mainly two existing strategies for applying pre-trained contextual representations to downstream tasks: 1) *feature-based* and 2) *fine-tuning*. For 