GPT-1 [@GPT], BERT [@bert] and GPT-2 [@GPT2], have been developed to provide generic contextual representations for the natural language, which have led to large improvements to various downstream tasks. The key idea underneath is to train a contextual encoder with a language model objective on a large unannotated text corpus. During the training, part of the text is masked and the goal is to encode the remaining context and predict the missing part. Specifically, given a corpus with vocabulary size $V$, the model consists of two parts: 1) an encoder is learned to embed the context and output an $m$-dimensional vector, and 2) the output vector goes through a softmax layer, where it is multiplied by an embedding matrix $\bm{W} \in \mathcal{R}^{V\times m}$ and is fed into a softmax function, to produce a conditional distribution of the missing word. The encoders trained in such a way are able to capture generic contextual information of the input text and have been used in a variety of benchmark tasks to establish state-of-the-art results.

However, training contextual representations is known to be a resource-hungry process. For example, ELMo was reported to take about two weeks to train on a one-billion-token corpus with a 