vocabulary of 800,000 words using three GPUs[^1]. This slow training procedure hinders the development cycle, prevents fine-grained parameter tuning, and makes training contextual representations inaccessible to a broader community. More importantly, the success of these models stems from the large amount of data they used. It is challenging, if not impossible, to train a contextual representation model on a larger corpus with tens or hundreds of billions of tokens.

In this work, we explore how to accelerate contextual representation learning. We target the softmax layer as the major cause of inefficiency. This component takes up a huge portion of all trainable parameters (80% for ELMo) and consumes a huge amount of training time. However, this layer will be discarded in the final model as the goal of contextual representation learning is to build a generic encoder. Therefore, it is rather a waste to allocate extensive computational resources to the softmax layer to obtain the best prediction of masked words. As mentioned, language modeling can be viewed as predicting the missing words based on a context vector generated by the contextual encoder and an embedding matrix $\bm{W}$ for target words [@tie_embedding; @tie_embedding_2]. Learning the contextual encoder is difficult while learning word 