hierarchy to avoid expensive normalization. The adaptive softmax, in particular, group words in the vocabulary into either a short-list or clusters of rare words. For frequent words, a softmax over the short-list would suffice, which reduces computation and memory usage significantly. The adaptive softmax has been shown to achieve results close to that of the full softmax whilst maintaining high GPU efficiency [@Merity2018LMScale].

Regarding contextual representation models, ELMo used the sampled softmax while GPT and BERT resorted to subword methods. Specifically, they used WordPiece [@google_wordpiece] or BPE [@subword] to split the words into subwords and the language models were trained to consume and predict these subwords. This method is efficient and scalable, as the subword vocabulary can be kept small. Language models trained in this way are neither strictly *word-level* nor *character-level*. Hence we categorize them as *subword-level* language models. One potential drawback, however, is that these models produce representations for fragments of words, and it takes extra efforts to generate word-level representations from them. In this paper, we focus on *word-level* language models and we will discuss *subword-level* language models in Section \[Exp:Subword\].

Approach {#sec:apprach}
========

In this section, we illustrate our approach to accelerating the training process of 