style="font-variant:small-caps;">Semfit</span>]{} layer has $O(1)$ time complexity, which grants scalability with respect to the vocabulary size. The time spent on the [<span style="font-variant:small-caps;">Semfit</span>]{} layer remains constant and negligible regardless of the vocabulary size. In comparison, the time complexity of the sampled softmax is proportional to the number of negative samples per batch. When the vocabulary is huge, a large number of negative samples are needed [@limitslm]. For the adaptive softmax, the time complexity is determined by the capacities of the short-list and rare-word clusters, which grows sub-linearly with respect to the vocabulary size.

#### Trainable Parameter Size

The softmax layer takes up a huge part of the parameters of a language model. Here, we list the parameter sizes of models reported in the literature trained on the One Billion Word Benchmark [@OneBillionWord], which is also the corpus ELMo was trained on. For ELMo, the character-CNN and LSTM have about 100 million parameters while the softmax layer has 400 million parameters. For the bigLSTM in , the softmax takes up 840 million parameters, while all other parts have 182 million parameters. These models used the sampled softmax, which is only designed to accelerate calculating the normalization term and does not reduce the 