\[table:models\].

**[<span style="font-variant:small-caps;">ELMo</span>]{}**: The input layer is a character-CNN, and the contextualizing structure is an LSTM with projection [@SakSB14LSTMProjection]. The output layer is a sampled softmax with 8192 negative samples per batch. This model is provided in AllenNLP by .

**[<span style="font-variant:small-caps;">ELMo</span>*-S*$_{\small \textsc{}}$]{}**: The input layer is the FastText embedding trained on Common Crawl [@fasttext_resource], denoted as [<span style="font-variant:small-caps;">FastText$_{\small \textsc{cc}}$</span>]{}[^9]. The contextualizing structure is an LSTM with projection the same size as the one in [<span style="font-variant:small-caps;">ELMo</span>]{}, but we added layer norm [@LayerNormalization] after the projection layer as we find it beneficial. The output layer is the [<span style="font-variant:small-caps;">Semfit</span>]{} layer with [<span style="font-variant:small-caps;">FastText$_{\small \textsc{cc}}$</span>]{} embedding as the target embedding. We use the cosine distance as the distance function of the [<span style="font-variant:small-caps;">Semfit</span>]{} layer because it is free of hyper-parameters and we find that it gives a satisfying and stable performance in preliminary experiments. The learning rate schedule from is used to aid large-batch training.

**[<span style="font-variant:small-caps;">ELMo*-A*$_{\small \textsc{}}$</span>]{}**: The input layer, the contextualizing structure and the training recipe of [<span style="font-variant:small-caps;">ELMo*-A*$_{\small \textsc{}}$</span>]{} is kept the same as [<span style="font-variant:small-caps;">ELMo</span>*-S*$_{\small \textsc{}}$]{}. The only difference is that the output layer of [<span style="font-variant:small-caps;">ELMo*-A*$_{\small \textsc{}}$</span>]{} is an adaptive softmax with 120 million parameters, half 