style="font-variant:small-caps;">Semfit</span>]{} w/ [<span style="font-variant:small-caps;">FastText$_{\small \textsc{cc}}$</span>]{}
  <span style="font-variant:small-caps;">[[<span style="font-variant:small-caps;">ELMo*-A*$_{\small \textsc{}}$</span>]{}]{}</span>       [<span style="font-variant:small-caps;">FastText$_{\small \textsc{cc}}$</span>]{}     LSTM w/ LN        Adaptive Softmax
  <span style="font-variant:small-caps;">[[<span style="font-variant:small-caps;">ELMo*-S*$_{\small \textsc{OneB}}$</span>]{}]{}</span>   [<span style="font-variant:small-caps;">FastText$_{\small \textsc{OneB}}$</span>]{}   LSTM w/ LN        [<span style="font-variant:small-caps;">Semfit</span>]{} w/ [<span style="font-variant:small-caps;">FastText$_{\small \textsc{One}}$</span>]{}
  <span style="font-variant:small-caps;">[[<span style="font-variant:small-caps;">ELMo*-S*$_{\small \textsc{CNN}}$</span>]{}]{}</span>    Trained CNN                                                                           LSTM w/ LN        [<span style="font-variant:small-caps;">Semfit</span>]{} w/ Trained CNN
  <span style="font-variant:small-caps;">[[<span style="font-variant:small-caps;">ELMo*-Sub*</span>]{}]{}</span>                          Subword                                                                               LSTM w/ LN        Softmax

To verify the efficacy of the proposed method, we introduce an ELMo model trained with our acceleration approach ([<span style="font-variant:small-caps;">ELMo</span>*-S*$_{\small \textsc{}}$]{}). We also include the original ELMo ([<span style="font-variant:small-caps;">ELMo</span>]{}) for comparison. Our main proposal is to use the [<span style="font-variant:small-caps;">Semfit</span>]{} layer as the output layer but there are substantial differences between [<span style="font-variant:small-caps;">ELMo</span>]{} and [<span style="font-variant:small-caps;">ELMo</span>*-S*$_{\small \textsc{}}$]{} besides the output layer. To make a fair comparison, we introduce an ELMo model with the adaptive softmax ([<span style="font-variant:small-caps;">ELMo*-A*$_{\small \textsc{}}$</span>]{}). [<span style="font-variant:small-caps;">ELMo*-A*$_{\small \textsc{}}$</span>]{} is designed to differ from [<span style="font-variant:small-caps;">ELMo</span>*-S*$_{\small \textsc{}}$]{} only in the output layer so that we could study the effect of the [<span style="font-variant:small-caps;">Semfit</span>]{} layer in isolation.

In the following, we describe the details of these models mainly from three aspects: 1) the input layer, 2) the contextualizing structure and 3) the output layer. A brief summary of these models are available in Table 