truncating the vocabulary or keeping one because we could use multi-process to dynamically prepare the embedding.

According to , the ability to model rare words is an essential advantage of the neural models against N-gram models. Now with an untruncated vocabulary, we possess the power to model unlimited rare words. This feature is especially attractive if we are training on domains or languages with a long tail, for example, the biomedical domain where truncating the vocabulary may not be acceptable.

#### Open-vocabulary Input Layer

To be easily applied in various tasks, the contextual encoder usually has an open-vocabulary input layer. ELMo used a character-CNN but it is relatively slow. Thus we propose to reuse the pre-trained open-vocabulary word embedding as the input layer of the contextual encoder, reducing the time complexity of the input layer to $O(1)$. This also aligns with the main spirit of our approach, which is to spend computational resources on the most important part, the contextualizing structure like LSTM.

The [<span style="font-variant:small-caps;">Semfit</span>]{} Layer {#sec:semfit}
==================================================================

Though the [<span style="font-variant:small-caps;">Semfit</span>]{} layer is intuitive, its properties are less known. In this section, we provide an analysis on the [<span style="font-variant:small-caps;">Semfit</span>]{} layer, extending the work of . We investigate 