kind of word representation, the open-vocabulary word embedding, such as the FastText embedding and the <span style="font-variant:small-caps;">mimick</span> model [@mimick]. These embeddings utilize character or subword information to provide embedding for out-of-vocabulary (OOV) words. Combining the [<span style="font-variant:small-caps;">Semfit</span>]{} layer with open-vocabulary embedding, we can train contextual encoders with untruncated vocabulary while making substantial simplifications to the input layer.

#### Scalable Word Representation

In the vanilla softmax, a simple $V$-by-$m$ matrix $\bm{W}$ is used to provide word representation. However, when we further scale up current language models, the parameter size of this matrix could become intractable. The adaptive softmax proposes to reduce $m$ for rare words such that the parameter size grows only sub-linearly. Here, we take another route and propose to utilize the open-vocabulary word embedding, which could represent an unlimited number of words with a fixed number of parameters, providing the much-needed scalability.

#### Learning with An Untruncated Vocabulary

Combining the $O(1)$ time complexity and scalable word representation, we can conduct training with an untruncated vocabulary. Softmax-based methods keep a vocabulary to calculate the normalization term. With the [<span style="font-variant:small-caps;">Semfit</span>]{} layer, we only need the target word embedding and the context vector to conduct training. There is no need for 