representation has been extensively studied [@word2vec]. It is natural to use a pre-trained word embedding to replace $\bm{W}$ and thus decouple learning contexts and words.

In this paper, we propose an efficient framework to learn the contextual encoder by leveraging pre-trained word embeddings[^2]. Instead of using a softmax layer to predict the distribution of the missing word, we utilize and extend the [<span style="font-variant:small-caps;">Semfit</span>]{} layer [@kumar2018mises] to predict the embedding of the missing word. In the training process, the contextual encoder is learned by minimizing the distance between its output and a pre-trained target word embedding. The [<span style="font-variant:small-caps;">Semfit</span>]{} layer, with constant time complexity and small memory footprint, perfectly serves our desire to decouple learning contexts and words and devote most computational resources to the contextual encoder. Our contributions are as follows:

We introduce the [<span style="font-variant:small-caps;">Semfit</span>]{} layer into contextual representation learning and further improve it with open-vocabulary word embeddings. The resulting model is computationally efficient and can be trained with an untruncated vocabulary. (Section \[sec:apprach\])

We discuss the global objective of the [<span style="font-variant:small-caps;">Semfit</span>]{} layer and draw a connection to dimension reduction. We show that the [<span style="font-variant:small-caps;">Semfit</span>]{} layer is particularly suitable for contextual representation learning. (Section \[sec:semfit\])

We 