is large, the vanilla softmax layer becomes the bottleneck. Many approaches (Section \[sec:relatedwork\]) have been proposed to address this issue, but we think there is still room for improvement.

We accelerates the training by replacing the softmax layer with a [<span style="font-variant:small-caps;">Semfit</span>]{} layer. Instead of maximizing the log-likelihood, after embedding the context into $\bm{c}$, we directly minimize the distance between the context vector $\bm{c}$ and a **target word embedding** $\bm{w}$, $$l(c,w) = d(\bm{c}, \bm{w}).$$ Notice that the target word embedding $\bm{w}$ is pre-trained and fixed. The distance function $d$ could be the L2 distance, cosine distance or some probabilistic distance metrics (see discussions in Section \[sec:semfit\]). For the rest of this section, we will discuss the advantages of the proposed approach in contextual representation learning.

Computational Efficiency {#subsec:computational_efficiency}
------------------------

We analyze the computational efficiency of the proposed approach against existing acceleration approaches for the softmax layer. In particular, we discuss the sampled softmax and the adaptive softmax, both being popular choices for speeding up softmax [@limitslm; @adaptive_softmax; @Merity2018LMScale].

#### Computational Time Complexity

In the [<span style="font-variant:small-caps;">Semfit</span>]{} layer, we only need to calculate the distance between two $m$-dimensional vectors. Without the normalization term or the need to sample words, the [<span 