a context vector $\bm{c}$ and using it to predict a missing word[^4]. The conventional way is to attach a softmax layer to the contextual encoder. The softmax layer multiplies $\bm{c}$ with an embedding matrix $\bm{W} \in \mathcal{R}^{V \times m}$ and then uses a softmax function to produce a conditional distribution over the vocabulary. Suppose the target word is $w$ and the context is $c$, the learning objective $l(c,w)$ for each instance is maximizing the negative log-likelihood of $w$ under $c$: $$\begin{aligned}
    l(c,w) & = &- \log p(w|c) \nonumber \\
      & = &- \log softmax(\bm{cW^{T}}) \nonumber \\
      & = & - \bm{c} \cdot \bm{w} + \log\sum\nolimits_{w'}{\exp{\bm{c} \cdot \bm{w'}}} \label{Eq:LMObj}\end{aligned}$$ $\bm{w} \in \mathcal{R}^{m}$ is a row from $\bm{W}$ corresponding to $w$ and the second term sums over the vocabulary. $V$ is the size of the vocabulary while $m \ll V$ is the size of the context vector. The overall learning objective is $$\mathcal{L} = \sum\nolimits_{(c,w)} \#(c,w) l(c,w)$$ where $\#(c,w)$ is the number of occurrences of the pair $(c,w)$ in the corpus. Note that the size of $W$ and the computational complexity of the normalization term (the second term in Eq. ) scale linearly to the vocabulary size. Therefore, when the vocabulary size 