adds different constraints on the encoder output’s norm. Although for different losses the norm of the optimal $\bm{c}$ might be different, the direction of the optimal $\bm{c}$ should be the same as $\Sigma_{w} p(w|c) \bm{w}$. Simply put, the [<span style="font-variant:small-caps;">Semfit</span>]{} layer models the weighted average word embedding under a context.

Dimension Reduction {#subsec:DimensionReduction}
-------------------

We seek to justify the very idea of the [<span style="font-variant:small-caps;">Semfit</span>]{} layer. We want to answer why minimizing the distance between the encoder output and the target word embedding should work.

We first show that the [<span style="font-variant:small-caps;">Semfit</span>]{} layer is essentially performing language modeling after dimension reduction. Following and , language modeling can be viewed as modeling a conditional probability matrix $\bm{P} \in \mathcal{R}^{N \times V}$, where $N$ is the number of all possible contexts and $V$ is the vocabulary size. Each row of $\bm{P}$ corresponds to the conditional distribution of the word under a certain context. Softmax-based methods seek to find a $\bm{C} \in \mathcal{R}^{N \times m}$ and $\bm{W} \in \mathcal{R}^{V \times m}$, such that $softmax(\bm{C}\bm{W^{T}})$ best approximates $\bm{P}$. For the [<span style="font-variant:small-caps;">Semfit</span>]{} Layer, we are modeling $\Sigma_{w} p(w|c) \bm{w}$, which translates to $\bm{P}\bm{W}$ in the matrix form. $\bm{W}$ is pre-trained and fixed. We are 