the global objective of different distance functions and link the [<span style="font-variant:small-caps;">Semfit</span>]{} layer to probabilistic language modeling and dimension reduction, which further justifies the intuition behind the [<span style="font-variant:small-caps;">Semfit</span>]{} layer. Finally, we discuss why the [<span style="font-variant:small-caps;">Semfit</span>]{} layer is particularly suited for contextual representation learning.

Global Objective {#subsec:GlobalObjective}
----------------

Recall that the [<span style="font-variant:small-caps;">Semfit</span>]{} layer minimizes the distance between the context vector $\bm{c}$ and the target word embedding $\bm{w}$, $ l = d(\bm{c}, \bm{w})$. There are several choices regarding the distance function $d$. We investigate these losses with different distance functions from [^6]:

L2: $(\bm{{c}} - \bm{{w}})^2$

Cosine: $-{\bm{\bar{c}} \cdot \bm{{w}}} $

NLLvMF: $ -\log C_m(\|\bm{c}\|) -
\lambda_2 \bm{{c}} \cdot \bm{{w}} + \lambda_1 \|\bm{c}\| $

Although the above losses have different interpretations, we find that they are similar. Specifically, we can rewrite the global objective in an uniform way[^7]: [$$\begin{aligned}
& \mathcal{L}  = \sum_{(c, w)} \#(c, w) (-\lambda \bm{\tilde{c}} \cdot \bm{{w}} + f(\|\bm{\tilde{c}}\|)) \\
  & = -\lambda \sum_{c} \bm{\tilde{c}} \cdot\sum_{w}\#(c, w) \bm{{w}} +  \sum_{(c,w)}\#(c, w)f(\|\bm{\tilde{c}}\|) \\
  & = -\lambda \sum_{c} \#(c)\bm{\tilde{c}}\! \cdot\! \sum_{w} p(w|c) \bm{w} \!+\!  \sum_{(c,w)}\!\#(c, w)\!f(\|\bm{\tilde{c}}\|) \\
\end{aligned}$$ ]{}

The first term minimizes the inner-product between the contextual encoder output and the target embedding, and the second term 