PMI matrix or the conditional probability matrix, $ (\bm{\Sigma_d})^\alpha \bm{V_d}$ serves as a good word embedding [@levy2014neural], where $\alpha \in [0, 1]$ is a tunable parameter.

This finding shows that there exists a natural link between the [<span style="font-variant:small-caps;">Semfit</span>]{} layer and SVD word embedding. Moreover, we could intuitively see that other kinds of word embedding might also serve well as the projection matrix $\bm{W}$. Suppose we want to preserve the variance of $\bm{P}$ in the spirit of PCA. If two contexts have very different conditional distribution, then $\Sigma_{w} p(w|c) \bm{\tilde{w}}$ for these two contexts should have very different directions, assuming that dissimilar words have dissimilar word vectors. Thus, a pre-trained word embedding might help us preserve much of the variance.

Decoding Algorithm {#subsec:DecodingAlgorithm}
------------------

The above analysis strengthens our argument that the [<span style="font-variant:small-caps;">Semfit</span>]{} layer especially suits contextual representation learning. To illustrate, one could see that the [<span style="font-variant:small-caps;">Semfit</span>]{} layer may not excel at predicting the next word, which in theory is determined by its ability to approximate $\bm{P}$. The [<span style="font-variant:small-caps;">Semfit</span>]{} layer models $\bm{PW}$, and unless we get the optimal $\bm{W}$, it is hard to get an approximated $\bm{P}$ from $\bm{PW}$; therefore, we cannot calculate perplexity from the [<span 