<p>There can be various reasons.</p>
<p>One reason is that this is due to the so-called <a href="https://arxiv.org/pdf/1412.6568.pdf" rel="nofollow noreferrer">hubness problem</a> of embedding spaces, which is an artifact of the high-dimensional space. Some words end up close to a large part of the space and act as sort of hubs in the nearest neighbor search, so through these words, you can get quickly from everywhere to everywhere.</p>
<p>Another reason might be that the model is just undertrained for this particular word. Word embeddings are typically trained on very large datasets, such that every word appears in sufficiently many contexts. If a word does not appear frequently enough or in too ambiguous contexts, then it also ends up to be similar to basically everything.</p>
Word2Vec Tensorflow tutorial weird output
<p>I'm trying out the Word2Vec tutorial at tensorflow (see here: <a href="https://www.tensorflow.org/tutorials/text/word2vec" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/text/word2vec</a>)</p>
<p>While all seems to work fine, the output is somewhat unexpected to me, especially the small cluster in the PCA. The 'closet' words in the embedding dimension also don't make much sense, especially compared to other examples.</p>
<p>Am I doing something (trivially) wrong? Or is this expected?</p>
<p>For completeness, I run this in the nvidia-docker image, but also found similar results running cpu only.</p>
<p>Here is the projected embedding showing the cluster.
<a href="https://i.stack.imgur.com/SnTCH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SnTCH.png" alt="enter image description here" /></a></p>