<p>Real datasets will have many <em>other</em> examples of contextual usage all of <code>'beautiful'</code>, <code>'boy'</code>, and <code>'lady'</code>, each somewhat different.</p>
<p>In fact, in order for a dataset to train useful high-dimensional vectors, they must have many varied uses - &amp; far more variety than your synthetic examples. As a result, you couldn't simply look at a real dataset, check that <code>'lady'</code> &amp; <code>'beautiful'</code> appear near-each-other 5x more than <code>'boy'</code> &amp; <code>'beautiful'</code>, &amp; be sure that <code>'lady'</code> and <code>'beautful'</code> will have more-similar word-vectors. Their actual positions will depend on all the <em>other</em> uses of those words, with all the <em>other</em> words in the corpus, which could override the simple relationship you've observed.</p>
<p>The value of word-vectors is simply in how well they perform on downstream tasks, and no set of word-vectors is best for all tasks. That is, there's no one 'right' way to create them - just a bunch of different algorithms, each also parameterizable, which might yield vectors better or worse for a particular purpose.</p>
<p>If you've decided, or have as the external constraint for some reason, that <code>'beautiful'</code> must be equidistant from both <code>'lady'</code> and <code>'boy'</code>, no matter what the training data teaches about usage, there are lots of things you could try.</p>
<p>Some are crude, like just patching the vectors at the very end. Others could add extra steps to training that maintain the desired relationship, or at least give it an advantage that the data &amp; other parameters could still weaken, at every update.</p>
<p>As you seem to be interested in equalizing relationships between words of different genders, you'd likely be interested in a paper which highlighted gender-biases in word-vectors, and proposed ways to modify word-vectors in response:</p>
<p><a href="https://arxiv.org/abs/1607.06520" rel="nofollow noreferrer">&quot;Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings&quot;</a></p>
<p>Note, though, that this paper, and much other work noticing the same effects, has overclaimed the strength of such biases by overlooking that typical routines for solving analogies tend to <em>rule out</em> any word that appears as a prompt.</p>
<p>For example, consider the headline claim of the paper, that the <code>GoogleNews</code> vectors released by Google along with the original word2vec paper will answer the analogy...</p>
<p><code>man : computer_programmer :: woman : ___?___</code></p>
<p>with <code>homemaker</code>.</p>
<p>Using something like the original <code>word-analogy</code> tool in the Google <code>word2vec</code> release, or the similar functionality of the <code>.most_similar()</code> method of Gensim Python library for working with word-vectors, you can naively reproduce that claim:</p>
<pre class="lang-py prettyprint-override"><code>from gensim.models import KeyedVectors
gkv = KeyedVectors.load_word2vec_format(
        '/home/gojomo/Downloads/GoogleNews-vectors-negative300.bin', 
        binary=True
)
gkv.most_similar(
    positive=['computer_programmer', 'woman'], 
    negative=['man']
)
</code></pre>
<pre class="lang-py prettyprint-override"><code>[('homemaker', 0.5627118945121765),
 ('housewife', 0.5105046629905701),
 ('graphic_designer', 0.505180299282074),
 ('schoolteacher', 0.497949481010437),
 ('businesswoman', 0.493489146232605),
 ('paralegal', 0.49255111813545227),
 ('registered_nurse', 0.4907974600791931),
 ('saleswoman', 0.48816272616386414),
 ('electrical_engineer', 0.4797726571559906),
 ('mechanical_engineer', 0.4755399227142334)]
</code></pre>
<p>(To solve an analogy of the form <code>A : B :: C : _?_</code> using <code>.most_similar()</code>, the <code>B</code> &amp; <code>C</code> should become <code>positive</code> examples, and <code>A</code> the <code>negative</code> example.)</p>
<p>But, each of these analogy-solving routines have coded-in as an assumption that the asker <em>never</em> wants to see <em>any</em> of the prompt-words. So even if <code>computer_programmer</code> was still the top-analogy-answer, it would be filtered from the results.</p>
<p>With gensim, we can override that behavior by supplying the prompts as raw vectors, rather than lookup keys - and then we get a different result:</p>
<pre class="lang-py prettyprint-override"><code>gkv.most_similar(
    positive=[gkv.get_vector('computer_programmer', norm=True), 'woman'], 
    negative=['man']
)
</code></pre>
<pre class="lang-py prettyprint-override"><code>[('computer_programmer', 0.8276970982551575),
 ('homemaker', 0.5627118945121765),
 ('housewife', 0.5105046629905701),
 ('graphic_designer', 0.505180299282074),
 ('schoolteacher', 0.497949481010437),
 ('businesswoman', 0.493489146232605),
 ('paralegal', 0.49255111813545227),
 ('registered_nurse', 0.4907974600791931),
 ('saleswoman', 0.48816272616386414),
 ('electrical_engineer', 0.4797726571559906)]
</code></pre>
<p>Now, the model recognizes <code>computer_programmer</code> as the top match, with the unrestricted analogical-answer not massively shifted by the gender-influence. The results retain a gendered-character, suggesting the 'target point' found by the analogy-calculations is on-the-side of <code>'computer_programmer'</code> towards a neighborhood of stereotypically-female roles.</p>
<p>Also interesting to note: if you check the top-10 most-similar words to <code>'computer_programmer'</code> in the <code>GoogleNews</code> vectors, <code>'homemaker'</code> is among them, as well as a woman author/programmer by name. So the gender contrasts among these roles and words, as captured by that particular vector-set, may be more subtle than some of the headline claims of papers emphasizing bias.</p>
word distribution for vector generation
<p>Distributional semantic models such as word2vec and Glove generate the word vector by capturing the context occurrence of words with each other. Then if one word appears more than others next to the focus word, it has a closer vector to the focus word, and it dominates the directions of the focus word's vector. For example:</p>
<p>Consider beautiful as a focus word, and the sequences are as follows:<br />
<strong>You are a beautiful lady. (this sequence appear 100 times)<br />
You are a beautiful boy. (this sequence appear 10 times)<br />
As a result, beautiful is closer to the lady than the boy.</strong></p>
<p>If the distribution of two context words (co-occur with the focus word) is the same in the entire dataset, then the focus word's vector might have the same closeness to those two context word's vector.</p>
<p>Consider the previous example:<br />
<strong>You are a beautiful lady. (this sequence appear 10 times)<br />
You are a beautiful boy. (this sequence appear 10 times)<br />
As a result, beautiful has same closeness to lady and boy.</strong></p>
<p>My question is related to this point, How close should be the word distribution to get the second example situation ( a vector that has same closeness to other two vectors)? Is there literature research that covers words distribution domination in generating the vector?</p>
<p>Update: Imagine beautiful has only two contexts, lady and boy in the entire dataset.</p>