<p>If you've already been able to perform sense-disambiguation outside word2vec, then you can change the word-tokens to reflect your external judgement. For example, change some appearances of the token <code>'jaguar'</code> to <code>'jaguar*car'</code> and others to <code>'jaguar*animal'</code>. Proceeding with normal word2vec training will then get your two different tokens two different word-vectors.</p>
<p>If you're hoping for the training to discover these itself, as ~Erwan mentioned in a comment, that seems like an open research question, without a standard or off-the-shelf solution that a beginner could drop-in.</p>
<p>I'd once seen a paper (around the time of the original word2vec papers, but can't find the link now) that tried to do this in a word2vec-compatible way by 1st proceeding with traditional polysemy-oblivious training. Then, for every appearance of a word X, model its surrounding context via some combination of the word-vectors of neighbors within a certain number of positions. (That in itself is very similar to the preparation of a context-vector in the CBOW mode of word2vec.) Perform some clustering on that collection-of-all-contexts to come up with some idea of alternate senses – each associated with one cluster. Then, in a followup pass on the original corpus, replace word-tokens with those that also reflect their nearby-context cluster. (EG: <code>'jaguar'</code> might be replaced with <code>'jaguar*1'</code>, <code>'jaguar*2'</code>, etc based on which discrete cluster its context suggested.) Then, repeat (or continue) word2vec training to get sense-specific word-vectors. Of course, the devil would be in the details of how contexts are defined, how clusters are deduced, and tough edge-cases (where potentially the text's author is themselves deploying the multiple senses).</p>
<p>Some other interesting efforts to model or deduce polysemy in word2vec models:</p>
<ul>
<li><a href="http://www.offconvex.org/2016/07/10/embeddingspolysemy/" rel="nofollow noreferrer">&quot;Linear Algebraic Structure of Word Meanings&quot;</a></li>
<li><a href="https://arxiv.org/abs/1707.01793" rel="nofollow noreferrer">&quot;A Simple Approach to Learn Polysemous Word Embeddings&quot;</a></li>
</ul>
<p>But per above, I've not seen these sorts of techniques widely implemented/adopted in a form that's easy to drop-in to another project.</p>
Contextual word embeddings from pretrained word2vec vectors
<p>I would like to create word embeddings that take <strong>context</strong> into account, so the vector of the word <em>Jaguar [animal]</em> would be different from the word <em>Jaguar [car brand]</em>.</p>
<p>As you know, word2vec only gives one representation for a given word, and I would like to take already pretrained embeddings and enrich them with context. So far I've tried a simple way with taking an average vector of the word and category word, for example <a href="https://i.stack.imgur.com/NtMaT.png" rel="nofollow noreferrer">like this</a>.</p>
<p>Now I would like to try to create and train a neural network that would take entire sentences, e.g.</p>
<ol>
<li><em>Jaguar F-PACE is a great SUV sports car.</em></li>
<li><em>Among cats, only tigers and lions are bigger than jaguars.</em></li>
</ol>
<p>And then it would undertake the task of text classification (I have a dataset with several categories like animals, cars, etc.), but the result would be new representations for the word jaguar, but in different contexts, so two different embeddings.</p>
<p>Does anyone have any idea how I could create such a network? I don't hide that I'm a beginner and have no idea how to go about it.</p>