<p>There are many ways to achieve this. One way would be to create the embeddings (vectors) yourself. This would have two advantages: first, you would be able to use bi-, tri-, and beyond (n-) grams as your tokens, and secondly, you are able to define the space that is best suited for your needs --- Wikipedia data is general, but, say, children's stories would be a more niche dataset (and more appropriate / "accurate" if you were solving problems to do with children and/or stories). There are several methods, of course <code>word2vec</code> being the most popular, and several packages to help you (e.g. <code>gensim</code>).</p>

<p>However, my guess is you would like something that's already out there. The best word embeddings right now are:</p>

<ul>
<li><a href="https://github.com/commonsense/conceptnet-numberbatch" rel="nofollow noreferrer">Numberbatch</a> ('classic' best-in-class ensemble);</li>
<li><a href="https://fasttext.cc/" rel="nofollow noreferrer">fastText</a>, by Facebook Research (created at the character level --- some words that are out of vocabulary can be "understood" as a result);</li>
<li><a href="https://github.com/explosion/sense2vec" rel="nofollow noreferrer">sense2vec</a>, by the same guys behind Spacy (created using parts-of-speech (POS) as additional information, with the objective to disambiguate).</li>
</ul>

<p>The one we are interested in for a quick resolve of your problem is <code>sense2vec</code>. You should read the <a href="https://arxiv.org/abs/1511.06388" rel="nofollow noreferrer">paper</a>, but essentially these word embeddings were created using Reddit with additional POS information, and (thus) able to discriminate entities (e.g. nouns) that span multiple words. <a href="https://explosion.ai/blog/sense2vec-with-spacy" rel="nofollow noreferrer">This blog post</a> describes <code>sense2vec</code> very well. Here's some code to help you get started (taken from the prior links):</p>

<p>Install:</p>

<pre><code>git clone https://github.com/explosion/sense2vec
pip install -r requirements.txt
pip install -e .
sputnik --name sense2vec --repository-url http://index.spacy.io install reddit_vectors
</code></pre>

<p>Example usage:</p>

<pre><code>import sense2vec
model = sense2vec.load()
freq, query_vector = model["onion_rings|NOUN"]
freq2, query_vector2 = model["chicken_nuggets|NOUN"]
print(model.most_similar(query_vector, n=5)[0])
print(model.data.similarity(query_vector, query_vector2))
</code></pre>

<p>Important note, <code>sense2vec</code> <strong>requires</strong> <code>spacy&gt;=0.100,&lt;0.101</code>, meaning <em>it will downgrade your current <code>spacy</code> install</em>, not too much of a problem if you are only loading the <code>en</code> model. Also, here are the POS tags used:</p>

<pre><code>ADJ ADP ADV AUX CONJ DET INTJ NOUN NUM PART PRON PROPN PUNCT SCONJ SYM VERB X
</code></pre>

<p>You could use <code>spacy</code> for POS and dependency tagging, and then <code>sense2vec</code> to determine the similarity of resulting entities. Or, depending on the frequency of your dataset (not too large), you could grab n-grams in descending (n) order, and sequentially check to see if each one is an entity in the <code>sense2vec</code> model.</p>

<p>Hope this helps!</p>
Handling compound words (2-grams) using NLTK
<p>I'm trying to identify user similarities by comparing the keywords used in their profile (from a website). For example, <code>Alice = pizza, music, movies</code>, <code>Bob = cooking, guitar, movie</code> and <code>Eve = knitting, running, gym</code>. Ideally, <code>Alice</code> and <code>Bob</code> are the most similar. I put down some simple code to calculate the similarity. To account for possible plural/singular version of the keywords I use something like:</p>

<pre><code>from nltk.stem import WordNetLemmatizer
from nltk.tokenize import word_tokenize
wnl = WordNetLemmatizer()
w1 = ["movies", "movie"] 
tokens = [token.lower() for token in word_tokenize(" ".join(w1))]
lemmatized_words = [wnl.lemmatize(token) for token in tokens]
</code></pre>

<p>So that, <code>lemmatized_words = ["movie", "movie"]</code>. 
Afterwards, I do some pairwise keywords comparison using <a href="https://spacy.io/usage/vectors-similarity" rel="nofollow noreferrer"><code>spacy</code></a>, such as:</p>

<pre><code>import spacy
nlp = spacy.load('en')
t1 = nlp(u"pizza")
t2 = nlp(u"food")
sim = t1.similarity(t2)
</code></pre>

<p>Now, the problem starts when I have to deal with compound  words such as: <code>artificial intelligence</code>, <code>data science</code>, <code>whole food</code>, etc. By tokenizing, I would simply split those words into 2 (e.g. <code>artificial</code> and <code>intelligence</code>), but this would affect my similarity measure. What is (would be) the best approach to take into account those type of words?</p>