<p>If you are looking for something very accurate, you need to use some better tool than tf-idf. <a href="https://arxiv.org/abs/1803.11175" rel="noreferrer">Universal sentence encoder</a> is one of the most accurate ones to find the similarity between any two pieces of text. Google provided pretrained models that you can use for your own application without a need to train from scratch anything. First, you have to install tensorflow and tensorflow-hub:</p>

<pre><code>    pip install tensorflow
    pip install tensorflow_hub
</code></pre>

<p>The code below lets you convert any text to a fixed length vector representation and then you can use the dot product to find out the similarity between them</p>

<pre class="lang-py prettyprint-override"><code>import tensorflow_hub as hub
module_url = "https://tfhub.dev/google/universal-sentence-encoder/1?tf-hub-format=compressed"

# Import the Universal Sentence Encoder's TF Hub module
embed = hub.Module(module_url)

# sample text
messages = [
# Smartphones
"My phone is not good.",
"Your cellphone looks great.",

# Weather
"Will it snow tomorrow?",
"Recently a lot of hurricanes have hit the US",

# Food and health
"An apple a day, keeps the doctors away",
"Eating strawberries is healthy",
]

similarity_input_placeholder = tf.placeholder(tf.string, shape=(None))
similarity_message_encodings = embed(similarity_input_placeholder)
with tf.Session() as session:
    session.run(tf.global_variables_initializer())
    session.run(tf.tables_initializer())
    message_embeddings_ = session.run(similarity_message_encodings, feed_dict={similarity_input_placeholder: messages})

    corr = np.inner(message_embeddings_, message_embeddings_)
    print(corr)
    heatmap(messages, messages, corr)
</code></pre>

<p>and the code for plotting:</p>

<pre class="lang-py prettyprint-override"><code>def heatmap(x_labels, y_labels, values):
    fig, ax = plt.subplots()
    im = ax.imshow(values)

    # We want to show all ticks...
    ax.set_xticks(np.arange(len(x_labels)))
    ax.set_yticks(np.arange(len(y_labels)))
    # ... and label them with the respective list entries
    ax.set_xticklabels(x_labels)
    ax.set_yticklabels(y_labels)

    # Rotate the tick labels and set their alignment.
    plt.setp(ax.get_xticklabels(), rotation=45, ha="right", fontsize=10,
         rotation_mode="anchor")

    # Loop over data dimensions and create text annotations.
    for i in range(len(y_labels)):
        for j in range(len(x_labels)):
            text = ax.text(j, i, "%.2f"%values[i, j],
                           ha="center", va="center", color="w", 
fontsize=6)

    fig.tight_layout()
    plt.show()
</code></pre>

<p>the result would be:
<a href="https://i.stack.imgur.com/ftGqC.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ftGqC.png" alt="the similarity matrix between pairs of texts"></a></p>

<p>as you can see the most similarity is between texts with themselves and then with their close texts in meaning.</p>

<p><strong>IMPORTANT</strong>: the first time you run the code it will be slow because it needs to download the model. if you want to prevent it from downloading the model again and use the local model you have to create a folder for cache and add it to the environment variable and then after the first time running use that path:
</p>

<pre><code>tf_hub_cache_dir = "universal_encoder_cached/"
os.environ["TFHUB_CACHE_DIR"] = tf_hub_cache_dir

# pointing to the folder inside cache dir, it will be unique on your system
module_url = tf_hub_cache_dir+"/d8fbeb5c580e50f975ef73e80bebba9654228449/"
embed = hub.Module(module_url)
</code></pre>

<p>More information: <a href="https://tfhub.dev/google/universal-sentence-encoder/2" rel="noreferrer">https://tfhub.dev/google/universal-sentence-encoder/2</a></p>
How to compute the similarity between two text documents?
<p>I am looking at working on an NLP project, in any programming language (though Python will be my preference).</p>
<p>I want to take two documents and determine how similar they are.</p>