<p><strong>You should NOT use BERT's output as sentence embeddings for semantic similarity.</strong> BERT is not pretrained for semantic similarity, which will result in poor results, even worse than simple Glove Embeddings. See below a comment from Jacob Devlin (first author in BERT's paper) and a piece from the Sentence-BERT paper, which discusses in detail sentence embeddings.</p>
<blockquote>
<p><strong>Jacob Devlin's comment:</strong> I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is is doing average pooling over the word tokens to get a sentence vector, but we never suggested that this will generate meaningful sentence representations. And even if they are decent representations when fed into a DNN trained for a downstream task, it doesn't mean that they will be meaningful in terms of cosine distance. (Since cosine distance is a linear space where all dimensions are weighted equally). (<a href="https://github.com/google-research/bert/issues/164#issuecomment-441324222" rel="nofollow noreferrer">https://github.com/google-research/bert/issues/164#issuecomment-441324222</a>)</p>
</blockquote>
<blockquote>
<p><strong>From Sentence-BERT paper:</strong> The results show that directly using the output of BERT leads to rather poor performances. Averaging the BERT embeddings achieves an average correlation of only 54.81, and using the CLS token output only achieves an average correlation of 29.19. Both are worse than computing average GloVe embeddings. (<a href="https://arxiv.org/pdf/1908.10084.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1908.10084.pdf</a>)</p>
</blockquote>
<p><strong>You should use instead a model pre-trained specifically for sentence similarity</strong>, such as Sentence-BERT. Sentence-BERT and several other pretrained models for sentence similarity are available in the sentence-transformers library (<a href="https://www.sbert.net/docs/pretrained_models.html" rel="nofollow noreferrer">https://www.sbert.net/docs/pretrained_models.html</a>), which is fully compatible with the amazing HuggingFace transformers library. With these libraries, you can obtain sentence embeddings in just a line of code.</p>
How to compare sentence similarities using embeddings from BERT
<p>I am using the HuggingFace Transformers package to access pretrained models. As my use case needs functionality for both English and Arabic, I am using the <a href="https://github.com/google-research/bert/blob/master/multilingual.md" rel="noreferrer">bert-base-multilingual-cased</a> pretrained model. I need to be able to compare the similarity of sentences using something such as cosine similarity. To use  this, I first need to get an embedding vector for each sentence, and can then compute the cosine similarity.</p>

<p>Firstly, what is the best way to extratc the semantic embedding from the BERT model? Would taking the last hidden state of the model after being fed the sentence suffice?</p>

<pre><code>import torch
from transformers import BertModel, BertTokenizer

model_class = BertModel
tokenizer_class = BertTokenizer
pretrained_weights = 'bert-base-multilingual-cased'

tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)

sentence = 'this is a test sentence'

input_ids = torch.tensor([tokenizer.encode(sentence, add_special_tokens=True)])
with torch.no_grad():
    output_tuple = model(input_ids)
    last_hidden_states = output_tuple[0]

print(last_hidden_states.size(), last_hidden_states)
</code></pre>

<p>Secondly, if this is a sufficient way to get embeddings from my sentence, I now have another problem where the embedding vectors have different lengths depending on the length of the original sentence. The shapes output are <code>[1, n, vocab_size]</code>, where <code>n</code> can have any value. </p>

<p>In order to compute two vectors' cosine similarity, they need to be the same  length. How can I do this here? Could something as naive as first summing across <code>axis=1</code> still work? What other options do I have? </p>