<p>In addition to an already great accepted answer, I want to point you to <a href="https://arxiv.org/abs/1908.10084" rel="noreferrer"><code>sentence-BERT</code></a>, which discusses the similarity aspect and implications of specific metrics (like cosine similarity) in greater detail.
They also have a <a href="https://github.com/UKPLab/sentence-transformers" rel="noreferrer">very convenient implementation</a> online. The main advantage here is that they seemingly gain a lot of processing speed compared to a "naive" sentence embedding comparison, but I am not familiar enough with the implementation itself.</p>

<p>Importantly, there is also generally a more fine-grained distinction in <em>what kind of similarity</em> you want to look at. Specifically for that, there is also a great discussion in one of the <a href="https://www.aclweb.org/anthology/S14-2001.pdf" rel="noreferrer">task papers</a> from SemEval 2014 (SICK dataset), which goes into more detail about this. From your task description, I am assuming that you are already using data from one of the later SemEval tasks, which also extended this to multilingual similarity.</p>
How to compare sentence similarities using embeddings from BERT
<p>I am using the HuggingFace Transformers package to access pretrained models. As my use case needs functionality for both English and Arabic, I am using the <a href="https://github.com/google-research/bert/blob/master/multilingual.md" rel="noreferrer">bert-base-multilingual-cased</a> pretrained model. I need to be able to compare the similarity of sentences using something such as cosine similarity. To use  this, I first need to get an embedding vector for each sentence, and can then compute the cosine similarity.</p>

<p>Firstly, what is the best way to extratc the semantic embedding from the BERT model? Would taking the last hidden state of the model after being fed the sentence suffice?</p>

<pre><code>import torch
from transformers import BertModel, BertTokenizer

model_class = BertModel
tokenizer_class = BertTokenizer
pretrained_weights = 'bert-base-multilingual-cased'

tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)

sentence = 'this is a test sentence'

input_ids = torch.tensor([tokenizer.encode(sentence, add_special_tokens=True)])
with torch.no_grad():
    output_tuple = model(input_ids)
    last_hidden_states = output_tuple[0]

print(last_hidden_states.size(), last_hidden_states)
</code></pre>

<p>Secondly, if this is a sufficient way to get embeddings from my sentence, I now have another problem where the embedding vectors have different lengths depending on the length of the original sentence. The shapes output are <code>[1, n, vocab_size]</code>, where <code>n</code> can have any value. </p>

<p>In order to compute two vectors' cosine similarity, they need to be the same  length. How can I do this here? Could something as naive as first summing across <code>axis=1</code> still work? What other options do I have? </p>