<p>You can use the <code>[CLS]</code> token as a representation for the entire sequence. This token is typically prepended to your sentence during the preprocessing step. This token that is typically used for classification tasks (see figure 2 and paragraph 3.2 in the <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="noreferrer">BERT paper</a>).</p>

<p>It is the very first token of the embedding.</p>

<p>Alternatively you can take the average vector of the sequence (like you say over the first(?) axis), which can yield better results according to the <a href="https://huggingface.co/transformers/model_doc/bert.html" rel="noreferrer">huggingface documentation</a> (3rd tip).</p>

<p>Note that BERT was not designed for sentence similarity using the cosine distance, though in my experience it does yield decent results. </p>
How to compare sentence similarities using embeddings from BERT
<p>I am using the HuggingFace Transformers package to access pretrained models. As my use case needs functionality for both English and Arabic, I am using the <a href="https://github.com/google-research/bert/blob/master/multilingual.md" rel="noreferrer">bert-base-multilingual-cased</a> pretrained model. I need to be able to compare the similarity of sentences using something such as cosine similarity. To use  this, I first need to get an embedding vector for each sentence, and can then compute the cosine similarity.</p>

<p>Firstly, what is the best way to extratc the semantic embedding from the BERT model? Would taking the last hidden state of the model after being fed the sentence suffice?</p>

<pre><code>import torch
from transformers import BertModel, BertTokenizer

model_class = BertModel
tokenizer_class = BertTokenizer
pretrained_weights = 'bert-base-multilingual-cased'

tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)

sentence = 'this is a test sentence'

input_ids = torch.tensor([tokenizer.encode(sentence, add_special_tokens=True)])
with torch.no_grad():
    output_tuple = model(input_ids)
    last_hidden_states = output_tuple[0]

print(last_hidden_states.size(), last_hidden_states)
</code></pre>

<p>Secondly, if this is a sufficient way to get embeddings from my sentence, I now have another problem where the embedding vectors have different lengths depending on the length of the original sentence. The shapes output are <code>[1, n, vocab_size]</code>, where <code>n</code> can have any value. </p>

<p>In order to compute two vectors' cosine similarity, they need to be the same  length. How can I do this here? Could something as naive as first summing across <code>axis=1</code> still work? What other options do I have? </p>