<p>There are 3 ways you can approach your problem-</p>
<ol>
<li>There exists a very cool tool called <a href="https://github.com/hanxiao/bert-as-service" rel="nofollow noreferrer">bert-as-service</a>. It maps a sentence to a fixed length word embeddings based on the model you choose to use. The documentation is very well written.
Install</li>
</ol>
<pre><code>pip install bert-serving-server  # server
pip install bert-serving-client  # client, independent of bert-serving-server
</code></pre>
<p>Download one of the pre-trained models available at official BERT repo- <a href="https://github.com/google-research/bert" rel="nofollow noreferrer">link</a></p>
<p>Start the server</p>
<pre><code>bert-serving-start -model_dir /model_directory/ -num_worker=4 
</code></pre>
<p>Generate embedding</p>
<pre><code>from bert_serving.client import BertClient
bc = BertClient()
vectors=bc.encode(your_list_of_sentences)
</code></pre>
<ol start="2">
<li><p>There exist an academic paper by name of <a href="https://arxiv.org/abs/1908.10084" rel="nofollow noreferrer">Sentence-BERT</a> and their <a href="https://github.com/UKPLab/sentence-transformers" rel="nofollow noreferrer">github repo</a></p>
</li>
<li><p>You are doing a lot of manual work- padding attn mask etc.
Toeknizer does it for you automatically, check the <a href="https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.__call__" rel="nofollow noreferrer">documentation</a>. And, if you see the implementation of the <a href="https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L1012" rel="nofollow noreferrer">forward()</a> call of the model, it returns-</p>
</li>
</ol>
<pre><code> return (sequence_output, pooled_output) + encoder_outputs[1:]
</code></pre>
<p>For the bert base (768 hidden states), sequence output is the embedding of all token of the sequence, so if your input size[ max_len] is 510, then each token is embedded is a space of 768 dimension making sequence output of size- 768 * 510 *1</p>
<p>Pooled output is the one where all the embeddings are squished into a space of 768*1 dimension.</p>
<p>So I think you will want to use Pooled output for simple embeddings.</p>
How to extract Sentence Embedding Using BERT model from [CLS] token
<p>I am following this link:</p>
<p><a href="https://stackoverflow.com/questions/63209960/bert-document-embedding">BERT document embedding</a></p>
<p>I want to extract sentence-embedding using <code>BERT</code> model using <code>CLS</code> token. Here is the code:</p>
<pre><code>import torch
from keras.preprocessing.sequence import pad_sequences
import tensorflow as tf

def text_to_embedding(tokenizer, model, in_text):
    '''
    Uses the provided BERT 'model' and 'tokenizer' to generate a vector
    representation of the input string, 'in_text'.

    Returns the vector stored as a numpy ndarray.
    '''

    # ===========================
    #   STEP 1: Tokenization
    # ===========================

    MAX_LEN = 510

    # 'encode' will:
    #  (1) Tokenize the sentence
    #  (2) Prepend the '[CLS]' token to the start.
    #  (3) Append the '[SEP]' token to the end.
    #  (4) Map tokens to their IDs.
    input_ids = tokenizer.encode(
        in_text,                         # sentence to encode.
        add_special_tokens = True,       # Add '[CLS]' and '[SEP]'
        max_length = MAX_LEN,            # Truncate all sentences.
        #return_tensors = 'pt'           # Return pytorch tensors.
    )

    print(input_ids)
    print(tokenizer.decode(input_ids))

    # Pad our input tokens. Truncation was handled above by the 'encode'
    # function, which also makes sure that the '[SEP]' token is placed at the
    # end *after* truncating.
    # Note: 'pad_sequences' expects a list of lists, but we only have one
    # piece of text, so we surround 'input_ids' with an extra set of brackets.

    results = tokenizer(in_text, max_length=MAX_LEN, truncation=True)
    input_ids = results.input_ids
    attn_mask = results.attention_mask
    
    print(results)

    # Cast to tensors.
    input_ids = torch.tensor(input_ids)
    attn_mask = torch.tensor(attn_mask)

    # Add an extra dimension for the &quot;batch&quot; (even though there is only one
    # input in this batch)
    input_ids = input_ids.unsqueeze(0)
    attn_mask = attn_mask.unsqueeze(0)


    # ===========================
    #   STEP 1: Tokenization
    # ===========================

    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    
    #model.eval()

    # Copy the inputs to the GPU
    #input_ids = input_ids.to(device)
    #attn_mask = attn_mask.to(device)

    # telling the model not to build the backward graph will make this
    # a little quicker.
    with torch.no_grad():

        # Forward pass, returns hidden states and predictions
        # This will return the logits rather than the loss because we have
        # not provided labels.
        outputs = model(input_ids = input_ids,token_type_ids = None,attention_mask = attn_mask)
        

        hidden_states = outputs[2]

        #Sentence Vectors
        #To get a single vector for our entire sentence we have multiple 
        #application-dependent strategies, but a simple approach is to 
        #average the second to last hiden layer of each token producing 
        #a single 768 length vector.
        # `hidden_states` has shape [13 x 1 x ? x 768]

        # `token_vecs` is a tensor with shape [? x 768]
        token_vecs = hidden_states[-2][0]

        # Calculate the average of all ? token vectors.
        sentence_embedding = torch.mean(token_vecs, dim=0)
        # Move to the CPU and convert to numpy ndarray.
        sentence_embedding = sentence_embedding.detach().cpu().numpy()

        return(sentence_embedding)


from transformers import BertTokenizer, BertModel
# Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased',output_hidden_states = True), # Whether the model returns all hidden-states.
#model.cuda()

from transformers import BertTokenizer

# Load the BERT tokenizer.
print('Loadin BERT tokenizer...')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

k=text_to_embedding(tokenizer, model, &quot;I like to play cricket&quot;)
</code></pre>
<p>Output:</p>
<pre><code>&lt;ipython-input-14-f03410b60544&gt; in text_to_embedding(tokenizer, model, in_text)
     77         # This will return the logits rather than the loss because we have
     78         # not provided labels.
---&gt; 79         outputs = model(input_ids = input_ids,token_type_ids = None,attention_mask = attn_mask)
     80 
     81 

TypeError: 'tuple' object is not callable
</code></pre>
<p>I get an error in this line <code>outputs = model(input_ids = input_ids,token_type_ids = None,attention_mask = attn_mask)</code></p>
<p>Instead of using average of hidden layer, I want to modify code to get embedding for input sentence using <code>CLS </code> token.</p>