<p>There is a paper <a href="https://arxiv.org/abs/1910.14659" rel="noreferrer">Masked Language Model Scoring</a> that explores pseudo-perplexity from masked language models and shows that pseudo-perplexity, while not being theoretically well justified, still performs well for comparing &quot;naturalness&quot; of texts.</p>
<p>As for the code, your snippet is perfectly correct but for one detail: in recent implementations of Huggingface BERT, <code>masked_lm_labels</code> are renamed to simply <code>labels</code>, to make interfaces of various models more compatible. I have also replaced the hard-coded <code>103</code> with the generic <code>tokenizer.mask_token_id</code>. So the snippet below should work:</p>
<pre class="lang-py prettyprint-override"><code>from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch
import numpy as np

model_name = 'cointegrated/rubert-tiny'
model = AutoModelForMaskedLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

def score(model, tokenizer, sentence):
    tensor_input = tokenizer.encode(sentence, return_tensors='pt')
    repeat_input = tensor_input.repeat(tensor_input.size(-1)-2, 1)
    mask = torch.ones(tensor_input.size(-1) - 1).diag(1)[:-2]
    masked_input = repeat_input.masked_fill(mask == 1, tokenizer.mask_token_id)
    labels = repeat_input.masked_fill( masked_input != tokenizer.mask_token_id, -100)
    with torch.inference_mode():
        loss = model(masked_input, labels=labels).loss
    return np.exp(loss.item())

print(score(sentence='London is the capital of Great Britain.', model=model, tokenizer=tokenizer)) 
# 4.541251105675365
print(score(sentence='London is the capital of South America.', model=model, tokenizer=tokenizer)) 
# 6.162017238332462
</code></pre>
<p>You can try this code in Google Colab by running <a href="https://gist.github.com/avidale/f574c014cd686709636b89208f2259ce" rel="noreferrer">this gist</a>.</p>
How to calculate perplexity of a sentence using huggingface masked language models?
<p>I have several masked language models (mainly Bert, Roberta, Albert, Electra). I also have a dataset of sentences. How can I get the perplexity of each sentence?</p>
<p>From the huggingface documentation <a href="https://huggingface.co/docs/transformers/perplexity" rel="noreferrer">here</a> they mentioned that perplexity &quot;is not well defined for masked language models like BERT&quot;, though I still see people somehow calculate it.</p>
<p>For example in this <a href="https://stackoverflow.com/questions/63030692/how-do-i-use-bertformaskedlm-or-bertmodel-to-calculate-perplexity-of-a-sentence">SO</a> question they calculated it using the function</p>
<pre><code>def score(model, tokenizer, sentence,  mask_token_id=103):
  tensor_input = tokenizer.encode(sentence, return_tensors='pt')
  repeat_input = tensor_input.repeat(tensor_input.size(-1)-2, 1)
  mask = torch.ones(tensor_input.size(-1) - 1).diag(1)[:-2]
  masked_input = repeat_input.masked_fill(mask == 1, 103)
  labels = repeat_input.masked_fill( masked_input != 103, -100)
  loss,_ = model(masked_input, masked_lm_labels=labels)
  result = np.exp(loss.item())
  return result

score(model, tokenizer, '我爱你') # returns 45.63794545581973
</code></pre>
<p>However, when I try to use the code I get <code>TypeError: forward() got an unexpected keyword argument 'masked_lm_labels'</code>.</p>
<p>I tried it with a couple of my models:</p>
<pre><code>from transformers import pipeline, BertForMaskedLM, BertForMaskedLM, AutoTokenizer, RobertaForMaskedLM, AlbertForMaskedLM, ElectraForMaskedLM
import torch

1)
tokenizer = AutoTokenizer.from_pretrained(&quot;bioformers/bioformer-cased-v1.0&quot;)
model = BertForMaskedLM.from_pretrained(&quot;bioformers/bioformer-cased-v1.0&quot;)
2)
tokenizer = AutoTokenizer.from_pretrained(&quot;sultan/BioM-ELECTRA-Large-Generator&quot;)
model = ElectraForMaskedLM.from_pretrained(&quot;sultan/BioM-ELECTRA-Large-Generator&quot;)
</code></pre>
<p><a href="https://stackoverflow.com/questions/61470768/how-does-masked-lm-labels-argument-work-in-bertformaskedlm">This</a> SO question also used the <code>masked_lm_labels</code> as an input and it seemed to work somehow.</p>