title: How do I use BertForMaskedLM or BertModel to calculate perplexity of a sentence?
body: <p>I want to use BertForMaskedLM or BertModel to calculate perplexity of a sentence, so I write code like this:</p>
  <pre class="lang-py prettyprint-override"><code>import numpy as np
  import torch
  import torch.nn as nn
  from transformers import BertTokenizer, BertForMaskedLM
  # Load pre-trained model (weights)
  with torch.no_grad():
      model = BertForMaskedLM.from_pretrained('hfl/chinese-bert-wwm-ext')
      model.eval()
      # Load pre-trained model tokenizer (vocabulary)
      tokenizer = BertTokenizer.from_pretrained('hfl/chinese-bert-wwm-ext')
      sentence = &quot;我不会忘记和你一起奋斗的时光。&quot;
      tokenize_input = tokenizer.tokenize(sentence)
      tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
      sen_len = len(tokenize_input)
      sentence_loss = 0.
  
      for i, word in enumerate(tokenize_input):
          # add mask to i-th character of the sentence
          tokenize_input[i] = '[MASK]'
          mask_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
  
          output = model(mask_input)
  
          prediction_scores = output[0]
          softmax = nn.Softmax(dim=0)
          ps = softmax(prediction_scores[0, i]).log()
          word_loss = ps[tensor_input[0, i]]
          sentence_loss += word_loss.item()
  
          tokenize_input[i] = word
      ppl = np.exp(-sentence_loss/sen_len)
      print(ppl)
  </code></pre>
  <p>I think this code is right, but I also notice BertForMaskedLM's paramaters <code>masked_lm_labels</code>, so could I use this paramaters to calculate PPL of a sentence easiler?
  I know the input_ids argument is the masked input, the masked_lm_labels argument is the desired output. But I couldn't understand the actual meaning of its output loss, its code like this:</p>
  <pre><code>if masked_lm_labels is not None:
      loss_fct = CrossEntropyLoss()  # -100 index = padding token
      masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), 
      masked_lm_labels.view(-1))
      outputs = (masked_lm_loss,) + outputs
  </code></pre>