title: Where is perplexity calculated in the Huggingface gpt2 language model code?
body: <p>I see some github comments saying the output of the model() call's loss is in the form of perplexity:
  <a href="https://github.com/huggingface/transformers/issues/473" rel="nofollow noreferrer">https://github.com/huggingface/transformers/issues/473</a></p>
  
  <p>But when I look at the relevant code...
  <a href="https://huggingface.co/transformers/_modules/transformers/modeling_openai.html#OpenAIGPTLMHeadModel.forward" rel="nofollow noreferrer">https://huggingface.co/transformers/_modules/transformers/modeling_openai.html#OpenAIGPTLMHeadModel.forward</a></p>
  
  <pre><code>    if labels is not None:
          # Shift so that tokens &lt; n predict n
          shift_logits = lm_logits[..., :-1, :].contiguous()
          shift_labels = labels[..., 1:].contiguous()
          # Flatten the tokens
          loss_fct = CrossEntropyLoss()
          loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
          outputs = (loss,) + outputs
  
      return outputs  # (loss), lm_logits, (all hidden states), (all attentions)
  </code></pre>
  
  <p>I see cross entropy being calculated, but no transformation into perplexity. Where does the loss finally get transformed? Or is there a transformation already there that I'm not understanding?</p>