section_path: Evaluation/Perplexity
section_name: Perplexity
paragraph_text: Perplexity
  
  The canonical measure of the performance of an LLM is its perplexity on a given text corpus. Perplexity measures how well a model predicts the contents of a dataset; the higher the likelihood the model assigns to the dataset, the lower the perplexity. In mathematical terms, perplexity is the exponential of the average negative log likelihood per token.
  
  log (Perplexity) = − 1 N ∑ i = 1 N log (Pr (token i ∣ context for token i)) {\displaystyle \log({\text{Perplexity}})=-{\frac {1}{N}}\sum _{i=1}^{N}\log(\Pr({\text{token}}_{i}\mid {\text{context for token}}_{i}))}
  
  Here, N {\displaystyle N} is the number of tokens in the text corpus, and "context for token i {\displaystyle i} " depends on the specific type of LLM. If the LLM is autoregressive, then "context for token i {\displaystyle i} " is the segment of text appearing before token i {\displaystyle i}. If the LLM is masked, then "context for token i {\displaystyle i} " is the segment of text surrounding token i {\displaystyle i}.
  
  Because language models may overfit to training data, models are usually evaluated by their perplexity on a test set. This evaluation is potentially problematic for larger models which, as they are trained on increasingly large corpora of text, are increasingly likely to inadvertently include portions of any given test set.