<p>There are 2 ways to compute the perplexity score: non-overlapping and sliding window. <a href="https://arxiv.org/pdf/2012.15832.pdf" rel="nofollow noreferrer">This paper</a> describes the details.</p>
How to measure performance of a pretrained HuggingFace language model?
<p>I am pretraining a <code>GPT2LMHeadModel</code> using <code>Trainer</code> as follows:</p>
<pre class="lang-py prettyprint-override"><code>training_args = TrainingArguments(
    output_dir=str(project_root / 'models/bn-gpt2/'),
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=4,
    fp16=True,
    optim=&quot;adafactor&quot;,
    eval_steps=400,
    save_steps=800,
    warmup_steps=500,
    evaluation_strategy=&quot;steps&quot;,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=tokenized_dataset['train'],
    eval_dataset=tokenized_dataset['test'],
)

trainer.train()
</code></pre>
<p>I want to measure the performance of my pre-trained model using perplexity or accuracy metrics during and after training. I have found some ways to measure these for individual sentences, but I cannot find a way to do this for the complete model. My goal is to create a next word prediction model for my native language using GPT2 training from scratch.</p>