nicoberk/GermanNewsCrime
Text Classification
•
0.1B
•
Updated
Nicolai Berk
nicoberk
chkla's profile picture
musashihinck's profile picture
6b4b86ec-928a-4b7e-9c1e-8d5f009e3272's profile picture
·
AI & ML interests
NLP, Political Communication, Media Effects
Recent Activity
reacted
to
MoritzLaurer's
post
with ❤️
1 day ago
Prompts are hyperparameters. Every time you test a different prompt on your data, you become less sure if the LLM actually generalizes to unseen data.
Issues of overfitting to a test set seem like concepts from boring times when people still fine-tuned models, but it's just as important for "zeroshot prompting". Using a separate validation split to tune the main hyperparameter of LLMs (the prompt) is just as important as train-val-test splitting for fine-tuning. The only difference is that you don't have a training dataset anymore and it somehow feels different because there is no training / no parameter updates.
Its easy to trick yourself into believing that an LLM performs well on your task, while you've actually overfit the prompt on your data. Every good "zeroshot" paper should clarify that they used a validation split for finding their prompt before final testing.
updated
a Space
29 days ago
PubPol/FrenchTutor
published
a Space
29 days ago
PubPol/FrenchTutor
Organizations
