City, University of London; University of Cambridge; Google Research
Lora Aroyo, Alex Taylor, Mark Díaz, Christopher Homan, Alicia Parrish, Gregory Serapio-García, Vinodkumar Prabhakaran, Ding Wang
conversational AI;human evaluation;human annotation;safety task;disagreement;variance in human annotations;diversity of rater pool
DICES Dataset: Diversity in Conversational AI Evaluation for Safety