Stanford University; Google DeepMind
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, Adams Wei Yu
language models;pretraining;domain reweighting;data curation
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining