Google Research
Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, SHARAN NARANG, Noah Constant
Keywords: multilingual;pretraining;language models;language sampling;language distribution;low-resource languages;overfitting
UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining