National University of Singapore; ETH Zurich; University of Edinburgh
Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, Zangwei Zheng, Yang You
Large Language Model;Transformer Scaling;Foundation Model Pre-training
To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis