Microsoft
Conglong Li, Minjia Zhang, Yuxiong He
natural language processing;language model pre-training;autoregressive language model;training instability
The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models