Hong Liu, Zhiyuan Li, David Hall, Percy Liang, Tengyu Ma
large language models;pretraining;optimization in deep learning
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training