Paper under double-blind review
optimization;asymptotic behavior of stochastic optimization;learning-rate decay;weight decay;language model pre-training;Transformer pre-training
Amos: An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale