Ant Group, Hangzhou, Zhejiang, China; Ant Group, Beijing, China
Yun Yue, Zhiling Ye, Jiadi Jiang, Yongchao Liu, Ke Zhang
adaptive optimizer;gradient difference;auto switch;AGD
AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix