Naver AI Lab, Naver Clova; Applied Information Engineering, Yonsei University; Naver AI Lab
Byeongho Heo, Sanghyuk Chun, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Gyuwan Kim, Youngjung Uh, Jung-Woo Ha
momentum optimizer;scale-invariant weights;normalize layer;effective learning rate
AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights