RIKEN Center for AIP; The University of Tokyo; Tsinghua University; MBZUAI
Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, Masashi Sugiyama
Weight Decay;Regularization;Optimization;Deep Learning
On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective