dropout that randomly zeros the activation of nodes [@srivastava2014dropout], give implicit regularization effects for the training process. The weight-decay is an explicit regularization such that an additional penalty term is defined in the energy function and the regularization effect can be tuned by its coefficient. In contrast to the recent development of adaptive methods on deep optimization, a constant weight-decay has been employed while it played an important role in both classic and modern deep neural networks [@krogh1992simple; @simonyan2014very; @chollet2017xception; @zhang2016understanding]. Layer-wise weight-decay was considered in [@ishii2017layer; @bengio2012practical] but it is limited to classical neural networks without the skip-connection.

We propose an adaptive regularization method of weight-decay, called Adaptive Weight-Decay (AdaDecay), that varies in spatio-temporal domains during the training process and is beneficial to both shallow and deep neural network models. The proposed AdaDecay determines the weight-decay rate in parameter-wise at each optimization iteration using the magnitude of the loss gradient that measures the difference between the current state of the model with the mini-batch data. We normalize the gradient norm in each layer in order to make the algorithm independent to the model architecture and robust to hyper-parameter selection. We also present experimental results of the presented AdaDecay in 