[@snoek2015scalable; @shahriari2016unbounded]. However, in contrast to recent development of adaptive methods of dropout [@ba2013adaptive; @kingma2015variational; @srinivas2016generalized; @huang2016deep; @noh2017regularizing] and learning-rate [@duchi2011adaptive; @tieleman2012lecture; @kingma2014adam] in deep optimization, a constant weight-decay coefficient has been employed in usual. Layer-wise weight-decay has been considered in [@ishii2017layer; @bengio2012practical] where different weight-decay coefficients are given for different layers of network model using the variance of gradients in layer. The drawback of the layer-wise method [@ishii2017layer; @bengio2012practical] is that it assumes that layers are aligned in a single sequence. The skip-connection [@he2016deep; @he2016identity; @Balduzzi2017shattered], that is one of the key architectures in the recent deep networks, makes it non-trivial.

The main contributions of this work are three folds: First, we propose an adaptive regularization method (AdaDecay) in which parameter-wise weight-decay varies in spatio-temporal domains reflecting the currant state of model and the mini-batch data. Second, the proposed AdaDecay determines the weight-decay rate of each parameter based on the norm of gradient normalized in layer. This makes the algorithm independent to model architecture and beneficial to both shallow and deep neural network models. Third, we empirically demonstrate the robustness and effectiveness of the presented AdaDecay in comparison to the state-of-the-art optimization algorithm using both shallow neural network models 