of optimization. The complexity of AdaDecay shown in Algorithm \[alg:adadecay\] remains $O(1)$ to the number of training examples as SGD with constant weight-decay.

$\{g_j^t\}_j$ : gradients of the parameters $j$ computed by back-propagation at iteration $t$ $\eta^t$ : learning rate at iteration $t$ $\lambda$ : global weight-decay coefficient $\alpha$ : hyper-parameter for the adaptation to the gradient norm $\mu_l^t$ : compute the mean of gradient norms $|g_j^t|$ in the layer $l$ $\sigma_l^t$ : compute the std of gradient norms $|g_j^t|$ in the layer $l$ $\tilde{g}_j^t = (|g_j^t| - \mu_l^t) / \sigma_l^t$ $\theta_j^t = S(\tilde{g}_j^t ; \alpha)$ using Eq.  $w_j^{t+1} \coloneqq (1 - \eta^{t} \lambda \theta_j^{t}) \, w_j^{t} - \eta^{t} \, g_j^t$

Experimental Results {#sec:experimental}
====================

We provide quantitative evaluation of the presented AdaDecay in comparison to the state-of-the-art optimization algorithms. For experiments, we use three of major benchmark datasets on image recognition: MNIST, Fashon-MNIST, and CIFAR-10. MNIST [@lecun1998gradient] is a simple yet fundamental dataset that consists of 60K training and 10K test gray images of hand-written 10 digits. Fashion-MNIST [@xiao2017online] is a modern dataset that consists of 60K training and 10K test gray images with 10 categories of clothes and fashion items. CIFAR-10 [@krizhevsky2009learning] is a more challenging task that 