& 98.48 & 98.55 & 98.55 & 98.56 & 98.55 & 89.01 & 89.27 & 89.31 & 89.49 & 89.58 & 93.92 & 94.79 & 94.78 & 94.80 & 94.74\
max & 98.57 & 98.67 & 98.70 & 98.72 & 98.69 & 89.35 & 89.56 & 89.68 & 89.84 & 89.92 & 94.24 & 95.05 & 94.98 & 95.04 & 94.94\

Adaptation to Gradient Norm {#sec:alpha}
---------------------------

We empirically demonstrates the effect of hyper-parameter $\alpha$ in Eq.  that determines the adaptation to the gradient norm normalized in layers. We present the $10\%$-trimmed average and the maximum accuracy over the trials in Table \[tab.alpha.tuning\] where $\alpha$ is set as -1, 1, 2, 4, 8 and we trained NN-2 using MNIST and Fashion-MNIST datasets and ResNet-18 using CIFAR-10 for instances. As shown in Table \[tab.alpha.tuning\], $\alpha=-1$ is inferior to $\alpha=1$ in accuracy. This allows our algorithm to impose higher weight-decay rate due to larger gradient-norms. Table \[tab.alpha.tuning\] also demonstrates the presented AdaDecay is robust to the choice of the hyper-parameter $\alpha>0$. We use $\alpha=4$ throughout the following experiment, that has achieved the best result for MNIST and CIFAR10 and second best for Fashion-MNIST in Table \[tab.alpha.tuning\]

Comparison to Randomized Weight-Decay {#sec:comparison.to.randomized.wd}
-------------------------------------

