Since we normalize the gradient norm in layer with mean of 0 and std of 1, one may argue that AdaDecay involves a randomization effect to weight-decay. We thus compare AdaDecay with a noise injection to the weight-decay, namely randomized weight-decay, that follows Algorithm \[alg:adadecay\] but replaces Eq.  by $$\theta_j^t = S(\mathcal{N}_{(0,1)}; \alpha) = \frac{2}{1 + \exp(- \alpha \mathcal{N}_{(0,1)})}, \label{eq:RND}$$ where $\mathcal{N}_{(0,1)}$ is a random variable following the Normal distribution with mean of 0 and std of 1. Table \[tab:randomized-wd\] presents the validation accuracy for MNIST, Fashion-MNIST, and CIFAR-10 by fundamental models of NN-2, NN-2, and ResNet-18 respectively, trained by SGD with constant weight-decay, the randomized weight-decay, and the proposed AdaDecay with $\lambda=5\times10^{-4}$ and $\alpha=4$. It is successfully demonstrated that the benefit of our AdaDecay is not due to the randomization effect to the weight-decay but the use of adaptive weight-decay based on the gradient norm.

[P[4mm]{} | PPP | PPP | PPP]{} & & &\
& const & rnd & ours & const & rnd & our & cons & rnd & ours\
ave & 98.53 & 98.53 & **98.56** & 89.23 & 89.25 & **89.49** & 94.70 & 94.70 & **94.80**\
max & 98.63 & 98.65 & **98.72** 