consists of 50K training and 10K test object images with 10 categories. Regarding the network architecture, we employ four of shallow networks and four of deep networks: The shallow networks include: fully-connected neural networks with two hidden layers (NN-2) and with three hidden layers (NN-3) [@blum1991approximation], LeNet-4 [@lecun1998gradient] with two convolution layers followed by two of fully-connected layers, and VGG-9 [@simonyan2014very]. The deep networks used in our experiments are: ResNet-18 [@he2016deep; @he2016identity], ResNet-50 [@Balduzzi2017shattered], GoogLeNet [@szegedy2015going], and the densely connected convolutional networks (DenseConv) [@huang2017densely]. The batch normalization [@ioffe2015batch] is used in VGG-9, ResNet-18, ResNet-50, GoogLeNet and DenseConv.

Our comparative analysis involves the following optimization algorithms: the stochastic gradient descent with the constant weight-decay (SGD), SGD with RMSprop [@tieleman2012lecture] (RMS), SGD with Adam [@kingma2014adam] (Adam), Entropy-SGD [@Chaudhari2017EntropySGD] (eSGD), Accelerated-SGD [@Kidambi2018Acc] (aSGD), and SGD with the presented adaptive weight-decay (AdaDecay). Regarding hyper-parameters in our experiments, we use a practical condition including the mini-batch size of $B=128$ with the momentum of 0.9. The weight-decay coefficient is inspected in Section \[sec:wd-tuning\] and fixed at $\lambda=5\times10^{-4}$ for all the algorithms. The hyper-parameter $\alpha$ of AdaDecay that determines the adaptation to the gradient norm is inspected in Section \[sec:alpha\] and fixed at $\alpha=4$. The learning-rate annealing 