with our AdaDecay (Ours) with $\lambda=5\times10^{-4}$ and $\alpha=4$. Figure \[fig:dropout\] demonstrates that our AdaDecay overtakes the constant weight-decay irrespective of the use of dropout. We employ the dropping rate of $P=0$ for LeNet-4 in the other experiment for a fair comparison with other network models.

Comparison to State-of-the-Arts {#sec:compariosn.to.soa}
-------------------------------

We now compare SGD with the presented adaptive weight-decay (AdaDecay) to SGD with the constant weight-decay (SGD), SGD with RMSprop [@tieleman2012lecture] (RMS), SGD with Adam [@kingma2014adam] (Adam), Entropy-SGD [@Chaudhari2017EntropySGD] (eSGD), and Accelerated-SGD [@Kidambi2018Acc] (aSGD). We fix the weight-decay coefficient at $\lambda=5\times10^{-4}$ for all the algorithms and $\alpha=4$ for ours. In Table \[tab:accuracy\], we present the $10\%$-trimmed average (upper) and the maximum (lower) validation accuracy with the shallow networks: NN-2, NN-3, LeNet-4, and VGG-9 for MNIST (1) and Fashion-MNIST (2) over 50 trials, and the deep models: ResNet-18, ResNet-50, GoogLeNet, and DesnseConv for CIFAR-10 (3) over 32 trials, respectively. It is shown that SGD powered by our AdaDecay outperforms all the others in the average accuracy consistently regardless of model and dataset. The visualization of the average accuracy curve over epochs in Figure \[fig:accuracy\_curve\] indicating that SGD with our AdaDecay (red) achieves better accuracy than SGD with the constant weight-decay (black), 