average of the last $10\%$-epoch accuracy over the trials.

Selection of Weight-Decay Coefficient {#sec:wd-tuning}
-------------------------------------

The weight-decay coefficient $\lambda$ determines the balance between the stochastic loss with the regularization and thus plays a critical role in both the standard weight-decay and the presented AdaDecay. In Figure \[fig:wd-tuning\], we test the coefficient ranging in $\lambda = 1\times10^{-3}, 7\times10^{-4}, 5\times10^{-4}, ... , 1\times10^{-4}$, using MNIST with NN-2, Fashion-MNIST with NN-2, and CIFAR-10 with ResNet-18 as instances. We compare SGD using the constant weight-decay (constant) with SGD using our AdaDecay with fixed $\alpha=4$ (ours) where the two algorithms share the same weight-decay coefficient. Figure \[fig:wd-tuning\] successfully demonstrates that the presented AdaDecay overtakes the constant weight-decay irrespective of the weight-decay coefficient $\lambda$ across both datasets and the model architecture. Based on the these results and the related works [@simonyan2014very; @chollet2017xception], we employ $\lambda=5\times10^{-4}$ in our experiments.

\[htb\]

  ------------------------------------------------- -------------------------------------------------------- ------------------------------------------------
   ![image]({{wd.MNIST.FFNet_MNIST}}){width="\fw"}   ![image]({{wd.FashionMNIST.FFNet_MNIST}}){width="\fw"}   ![image]({{wd.cifar10.ResNet18}}){width="\fw"}
                     \(1) MNIST                                        \(2) Fashion-MNIST                                     \(3) CIFAR-10
  ------------------------------------------------- -------------------------------------------------------- ------------------------------------------------

[p[6mm]{} | PPPPP | PPPPP | PPPPP]{} & & &\
$\alpha$ & -1 & 1 & 2 & 4 & 8 & -1 & 1 & 2 & 4 & 8 & -1 & 1 & 2 & 4 & 8\
ave 