& 89.50 & 89.59 & **89.84** & 94.98 & 95.00 & **95.04**\

\[htb\]

[PP]{} ![Validation accuracy over drop-out rate $P = 0, 2^{-4}, 2^{-3}, 2^{-2}$ by SGD with the constant weight-decay (black) and the proposed AdaDecay with fixed $\alpha=4$ (red) for MNIST (left) and Fashion-MNIST (right) using LeNet-4. The weight-decay coefficient is fixed at $\lambda=5\times10^{-4}$. The $10\%$-trimmed average of the last $10\%$-epoch mean accuracy over 50 trials for MNIST and Fashion-MNIST is shown. []{data-label="fig:dropout"}]({{dropout.mnist.ConvNet_MNIST}} "fig:"){width="\fw"} & ![Validation accuracy over drop-out rate $P = 0, 2^{-4}, 2^{-3}, 2^{-2}$ by SGD with the constant weight-decay (black) and the proposed AdaDecay with fixed $\alpha=4$ (red) for MNIST (left) and Fashion-MNIST (right) using LeNet-4. The weight-decay coefficient is fixed at $\lambda=5\times10^{-4}$. The $10\%$-trimmed average of the last $10\%$-epoch mean accuracy over 50 trials for MNIST and Fashion-MNIST is shown. []{data-label="fig:dropout"}]({{dropout.FashionMNIST.ConvNet_MNIST}} "fig:"){width="\fw"}\
(1) MNIST & (2) Fashion-MNIST\

Effect of Dropout {#sec:effect.of.dropout}
-----------------

We demonstrate that our AdaDecay can be combined with dropout that gives an implicit regularization effect. We present accuracy curve over the dropping rate ($P$) within 50 trials in Figure \[fig:dropout\] where we trained LeNet-4 that supports the dropout for MNIST and Fashion-MNIST using SGD with the constant weight-decay of $\lambda=5\times10^{-4}$ (SGD) and 