comparison to the state-of-the-art optimization algorithms using major benchmark datasets on the image classification with shallow and deep networks, where our AdaDecay has overtakes the others in validation accuracy.

In the remaining of this paper we summarize the related studies with our contributions in Section \[sec:works\] and then describe notations in Section \[sec:prelimminary\]. We present AdaDecay in Section \[sec:adadecay\], followed by experimental results in Section \[sec:experimental\] and conclusion in Section \[sec:conclusion\]

Related Work {#sec:works}
============

[**Learning Rate Annealing:**]{} The stochastic gradient, calculated by a subset of data, gives a noise to gradient and provides an implicit regularization effect [@zhu2019anisotropic]. In SGD, parameters are updated by subtracting the gradient with the stochastic noise multiplied by the learning rate. The learning rate should shrink in order to reduce the noise and converge the algorithm. To this aim, a variety of learning rate annealing, e.g. exponential [@george2006adaptive] and staircase [@smith2017don], and the adaptive learning rates, e.g., AdaGrad [@duchi2011adaptive], have been proposed, The sophisticated adaptive techniques, e.g., RMSprop [@tieleman2012lecture] and Adam [@kingma2014adam], enable parameter-wise control of the learning rates. The drawback of learning rate techniques on the regularization is that it reduces or increases both the step-size and the noise.

[**Dropout**]{} is another regularization 