we denote by $\nabla f_{i}(w^{t})$ gradient of $f_i$ with respect to $w$ at iteration $t$, and by $\eta^{t}$ the learning rate at iteration $t$. The computation of the above full gradient over the entire training data is often intractable due to a large number of data, which leads to the use of stochastic gradient that is computed using a subset uniformly selected at random from the training data. The iterative step of the stochastic gradient descent algorithm at iteration $t$ reads: $$\begin{aligned}
        w^{t+1} \coloneqq w^{t} - \eta^{t} \left( \frac{1}{B} \sum_{i \in \beta^t} \nabla f_{i}(w^{t}) + \lambda \nabla \gamma(w^t) \right), \label{eq:update:std}
    \end{aligned}$$ where $\beta^t$ denotes a mini-batch that is the index set of a subset uniformly selected at random from the training data. The mini-batch size $B = |\beta^t|$ is known to be related to the variance of the gradient norms, and thus to the regularization of the model. We assume that the mini-batch size is fixed in the optimization procedure to simplify the problem and emphasize the role of regularization parameter $\lambda$.

Regularization via Adaptive Weight-Decay (AdaDecay) {#sec:adadecay}
===================================================

We present a regularization algorithm that is designed to determine the degree of regularization for each model parameter considering its current 