regularity: $$\begin{aligned}
    \theta_j^t \propto \tilde{g}_j^t, \label{eq:weight:distribution}
    \end{aligned}$$ where the degree of regularization for each parameter is proportional to the norm of its gradient. In the determination of our adaptive regularization, we use the scaled sigmoid function defined by: $S(x; \alpha) = 2 / (1 + \exp(- \alpha x))$, where $\alpha \in \R$ is a control parameter for the steepness of function value transition. Then, the relative degree of regularization $\theta_j^t$ for each parameter $w_j$ at iteration $t$ is determined by the scaled sigmoid function $S$ of the normalized gradient norm $\tilde{g}_j^t$ as follows: $$\begin{aligned}
    \theta_j^t = S(\tilde{g}_j^t; \alpha) = \frac{2}{1 + \exp(- \alpha \tilde{g}_j^t)}, \label{eq:theta}
    \end{aligned}$$ where $\alpha$ determines the slope of the decay rate transition according to the gradient norm, and $\theta_j^t$ ranges from $0$ to $2$ and its average is $1$ since $\tilde{g}_j^t$ is normalized to have mean $0$ and standard deviation $1$.

The pseudo code of the proposed algorithm for the adaptive weight-decay is described in Algorithm \[alg:adadecay\] where the degree of regularization for each parameter is determined based on the scaled sigmoid function of the norm of its gradient leading to the adaptive regularization in both the spatial domain of model parameters and the temporal domain 