&= \rho( w ) + \frac{\lambda}{2} \, \| w \|_2^2, \label{eq:objective:l2}
    \end{aligned}$$ where $\lambda \in \R$ denotes the coefficient for the regularization term. Then, the gradient descent step at each iteration $t$ by a first-order optimization algorithm reads: $$\begin{aligned}
    w^{t+1} & \coloneqq w^{t} - \eta^{t} \left( \nabla \rho( w^t ) + \lambda w^t \right) \nonumber \\ 
            & = (1 - \eta^{t} \lambda) \, w^{t} - \eta^{t} \, \nabla \rho(w^{t}), \label{eq:update:weightdecay}
    \end{aligned}$$ where $\eta^t \lambda$ is constrained to be $[0, 1)$ leading to the shrinkage of the unknown model parameters $w$ in iteration $t$ and this regularization scheme based on the $L_2^2$ norm is called weight-decay. In contrast to the static coefficient $\lambda \in \R$ in the conventional weight-decay regularization, we propose a regularization scheme that is designed to impose adaptive regularity to each model parameter $w_j$ with an additional term $\theta_j$ as follows: $$\begin{aligned}
    \L(w) &= \rho( w ) + \frac{\lambda}{2} \, \| \theta \odot w \|_2^2, \label{eq:objective:l2:parameterwise}
    \end{aligned}$$ where $\theta = (\theta_1, \theta_2, \cdots, \theta_m) \in \R^{m}, w = (w_1, w_2, \cdots, w_m) \in \R^{m}$ and the symbol $\odot$ denotes the Hadamard product defined by: $\theta \odot w = (\theta_1 w_1, \theta_2 w_2, \cdots, \theta_m w_m)$. The degree of 