regularization for each model parameter $w_j^t$ at iteration $t$ is determined by the adaptive term $\theta_j^t$, leading to the following modified iterative update step: $$\begin{aligned}
 \label{eq:update:weightdecay:adaptive}
    w_j^{t+1} & \coloneqq (1 - \eta^{t} \lambda \theta_j^{t}) \, w_j^{t} - \eta^{t} \, g_j^t,
    \end{aligned}$$ where $g_j^t = \frac{\partial \rho(w^t)}{\partial w_j}$ is the gradient of the data fidelity $\rho$ with respect to the parameter $w_j$ at iteration $t$. The weight-decay coefficient $\lambda \ge 0$ determines the constant degree of regularity for all the model parameters whereas the adaptive term $\theta$ determines the relative significant of each model parameter at each step in the course of optimization, i.e., the decay rate for each model parameter $w_j^t$ at iteration $t$ is determined by the global regularization parameter $\lambda$ multiplied by the adaptive term $\theta_j^t$. Note that Eq.  with $\theta_j^t = 1$ for all $j$ and $t$ becomes the same as Eq.  with a constant weight-decay.

Adaptive Weight-Decay based on Residual
---------------------------------------

We now consider the parameter-wise adaptive term $\theta_j^{t}$ in Eq. . Our proposed regularization scheme is designed to impose an adaptive regularity to each model parameter based on its associated residual at each iteration of optimization leading to an adaptive regularization in both the spatial 