state of solution in the course of optimization procedure in an adaptive way. The optimization of interest aims to minimize the objective function that consists of a data fidelity term, a regularization term, and a control parameter for their relative weight. The control parameter that determines the relative significance between the data fidelity and the regularization is generally chosen to be constant based on the assumption that the underlying distributions of the residual and the prior smoothness follow uni-modal distributions. However, it is often ineffective to model the trade-off between the data fidelity and the regularization distributions using a static control parameter based on the ratio between the variances of their distributions. Thus, we propose an adaptive regularization scheme that considers residual in the determination of regularity for both the spatial domain of model parameters and the temporal domain of optimization.

Weight-Decay for Individual Model Parameter
-------------------------------------------

The computation of empirical stochastic gradient involves the noise process following a certain distribution with zero mean, and its variance is related to the degree of regularization that is desired to be imposed. We consider the regularization $\gamma(w)$ in Eq.  by the squared Euclidean norm leading to the following objective function: $$\begin{aligned}
    \L(w) 