domain of the model parameter and the temporal domain of the optimization. The degree of regularity for each model parameter is determined in consideration of residual, or norm of gradient, that determines a discrepancy between the current state of model and the observation. The gradient norm $|g_j^t|$, known as Gauss-Southwell rule, has been used in importance sampling of parameters, e.g.,  [@glasmachers2013accelerated; @nutini2017let; @namkoong2017adaptive]. This is, however, not directly applicable to our case since the magnitude of gradients varies exponentially over layers in deep model. We thus normalize the gradient norm $|g_j^t|$ to have mean $0$ and standard deviation (std) $1$ within each layer at each iteration in order to consider the relative significance of the local parameters within the layer. The normalized gradient-norm $\tilde{g}_j^t$ is given by: $$\begin{aligned}
    \tilde{g}_j^t &= \frac{|g_j^t| - \mu_l^t}{\sigma_l^t}, \label{eq:grad:normalized}
    \end{aligned}$$ where $l$ denotes the index of the layer that includes the parameter $w_j$, and $\mu_l^t$ and $\sigma_l^t$ denotes the mean and standard deviation of all the gradient norms for the parameters within the layer $l$ at iteration $t$, respectively. We assume that the degree of regularity $\theta_j^t$ for each parameter $w_j$ at iteration $t$ follows a distribution of the residual leading to the following data-driven 