On the limits of neural network explainability via descrambling
We characterize the exact solutions to neural network descrambling--a
mathematical model for explaining the fully connected layers of trained neural
networks (NNs). By reformulating the problem to the minimization of the
Brockett function arising in graph matching and complexity theory we show that
the principal components of the hidden layer preactivations can be
characterized as the optimal explainers or descramblers for the layer weights,
leading to descrambled weight matrices. We show that in typical deep learning
contexts these descramblers take diverse and interesting forms including (1)
matching largest principal components with the lowest frequency modes of the
Fourier basis for isotropic hidden data, (2) discovering the semantic
development in two-layer linear NNs for signal recovery problems, and (3)
explaining CNNs by optimally permuting the neurons. Our numerical experiments
indicate that the eigendecompositions of the hidden layer data--now understood
as the descramblers--can also reveal the layer's underlying transformation.
These results illustrate that the SVD is more directly related to the
explainability of NNs than previously thought and offers a promising avenue for
discovering interpretable motifs for the hidden action of NNs, especially in
contexts of operator learning or physics-informed NNs, where the input/output
data has limited human readability.
Shashank Sule, Richard G. Spencer, Wojciech Czaja
# Emergence of the SVD as an interpretable factorization in deep learning for inverse problems

###### Abstract.

We demonstrate the emergence of weight matrix singular value decomposition (SVD) in interpreting neural networks (NNs) for parameter estimation from noisy signals. The SVD appears naturally as a consequence of initial application of a descrambling transform - a recently-developed technique for addressing interpretability in NNs [1]. We find that within the class of noisy parameter estimation problems, the SVD may be the means by which networks memorize the signal model. We substantiate our theoretical findings with empirical evidence from both linear and non-linear settings. Our results also illuminate the connections between a mathematical theory of semantic development [2] and neural network interpretability.

Interpreting the nature of the mapping between inputs and outputs of trained neural networks (NNs) remains one of the major unsolved problems in machine learning and is the focus of significant research effort [3, 4, 5, 6, 7, 8, 9, 10]. More recently, such efforts have succeeded in interpreting NN's in image classification by applying interpretation heuristics to the singular value decompositions (SVDs) of trained weights [1, 11, 12, 13]. The SVD has innumerable applications in statistics [14, 15], dimensionality-reduction [16, 17, 18], and geometric data science [19, 20]. It has also found previous application in specific deep learning settings, including sensing neural network topology [21], isotropic pattern recognition in control charts [22], neural network weight compression [23]. The key idea behind using SVD for NN interpretation is that the singular subspaces of NN-associated matrices, such as weight and data correlation matrices, encode the learning that occurs during training. This is particularly appealing when interpretation relies upon recognition of readily-identifiable patterns, such as dog or cat labels. Consequently, the hidden semantics of a trained NN may be uncovered by taking the SVD as a starting point and then applying a (possibly complicated) heuristic such as intertwiner groups [12] or hypergraph rearrangements [13] of these SVDs.

In this paper we show that this link between singular spaces and NN interpretation is more fundamental: an interpretation heuristic can be the starting point and the singular value decomposition can naturally emerge as an interpretation. We specifically exhibit this phenomenon for _smoothness descrambling transformations_ - or simply, descrambling transformations - for interpretations of NNs used in signal processing [1]. Neural networks are routinely used for such regression problems in the sciences [24, 25, 26, 27] where network outputs are no longer simple class labels but are objects in a desired function space, and the network is trained to learn the underlying mapping between the input and output function classes. While these networks often match the prior state-of-the-art, their interpretability in the context of these problems remains an open question since the applicability of NN interpretation techniques from image classification is not established yet. To that end, we have focused on the SVD and the latent orthogonal transformations learned by the network motivated by the observation that many well-known transformations, such as the Fourier transform, are unitary and thus serve as a standard for human readability. This approach is quite promising: an analysis of linear two-layer neural networks [2] showed that the first and last weight matrices interact with the SVD of the covariance matrices from the distributions of training data. In particular, if the network structure is given by \(f(x)=W_{2}W_{1}x\), then the weights \(W_{1}\) and \(W_{2}\) in the large data limit obey the continuous-time evolution law:

\[W_{2}(t)=UA(\Lambda,t)Q^{-1}\quad W_{2}(t)=QA(\Lambda,t)V^{\top} \tag{1}\]

Here \(U\) and \(V\) are the right and left singular vectors of the input-output correlation matrix \(\Sigma_{yx}=\mathbb{E}[yx^{\top}]=U\Lambda V^{\top}\). While this simple model shows that the singular vectors closest to both input and output data learn semantics from the data distribution, an understanding of the intermediate weight matrices (matrices of the type \(Q\) in (1)) and the case of nonlinear and deep networks, remains unresolved. A major step in addressing this problem was taken in [28], in the context of digital signal processing, where a fully-connected network
## 1. Introduction

The _macroscopic_ of a graph \(G\) is a graph \(G\) with vertices \(v\in V(G)\) and edges \(v\in V(G)\), where \(v\) is the vertex of \(G\). The _macroscopic_ of a graph \(G\) is a graph \(G\) with vertices \(v\in V(G)\) and edges \(v\in V(G)\) are called _macroscopic_.
and biexponential parameter estimation-noting that right singular vectors of the first weight matrix closely resemble the training data model independently of the descrambling.

## 1. Smoothness Criterion Descrambling

In this section we formulate the main mathematical problem that we address in this paper. Let \(\eta\) - the objective function, be the descrambler criterion and let \(\widehat{P}\) denote the descrambler. The goal of this procedure is design \(\eta\) which yields a simple, human readable explanation of the descrambled weight matrices \(P^{*}(k,X)W_{k}\). In this context the _smoothness criterion_

\[\eta(P)=\|D^{2}PW_{k}\circ\sigma\circ\cdots W_{1}X\|_{F}^{2} \tag{4}\]

is well suited for interpretation of linear layers of feedforward networks. The motivation behind this criterion is that while the network's incoming and outgoing data is usually smooth and has an intelligible time-ordered structure, the intermediate data often loses this structure. As such, the role of the descrambler is to change basis so that the intermediate signal is itself smooth -- and thus ordered with respect to the indices of the output or input dimension. This strategy was used successfully to interpret the first layer of DEERNet [1, 28]. Our goal is to study the minimizers

\[\widehat{P}(k,X):=\operatorname{argmin}\,\|DPf_{k}(X)\|_{F}^{2}\quad\text{w.r.t}\,P^{\top}P=I. \tag{5}\]

Here \(X\) denotes the matrix of the training data, which we assume is drawn from a context-dependent prior distribution. Different choices of this prior will lead to different solutions of the descrambling problem 3. However, for any general prior it is difficult to say anything about the structure of \(\widehat{P}(k,X)\). As a consequence we focus on the case where \(X\) models data used in inverse problems - the original use-case of descrambling transformations. Thus each column of \(X\) can be written as a noisy measurement \(s(z)+\alpha^{-1}y\) where \(z\) is a to-be-recovered variable, \(y\) is noise, and \(\alpha^{-1}\) is the SNR. We will first understand the case when the network is linear and the input is only noise. Then we will generalize it to the case where the output has an underlying signal, and finally when the network is non-linear. All proofs have been outlined in the SI appendix.

## 2. Results

We proceed to characterize the minimizers of (5) - the smoothness criterion descrambling problem. As stated above, the minimizer \(\widehat{P}(k,X)\) depends on the matrix \(X\) and the wiretapped network \(f_{k}\). Our main mathematical innovation is to separate the cases \(k=1\) (when \(f_{k}\) is affine) and \(k\geq 2\) to consider \(X\) as sampled over a distribution. This enables us to model the real-life settings in which neural networks are trained and evaluated. With these assumptions in mind, we state our main theoretical results below.

**Linear Network, Only Noise.** We first address the case when \(k=1\) in (5). In this case, \(f_{1}(x)=W_{1}x\) so the wiretapped network \(f_{1}\) is linear. Additionally, to approach the data models present in inverse problems, we first consider the case where \(X\) is pure noise, i.e \(x_{i}\sim\xi\) where \(\xi\) is some isotropic random variables. The canonical example of such a random variable is if \(\xi\sim N(0,I_{d})\). Our strategy to understand the minimizers of (5) is to study the convergence of \(\widehat{P}\) as \(N\to\infty\). As such, we denote \(\widehat{P}(k,X_{N}):=\widehat{P}(N)\) and \(X:=X_{N}\) to keep track of the number of data points and implicitly assume \(k=1\). We shall see that these minimizers \(\widehat{P}(N)\) have a limit \(\mathcal{P}\) given by the (strong) law of large numbers (SLLN). However, to use the SLLN to swap a limit and a minimum we need to ensure the stability of the singular spaces of the random matrices \(WX_{N}\). Bearing this in mind we first introduce the following definition.

_Definition 1_.: Let \(A_{n,m}\subseteq M_{n,m}(\mathbb{R})\) be the set of matrices with distinct non-zero singular values. Let \(A=\bigcup_{n,m}A_{n,m}\) be the set of _admissible matrices_. A fully-connected feedforward NN with weights \(W_{k}\in A\) is termed an _admissible neural network_.

Using the definition of admissibility, the following lemma can be proved using the continuity of the simple eigenvalues of a matrix in its entries [29]:

_Lemma 1_.: Let \(W_{n}\) be a sequence of matrices such that \(W_{n}\to W\) in the Frobenius norm, where \(W\in A\). Then, the left and right singular vectors \(\{u_{i}^{n}\}\) and \(\{v_{i}^{n}\}\) of \(W_{n}\) converge to the right and left singular vectors of \(W\), respectively.
Henceforth, we shall only work with admissible neural networks. Note that since \(A_{n,m}\) is a dense subset of \(M_{n,m}\), admissible neural networks are dense in the set of neural networks. Note that for \(k=1\), we can simplify (5) to

\[\widehat{P}(N)=\operatorname{argmin}\|DPW_{1}X_{N}\|_{F}^{2}\qquad\text{w.r.t }P^{\top}P=I. \tag{6}\]

We make the following assumptions modelling the empirical setup in [1]:

1. \(D\) is a Fourier Differentiation Stencil
2. \(X\) is a random \(d\times N\) matrix such that each column \(x_{i}\) is drawn independently from a zero-mean isotropic density.
3. \(T\) is the \(m\times m\) matrix such that every \(k\)th column has \(l\)th entry \(t_{k}(l)\) given by \[t_{k}(l)=\begin{cases}\cos\frac{\pi lk}{m}&k=2n\\ \sin\frac{\pi l(k+1)}{m}&k=2n-1.\end{cases}\]

With these assumptions we can show that as the number of points \(N\) tends to infinity, \(\widehat{P}(N)\) can be expressed in terms of the matrix \(T\) and the SVD of \(W_{1}\):

_Theorem 1_.: Let \(f\) be an admissible NN with \(W_{1}\) as its first layer weight matrix with rank-\(r\) SVD \(W=U_{r}\Sigma_{r}V_{r}^{\top}\). Additionally, let \(T_{r}\) be the matrix formed by selecting the first \(r\) columns of \(T\). Then,

\[\widehat{P}(N)\longrightarrow\widehat{\mathcal{P}}:=T_{r}U^{\top}\,\text{a.s. }\,with\,N\to\infty. \tag{7}\]

In particular, the descrambled weight matrix converges to \(T_{r}\Sigma V^{\top}\) almost surely in the measure induced by \(x\).

Thus, under certain geometric assumptions for the data \(X\) (isotropic, finite second moment), weight matrices descrambled for smooth propagation of data choose to propagate that data in a sinusoidal basis. Here this choice of \(X\) corresponds to the noise track taken in many signal processing applications. In particular if the SNR \(\alpha^{-1}\) "equals" zero then the smoothness descramblers \(\widehat{P}_{N}\) tend to \(T_{r}U_{1}^{\top}\). This limit produces a scaling law \(P\to T_{r}^{P}\top U\). Under this scaling law, the minimizers \(\widehat{P}(N)=T_{r}U_{1}^{\top}\) are scaled to \(I\) as \(N\to\infty\). This scaling law will be used to justify the use of Fourier domain visualizations in [1]. We provide a more general version of the theorem in the Supplementary Information; here the choice of \(D\) as a finite difference stencil was taken to align the result with the experimental setup used in [1]

In more realistic applied settings, training data is usually a mixture of signal and noise; we extend Theorem 1 to this case below. We recall that \(k=1\), so it will be dropped from the notation for clarity.

Figure 1. We sampled \(X\) in Equation 5 from a standard Gaussian ensemble and computed the left singular vectors of the first layer pre- and post-descrambling of the network in [30]. Note that the left singular vectors of the descrambled weight matrix are perfectly oscillating: this is because \(X\) as a standard Gaussian ensemble together with a linear wiretapped layer yields a descrambler \(\widehat{P}(N)\approx T_{r}U^{\top}\), which results in \(\widehat{P}W\approx T_{r}\Sigma V^{\top}\) so that the left singular vectors of the descrambled weight matrix are given by \(T_{r}\)
**Linear Network, Noise and Signal.** We consider the case when each training sample is \(x=s(z)+\alpha^{-1}y\). First we observe that given an unbiased noise we have

\[\eta(P)\to\mathbb{E}[\|DPW(s(z))\|_{2}^{2}]+\alpha^{-2}\mathbb{E}[\|DPWy\|_{2}^ {2}]]. \tag{8}\]

as \(N\to\infty\) (see SI appendix). Thus, the objective function \(\eta\)_splits_ into the sum of the _signal term_\(\mathbb{E}[\|DPW(s(z))\|_{2}^{2}]\) and a noise term \(\mathbb{E}[\|DPWy\|_{2}^{2}]]\) weighted by the SNR \(\alpha\). Using (8) we show the following result for a noisy signal model.

_Theorem 2_.: Borrowing notation from Theorem 1, let \(X\) be a \(d\times N\) matrix of training data where each column is sampled from \(x=s(z)+\alpha^{-1}y\). Denote \(\widehat{\mathcal{P}}(\alpha):=\lim_{N\to\infty}\widehat{P}(N)\) and let \(\widehat{\mathcal{U}}(\alpha)\) be the matrix of left singular vectors of the descrambled weight matrix \(\widehat{\mathcal{P}}(\alpha)W\). Then \(\mathcal{U}(\alpha)\) is continuous in \(\alpha\) and as \(\alpha\to 0\)

\[\|\widehat{\mathcal{U}}(\alpha)-T_{r}\|_{F}\to 0. \tag{9}\]

Theorem 2 shows that the left singular vector basis of the descrambled matrix are "close" to a trigonometric basis, and extent of this proximity is controlled by the signal to noise ratio \(\alpha^{-1}\). As \(\alpha\to 0\) this basis converges to the trigonometric basis (which is consistent with the case of Theorem 1).

**Non-linear network.** Finally, we proceed to discuss the non-linear case when \(k\geq 2\). In this situation we choose to deal with the non-linearities using the Taylor expansion at the sample mean \(\overline{X}\):

\[\widehat{P}(k,N)\approx\operatorname*{argmin}_{P^{\top}P}\mathbb{E}[\|DP(f_{ k}(\overline{x})+Jf(\overline{x})^{\top}(X-\overline{x}))\|_{2}^{2}] \tag{10}\]

Now the RHS can be characterized through Theorems 1 and 2 applied to \(Jf(\overline{X})\). We were able to also provide empirical evidence for the quality of the Jacobian approximation to the descrambling problem in Figure 2.

## 3. Applications

**Interpretable neural networks.** We use the results from the preceding discussion to study two cases in which descrambling reveals information already available to us through some other well-understood transformation of the weights. The first example comes from convolutional neural networks (CNN's).

_Corollary 1_.: Adopting the assumptions of Theorem 1 with samples of noise \(X\) and \(k=1\), let \(f\) be a 1-D CNN with stride 1. Then \(\widehat{\mathcal{P}}(k)W=W\), i.e., descrambling acts as identity transformation on the weight matrix.

Corollary 1 presents a simple case where descrambling becomes ineffective. This sheds light onto the limitations of this method when it comes to network interpretation. More pertinently, however, it shows that every choice of \(\eta\), \(G\), and \(X\) from (3) defines a class of \(k\)-_interpretable_ weights \(\mathcal{I}_{k}\) such that the minimizers \(\widehat{P}(\eta,G,k,N)\) converge to \(I\) whenever \(W_{k}\in\mathcal{I}_{k}\) in the large data limit (here we change notation slightly

Figure 2. We compare the descramblers computed for \(k=2\) in DEERnet with \(f_{2}(X)\) (termed full) and with the approximation in (10) (termed jacobian). We find that the overall descrambled matrices are visually similar (albeit not very interpretable). We also see, via the action of \(\widehat{P}\) on a singular vector, that the Jacobian descramblers \(\widehat{P}^{j}\) are an intermediate between full descrambling and no descrambling (bottom right).
from (3) to include the effect of group \(G\) and criterion \(\eta\)). We formalize this intuition by defining a class of _interpretable_ neural networks. In what follows we assume that \(\mathcal{N}\) is the set of fully connected neural networks.

_Definition 2_.: An interpretability class \(\mathcal{I}(\eta,G,k,x\sim\mu)\subset\mathcal{N}\) is the set of neural networks such that \(f\in\mathcal{I}(\eta,G,k,x\sim\mu)\) if, for the \(kth\) layer, a group \(G\), criterion \(\eta\), and data \(X\) the solution to (3) where \(X\) has \(N\) i.i.d columns sampled by \(\mu\) is \(\widehat{\mathcal{P}}(\eta,G,k,x\sim\mu)=I\) as \(N\to\infty\). If \(f\in\mathcal{I}(\eta,G,k,x)\) then \(f\) is said to be \(\mathcal{I}(\eta,G,k,x)\)-interpretable, or simply \(\mathcal{I}\) interpretable if \(\eta,G,k\) and \(x\) are clear from context.

Interpretability classes quantify those neural networks which are interpretable after training and do not require any descrambling to be intelligible to humans/experts. In describing such neural networks,\(\mathcal{I}\)-classes account for the underlying problem (such as signal recovery/image classification) by tracking the distribution on the training data. Furthermore, definition 2 models a simple but natural setting used for training various types of neural networks: the training inputs are an incoming stream of samples of \(x\) forming the matrix of samples \(X\). Under this training data regime, \(\mathcal{I}\)-interpretability record the large-data behaviour of the descrambling matrix \(\widehat{P}\) for a specific choice of wiretapped layer and descrambling criterion. _A priori_ interpretability classes have a rather complicated structure and it is a mathematically interesting problem to provide some better understanding of different cases. In fact, we can reformulate Corollary 1 to give an explicit presentation for \(\mathcal{I}(\eta,O(n),1,x\sim N(0,\alpha^{-1}I))\) as the class of \(1-D\) neural networks which are convolutional in the first layer.

_Proposition_.: Let \(k=1\), \(G=O(n)\), \(\eta\) be the smoothness criterion descrambling functional ((5)) and let \(x\sim\alpha^{-1}N(0,I))\) - the multivariate zero-mean normal distribution with variance \(\alpha^{-2}I\). Let \(\mathcal{C}_{1}^{n}\subset\mathcal{N}\) with a convolutional first layer with \(n\) nodes in the first hidden layer and let \(\mathcal{N}\) be the class of \(1-D\) NNs in \(C(\mathbb{R}^{n})^{m}\). Then, we have

\[\mathcal{I}(\eta,O(n),1,(x,N(0,\alpha^{-1}I))=\mathcal{C}_{1}^{n}. \tag{11}\]

As such, characterizing \(\mathcal{I}(\eta,O(n),1,(x,N(0,\alpha^{-1}I))\) can seem unrealistic because most real-life instances of neural network training do not utilize purely noisy data. Recalling the context of inverse problems, it is of interest to know whether we can characterize any interpretability class when the input data is in the form of noisy measurements of a parameter. It turns out, for instance, that a simple phase identification problem in oscillatory data analysis (ODA) can be the right setting for such a question. A central problem in ODA is to identify the phases \(\phi(t)\) and the trend \(T(t)\) from the measurements of the noisy signal

\[f(t)=\exp 2\pi i\phi(t)+T(t)+y. \tag{12}\]

The signal model in (12) is ubiquitous in many applications of science and engineering, including clinical, seismic, and climate data, and investigative art [31, 32, 33, 34, 35, 36, 37, 38]. Unsurprisingly, this type of inverse problem has attracted neural network approaches [39]. Here we show that the simple case when the trend \(T=0\) and the phase \(\phi=z\), a constant to be estimated under a uniform prior, thee oscillatory data modeled by (12) defines a setting where we can characterize an interpretability class.

_Corollary 2_.: Let \(z\sim\text{Unif}[-uN,vN]\) be the phase parameter with uniform prior for \(u,v\in\mathbb{Z}\) and the oscillating signal \(s(z)=\left(\exp 2\pi ikT/N\right)_{k=0}^{N-1}\) be measured as \(x=s(z)+\alpha^{-1}y\) where \(y\) is a standard normal variable. If \(\eta\) is the smoothness descrambling criterion then

\[\mathcal{I}(\eta,O(n),1,x)=\mathcal{C}_{1}^{n}. \tag{13}\]

Here \(\mathcal{C}_{1}^{n}\) is the class of neural networks with a convolutional first layer and \(n\) nodes in the first hidden layer.

Corollary 2 has multiple implications. For example, it shows that convolutional neural networks reside in the class of interpretable networks for the problem of recovering the phase from equispaced samples of an oscillating signal in time. Second, it shows shows that descrambling the first layer for a network trained to solve this inverse problem might be inconclusive in yielding interpretations, because the descrambler transformation converges to the identity matrix. Thus, using smoothness descrambling for network interpretation reveals to us only the information already available in the raw weights of the first layer.
**DEERNet.** We use our theoretical analysis to furnish quantitative justifications for the visual descrambling analysis of DEERNet [1], the neural network on which descrambling was first tested. This theroetical perspective will illuminate why the SVD emerges as an "interpretable" factorization. DEERNet is a feedforward neural net trained to solve a noisy signal recovery problem in deep electron-electron resonance spectroscopy. In particular, it solves the following Fredholm equation of the first kind:

\[\Gamma(t)=\int_{\Omega}p(r)\gamma(r,t)\,dr+\xi. \tag{14}\]

Here \(\gamma(r,t)\) is the DEER kernel given by

\[\gamma(r,t)\!:=\!\sqrt{\frac{\pi r^{3}}{6Dt}}\left[\cos[Dt]C\!\left[\sqrt{ \frac{6Dt}{\pi}}\right]+\!\sin[Dt]S\!\!\left[\sqrt{\frac{6Dt}{\pi}}\right]\right] \tag{15}\]

\[D\!:=\frac{\mu_{0}}{4\pi}\frac{\gamma_{1}\gamma_{2}h}{r^{3}};\,FrC(x)=\int_{0} ^{x}\cos(t^{2})\,dt;\,FrS(x)=\int_{0}^{x}\sin(t^{2})\,dt. \tag{16}\]

The training inputs and outputs are of the form \(\{d_{i},p_{i}\}_{i=1}^{N}\), where \(p_{i}\) are distributions and \(\Gamma_{i}\) is the distance detected at times \(\{t_{j}\}_{j=1}^{256}\), with Gaussian noise. DEERNet obtains \(p_{i}\) from \(d_{i}\) and hence can be formulated as a network that solves problem of recovering \(z\) from noisy observations of \(x=s(z)+\alpha^{-1}y\). Here \(s\) represents the integral operator corresponding to integration against the DEER kernel and \(z\) is the input probability distribution.

Why Fourier domain visualization worksIn [1] a descrambling analysis of the first layer weights of a shallow 2-layer DEERnet uncovered a notch filter and a bandpass filter in the first weight matrix after visualizing the 2-D DFT \(\mathcal{F}\widehat{P}(1,N)W_{1}\) (Figure 3). Fourier domain visualization in [1] was justified experimentally because of the interlocking wave patterns seen in the descrambled weights \(\widehat{P}(1,N)W_{1}\). We now provide a more quantitative justification for this heuristic based on Theorem 1, and on the correspondence between

Figure 3. Left: Descrambling the first layer of a two-layer DEERNet reveals a notch and bandpass filter. A powerful plausibility argument is mounted in [1] to indicate that this notch resembles a cube root on the time axis/input dimension. Center: Visualize the fourier transform of \(\Sigma_{1}V_{1}^{\top}\) (right) handily reveals repeating streaks along the singular vectors; in fact along the time axis we see these streaks arranged in the shape of a cubic. Right: We visualize the Fourier transform of the right singular vectors of the integral kernel \(K\) scaled by the singular values \(\Sigma_{K}\) and observe cubic streaks similar to the central panel. This provides a more quantitative justification for the first layer’s function as inverting the integral operator.
the SNR \(\alpha^{-1}\) and the interpretation matrix \(\widehat{P}(1,N)\). When the descrambling data \(X\) is purely noisy, \(\widehat{P}(N)\to T_{r}U_{1}^{\top}\iff T_{r}^{\top}\widehat{P}(N)U_{1}\to I\) due to Theorem 1. Thus, the group homomorphism

\[P\to\varphi(P)=T_{r}^{\top}PU_{1} \tag{17}\]

provides an appropriate rescaling for the limiting descrambling matrix \(\widehat{\mathcal{P}}(1)\). For example, when \(\alpha^{-1}\to 0\), \(\varphi(\widehat{P}(N))\to I\), so that the group homomorphism (17) simplifies the correspondence between the SNR and the rescaled large-limit descrambler \(\varphi(\widehat{P}(N))\). In this case specifically we have

\[\text{SNR}=0\iff\varphi(\widehat{\mathcal{P}})=I. \tag{18}\]

As a consequence, post-descrambling visualization of the 2D DFT of \(W_{1}\) approximates \(\varphi(\widehat{\mathcal{P}})\Sigma_{1}(\mathcal{F}V_{1})^{\top}\). In fact, in the case when \(\varphi(\mathcal{P})=I\), \(\mathcal{F}\widehat{P}_{1}W_{1}\approx\Sigma_{1}(\mathcal{F}V_{1})^{\top}\) - the Fourier transform of the singular vectors rescaled by the singular values. Thus, visualizing the matrix by setting \(\varphi(P)=I\) is quite informative: the result is \(\Sigma_{1}V_{1}^{\top}\) and it reveals a repeating pattern of cubic streaks, uncovering both the notch filter and the distance cube root as found in the integral kernel from (14). This shows that descrambling uncovers - to a large extent - the information within the SVD and that multiplication by \(\widehat{P}_{1}\) moves \(W_{1}\) closer to the integral kernel. Singular vectors themselves can be interpretable without necessitating any additional processing such as descrambling, intertwiner groups, or hypergraph arrangements [13, 12] in the context of noisy signal estimation.

Connection to semantic development.The emergence of the DEER kernel in the right singular vector matrix \(V_{1}\) is not entirely surprising due to a heuristic motivated by (1). Indeed, if the network did not have a non-linearity, then \(W_{1}(t)=Q^{-1}A(\Lambda,t)V^{\top}\) where \(V\) is the matrix of the right singular vectors of \(\Sigma_{yx}=\mathbb{E}[yx^{\top}]\), the output-input covariance matrix. But in the signal estimation case where \(y=s(x)+\alpha^{-1}\xi\), this covariance matrix has a very simple form, especially when \(s\) is a linear map. In this case, we have \(\Sigma_{yx}=\mathbb{E}_{yx}[Kxx^{\top}K^{\top}]=K\mathbb{E}[xx^{\top}]K^{\top}\). With the assumption that \(\mathbb{E}[xx^{\top}]=I\), we have \(\Sigma_{yx}=KK^{\top}\) so the matrix \(V\) corresponds to the spectral decomposition of the kernel \(KK^{\top}\). We confirm this heuristic in the right panel of Figure 3

**Noise isn't always meaningless.** Our analysis of DEERnet shows that descrambling goes hand-in-hand with semantic development in the SVD, particularly when a network is trained to learn a signal from noisy measurements. However, this interplay with semantic development is only approximate. This is because the linear network training dynamics (1) merely approximates the real training dynamics of nonlinear networks. This necessitates the discovery of nonlinear features via descrambling. For example, the notch filter in the first layer of DEERnet cannot be discovered by noise-only descrambling, and we require the presence of the forward map \(s\) in the propagating data to discover a notch that is latent in \(W_{1}\).

Here we describe a case where noise descrambling, i.e the neural network interpretation via its singular vectors, is useful in and of itself. We demonstrate this phenomenon for a NN in which the forward map \(s\) is non-linear, so that the nature of matrices \(U\) and \(V\) in (1) is entirely unspecified. In particular, following [30] we train a four layer fully-connected NN to learn the exponential parameters \((T_{2,1},T_{2,2})\) generating a biexponential model:

\[y(t)=0.6\exp(-t/T_{2,1})+0.4\exp(-t/T_{2,2},) \tag{19}\]

corrupted by noise. The recovery of exponential parameters \(T_{2,1},T_{2,2}\) from a noisy decay curve is a central problem in magnetic resonance relaxometry [40], and is well-known to be ill-posed with parameter estimates strongly dependent on the noise. This problem has been investigated with a number of neural network-based approaches [41, 42]; the novelty of the network in [30] is that it is trained to solve this problem on both noisy and smooth forms of the same data, as a form of input data transformation to incorporate high-fidelity and high-stability and generalizability characteristics into the solutions. To achieve this, the noisy input data is first processed with regularized non-linear least squares parameter estimation, with these estimates used to generate smooth decay curves. These smooth curves are concatenated with the noisy samples to form a single input sample for presentation to the NN. The NN with just the native input, concatenated with itself, is termed (ND, ND), where ND indicates noisy decay. The NN with the concatenated native and smoothed versions of the decay curve is termed (ND, Reg), with Reg indicating the smooth decay generated by the regularized nonlinear least squares analysis. This strategy of training on both noisy and smooth data is termed _input layer regularization_, and improves parameter estimation by 5-10 percent as compared to
the more conventional NN estimation of parameters from noisy decay curves [30]. We find that the right singular vectors corresponding to the largest singular values of the first layer are biexponential curves, so that the network learns an input signal library in the class of its training data. Most notably, the (ND, Reg) network learns two very different shapes of biexponential for the noisy and smooth case; we attribute its higher test accuracy as compared to (ND, ND) to this result, indicating that (ND, Reg) learns a larger set of functions within the signal model class of biexponential functions. Thus, in the regression setting the descrambling guides us to finding the location of NN learning, namely within the SVD. In addition to explaining generalization, learning of the model in the SVD may have adversarial implications since we are able to learn samples of the input data model directly from a trained network. While the SVD has been used as a compression technique in feedforward neural networks [23] and deep layer interpretation in classification [11, 13], our results appear to be the first demonstration that data model learning occurs in singular vectors for nonlinear networks.

## 4. Conclusion

A key novelty of descrambling is that it leverages problem-dependent interpretation, akin to the manner in which intepretability methods in image classification concern the most influential pixels, image substructures, and, decision boundaries. We take this concept further: our characterization of solutions to (5) and the subsequent theoretical explanations to the experimental results in [1] show that the interpretation matrices \(\widehat{P}\) can admit simple large-data limits that depend on the underlying problem. In fact, we observe that the mathematical formalism in inverse problems allows us to obtain strong characterizations of these limits with practical implications. For example, descrambling implicitly defines problem-dependent classes of neural networks - and these classes can be explicitly characterized, for instance, in a simple problem from oscillatory data analysis. Furthermore, while [1] interprets DEERNet through the SVD of the descrambled weights, we show that even the SVDs of the _scrambled_, i.e., raw, weights can themselves be informative. We give an explanation for this phenomenon - if the data is highly noisy, the Fourier domain visualization of the

Figure 4. We visualize the singular vectors of the first layer of ILR networks trained on concatenations of noisy time-series data with the functional form \(c_{1}\exp(-t/T_{2,1})+c_{2}\exp(-t/T_{2,2}\) for \(c_{i}\geq 0\), discovering that these singular vectors themselves can be fit by curves of the same algebraic form as the noise-less version of the training data, allowing for the \(c_{i}\)’s to be negative. Note that this phenomenon persists for a variety of SNR’s and for both classes of networks–(ND, ND) and (ND, Reg). Here (ND, Reg) refers to networks trained on data points given by a concatentation of noisy and Tikhonov regularized copies of the same data (right panel).
descrambled weights approximates the Fourier domain visualization of the orthogonal row basis vectors provided by the right singular vectors through the rescaling formula motivated by Theorem 1. We remark, however, that this does not imply that descrambling is equivalent to the SVD -indeed, in the cases when the data \(X\) follows the more realistic distribution of a signal under noisy measurements and the network is wiretapped at a higher layer (\(k\geq 2\)) we can no longer exactly characterize the solutions to (5) and the minimizers do indeed reflect the latent transformations in the underlying weights \(W_{k}\). We find, at least empirically that the action of descrambler matrices can be approximated through descrambling of the network linearized its Jacobian at the empirical average. This does not mean the choice of the Jacobian as a linear approximation is canonical-there are many different linear models for NN's [43, 44, 45] and it is yet unclear which approximation serves best as a surrogate for modeling the action of descrambling transformations.

A surprising and highly significant aspect of our results is that when the SVD is uncovered indirectly by descrambling, it _still_ remains informative of the transformations in the data. This observation closely resembles examples [13, 12, 46] where SVDs of weights in CNNs are used to interpret features of an image classification network. Unlike these approaches - which require highly non-trivial post-processing of the singular vectors - we demonstrate that a simple visual analysis of singular vectors themselves can inform the post hoc analysis of the network's performance. Moreover we show that in noisy estimation problems it is the SVD where the network memorizes the underlying function class of the problem. This demonstrates not only that the training dynamics given by (1) for linear models can approximate non-linear settings but more also that the SVD by itself can illuminate the network's generalizability and interpretability.

## Materials and Methods

The proofs all our results and supporting figures have been provided in the SI document. Code for the figures and additional experiments can be found at [https://github.com/ShashankSule/descrambling-NN](https://github.com/ShashankSule/descrambling-NN)

## Acknowledgements

This work was supported in part by the Intramural Research Program of the National Institute on Aging of the National Institutes of Health (NIH).
## Appendix

Theoretical Results

Here we provide proofs for the results in our submitted article.

**Lemma 1**.: Let \(W_{n}\) be a sequence of matrices such that \(W_{n}\to W\) in the Frobenius norm where \(W\in A\), where \(A\) is the set of matrices with distinct non-zero singular values. Then the (right and left) singular vectors \(\{u_{i}^{n}\}\) and \(\{v_{i}^{n}\}\) of \(W_{n}\) converge to the right (resp. left) singular vectors of \(W\).

Proof.: Since \(u_{i}^{n}\) and \(v_{i}^{n}\) are the eigenvectors of \(WW^{\top}\) and \(W^{\top}W\) respectively, we drop without losing generality to the case where \(W\) is symmetric, so it has real eigenvalues. But from [29, Theorem 8, pp. 130], the eigenvectors of a matrix with simple eigenvalues are differentiable in its entries. Consequently, if \(u_{i}^{n}\) is the \(i\)th eigenvector of \(W_{n}\) then \(u_{i}^{n}\to u_{i}\) where \(u_{i}\) is the \(i\)th eigenvector of \(W\). 

**Theorem 1**.: For the reader's convenience we repeat the statement of the theorem here. Let

1. \(D\) be a Fourier/finite difference stencil [47]
2. \(X_{N}\) a random \(d\times N\) matrix such that each column \(x_{i}\) is drawn independently from an isotropic density.
3. \(W\) any \(m\times d\) weight matrix with ordered rank \(r\) SVD \(W=U\Sigma V^{\top}\)
4. \(T\) the \(m\times m\) matrix such that every \(k\)th column has \(l\)th entry \(t_{k}(l)\) given by (20) \[t_{k}(l)=\begin{cases}\cos\frac{\pi lk}{m}&k\text{ even}\\ \sin\frac{\pi l(k+1)}{m}&k\text{ odd}\end{cases}\]
5. \(T_{r}\) the submatrix of \(T\) given by picking the last \(r\) columns.

Then we have that

\[\widehat{P}_{N}=\text{argmin}\left\|DPWX_{N}\right\|_{F}^{2}\text{w.r.t}\,P^{ \top}P=I\longrightarrow T_{r}U^{\top}\]

almost surely as \(N\to\infty\). In particular the descrambled weight matrix converges to \(T_{r}\Sigma V^{\top}\) almost surely.

Proof.: First to clean up some notation we set \(S_{N}:=WX_{N}\). Now note that

\[\underset{P^{\top}P=I}{\text{argmin}}\|DPWX\|_{F}^{2}=\underset{P^{\top}P=I} {\text{argmin}}\frac{1}{N}\|DPWX\|_{F}^{2}\]

So we can switch to analyzing the minimum of \(\frac{1}{N}\|DPWX\|_{F}^{2}\). But now,

\[\frac{1}{N}\|DPWX_{N}\|_{F}^{2} =\frac{1}{N}\text{Tr}(DPWX(DPWX)^{\top}) \tag{22}\] \[=\frac{1}{N}\text{Tr}(DPWXX^{\top}W^{\top}P^{\top}D^{\top})\] (23) \[=\text{Tr}\left(DPW\left(\frac{1}{N}X_{N}X_{N}^{\top}\right)W^{ \top}P^{\top}D^{\top}\right)\] (24) \[=\text{Tr}(DPS_{N}S_{N}^{\top}P^{\top}D^{\top})\] (25) \[=\text{Tr}(S_{N}^{\top}P^{\top}D^{\top}DPS_{N}) \tag{21}\]

To find the minimizer to (25) w.r.t \(P^{\top}P=I\) we make an interesting change of variable: let \(S_{N}=U_{N}\Sigma_{N}V_{N}\) be the SVD of \(S_{N}:=W\left(\frac{1}{N}X_{N}X_{N}^{\top}\right)W^{\top}\). Then we let \(Y_{N}=PS_{N}V_{N}\). Note that this means \(Y_{N}^{\top}Y=\Sigma_{N}^{2}\) given the constraint \(P^{\top}P\). With this change of variables (25) can be written as

\[\frac{1}{N}\|DPWX_{N}\|_{F}^{2}=\text{Tr}(Y^{\top}D^{\top}DY) \tag{26}\]

Consequently, the smoothness descrambling problem

\[\min\|DPWX_{N}\|_{F}^{2}\qquad\text{w.r.t}P^{\top}P=I \tag{27}\]

can be relaxed to

\[\min\text{Tr}(Y^{\top}D^{\top}DY)\qquad\text{w.r.t}Y^{\top}Y=\Sigma_{N}^{2}. \tag{28}\]
Note that the optimization problem 27 is a relaxation of 28 because of the transformation \(P\to PWV\). But now the solution to the relaxed problem 28 is well-known: it is a generalized eigenvalue problem whose solutions correspond to the first \(R_{N}=\min(r_{N},d-1)\) eigenvectors of \(D^{\top}D\) where \(r_{N}\) is the number of non-zero diagonal entries in \(\Sigma_{N}^{2}\) (this is because notably any finite difference stencil has rank \(d-1\)). Since \(D^{\top}D\) is diagonalized by \(T\), we pick the largest \(R_{N}\) eigenvectors (indexed by the matrix \(T_{R_{N}}\)) and scale them by \(\Sigma_{N}\) to get \(Y\). Thus, \(Y=T_{R_{N}}\Sigma_{R_{N}}\). Now using the change of variables \(Y=PS_{N}V_{R_{N}}\) we get \(PS_{N}=T_{R_{N}}\Sigma_{R_{N}}V_{R_{N}}^{\top}\) so \(S_{N}=P^{\top}T_{R_{N}}\Sigma_{R_{N}}V_{R_{N}}^{\top}\). This is a singular value decomposition for \(S_{N}\), so as long as the non-zero singular values of \(S_{N}\) are distinct we get from the uniqueness of the SVD that \(\widehat{P}_{N}^{\top}T_{R_{N}}=U_{N}\iff\widehat{P}_{N}=T_{R_{N}}(U_{N})_{R_ {N}}\). Now this is where the assumption that \(W\in A\) comes in: due to the strong law of large numbers we have that \(S_{N}\to WW^{\top}\) almost surely. Since the singular values of \(W\) are distinct and we get that the singular values of \(S_{N}\) are eventually distinct and ordered according to the ordered of \(W\) so the expression for \(\widehat{P}_{N}\) is valid; moreover from Lemma 1 we get that the matrices \((U_{N})_{R_{N}}\) will converge to the left singular vectors of \(U\) of \(W\). This proves our result. 

**Equation [10]**.: Let \(x=s(z)+\alpha^{-1}y\) where \(y\) is a zero-mean isotropic noise vector with finite second moment and \(z\) has some prior distribution independent of the noise \(y\). We stated the following convergence result:

\[\eta(P)\to\mathbb{E}[\|DPW(s(z))\|_{2}^{2}]+\alpha^{-2}\mathbb{E}[\|DPWy\|_{2 }^{2}]] \tag{29}\]

The proof of (8) is as follows: Note first that because of the law of large numbers,

\[\frac{1}{N}\|DPWX\|_{F}^{2}=\frac{1}{N}\sum_{i=1}^{N}\|DPWx_{i}\|_{2}^{2}\to \mathbb{E}_{z,y}[\|DPWx\|]\]

We directly switch to analyzing the LLN limit of \(\eta\) and write

\[\eta(P)=\mathbb{E}[\|DPWx\|_{2}^{2}]=\mathbb{E}[\mathbb{E}[\|DPWx\|_{2}^{2} \mid z]]=\mathbb{E}[\mathbb{E}[\|DPW(s(z)+\alpha^{-1}y)\|_{2}^{2}\mid z]]\]

Now setting \(A=DPW\) and using the zero-mean property of the additive noise we get

\[\eta(P) =\mathbb{E}\Big{[}\mathbb{E}[\|A(s(z))\|_{2}^{2}]\mid z]+ \mathbb{E}[\|A(\alpha^{-1}y)\|_{2}^{2}]\mid z]\] \[+2\alpha^{-1}\mathbb{E}[\langle\mathcal{A}^{\top}A(s(z)),y\rangle \mid z]\Big{]}\] \[=\mathbb{E}\Big{[}\mathbb{E}[\|A(s(z))\|_{2}^{2}]\mid z]+ \mathbb{E}[\|A(\alpha^{-1}y)\|_{2}^{2}]\mid z]\Big{]}\] \[=\mathbb{E}[\|A(s(z))\|_{2}^{2}]+\alpha^{-2}\mathbb{E}[\|Ay\|_{2 }^{2}]] \tag{30}\] \[=\mathbb{E}[\|DPW(s(z))\|_{2}^{2}]+\alpha^{-2}\mathbb{E}[\|DPWy\| _{2}^{2}]]\]

Now we state Theorem 2:

**Theorem 2**.: Borrowing notation from Theorem 1, let \(X\) be a \(d\times N\) matrix of training data where each column is sampled from \(x=s(z)+\alpha^{-1}y\). Denote \(\widehat{\mathcal{P}}(\alpha):=\lim_{N\to\infty}\widehat{P}(N)\) and let \(\widehat{\mathcal{U}}(\alpha)\) be the matrix of left singular vectors of the descrambled weight matrix \(\widehat{\mathcal{P}}(\alpha)W\). Then \(\mathcal{U}(\alpha)\) is continuous in \(\alpha\) and as \(\alpha\to 0\)

\[\|\widehat{\mathcal{U}}(\alpha)-T_{r}\|_{F}\to 0. \tag{31}\]

Proof.: We can use the same analysis as the proof of Theorem 4 to conclude first that \(\widehat{P}_{N}=T_{R_{M}}(U_{N})_{R_{N}}\); here \(U_{N}\) is the matrix of left singular vectors of \(W\frac{1}{N}\left(X_{N}X_{N}^{\top}\right)W^{\top}\). Then \(\left(X_{N}X_{N}^{\top}\right)\to\mathbb{E}_{X}[xx^{\top}]\), the autocorrelation matrix of \(X\). But as \(\alpha\to 0\), this autocorrelation matrix converges to \(I\). Once again using the continuity of eigenvectors in the entries of the matrix, we get that \(U(\alpha)\to U\), where \(U(\alpha)\) is the matrix of singular vectors of \(W\mathbb{E}[xx^{\top}]W^{\top}\) and \(U\) is the matrix of singular vectors of \(W\). 

Finally, we state and prove Corollary 1 and 2 pertaining to convolutional neural networks and oscillatory data analysis:
**Corollary 1**.: Adopting the assumptions of Theorem 1 with samples of noise \(X\) and \(k=1\), let \(f\) be a 1-D CNN with stride 1. Then \(\widehat{\mathcal{P}}(k)W=W\), i.e., descrambling acts as identity transformation on the weight matrix.

Proof.: If \(f\) is a 1-D CNN where the first filter \(W\) has stride 1 then \(W\) is symmetric and circulant so \(W=T\Sigma T^{\top}\) where \(T\) is the matrix of samples of a trigonometric basis from Theorem 1. Then \(P=T_{r}^{\top}T_{r}=I_{r}\) 

**Corollary 2**.: Adopting the assumptions of Theorem 1, let \(z\sim\text{Unif}[-uN,vN]\) for \(u,v\in\mathbb{Z}\) and \(s(z)=(\exp 2\pi ikT/N)_{k=0}^{N-1}\). Then \(\widehat{P}(N)\to\widehat{\mathcal{P}}=T_{r}U^{\top}\) where \(T_{r}\) is the trigonometric basis from Theorem 1 and \(U\) is the left singular vector matrix of the weights \(W\).

Proof.: We show that the signal term and noise term reduce to the same minimization problem. Let \(A:=DPW\). Then we have

\[\mathbb{E}_{z}[DPWs(z)]=\mathbb{E}_{z}[As(z)]=\frac{1}{N(u+v)}\int_{-uN}^{vN} \|As(z)\|_{2}^{2}\,dz\]

But now, \((As(z))_{j}\overline{(As(z))_{j}}=\sum_{k,l}a_{jk}a_{jl}\exp 2\pi i(k-l)z/N\), and

\[(1/N(u+v))\int_{-uN}^{vN}\exp 2\pi i(k-l)z/N\,dz=\delta_{l,k}\]

Here \(\delta_{k,l}\) is the Kronecker delta. Now,

\[\frac{1}{N(u+v)}\int_{-uN}^{vN}\|As(z)\|_{2}^{2}\,dz =\frac{1}{N(u+v)}\int_{-uN}^{vN}\sum_{j}(As(z))_{j}\overline{(As(z ))_{j}}\,dz\] \[=\sum_{j}\frac{1}{N(u+v)}\int_{-uN}^{vN}(As(z))_{j}\overline{(As(z ))_{j}}\,dz\] \[=\sum_{j}\sum_{k}|a_{j,k}|^{2}\] \[=\|A\|_{F}^{2}=\|DPW\|_{2}^{2}\]

Putting the above calculation together with the calculation for isotropic noise from Theorem 1 and using Equation (8) we get

\[\eta(P) =\mathbb{E}_{z}[DPWs(z)]+\alpha^{-2}\mathbb{E}[\|DPWy\|_{2}^{2}]]\] \[=\|DPW\|_{2}^{2}+\alpha^{-2}\|DPW\|_{2}^{2}\]

Minimizing the above function for \(P\) orthogonal leads to \(P=T_{r}^{\top}U\) (just like Theorem 1). 

**MDS Criterion.** The _maximum diagonal sum_ (MDS) criterion is suggested in [1] for NN interpretation of frequency domain data:

\[\widehat{P}_{MDS}=\underset{P^{\top}P}{\operatorname{argmax}}\,\text{Tr}(PW)\]

We show that for this criterion \(\widehat{P}\) can be given explicitly in terms of the SVD of weights \(W\):

_Proposition_.: Let \(W\in M_{n}(\mathbb{R})\). Then \(\underset{P^{\top}P=I}{\operatorname{argmax}}Tr(PW)=VU^{\top}\) where \(W=U\Sigma V^{\top}\) is the SVD of \(W\).

Proof.: Let \(W=\sum_{i=1}^{r}\sigma_{i}u_{i}v_{i}^{\top}\) be the SVD of \(W\) and let \(P\) be orthogonal. Then

\[Tr(PW)=\sum_{i=1}^{r}\sigma_{i}Tr(Pu_{i}v_{i}^{\top})\]

But note that \(Tr(xy^{\top})=\langle x,y\rangle\) so \(Tr(Pu_{i}v_{i}^{\top})=\langle Pu,v\rangle\leq||Pu||^{2}||v||^{2}=1\) using Cauchy-Schwarz. So
\[Tr(PW)=\sum_{i=1}^{r}\sigma_{i}Tr(Pu_{i}v_{i}^{\top})\leq\sum_{i=1}^{r}\sigma_{i}= Tr(\sqrt{W^{\top}W})\]

Thus \(\widehat{P}=VU^{\top}\) yields the conclusion. 

**DEERNet.** We descrambled DEERNet according to the specifications in [1]. In partciular we used the shallow architecture Input \(\rightarrow\) Fully connected \(256\times 80\rightarrow\) tanh activation \(\rightarrow\) Fully connected \(80\times 256\rightarrow\) sigmoid activation \(\rightarrow\) renormalization \(\rightarrow\) Output. Additionally, we made use of the MATLAB modules in the Spinach package to train the network and descramble it on a Quadro GV100 GPU to reproduce the experiments in [1] as closely as possible. We were able to reproduce these experiments only up to an amount of noise-the recovery of the cubic notch filter is much clearer in [1] and may have contributed to the interpretation of the first layer as notch filter/baseline elimination. Nonetheless, in our main article we have also given a quantitative explanation for why the first layer represents the DEER kernel based on a rescaling analysis motivated by Theorem 1. We also confirmed the existence of cubic conversion by changing the cubic factor in the DEER kernel to a quartic factor and finding a narrowing in the notch presumably to account for the fact that a quartic curve is flatter around the origin than a cubic. This experiment has been provided in our data repository [https://github.com/ShashankSule/descrambling-NN](https://github.com/ShashankSule/descrambling-NN).

**ILR network.** We trained neural networks with a \(128\to 32\to 256\to 256\to 2\) input layer regularization structure with ReLU non-linearities to solve the problem of recovering \((T_{2,1},T_{2,2})\) from 64 noisy time-equispaced measurements of the signal

\[0.6\exp-t/T_{2,1}+0.4\exp\left(-t/T_{2,2}\right) \tag{32}\]

The raw noisy data track (termed ND) was processed through a non-linear least squares problem and the resulting NLLS-recovered parameters \((T_{2,1}^{NLLS},T_{2,2}^{NLLS})\) were used to generate a new track (termed Reg) using equispaced time samples. The concatenated data, now of 128 measurements, was fed through the network with RMSE loss to recover the original \((T_{2,1},T_{2,2})\). Following the recommendations in [30] we used a Tikhonov regularization procedure in the NLLS pre-processing step with \(\lambda=1.6\times 10^{-4}\). We also trained the same architecture by concatenating the same sample of noisy data into a size 128 vector; this architecture is termed (ND,ND) and it represents the traditional neural-network approach of recovering parameters from noisy samples of a signal. Post-training we compute the singular vectors of the weight matrices \(W_{1}\), finding that these singular vectors themselves are noisy samples of biexponential curves. The caveat here, however, is that we required biexponential functions of the form \(c_{1}\exp\left(-t/T_{2,1}\right)+c_{2}\exp\left(-t/T_{2,2}\right)\) where \(c_{1},c_{2},T_{2,1},T_{2,2}\) could be any real number. As a consequence, these singular vector curves cannot necessarily fit a _decaying_ biexponential model. We infer from this that the first layer seems to learn the general signal model rather than training examples themselves.
Figure 5: Descrambling the first layer of DEERNet reveals interlocking wave patterns, hinting at the transformation underlying this weight matrix
Figure 6. Visualizing the three right singular vectors corresponding to the largest singular values for the first weight matrix in (ND, Reg) networks for SNR = 100,30,1. We find that these vectors can be fit with a (not necesarily decaying biexponential model). In fact the singular vectors themselves are split along the 64th entry (akin to the training data) with a different biexponential curve being learned for each half. The curves learned for the noisy half vs the smooth half are markedly different in shape, enabling us to better understand why (ND,Reg) generalizes better than (ND,ND)
Figure 7. Visualizing the three right singular vectors corresponding to the largest singular values for the first weight matrix in (ND, ND) networks for SNR = 100,30,1. We witness a similar pattern to the (ND, Reg) networks where the singular vectors can be fit with a biexponential model, concluding that singular vector model learning is truly a consequence of the data and not the concatenation procedure. Clearly, since both halves of the data are the same, the singular vectors on both halves the data are also nearly identical. Thus the signal model library learned by the weights of the NN is less diverse, and may be reponsible for the lower test accuracy of the NN
Figure 8. We also descrambled the first layer of an SNR 10, (ND, Reg) network with its input data. However, unlike what was done for DEERNet, we were not able to discern the underlying transformation of this layer from its Fourier signature, or from the descrambled singular vectors. In fact, after descrambling with data only from the noise track we found the descrambled matrices much more interpretable; Theorem 4 explained why–this was because we were actually visualizing the right singular vectors, which turn out to be samples of a biexponential model.