On the Equivalence between Implicit and Explicit Neural Networks: A
  High-dimensional Viewpoint
Implicit neural networks have demonstrated remarkable success in various
tasks. However, there is a lack of theoretical analysis of the connections and
differences between implicit and explicit networks. In this paper, we study
high-dimensional implicit neural networks and provide the high dimensional
equivalents for the corresponding conjugate kernels and neural tangent kernels.
Built upon this, we establish the equivalence between implicit and explicit
networks in high dimensions.
Zenan Ling, Zhenyu Liao, Robert C. Qiu
# On the Equivalence between Implicit and Explicit Neural Networks:

###### Abstract

Implicit neural networks have demonstrated remarkable success in various tasks. However, there is a lack of theoretical analysis of the connections and differences between implicit and explicit networks. In this paper, we study high-dimensional implicit neural networks and provide the high dimensional equivalents for the corresponding conjugate kernels and neural tangent kernels. Built upon this, we establish the equivalence between implicit and explicit networks in high dimensions.

## 1 Introduction

Implicit neural networks (NNs) [2] have recently emerged as a new paradigm in neural network design. An implicit NN is equivalent to an infinite-depth weight-shared explicit NN with input-injection. Unlike explicit NNs, implicit NNs generate features by directly solving for the fixed point, rather than through layer-by-layer forward propagation. Moreover, implicit NNs have the remarkable advantage that gradients can be computed analytically only through the fixed point with _implicit differentiation_. Therefore, training implicit NNs only requires constant memory.

Despite the empirical success achieved by implicit NNs [3, 11], our theoretical understanding of these models is still limited. In particular, there is a lack of theoretical analysis of the training dynamics and generalization performance of implicit NNs, and possibly more importantly, whether these properties can be connected to those of explicit NNs. [2] demonstrates that any deep NN can be reformulated as a special implicit NN. However, it remains unknown whether general implicit NNs have advantages over explicit NNs. [6] extends previous neural tangent kernel (NTK) studies to implicit NNs and give the exact expression of the NTK of the ReLU implicit NNs. However, the differences between implicit and explicit NTKs are not analyzed. Moreover, previous works [9, 10] have proved the global convergence of gradient descent for training implicit NNs. However, it is still unclear what distinguishes the training dynamic of implicit NNs and that of explicit NNs.

In this paper, we investigate implicit NNs from a high-dimensional view. Specifically, we perform a fine-grained asymptotic analysis on the eigenspectra of conjugate kernel (CKs) and NTKs of implicit NNs, which play a fundamental role in the convergence and generalization high dimensional NNs [8]. By considering input data uniformly drawn from the unit sphere, we derive, with recent advances in random matrix theory, high-dimensional (spectral) equivalents for the CKs and NTKs of implicit NNs, and establish the equivalence between implicit and explicit NNs by matching the coefficients of the corresponding asymptotic spectral equivalents. Surprisingly, our results reveal that a _single-layer_ explicit NN with carefully designed activations has the same CK or NTK eigenspectra as a ReLU implicit NN, whose depth is essentially _infinite_.
## 2 Preliminaries

### Implicit and Explicit NNs

Implicit NNs.In this paper, we study a typical implicit neural network, the deep equilibrium model (DEQ) [2]. Let \(\mathbf{X}=[\mathbf{x}_{1},\cdots,\mathbf{x}_{n}]\in\mathbb{R}^{d\times n}\) denote the input data. We define a vanilla DEQ with the transform at the \(l\)-th layer as

\[\mathbf{h}_{i}^{(l)}=\sqrt{\frac{\sigma_{a}^{2}}{m}}\mathbf{A}\mathbf{z}_{i}^{(l-1)}+\sqrt{ \frac{\sigma_{b}^{2}}{m}}\mathbf{B}\mathbf{x}_{i},\quad\mathbf{z}_{i}^{(l)}=\phi(\mathbf{h}_{i }^{(l)}) \tag{1}\]

where \(\mathbf{A}\in\mathbb{R}^{m\times m}\) and \(\mathbf{B}\in\mathbb{R}^{m\times d}\) are weight matrices, \(\sigma_{a},\sigma_{b}\in\mathbb{R}\) are constants, \(\phi\) is an element-wise activation, \(\mathbf{h}_{i}^{(l)}\) is the pre-activation and \(\mathbf{z}_{i}^{(l)}\in\mathbb{R}^{m}\) is the output feature of the \(l\)-th hidden layer corresponding to the input data \(\mathbf{x}_{i}\). The output of the last hidden layer is defined by \(\mathbf{z}_{i}^{*}\triangleq\lim_{l\to\infty}\mathbf{z}_{i}^{(l)}\) and we denote the corresponding pre-activation by \(\mathbf{h}_{i}^{*}\). Note that \(\mathbf{z}_{i}^{*}\) can be calculated by directly solving for the equilibrium point of the following equation

\[\mathbf{z}_{i}^{*}=\phi\left(\sqrt{\frac{\sigma_{a}^{2}}{m}}\mathbf{A}\mathbf{z}_{i}^{*}+ \sqrt{\frac{\sigma_{b}^{2}}{m}}\mathbf{B}\mathbf{x}_{i}\right). \tag{2}\]

We are interested in the conjugate kernel and neural tangent kernel (Implicit-CK and Implicit-NTK, for short) of implicit neural networks defined in Eq. (2). Following [6], we denote the corresponding Implicit-CK by \(\mathbf{G}^{*}=\lim_{l\to\infty}\mathbf{G}^{(l)}\) where the \((i,j)\)-th entry of \(\mathbf{G}^{(l)}\) is defined recursively as

\[\mathbf{G}_{ij}^{(0)} =\mathbf{x}_{i}^{\top}\mathbf{x}_{j},\quad\mathbf{\Lambda}_{ij}^{(l)}=\left[ \begin{array}{cc}\mathbf{G}_{ii}^{(l-1)}&\mathbf{G}_{ij}^{(l-1)}\\ \mathbf{G}_{ji}^{(l-1)}&\mathbf{G}_{jj}^{(l-1)}\end{array}\right], \tag{3}\] \[\mathbf{G}_{ij}^{(l)} =\sigma_{a}^{2}\mathbb{E}_{(\text{u},\text{v})\sim\mathcal{N}(0, \mathbf{\Lambda}_{ij}^{(l)}}[\phi(\text{u})\phi(\text{v})]+\sigma_{b}^{2}\mathbf{x}_{i }^{\top}\mathbf{x}_{j},\] \[\dot{\mathbf{G}}_{ij}^{(l)} =\sigma_{a}^{2}\mathbb{E}_{(\text{u},\text{v})\sim\mathcal{N}(0, \mathbf{\Lambda}_{ij}^{(l)}}[\phi^{\prime}(\text{u})\phi^{\prime}(\text{v})].\]

And the Implicit-NTK is defined as \(\mathbf{K}^{*}=\lim_{l\to\infty}\mathbf{K}^{(l)}\) whose the \((i,j)\)-th entry is defined as

\[\mathbf{K}_{ij}^{(l)}=\sum_{h=1}^{l+1}\left(\mathbf{G}_{ij}^{(h-1)}\prod_{h^{\prime}=h }^{l+1}\dot{\mathbf{G}}_{ij}^{(h^{\prime})}\right). \tag{4}\]

Explicit Neural Networks.We consider a single-layer fully-connected NN model defined as \(\mathbf{Y}=\sqrt{\frac{1}{p}}\sigma(\mathbf{W}\mathbf{X})\) where \(\mathbf{W}\in\mathbb{R}^{p\times d}\) is the weight matrix and \(\sigma\) is an element-wise activation function. Let \(\mathbf{w}\sim\mathcal{N}(0,\mathbf{I}_{d})\), the corresponding Explicit-CK matrix \(\mathbf{\Sigma}\) and Explicit-NTK matrix \(\mathbf{\Theta}\) are defined as follows:

\[\mathbf{\Sigma}=\mathbb{E}_{\mathbf{w}}[\sigma(\mathbf{w}^{\top}\mathbf{X})^{\top}\sigma(\mathbf{ w}^{\top}\mathbf{X})],\quad\mathbf{\Theta}=\mathbf{\Sigma}+\left(\mathbf{X}^{\top}\mathbf{X} \right)\odot\mathbb{E}_{\mathbf{w}}[\sigma^{\prime}(\mathbf{w}^{\top}\mathbf{X})^{\top} \sigma^{\prime}(\mathbf{w}^{\top}\mathbf{X})]. \tag{5}\]

### CKs and NTKs of ReLU Implicit NNs

We make the following assumptions on the random initialization, the input data, and activations.
**Assumption 1**: _(i) As \(n\to\infty\), \(d/n\to c\in(0,\infty)\). All data points \(\mathbf{x}_{i}\), \(i\in[n]\), are independent and uniformly sampled from \(\mathbb{S}^{d-1}\). (ii) \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{W}\) are independent and have i.i entries of zero mean, unit variance, and finite fourth kurtosis. Moreover, we require \(\sigma_{a}^{2}+\sigma_{b}^{2}=1\). (iii) The activation \(\phi\) of the implicit NN is the normalized ReLU, i.e., \(\phi(x)=\sqrt{2}\max(x,0)\). The activation \(\sigma\) of the explicit NN is a \(C^{3}\) function._

**Remark 1**: _(i) Despite derived here for uniform distribution on the unit sphere, we conjecture that our results extend the result to more general distributions by using the technique developed in [5, 7]. (ii) The additional requirement on the variance is to ensure the existence and uniqueness of the fixed point of the NTK and to keep the diagonal entries of the CK matrix at \(1\), see examples in [6]. (iii) It is possible to extend our results to implicit NNs with general activations by using the technique proposed in [10]. We defer the extension to more general data distributions and activation functions to future work._

Under Assumptions 1, the limits of Implicit-CK and Implicit-NTK exist, and one can have precise expressions of \(\mathbf{G}^{*}\) and \(\mathbf{K}^{*}\) as follows [6, 9].

**Lemma 1**: _Let \(f(x)=\frac{\sqrt{1-x^{2}}+(\pi-\arccos x)x}{\pi}\). Under Assumptions 1, the fixed point of Implicit-CK \(\mathbf{G}^{*}_{ij}\) is the root of_

\[\mathbf{G}^{*}_{ij}=\sigma_{a}^{2}f(\mathbf{G}^{*}_{ij})+(1-\sigma_{a}^{2})\mathbf{x}_{i} ^{\top}\mathbf{x}_{j}. \tag{6}\]

_The limit of Implicit-NTK is_

\[\mathbf{K}^{*}_{ij}=h(\mathbf{G}^{*}_{ij})\triangleq\frac{\mathbf{G}^{*}_{ij}}{1-\dot{ \mathbf{G}}^{*}_{ij}}\quad\text{where}\quad\dot{\mathbf{G}}^{*}_{ij}\triangleq\sigma_ {a}^{2}\pi^{-1}(\pi-\arccos\bigl{(}\mathbf{G}^{*}_{ij}\bigr{)}). \tag{7}\]

## 3 Main Results

In this section, we prove the high-dimensional equivalents for CKs and NTKs of implicit and explicit NNs. As a result, by matching the coefficients of the asymptotic spectral equivalents, we establish the equivalence between implicit and explicit NNs in high dimensions.

### Asymptotic Approximations

Cks.We begin by defining several quantities that are crucial to our results. Note that the unique fixed point of Eq. (6) exists as long as \(\sigma_{a}^{2}<1\). We define the implicit map induced from Eq. (6) as \(\mathbf{G}^{*}_{ij}\triangleq g(\mathbf{x}_{i}^{\top}\mathbf{x}_{j})\). Let \(\angle^{*}=g(0)\) be the solution of \(\angle^{*}=\sigma_{a}^{2}f(\angle^{*})\) when \(\mathbf{x}_{i}^{\top}\mathbf{x}_{j}=0\). Using implicit differentiation, one can obtain that

\[g^{\prime}(0)=\frac{1-\sigma_{a}^{2}}{1-\sigma_{a}^{2}f^{\prime}(\angle^{*})},\quad g^{\prime\prime}(0)=\frac{\sigma_{a}^{2}(1-\sigma_{a}^{2})^{2}f^{ \prime\prime}(\angle^{*})}{(1-\sigma_{a}^{2}f^{\prime}(\angle^{*}))^{3}}.\]

Now we are ready to present the asymptotic equivalent of the Implicit-CK matrix.

**Theorem 1** (Asymptotic approximation of Implicit-CKs): _Let Assumptions 1 hold. As \(n,d\to\infty\), the Implicit-CK matrix \(\mathbf{G}^{*}\) defined in Eq. (6) can be approximated consistently in operator norm, by the matrix \(\overline{\mathbf{G}}\), that is \(\|\mathbf{G}^{*}-\overline{\mathbf{G}}\|_{2}\to 0\), where_

\[\overline{\mathbf{G}}=\alpha\mathbf{1}\mathbf{1}^{\top}+\beta\mathbf{X}^{\top}\mathbf{X}+\mu\mathbf{ I}_{n},\]

_with \(\alpha=g(0)+\frac{g^{\prime\prime}(0)}{2d}\), \(\beta=g^{\prime}(0)\), and \(\mu=g(1)-g(0)-g^{\prime}(0)\)._
**Theorem 2** (Asymptotic approximation for Explicit-CKs): _Let Assumptions 1 hold. As \(n,d\to\infty\), the Explicit-CK matrix \(\mathbf{\Sigma}\) defined in Eq. (5) can be approximated consistently in operator norm, by the matrix \(\overline{\mathbf{\Xi}}\), that is \(\|\mathbf{\Sigma}-\overline{\mathbf{\Xi}}\|_{2}\to 0\), where_

\[\overline{\mathbf{\Xi}}=\alpha_{1}\mathbf{1}\mathbf{1}^{\top}+\beta_{1}\mathbf{X}^{\top}\mathbf{X}+ \mu_{1}\mathbf{I}_{n},\]

_with \(\alpha_{1}=\mathbb{E}[\sigma(z)]^{2}+\frac{\mathbb{E}[\sigma^{\prime\prime}(z) ]^{2}}{2d}\), \(\beta_{1}=\mathbb{E}[\sigma^{\prime}(z)]^{2}\), and \(\mu_{1}=\mathbb{E}[\sigma^{2}(z)]-\mathbb{E}[\sigma(z)]^{2}-\mathbb{E}[\sigma ^{\prime}(z)]^{2}\), for \(z\sim\mathcal{N}(0,1)\)._

NTKs.For the Implicit-NTK, we define \(\mathbf{K}^{*}_{ij}=k(\mathbf{x}^{\top}_{i}\mathbf{x}_{j})\), i.e., \(k(\mathbf{x}^{\top}_{i}\mathbf{x}_{j})=h(g(\mathbf{x}^{\top}_{i}\mathbf{x}_{j}))\), for \(i,j\in[n]\). It is easy to check that \(k(0)=h(\angle^{*})\) and \(k(1)=h(g(1))\). Using implicit differentiation again, we have

\[k^{\prime}(0)=\frac{(1-\sigma_{a}^{2})h^{\prime}(\angle^{*})}{\sigma_{a}^{2}f ^{\prime}(\angle^{*})-1},\;k^{\prime\prime}(0)=\frac{(1-\sigma_{a}^{2})^{2}(h ^{\prime\prime}(\angle^{*})-\sigma_{a}^{2}f^{\prime}(\angle^{*})h^{\prime \prime}(\angle^{*})+\sigma_{a}^{2}h^{\prime}(\angle^{*})f^{\prime\prime}( \angle^{*}))}{(1-\sigma_{a}^{2}f^{\prime}(\angle^{*}))^{3}}.\]

Now we are ready to present the asymptotic equivalent of the Implicit-NTK matrix.

**Theorem 3** (Asymptotic approximation for Implicit-NTKs): _Let Assumptions 1 hold. As \(n,d\to\infty\), the Implicit-NTK matrix \(\mathbf{K}^{*}\) defined Eq. (7) in can be approximated consistently in operator norm, by the matrix \(\overline{\mathbf{K}}\), that is \(\|\mathbf{K}^{*}-\overline{\mathbf{K}}\|_{2}\to 0\), where_

\[\overline{\mathbf{K}}=\dot{\alpha}\mathbf{1}\mathbf{1}^{\top}+\dot{\beta}\mathbf{X}^{\top}\bm {X}+\dot{\mu}\mathbf{I}_{n},\]

_with \(\dot{\alpha}=k(0)+\frac{k^{\prime\prime}(0)}{2d}\), \(\dot{\beta}=k^{\prime}(0)\), and \(\dot{\mu}=k(1)-k(0)-k^{\prime}(0)\)._

**Theorem 4** (Asymptotic approximation for Explicit-NTKs): _Let Assumptions 1 hold. As \(n,d\to\infty\), the Explicit-NTK matrix \(\mathbf{\Theta}\) defined in Eq. (5) can be approximated consistently in operator norm, by the matrix \(\overline{\mathbf{\Theta}}\), that is \(\|\mathbf{\Theta}^{*}-\overline{\mathbf{\Theta}}\|_{2}\to 0\), where_

\[\overline{\mathbf{\Theta}}=\dot{\alpha}_{1}\mathbf{1}\mathbf{1}^{\top}+\dot{\beta}_{1}\bm {X}^{\top}\mathbf{X}+\dot{\mu}_{1}\mathbf{I}_{n},\]

_with \(\dot{\alpha}_{1}=\mathbb{E}[\sigma(z)]^{2}+\frac{3\mathbb{E}[\sigma^{\prime \prime}(z)]^{2}}{2d}\), \(\dot{\beta}_{1}=2\mathbb{E}[\sigma^{\prime}(z)]^{2}\), and \(\dot{\mu}_{1}=\mathbb{E}[\sigma^{2}(z)]+\mathbb{E}[\sigma^{\prime}(z)^{2}]- \mathbb{E}[\sigma(z)]^{2}-2\mathbb{E}[\sigma^{\prime}(z)]^{2}\) for \(z\sim\mathcal{N}(0,1)\)._

**Remark 2**: _(i) Due to the homogeneity of the ReLU function, the Implicit-CK and the Implicit-NTK are essentially inner product kernel random matrices. Consequently, Theorem 1 and 3 can be built upon the results in [4]. We postpone the study on general activations to future work. (ii) The results in Theorem 2 and 4 generalize those of [1, 7] to the cases of "non-centred" activations, i.e., we do not require \(\mathbb{E}[\sigma(z)]=0\) for \(z\sim\mathcal{N}(0,1)\)._

### The Equivalence between Implicit and Explicit NNs

In the following corollary, we show a concrete case of a single-layer explicit NN with an quadratic activation, that matches the CK or NTK eigenspectra of a ReLU implicit NN. The idea is to utilize the results of Theorems 1-4 to match the coefficients of the asymptotic equivalents such that \(\alpha_{1}=\alpha,\beta_{1}=\beta,\mu_{1}=\mu\), or \(\dot{\alpha}_{1}=\dot{\alpha},\dot{\beta}_{1}=\beta,\dot{\mu}_{1}=\mu\). We implement numerical simulations to verify our theory. The numerical results are shown in Figure 1.
**Corollary 1**: _We consider a quadratic polynomial activation \(\sigma(t)=a_{2}t^{2}+a_{1}t+a_{0}\). Let Assumptions 1 hold. As \(n,d\rightarrow\infty\), the Implicit-CK matrix \(\mathbf{G}^{*}\) defined in Eq. (6) can be approximated consistently in operator norm, by the Explicit-CK matrix \(\mathbf{\Sigma}\) defined in Eq. (5), i.e., \(\|\mathbf{G}^{*}-\mathbf{\Sigma}\|_{2}\to 0\), as long as_

\[a_{2}=\pm\sqrt{\frac{\mu}{2}}\quad a_{1}=\pm\sqrt{\beta},\quad a_{0}=\pm\sqrt {\alpha-\frac{\mu}{d}}-a_{2},\]

_and the Implicit-NTK matrix \(\mathbf{K}^{*}\) defined in Eq. (7) can be approximated consistently in operator norm, by the Explicit-NTK matrix \(\mathbf{\Theta}\) defined in Eq. (5), i.e., \(\|\mathbf{K}^{*}-\mathbf{\Theta}\|_{2}\to 0\), as long as_

\[a_{2}=\pm\sqrt{\frac{\mu}{6}},\quad a_{1}=\pm\sqrt{\frac{\hat{\beta}}{2}}, \quad a_{0}=\pm\sqrt{\dot{\alpha}-\frac{\dot{\mu}}{d}}-a_{2}.\]

## 4 Conclusion

In this paper, we study the CKs and NTKs of high-dimensional ReLU implicit NNs. We prove the asymptotic spectral equivalents for Implicit-CKs and Implicit-NTKs. Moreover, we establish the equivalence between implicit and explicit NNs by matching the coefficients of the asymptotic spectral equivalents. In particular, we show that a single-layer explicit NN with carefully designed activations has the same CK or NTK eigenspectra as a ReLU implicit NN. For future work, it would be interesting to extend our analysis to more general data distributions and activation functions.

Figure 1: We independently generate \(n=1\,000\) data points from the \(d=1\,200\)-dimensional unit sphere. We use Gaussian initialization and \(\sigma_{a}^{2}\) is set as \(0.2\). Upper: the CK results. Bottom: the NTK results. (a) spectral densities of implicit kernels, (b) spectral densities of explicit kernels, (c) quadratic activations.
#### Acknowledgements

Z. Liao would like to acknowledge the National Natural Science Foundation of China (via fund NSFC-62206101) and the Fundamental Research Funds for the Central Universities of China (2021XXJS110) for providing partial support. R. C. Qiu and Z. Liao would like to acknowledge the National Natural Science Foundation of China (via fund NSFC-12141107), the Key Research and Development Program of Hubei (2021BAA037) and of Guangxi (GuiKe-AB21196034).
