# On the optimization and generalization of overparameterized implicit neural
networks

Tianxiang Gao
Department of Computer Science
Iowa State University
<EMAIL_ADDRESS>
&Hongyang Gao
Department of Computer Sceicen
Iowa State University
<EMAIL_ADDRESS>email

###### Abstract

Implicit neural networks have become increasingly attractive in the machine
learning community since they can achieve competitive performance but use much
less computational resources. Recently, a line of theoretical works
established the global convergences for first-order methods such as gradient
descent if the implicit networks are over-parameterized. However, as they
train all layers together, their analyses are equivalent to only studying the
evolution of the output layer. It is unclear how the implicit layer
contributes to the training. Thus, in this paper, we restrict ourselves to
only training the implicit layer. We show that global convergence is
guaranteed, even if only the implicit layer is trained. On the other hand, the
theoretical understanding of when and how the training performance of an
implicit neural network can be generalized to unseen data is still under-
explored. Although this problem has been studied in standard feed-forward
networks, the case of implicit neural networks is still intriguing since
implicit networks theoretically have infinitely many layers. Therefore, this
paper investigates the generalization error for implicit neural networks.
Specifically, we study the generalization of an implicit network activated by
the ReLU function over random initialization. We provide a generalization
bound that is initialization sensitive. As a result, we show that gradient
flow with proper random initialization can train a sufficient over-
parameterized implicit network to achieve arbitrarily small generalization
errors.

##  1 Introduction

Implicit neural networks El Ghaoui et al. (2021) have been renewed interest in
the machine learning community recently as it can achieve competitive or
dominated performance in many applications compared to traditional neural
networks while using significantly less memory resources Bai et al. (2019);
Dabre & Fujita (2019). Unlike traditional neural networks, feature vectors in
implicit layers are not provided recursively. Instead, they are solutions to
an equilibrium equation induced by implicit layers. Consequently, an implicit
neural network is equivalent to an infinite-depth neural network with weight-
tied and input-injected. Thus, the gradients can be computed through implicit
differentiation Bai et al. (2019) using constant memory.

Empirical success of implicit neural networks has been observed in a number of
applications such as natural language processing Bai et al. (2019), compute
vision Bai et al. (2022), optimization Ramzi et al. (2021), and time series
analysis Rubanova et al. (2019). However, the theoretical understanding of
implicit neural networks is still limited compared to conventional neural
networks. One of the essential questions in the deep learning community is
whether a simple first-order method can converge to a global minimum. This
question is even more significant and complicated in the case of implicit
neural networks. Since the network has infinitely many layers, it may not be
well-posed. Specifically, the equilibrium equation may admit zero or multiple
solutions, so the forward propagation is probably not well-posed or even
divergent. Many works in the literature Chen et al. (2018); Bai et al. (2019;
2021); Kawaguchi (2021) observe instability of forwarding pass along training
epochs: the number of iterations the forward pass uses to find the equilibrium
point grows with training epochs. Thus, a line of works put efforts into
dealing with this well-posedness issue Winston & Kolter (2020); El Ghaoui et
al. (2021); Xie et al. (2022); Gao et al. (2021). Then some recent studies
successfully show global convergence of gradient flow and gradient descent for
implicit networks. For example, Kawaguchi (2021) proves the convergence of
gradient flow for a linear implicit network, but its result cannot be applied
to nonlinear activation. Gao et al. (2021) obtains the global convergence
results for ReLU-activated implicit networks if the width of networks is
quadratic of the sample size. However, the output layer in their setup is
combined with the feature vector, so their results can only be applied in a
restricted range of applications. The output issue is solved in their
following work Gao & Gao (2022), and the network size is reduced to linear
widths. Since they train all layers together, it can be considered a perturbed
version of only training the output layer. It is hard to justify the
contribution of the implicit layer to the training process. In addition, their
work cannot be directly applied to generalization analysis which is another
essential problem in the machine learning community.

One essential mystery in the deep learning community is that the neural
networks used in practice are often heavily overparameterized such that they
can even fit random labels, while they can achieve small generalization errors
(i.e., test, error). Although this problem has been studied extensively in
standard feed-forward networks Arora et al. (2019); Cao & Gu (2020); Allen-Zhu
et al. (2019); Cao & Gu (2019); Jacot et al. (2018), the case of implicit
models is still intriguing because implicit networks have infinitely many
layers. Unfortunately, there is no study on the generalization theory for
learning an implicit neural network to our best knowledge. As more and more
successes of implicit networks are observed in practice, there is increasing
demand for theoretical analysis to support these observations. In this paper,
we initiate the exploration of generalization errors for implicit neural
networks. We study one main class implicit network called deep equilibrium
models Bai et al. (2019) that is activated by the ReLU activation function. By
coupling the implicit network with a kernel machine, we can show that the
implicit neural network trained by random initialized gradient flow can
achieve arbitrarily small generalization errors if the implicit network is
sufficiently overparameterized. Moreover, the generalization bound we obtained
in this paper is initialization sensitive. This type of generalization error
itself also contributes to the deep learning community. It justifies the
observation that initialization is essential for a network to generalize.
Consequently, it supports some start-of-the-techniques, such as pre-training,
to achieve better test performance. In addition, it could provide a more
accurate estimate of the test performance in practice.

Main contribution: In this paper, we conduct analyses on the optimization and
generalization of an implicit neural network activated by the ReLU activation
function.

  * •

We train the implicit neural network through random initialized gradient flow.
If the neural network is overparameterized, then gradient flow converges to a
global minimum at a linear rate (with high probability). Although this result
is obtained in some previous works Gao et al. (2021); Gao & Gao (2022), our
fine-gained analysis and proof method set our work apart from the previous
works as we only train the implicit layer and the convergence result can be
further used in the generalization analysis.

  * •

We couple the implicit neural network with a kernel machine. We use Rademacher
complexity theory to provide an initialization-sensitive generalization bound
for an overparameterized implicit neural network. As a result, the
generalization error of this implicit neural network can be reduced to
arbitrarily small if the width of the implicit layer is sufficiently large.

  * •

Some concrete examples of random initialization are provided to show that the
assumptions made in the previous contributions can easily be satisfied. Under
these specified random initializations, we provide another generalization
bound independent of initialization. This generalization bound is easy to
compute and independent of the size of the neural network.

##  2 Preliminaries of Implicit Deep Learning

Notation: We use $\|{\bm{x}}\|$ to denote its Euclidean norm of a vector
${\bm{x}}$ and $\|{\bm{A}}\|$ to denote the operator norm of a matrix
${\bm{A}}$. For a square matrix ${\bm{A}}$, $\lambda_{\min}({\bm{A}})$ denote
the smallest eigenvalue of ${\bm{A}}$. We use
$\text{vec}\left({\bm{A}}\right)$ to denote the vectorization operation
applied on the matrix ${\bm{A}}$. Given a function $Y=f(X)$, the derivative
$\partial f/\partial X$ is defined by
$\text{vec}\left(dY\right)=\left(\partial f/\partial X\right)^{T}dX$, where
$X$ and $Y$ can be either scalars, vectors, or matrices. We also denote
$[n]:=[1,2,\cdots,n]$ to simplify our notation.

In this paper, we consider a main class of implicit networks called deep
equilibrium model with $m$ neurons in the implicit layer defined by

$\displaystyle f_{\theta}({\bm{x}}):=\frac{1}{\sqrt{m}}{\bm{z}}^{T}{\bm{b}},$
(1)
$\displaystyle{\bm{z}}:=\lim_{\ell\rightarrow\infty}{\bm{z}}^{\ell}=\lim_{\ell\rightarrow\infty}\sigma(\gamma{\bm{A}}^{T}{\bm{z}}^{\ell-1}+\phi({\bm{x}})),$
(2) $\displaystyle\phi({\bm{x}}):=\phi({\bm{W}}^{T}{\bm{x}}),$ (3)

where ${\bm{x}}\in\mathbb{R}^{d}$ is the input, ${\bm{z}}\in\mathbb{R}^{m}$ is
an equilibrium point of the transition equation 2, ${\bm{b}}\in\mathbb{R}^{m}$
is the weight vector in the output layer, ${\bm{A}}\in\mathbb{R}^{m\times m}$
is the weight matrix shared among implicit layers,
${\bm{W}}\in\mathbb{R}^{d\times m}$ is the weight matrix in the first layer,
and $\sigma(z)=\max\\{0,z\\}$ is the ReLU activation function and $\phi$ is
another activation function that is not necessary to be ReLU, but we assume
$\phi$ is also $1$-Lipschitz continuous for simplicity. For convince, we
denote ${\bm{\theta}}:=\text{vec}\left({\bm{W}},{\bm{A}},{\bm{b}}\right)$ as
the parameter vector. Here the scalar $\gamma\in(0,1)$ is a fixed scalar that
ensures the transition equation 2 is well posed (Gao & Gao, 2022, Lemma 3.1.).

We are given $n$ input-output samples
$S:=\\{({\bm{x}}_{i},y_{i})\\}_{i=1}^{n}$ that are drawn i.i.d. from some
underlying distribution $\mathcal{D}$. We denote
${\bm{X}}:=[{\bm{x}}_{1},\ldots,{\bm{x}}_{n}]\in\mathbb{R}^{n\times d}$,
${\bm{y}}:=(y_{1},\ldots,y_{n})\in\mathbb{R}^{n}$,
${\bm{\Phi}}:=[{\bm{\phi}}_{1},\ldots,{\bm{\phi}}_{n}]\in\mathbb{R}^{n\times
m}$, and ${\bm{Z}}:=[{\bm{z}}_{1},\ldots,{\bm{z}}_{n}]\in\mathbb{R}^{n\times
m}$. For simplicity, we further assume for each sample $\|{\bm{x}}\|=1$ and
$\left|y\right|\leq 1$.

We train this implicit network through randomly initialized gradient flow (GF)
on the square loss over the sample $S$. Specifically, we initialize the
parameters as follows

$\displaystyle
b_{r}\overset{i.i.d.}{\sim}\mathcal{U}\\{-1,+1\\},\quad{\bm{a}}_{r}\overset{i.i.d.}{\sim}\textit{subG}(\textbf{{0}},{\bm{I}}_{m}),\quad{\bm{w}}_{r}\overset{i.i.d.}{\sim}\textit{subG}(\textbf{{0}},{\bm{I}}_{d}),\quad\forall
r\in[m].$ (4)

where $\mathcal{U}$ is the discrete uniform distribution with probability
$1/2$, subG is some sub-Gaussian distribution with zero mean and unit
variance. We fix the first layer ${\bm{W}}$ and output layer ${\bm{b}}$ and
only optimize the implicit layer ${\bm{A}}$ through gradient flow on the
objective function as follows

$\displaystyle L({\bm{A}}):=\frac{1}{2}\|{\bm{u}}-{\bm{y}}\|^{2},$ (5)

where we use ${\bm{u}}:=f_{\theta}({\bm{X}})$ to simplify notation.

Suppose the forward propagation is well-posed. Then equilibrium point
${\bm{Z}}$ is well defined. By using the implicit function theorem, we can
write the gradient flow update as

$\displaystyle\frac{d\text{vec}\left({\bm{A}}\right)}{dt}=-\frac{\partial
L}{\partial{\bm{A}}}=-\frac{\gamma}{\sqrt{m}}\left[{\bm{D}}({\bm{I}}_{m}\otimes{\bm{Z}})\right]^{T}{\bm{Q}}^{-T}\left({\bm{b}}^{T}\otimes{\bm{I}}_{n}\right)^{T}({\bm{u}}-{\bm{y}}),$
(6)

where ${\bm{D}}:=\mathop{\bf
diag}[\text{vec}\left(\sigma^{\prime}(\gamma{\bm{Z}}{\bm{A}}+{\bm{\Phi}})\right)]$
and ${\bm{Q}}:={\bm{I}}_{Nm}-\gamma{\bm{D}}({\bm{A}}^{T}\otimes{\bm{I}}_{n})$.

Let $({\bm{x}},y)$ be an unseen data following the underlying distribution
$\mathcal{D}$. Then the essential learning task is to find a mapping $f$ for
which it minimizes the generalization error or population risk

$\displaystyle
R(f):=\mathbb{E}_{({\bm{x}},y)\sim\mathcal{D}}\left[\ell(f({\bm{x}}),y)\right],$

where $\ell$ is a selected loss function.

##  3 Optimization

This section presents our convergence result for random initialized gradient
flow. The analysis of gradient flow is a stepping stone toward understanding
the discrete algorithms. A line of recent works for discrete algorithms such
as gradient descent and stochastic gradient descent is based on or inspired by
the analysis of gradient flow Arora et al. (2018); Du et al. (2018); Gao et
al. (2021).

Assume ${\bm{Z}}(t)$ is well defined at time step $t$. By using a simple chain
rule and equation 6, then dynamics of ${\bm{u}}(t)$ under gradient flow can be
written as follows

$\displaystyle\frac{d{\bm{u}}}{dt}=\left(\frac{\partial{\bm{u}}}{\partial{\bm{A}}}\right)^{T}\left(\frac{d\text{vec}\left({\bm{A}}\right)}{dt}\right)=-{\bm{H}}(t)({\bm{u}}(t)-{\bm{y}}),$
(7)

where ${\bm{H}}(t)\in\mathbb{R}^{n\times n}$ and
${\bm{J}}(t)\in\mathbb{R}^{n\times mn}$ are defined by

$\displaystyle{\bm{H}}(t):=$
$\displaystyle\left(\frac{\partial{\bm{u}}}{\partial{\bm{A}}}\right)^{T}\left(\frac{\partial{\bm{u}}}{\partial{\bm{A}}}\right)=\frac{\gamma^{2}}{m}{\bm{J}}(t)\left[{\bm{I}}_{m}\otimes{\bm{Z}}(t){\bm{Z}}(t)^{T}\right]{\bm{J}}(t)^{T},$
(8) $\displaystyle{\bm{J}}(t):=$
$\displaystyle\left[{\bm{b}}^{T}\otimes{\bm{I}}_{n}\right]{\bm{Q}}(t)^{-1}{\bm{D}}(t).$
(9)

Clearly, ${\bm{H}}(t)$ is a positive semi-definite matrix (PSD). If the
smallest eigenvalue ${\bm{H}}(t)$ is lower bounded by some positive constant
$\lambda_{0}>0$, then solving the ordinary differential equation (ODE) above
immediately implies $L(t):=L({\bm{A}}(t))$ converges to $0$ at an
exponentially rate, i.e., $L(t)\leq e^{-\lambda_{0}t}L(0)$. It is indeed the
case for overparameterized implicit neural networks. Suppose
$\lambda_{\min}[{\bm{H}}(0)]$ is lower bounded. Then we can show ${\bm{H}}(t)$
is close to ${\bm{H}}(0)$ so that $\lambda_{\min}[{\bm{H}}(t)]$ is also lower
bounded if the implicit network is sufficiently overparameterized.

On the other hand, one needs to ensure that forward propagation is well-posed
throughout the training. It follows from (Vershynin, 2018, Theorem 4.4.5) that
$\|{\bm{A}}(0)\|=\mathcal{O}\left(\sqrt{m}\right)$ with high probability.
Thus, by choosing $\gamma:=\gamma_{0}/\sqrt{m}$ for small
$\gamma_{0}\in(0,1)$, the forward propagation becomes a contraction mapping
and the equilibrium point ${\bm{Z}}(0)$ is uniquely determined. Although
${\bm{A}}(t)$ are varied during the training, we can show
$\|{\bm{A}}(t)\|=\mathcal{O}\left(\sqrt{m}\right)$ is always the case
throughout the training. Thus, ${\bm{Z}}(t)$ is always well defined.

###### Theorem 3.1

For any $\delta\in(0,1)$, choose
$m=\Omega\left(\frac{N^{4}}{\lambda_{0}^{4}\delta}\right)$ and
$\gamma=\gamma_{0}/\sqrt{m}$ for a fixed $\gamma_{0}\in(0,1)$. Assume
$\lambda_{\min}({\bm{H}}(0))\geq\lambda_{0}$ for some constant $\lambda_{0}>0$
with probability $1$ over the random initialization equation 4. Then with
probability at least $1-\delta$ over the random initialization equation 4:

  * •

$\|{\bm{u}}(0)-{\bm{y}}\|=\mathcal{O}\left(\sqrt{N/\delta}\right)$

  * •

${\bm{A}}(t)=\mathcal{O}\left(\sqrt{m}\right)$

  * •

$\|{\bm{u}}(t)-{\bm{y}}\|^{2}\leq
e^{-\lambda_{0}t}\|{\bm{u}}(0)-{\bm{y}}\|^{2}$

###  3.1 Proof sketch of Theorem 3.1

Several previous works Gao & Gao (2022); Gao et al. (2021) have successfully
established the global convergence results. However, as they train three
layers together, the Gram matrix ${\bm{H}}(t)$ is a sum of three PSD matrices
induced by each layer. Then they only focus their analyses on the evolution of
the output layer instead of the implicit layer. This case is similar to
solving a perturbed convex optimization problem. Instead, here we only train
the implicit layer while the rest is fixed at their initialization. The
corresponding Gram matrix is much more complicated. More fine-grained analysis
is required to bound ${\bm{H}}(t)-{\bm{H}}(0)$.

To bound the difference ${\bm{H}}(t)$ to its initialization, it is significant
to characterize the trajectory of ${\bm{Z}}(t)$ and ${\bm{J}}(t)$ during
training. Specifically, we derive the partial derivatives of ${\bm{Z}}$ and
${\bm{J}}$ with respect to ${\bm{A}}$ as follows.

###### Lemma 3.2

Suppose $\|{\bm{A}}\|\leq M$ and $\gamma:=\gamma_{0}/M$ for
$\gamma_{0}\in(0,1)$. Then the equilibrium point ${\bm{Z}}$ is uniquely
determined. Moreover, the partial derivatives of ${\bm{Z}}$ and ${\bm{J}}$
with respect to ${\bm{A}}$ are given by

$\displaystyle\frac{\partial{\bm{Z}}}{\partial{\bm{A}}}$
$\displaystyle=\gamma\left[{\bm{D}}({\bm{I}}_{m}\otimes{\bm{Z}})\right]^{T}{\bm{Q}}^{-T},$
(10) $\displaystyle\frac{\partial{\bm{J}}^{T}}{\partial{\bm{A}}}$
$\displaystyle=\gamma\left[({\bm{D}}{\bm{U}}^{-1}{\bm{B}})^{T}\otimes{\bm{D}}{\bm{U}}^{-1}\right]^{T}$
(11)

where we denote ${\bm{U}}:={\bm{Q}}^{T}$ and
${\bm{B}}:={\bm{b}}\otimes{\bm{I}}_{n}$ to simplify derivation.

Now we are ready to present our proof sketch. Here we only list the results
that are also useful for later generalization analysis. Please see the whole
proof in Appendix C.

We assume the results hold for all $0\leq s\leq t$:

  * •

${\bm{A}}(s)=\mathcal{O}\left(\sqrt{m}\right)$

  * •

$\|{\bm{u}}(s)-{\bm{y}}\|^{2}\leq
e^{-\lambda_{0}t}\|{\bm{u}}(0)-{\bm{y}}\|^{2}$

The gradient $\partial L/\partial{\bm{A}}$ can be bounded as follows

$\displaystyle\left\|\partial L(s)/\partial{\bm{A}}\right\|\leq
c\|{\bm{X}}\|_{F}\|{\bm{u}}(s)-{\bm{y}}\|.$

where the scalar $c$ absorbs constants and quantities related to $\gamma_{0}$,
and we use the facts $\|{\bm{W}}(0)\|=\mathcal{O}\left(\sqrt{m}\right)$ and
$\|{\bm{b}}(0)\|=\sqrt{m}$. Then we can bound ${\bm{A}}(t)-{\bm{A}}(0)$:

$\displaystyle\|{\bm{A}}(t)-{\bm{A}}(0)\|_{F}\leq$
$\displaystyle\int_{0}^{t}\left\|\partial
L/\partial{\bm{A}}(s)\right\|ds\leq\frac{c}{\lambda_{0}}\|{\bm{X}}\|_{F}\|{\bm{u}}(0)-{\bm{y}}\|\leq\sqrt{m},$

where the last inequality is because
$m=\Omega\left(\frac{N^{4}}{\lambda_{0}^{4}\delta}\right)$. As a result, we
have

$\displaystyle\|{\bm{A}}(t)\|\leq\|{\bm{A}}(t)-{\bm{A}}(0)\|_{F}+\|{\bm{A}}(0)\|=\mathcal{O}\left(\sqrt{m}\right).$

By Lemma 3.2, we can bound ${\bm{Z}}(t)$ and ${\bm{J}}(t)$ to their
initialization as follows

$\displaystyle\left\|{\bm{Z}}(t)-{\bm{Z}}(0)\right\|_{F}\leq\int_{0}^{t}\left\|\frac{d\text{vec}\left({\bm{Z}}\right)}{ds}\right\|ds\leq\frac{c}{\lambda_{0}}\|{\bm{X}}\|_{F}^{2}\|{\bm{u}}(0)-{\bm{y}}\|.$
(12)
$\displaystyle\|{\bm{J}}(t)-{\bm{J}}(0)\|_{F}\leq\int_{0}^{t}\left\|\frac{d\text{vec}\left({\bm{J}}^{T}\right)}{ds}\right\|ds\leq\frac{c}{\lambda_{0}}\|{\bm{X}}\|_{F}\|{\bm{u}}(0)-{\bm{y}}\|.$
(13)

Moreover, we have for all $0\leq s\leq t$

$\displaystyle\left\|{\bm{J}}(s)\right\|=\left\|\left({\bm{b}}^{T}\otimes{\bm{I}}_{n}\right){\bm{Q}}(s)^{-1}{\bm{D}}(s)\right\|=\mathcal{O}\left(\sqrt{m}\right),.$
(14)

Combining equation 12, equation 13, equation 14, and Lemma A.5, we have

$\displaystyle\left\|{\bm{H}}(t)-{\bm{H}}(0)\right\|\leq\frac{c\|{\bm{X}}\|_{F}^{3}\|{\bm{u}}(0)-{\bm{y}}\|}{\lambda_{0}\sqrt{m}}\leq\frac{cn^{2}}{\lambda_{0}\sqrt{\delta}\sqrt{m}},$
(15)

where the last inequality is due to $\|{\bm{X}}\|=\sqrt{n}$ and
$\|{\bm{u}}(0)-{\bm{y}}\|=\mathcal{O}\left(\sqrt{n}/\sqrt{\delta}\right)$. It
follows from $m=\Omega\left(\frac{n^{4}}{\lambda_{0}^{4}\delta}\right)$ that
$\left\|{\bm{H}}(t)-{\bm{H}}(0)\right\|\leq\lambda_{0}/2$. By Weyl’s
inequality, we have

$\displaystyle\lambda_{\min}\left[{\bm{H}}(t)\right]\geq\lambda_{\min}\left[{\bm{H}}(0)\right]-\left\|{\bm{H}}(t)-{\bm{H}}(0)\right\|\geq\lambda_{0}/2.$

Therefore, the dynamics of loss function $L(t)$ satisfies

$\displaystyle\frac{d}{dt}\|{\bm{u}}-{\bm{y}}\|^{2}=2({\bm{u}}-{\bm{y}})^{T}\frac{d{\bm{u}}}{dt}=-2({\bm{u}}-{\bm{y}})^{T}{\bm{H}}(t)({\bm{u}}-{\bm{y}})\leq-\lambda_{0}\|{\bm{u}}-{\bm{y}}\|^{2}.$

By solving the ordinary differential equation above, we have

$\displaystyle\|{\bm{u}}(t)-{\bm{y}}\|^{2}\leq
e^{-\lambda_{0}t}\|{\bm{u}}(0)-{\bm{y}}\|^{2}.$

##  4 Generalization

In this section, we study the generalization ability of implicit networks
trained by randomly initialized gradient flow. Let
$f_{t}({\bm{x}}):=f_{{\bm{\theta}}(t)}({\bm{x}})$ be the corresponding
implicit neural network at time $t$. Our result is based on the observation
made in Domingos (2020), that is, $f_{t}({\bm{x}})$ is equivalent to a kernel
machine whose kernel function is induced by the gradients along the training.

Specifically, let $({\bm{x}}_{0},y_{0})$ be a data point that could be either
training data or unseen data. Its prediction at time $t$ is given by
$f_{t}({\bm{x}}_{0})$. Then its dynamics can be written as

$\displaystyle\frac{df_{t}({\bm{x}}_{0})}{dt}=\left(\frac{\partial
f_{t}({\bm{x}}_{0})}{\partial{\bm{A}}}\right)^{T}\left(-\frac{\partial
L}{\partial{\bm{A}}}\right)=k_{t}({\bm{x}}_{0},{\bm{X}})({\bm{y}}-{\bm{u}}(t)),$

where we define a kernel function

$\displaystyle k_{t}({\bm{x}},{\bm{x}}^{\prime}):=\left\langle\frac{\partial
f_{t}({\bm{x}})}{\partial{\bm{A}}},\frac{\partial
f_{t}({\bm{x}}^{\prime})}{\partial{\bm{A}}}\right\rangle,$ (16)

and the mapping ${\bm{x}}\mapsto\partial f_{t}({\bm{x}})/\partial{\bm{A}}$ is
the corresponding feature map. Taking integration upto time $t$, the
prediction of $f_{t}({\bm{x}}_{0})$ can be written as follows

$\displaystyle
f_{t}({\bm{x}}_{0})=f_{0}({\bm{x}}_{0})+\int_{0}^{t}k_{s}({\bm{x}}_{0},{\bm{X}})({\bm{y}}-{\bm{u}}(s))ds.$
(17)

Here the kernel function $k_{t}$ changes along the training process, but by
applying some simple algebraic tricks, one can show $f_{t}$ is indeed a kernel
machine whose kernel is called path kernel introduced in Domingos (2020).
However, its Rademacher complexity is hard to compute, and its generalization
cannot be determined simply.

Instead, the main idea in this paper is to couple the trajectory of $f_{t}$
with another function $\hat{f}_{t}$ whose dynamics are given by

$\displaystyle\frac{d\hat{f}_{t}({\bm{x}}_{0})}{dt}=k_{0}({\bm{x}}_{0},{\bm{X}})({\bm{y}}-\hat{{\bm{u}}}(t)),$

where $\hat{{\bm{u}}}(t)$ is the corresponding estimate for training data
$({\bm{X}},{\bm{y}})$ with dynamics given by

$\displaystyle\frac{d\hat{{\bm{u}}}}{dt}=-{\bm{H}}(0)(\hat{{\bm{u}}}(t)-{\bm{y}}).$
(18)

Then the prediction of $\hat{f}_{t}$ for data $({\bm{x}}_{0},y_{0})$ is given
by

$\displaystyle\hat{f}_{t}({\bm{x}}_{0})=f_{0}({\bm{x}}_{0})+\int_{0}^{t}k_{0}({\bm{x}}_{0},{\bm{X}})({\bm{y}}-\hat{{\bm{u}}}(s))ds,$
(19)

where we use the fact $\hat{f}_{0}({\bm{x}}_{0})=f_{0}({\bm{x}}_{0})$.

We provide a generalization bound in the following theorem for any Lipschitz
continuous loss function $\ell$ by showing $f_{t}$ is close to $\hat{f}_{t}$
if the implicit network is sufficiently overparameterized.

###### Theorem 4.1

Fix $\delta\in(0,1)$. Suppose $m=\Omega(\lambda_{0}^{-8}\delta^{-1}N^{10})$
and $\gamma=\gamma_{0}/\sqrt{m}$ for some $\gamma_{0}\in(0,1)$. Assume
$\lambda_{\min}[{\bm{H}}(0)]\geq\lambda_{0}>0$ with probability $1$. Then with
probability at least $1-\delta$ over the random initialization and random
training sample $S$, the implicit neural network $f_{\infty}$ trained by
gradient flow has generalization error
$R(f_{\infty}):=\mathbb{E}\left[\ell(f_{\infty}({\bm{x}}),y)\right]$ bounded
by

$\displaystyle
R(f_{\infty})\leq\mathcal{O}\left(\sqrt{\frac{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))}{n}}+\sqrt{\frac{\log(1/\delta)}{n}}\right).$
(20)

The dominating term in equation 20 is :

$\displaystyle\sqrt{\frac{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))}{n}}.$

This can be used to predict the test accuracy of the learned neural network.
It is worth noting that our bound identifies the sensitivity of
initialization. If one has a good start, say $\|{\bm{y}}-{\bm{u}}(0)\|$ is
small, then a better test accuracy could probably be achieved. Moreover, let
${\bm{H}}(0)={\bm{Q}}{\bm{\Lambda}}{\bm{Q}}^{T}$ be the eigenvalue
decomposition of ${\bm{H}}(0)$. Then the dominating term can be written as

$\displaystyle\sqrt{\sum_{i=1}^{n}\lambda_{i}^{-2}({\bm{q}}_{i}^{T}({\bm{y}}-{\bm{u}}(0)))^{2}/n}\leq$
$\displaystyle\sum_{i=1}^{n}\lambda_{i}^{-1}\left|{\bm{q}}_{i}^{T}({\bm{y}}-{\bm{u}}(0))\right|/\sqrt{n}$
$\displaystyle\leq$
$\displaystyle\left(\sum_{i=1}^{n}\lambda_{i}^{-1}\right)\|{\bm{y}}-{\bm{u}}(0)\|/\sqrt{n}.$

To have better test performance, we may want to have ${\bm{y}}-{\bm{u}}(0)$ to
align the eigenspace spanned by the eigenvectors with large eigenvalues. Note
that
$\sum_{i=1}^{n}\lambda_{i}=\|{\bm{H}}(0)\|_{F}=\mathcal{O}\left(n\right)$.
Thus, the test performance can be further improved if eigenvalues
$\lambda_{i}$ are close to each other, which probably explains why gradient
descent has implicit regularization induced from discretization. We will
consider that as our future work.

###  4.1 Proof sketch of Theorem 4.1

Before showing $f_{t}$ is close to $\hat{f}_{t}$, we first provide some
essential properties of $\hat{f}_{t}$.

###### Lemma 4.2

Assume $\lambda_{\min}[{\bm{H}}(0)]\geq\lambda_{0}>0$. Then the function
$\hat{f}_{t}$ has the following properties for all $t\geq 0$:

  * •

${\bm{y}}-\hat{{\bm{u}}}(t)=e^{-{\bm{H}}(0)t}({\bm{y}}-{\bm{u}}(0))$.

  * •

Let $\hat{f}_{\infty}$ be the limit of $\hat{f}_{t}$. Then
$\hat{f}_{\infty}\in\mathcal{F}_{B}$ defined by

$\displaystyle\mathcal{F}_{B}:=\left\\{f:{\bm{x}}\mapsto
k_{0}({\bm{x}},{\bm{X}}){\bm{\alpha}}:{\bm{\alpha}}^{T}{\bm{H}}(0){\bm{\alpha}}\leq
B^{2}\right\\}$ (21)

where
$B^{2}:=({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))$.

  * •

The Rademacher complexity of $\mathcal{F}_{B}$ is given by

$\displaystyle\mathfrak{R}_{S}(\mathcal{F}_{B})\leq
c\sqrt{\frac{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))}{N}}.$
(22)

It is clear that both $f_{t}$ and $\hat{f}_{t}$ are close related to the
dynamics of training estimation ${\bm{u}}(t)$ and $\hat{{\bm{u}}}(t)$,
correspondingly. By equation 15, the following lemma shows ${\bm{u}}(t)$ can
be considered as a perturbed version of $\hat{{\bm{u}}}(t)$.

###### Lemma 4.3

Suppose the assumptions made in Theorem 3.1 holds. Then for all $t\geq 0$ we
have

$\displaystyle\|{\bm{u}}(t)-\hat{{\bm{u}}}(t)\|=\mathcal{O}\left(\frac{n^{3/2}}{\lambda_{0}\delta
m^{1/4}}e^{-\lambda_{0}t}\right),$ (23)

The following lemma show $f_{t}$ is close to $\hat{f}_{t}$ if the implicit
network is sufficiently overparameterized.

###### Lemma 4.4

Suppose the assumptions made in Theorem 3.1 holds. Then with probability at
least $1-\delta$ over random initialization equation 4, for all $t\geq 0$ and
$\|{\bm{x}}_{0}\|=1$ we have

$\displaystyle\left|f_{t}({\bm{x}}_{0})-\hat{f}_{t}({\bm{x}}_{0})\right|=\mathcal{O}\left(\frac{n^{2}}{\lambda_{0}^{2}\delta
m^{1/4}}\right).$ (24)

Since Theorem 4.1 assumes $m=\Omega(\lambda_{0}^{-8}\delta^{-1}n^{10})$, Lemma
4.4 immediately implies

$\displaystyle\left|f_{t}({\bm{x}}_{0})-\hat{f}_{t}({\bm{x}}_{0})\right|=\mathcal{O}\left(\sqrt{\frac{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))}{N}}\right)$

Clearly, the RHS in equation 24 is independent of time $t$. Let
$t\rightarrow\infty$ and we obtain the limiting functions $f_{\infty}$ and
$\hat{f}_{\infty}$. Combining the Rademacher complexity in Lemma 4.2 with the
bound in Lemma 4.4, we can complete the proof.

##  5 Discussion on initialization

In this section, we want to show that the assumptions made in Theorem 3.1 and
4.1 can be satisfied by some specified random initialization with
exponentially high probability. Unfortunately, these random initialization are
not commonly used in practice. In addition, to our best knowledge, no previous
works are studying the limiting neural tangent kernel
${\bm{H}}^{\infty}:=\lim_{m\rightarrow\infty}{\bm{H}}(0)$ for implicit neural
networks. Although this problem has been studied for some standard feed-
forward neural networks, the case of implicit neural networks is more
complicated since they have infinite depth. We will consider that as our
future work.

By using simple linear algebra results for PSD matrices, the Gram matrix
${\bm{H}}(0)$ satisfies the following inequality

$\displaystyle{\bm{H}}(0)\succeq$
$\displaystyle\|{\bm{b}}(0)\|^{2}\lambda_{\min}\left\\{{\bm{Q}}(0)^{-1}{\bm{Q}}(0)^{-T}\right\\}\cdot\frac{\gamma^{2}}{m}{\bm{D}}(0)\left[{\bm{I}}\otimes{\bm{Z}}(0){\bm{Z}}(0)^{T}\right]{\bm{D}}(0)$
$\displaystyle\succeq$
$\displaystyle\frac{c}{m}{\bm{D}}(0)\left[{\bm{I}}\otimes{\bm{Z}}(0){\bm{Z}}(0)^{T}\right]{\bm{D}}(0)$
(25)

where $c>0$ absorbs constants related to $\gamma_{0}$ and
$``{\bm{A}}\succeq{\bm{B}}"$ if ${\bm{A}}-{\bm{B}}$ is PSD. To show
$\lambda_{\min}[{\bm{H}}(0)]$ is lower bounded, it suffices to show the RHS is
strictly positive (with high probability). In the following, we provide a
simple example of random initialization by which the RHS in equation 25 is
strictly positive.

###  5.1 $\lambda_{\min}\\{\textbf{{H}}(0)\\}\geq\lambda_{0}>0$

We choose $\phi=\sigma$ to also be the ReLU activation function. The random
initialization equation 4 is specified as follows

$\displaystyle{\bm{a}}_{r}\overset{i.i.d.}{\sim}\left|\mathcal{N}\right|(0,{\bm{I}}_{m}),\quad{\bm{w}}_{r}\overset{i.i.d.}{\sim}\mathcal{N}(\textbf{{0}},{\bm{I}}_{d}),$
(26)

where $\mathcal{N}$ is Gaussian distribution and $\left|\mathcal{N}\right|$ is
half-normal distribution. Then ${\bm{A}}_{ij}(0)\geq 0$ for all $i,j$. It
follows from the nonnegativity of ReLU activation that ${\bm{\Phi}}_{ij}\geq
0$ for all $i,j$. By Neumann series, we can write the explicit form of the
equilibrium point ${\bm{Z}}(0)$ at initialization as follows

$\displaystyle{\bm{Z}}(0)=\sigma({\bm{X}}{\bm{W}}(0))\left[{\bm{I}}_{m}-\gamma{\bm{A}}(0)\right]^{-1},$

and ${\bm{D}}(0)={\bm{I}}_{nm}$. The inequality equation 25 becomes

$\displaystyle{\bm{H}}(0)\succeq\frac{c}{m}\sum_{r=1}^{m}\sigma({\bm{X}}{\bm{w}}_{r}(0))\sigma({\bm{X}}{\bm{w}}_{r}(0))^{T}.$
(27)

Let $m\rightarrow\infty$, then we obtain the kernel matrix given by

$\displaystyle{\bm{G}}^{\infty}:=\mathbb{E}_{{\bm{w}}\sim\mathcal{N}(0,{\bm{I}}_{d})}\left[\sigma({\bm{X}}{\bm{w}})\sigma({\bm{X}}{\bm{w}})^{T}\right]$
(28)

that is induced by the neural tangent kernel Jacot et al. (2018) defined by

$\displaystyle
k_{*}({\bm{x}},{\bm{x}}^{\prime}):=\mathbb{E}_{{\bm{w}}\sim\mathcal{N}(0,{\bm{I}}_{d})}\left[\sigma({\bm{w}}^{T}{\bm{x}})\sigma({\bm{w}}^{T}{\bm{x}}^{\prime})\right].$

Several previous works have shown that
$\lambda_{*}:=\lambda_{\min}\\{{\bm{G}}^{\infty}\\}>0$ under some mild data
distribution assumption. For example, Gao et al. (2021) shows that
$\lambda_{*}>0$ if no two training points are parallel. By using Matrix-
Chernoff inequality, one can easily show

$\displaystyle\lambda_{\min}\\{{\bm{H}}(0)\\}\geq\frac{c}{m}\lambda_{\min}\left[\sigma({\bm{X}}{\bm{W}}(0))\sigma({\bm{X}}{\bm{W}}(0))^{T}\right]\geq\frac{c}{m}\cdot
m\lambda_{*}/4=c\lambda_{*}/4:=\lambda_{0}.$

with probability at least $1-\delta$ (see Lemma 5.2 of Nguyen et al. (2021))
if $m=\tilde{\Omega}(n/\lambda_{*})$ holds, where $\tilde{\Omega}$ omits
logarithmic factors depending on $\delta$. Thus, the conditions on Theorem 3.1
and 4.1 are all satisfied with probability at least $1-\delta$. By using union
bound, the results in Theorem 3.1 and 4.1 are obtained.

###  5.2 Generalization bound induced by $\textbf{{H}}^{\infty}$

To simplify the analysis, we assume in this subsection that ${\bm{A}}(0)$ is a
randomly initialized diagonal matrix, e.g., ${\bm{A}}_{ij}(0)=\delta_{ij}z$
with $z\sim\mathcal{N}(0,1)$. Then

$\displaystyle{\bm{H}}(0)_{ij}=\frac{\gamma^{2}({\bm{z}}_{i}^{T}{\bm{z}}_{j})}{m}\sum_{r=1}^{m}\hat{b}_{ir}\hat{b}_{jr}\sigma^{\prime}(\gamma{\bm{a}}_{r}^{T}{\bm{z}}_{i}+\phi({\bm{w}}_{r}^{T}{\bm{x}}_{i}))\sigma^{\prime}(\gamma{\bm{a}}_{r}^{T}{\bm{z}}_{j}+\phi({\bm{w}}_{r}^{T}{\bm{x}}_{j}))$

where $\hat{b}_{ir}$ is the $r$-the entry of the vector
$\hat{{\bm{b}}}_{i}={\bm{Q}}^{-1}_{i}{\bm{b}}$. Since ${\bm{A}}(0)$ is
diagonal, we have $\hat{b}_{ir}=\Theta(1)$ for all $i\in[n],r\in[m]$. It
follows from (Gao et al., 2021, Lemma 2.2) that
$\|{\bm{z}}_{i}\|\leq\gamma^{-1}$ for all $i\in[n]$. Therefore,
${\bm{H}}(0)_{ij}$ is the average of $m$ random variables that are bounded in
$[-c,c]$ for some constant $c>0$. By Hoeffding’s inequality, we have

$\displaystyle\|{\bm{H}}(0)-{\bm{H}}^{\infty}\|_{F}=\mathcal{O}\left(\frac{n\sqrt{\log(n/\delta)}}{\sqrt{m}}\right).$

where ${\bm{H}}^{\infty}$ is the limiting neural tangent kernel matrix Jacot
et al. (2018). Therefore, the generalization bound in Theorem 4.1 becomes

$\displaystyle
R(f_{\infty})\leq\mathcal{O}\left(\sqrt{\frac{{\bm{y}}^{T}({\bm{H}}^{\infty})^{-1}{\bm{y}}}{n}}+\sqrt{\frac{\log(n/\delta)}{n}}\right),$

provided $\lambda_{\min}\\{{\bm{H}}^{\infty}\\}\geq\lambda_{*}$ for some
constant $\lambda_{*}>0$. This is one of the well-known generalization bounds
for finite-depth neural networks Arora et al. (2019); Cao & Gu (2020). Since
${\bm{H}}^{\infty}$ is constant and induced from the ReLU activation, one of
the advantages of this generalization bound is that it can be computed
directly by using the training sample
$S=\\{({\bm{x}}_{i},y_{i})\\}_{i=1}^{n}$. In contrast, the generalization
bound in Theorem 4.1 is initialization-dependent, which probably has the
potential to provide an estimate of the test performance closely related to
empirical results.

##  6 Experimental results

We conduct a series of experiments to verify the convergence and test the
performance of gradient flow. The experimental setup is identical to Gao et
al. (2021), but we only train the implicit layer: we draw 500 random samples
of classes 0 and 1 from the MNIST dataset. We can see from Figure 1 that the
training loss (i.e., square loss) and train classification error (i.e., 0-1
loss) is consistently decreased to $0$. In addition, a faster convergence rate
is obtained by using wider implicit layers. As we prove in Theorem 4.1,
overparameterized implicit neural network also generalizes. This can be seen
from Figure 1 as the test loss and test classification errors are reduced
consistently. It is worth noting that a wider implicit network also provides
better test performance.

Figure 1: We evaluate the impact of the width $m$ on the training loss, test
loss, and operator norm of the scaled matrix $\gamma A(k)$ on the modified
dataset of MNIST.

##  7 Conclusion

This paper establishes a convergence result of gradient flow for implicit
neural networks. We show that gradient flow with random initialization
converges to a global minimum at a linear rate, even if we only optimize the
implicit layer while the rest is untrained. Moreover, we prove that
overparameterized implicit network is closely related to a kernel machine. By
leveraging the Rademacher complexity theory, an initialization-sensitive
generalization bound is provided. Thus, an arbitrarily small generalization
error can be obtained if the implicit layer is sufficiently overparameterized.

## References

  * Allen-Zhu et al. (2019) Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang.  Learning and generalization in overparameterized neural networks, going beyond two layers.  _Advances in neural information processing systems_ , 32, 2019.
  * Arora et al. (2018) Sanjeev Arora, Nadav Cohen, and Elad Hazan.  On the optimization of deep networks: Implicit acceleration by overparameterization.  In _International Conference on Machine Learning_ , pp. 244–253. PMLR, 2018.
  * Arora et al. (2019) Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang.  Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks.  In _International Conference on Machine Learning_ , pp. 322–332. PMLR, 2019.
  * Bai et al. (2019) Shaojie Bai, J Zico Kolter, and Vladlen Koltun.  Deep equilibrium models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Bai et al. (2021) Shaojie Bai, Vladlen Koltun, and J Zico Kolter.  Stabilizing equilibrium models by jacobian regularization.  _arXiv preprint arXiv:2106.14342_ , 2021.
  * Bai et al. (2022) Shaojie Bai, Zhengyang Geng, Yash Savani, and J Zico Kolter.  Deep equilibrium optical flow estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 620–630, 2022.
  * Cao & Gu (2019) Yuan Cao and Quanquan Gu.  Generalization bounds of stochastic gradient descent for wide and deep neural networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Cao & Gu (2020) Yuan Cao and Quanquan Gu.  Generalization error bounds of gradient descent for learning over-parameterized deep relu networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 3349–3356, 2020.
  * Chen et al. (2018) Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud.  Neural ordinary differential equations.  _arXiv preprint arXiv:1806.07366_ , 2018.
  * Dabre & Fujita (2019) Raj Dabre and Atsushi Fujita.  Recurrent stacking of layers for compact neural machine translation models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 6292–6299, 2019.
  * Domingos (2020) Pedro Domingos.  Every model learned by gradient descent is approximately a kernel machine.  _arXiv preprint arXiv:2012.00152_ , 2020.
  * Du et al. (2018) Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh.  Gradient descent provably optimizes over-parameterized neural networks.  _arXiv preprint arXiv:1810.02054_ , 2018.
  * El Ghaoui et al. (2021) Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai.  Implicit deep learning.  _SIAM Journal on Mathematics of Data Science_ , 3(3):930–958, 2021.
  * Gao & Gao (2022) Tianxiang Gao and Hongyang Gao.  Gradient descent optimizes infinite-depth relu implicit networks with linear widths.  _arXiv preprint arXiv:2205.07463_ , 2022.
  * Gao et al. (2021) Tianxiang Gao, Hailiang Liu, Jia Liu, Hridesh Rajan, and Hongyang Gao.  A global convergence theory for deep relu implicit networks via over-parameterization.  _arXiv preprint arXiv:2110.05645_ , 2021.
  * Jacot et al. (2018) Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  _Advances in neural information processing systems_ , 31, 2018.
  * Kawaguchi (2021) Kenji Kawaguchi.  On the theory of implicit deep learning: Global convergence with implicit layers.  _arXiv preprint arXiv:2102.07346_ , 2021.
  * Nguyen et al. (2021) Quynh Nguyen, Marco Mondelli, and Guido F Montufar.  Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks.  In _International Conference on Machine Learning_ , pp. 8119–8129. PMLR, 2021.
  * Ramzi et al. (2021) Zaccharie Ramzi, Florian Mannel, Shaojie Bai, Jean-Luc Starck, Philippe Ciuciu, and Thomas Moreau.  Shine: Sharing the inverse estimate from the forward pass for bi-level optimization and implicit models.  _arXiv preprint arXiv:2106.00553_ , 2021.
  * Rubanova et al. (2019) Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud.  Latent ordinary differential equations for irregularly-sampled time series.  _Advances in neural information processing systems_ , 32, 2019.
  * Vershynin (2018) Roman Vershynin.  _High-dimensional probability: An introduction with applications in data science_ , volume 47.  Cambridge university press, 2018.
  * Winston & Kolter (2020) Ezra Winston and J Zico Kolter.  Monotone operator equilibrium networks.  _Advances in neural information processing systems_ , 33:10718–10728, 2020.
  * Xie et al. (2022) Xingyu Xie, Qiuhao Wang, Zenan Ling, Xia Li, Guangcan Liu, and Zhouchen Lin.  Optimization induced equilibrium networks: An explicit optimization perspective for understanding equilibrium models.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.

##  Appendix A Useful mathematical lemmas

######  Lemma A.1 (Gronwall’s inequality)

Let $u$, $\alpha$, $\beta$ be real-valued continuous functions that satisfies
the integral inequality

$\displaystyle u(t)\leq\alpha(t)+\int_{0}^{t}\beta(s)u(s)ds.$ (29)

Then

$\displaystyle
u(t)\leq\alpha(t)+\int_{0}^{t}\alpha(s)\beta(s)\exp\left(\int_{s}^{t}\beta(r)dr\right)ds.$
(30)

If, in addition, $\alpha(t)$ is non-decreasing, then

$\displaystyle u(t)\leq\alpha(t)\exp\left(\int_{0}^{t}\beta(s)ds\right)$ (31)

######  Lemma A.2 (Weyl’s inequality)

Let ${\bm{A}},{\bm{B}}\in\mathbb{R}^{m\times n}$ with singular values
$\sigma_{1}({\bm{A}})\geq\cdots\geq\sigma_{r}({\bm{A}})$, where
$r:=\min\\{m,n\\}$. Then

$\displaystyle\left|\sigma_{k}({\bm{A}})-\sigma_{k}({\bm{A}})\right|\leq\|{\bm{A}}-{\bm{B}}\|$
(32)

###### Theorem A.3

Let $k$ be a kernel and $S=\\{{\bm{X}},{\bm{y}}\\}$ be a sample of size $n$
for which $k({\bm{x}},{\bm{x}}^{\prime})\leq r^{2}$. Let
$\mathcal{F}_{B}=\\{{\bm{x}}\mapsto
k({\bm{x}},{\bm{X}}){\bm{\alpha}}:{\bm{\alpha}}^{T}k({\bm{X}},{\bm{X}}){\bm{\alpha}}\leq
B^{2}\\}$. Then

$\displaystyle\mathfrak{R}_{S}(\mathcal{F}_{B})\leq\frac{B}{n}\sqrt{\mathbf{tr}(k({\bm{X}},{\bm{X}}))}\leq\frac{Br}{\sqrt{n}}$
(33)

###### Theorem A.4

Suppose the loss function $\ell(y,\cdot)$ is $\rho$-Lipschitz continuous and
is bounded in $[0,c]$ for some constant $c>0$. Then with probability at least
$1-\delta$ over random sample $S$ of size $N$

$\displaystyle\sup_{f\in\mathcal{F}}\left[R(f)-R_{S}(f)\right]\leq
2\mathfrak{R}_{S}(\mathcal{F})+3c\sqrt{\frac{\log(2/\delta)}{2N}}.$

###### Lemma A.5

Suppose $\|{\bm{A}}\|\leq M$ for some constant $M>0$ and choose the scalar
$\gamma>0$ such that $\gamma_{0}:=\gamma M<1$. Then the existence of the fixed
point ${\bm{Z}}$ is uniquely determined. Moreover, we have
$\|{\bm{Z}}^{\ell}\|_{F}\leq\frac{1}{1-\gamma_{0}}\|\Phi\|_{F}$ for all
$\ell$, hence $\|{\bm{Z}}\|_{F}\leq\frac{1}{1-\gamma_{0}}\|{\bm{\Phi}}\|_{F}$.

##  Appendix B Proof of Lemma 3.2

It follows from Lemma A.5 that ${\bm{Z}}$ is uniquely determined. As a result,
${\bm{Z}}$ is also a root of the following function

$\displaystyle
f({\bm{Z}},{\bm{A}})={\bm{Z}}-\sigma(\gamma{\bm{Z}}{\bm{A}}+{\bm{\Phi}}).$

Then by using the implicit function theorem, (Gao & Gao, 2022, Lemma 3.2)
provides the derivative of ${\bm{Z}}$ with respect to ${\bm{A}}$ as follows

Then the differential of $f$ is given by

$\displaystyle df=$ $\displaystyle
d{\bm{Z}}-d\sigma(\gamma{\bm{Z}}{\bm{A}}+{\bm{\Phi}})$ $\displaystyle=$
$\displaystyle d{\bm{Z}}-\sigma^{\prime}({\bm{U}})\odot
d(\gamma{\bm{Z}}{\bm{A}}+{\bm{\Phi}})$ $\displaystyle=$ $\displaystyle
d{\bm{Z}}-\sigma^{\prime}({\bm{U}})\odot\gamma(d{\bm{Z}}){\bm{A}}-\sigma^{\prime}({\bm{U}})\odot\gamma{\bm{Z}}d{\bm{A}}$

where ${\bm{U}}:=\gamma{\bm{Z}}{\bm{A}}+{\bm{\Phi}}$. Taking vectorization on
both sides yields

$\displaystyle\text{vec}\left(df\right)=\text{vec}\left(d{\bm{Z}}\right)-\gamma{\bm{D}}\left({\bm{A}}^{T}\otimes{\bm{I}}_{n}\right)\text{vec}\left(d{\bm{Z}}\right)-\gamma{\bm{D}}\left({\bm{I}}_{m}\otimes{\bm{Z}}\right)\text{vec}\left(d{\bm{A}}\right)$

where $D:=\mathop{\bf diag}[\text{vec}\left(\sigma^{\prime}(U)\right)]$. Thus,
we obtain

$\displaystyle\frac{\partial f}{\partial{\bm{Z}}}=$
$\displaystyle\left[{\bm{I}}_{Nm}-\gamma{\bm{D}}({\bm{A}}^{T}\otimes{\bm{I}}_{n})\right]^{T}$
$\displaystyle\frac{\partial f}{\partial{\bm{A}}}=$
$\displaystyle-\gamma\left[{\bm{D}}({\bm{I}}_{m}\otimes{\bm{Z}})\right]^{T}$

By reverse triangle inequality, we have

$\displaystyle\left\|{\bm{I}}_{nm}-\gamma{\bm{D}}({\bm{A}}^{T}\otimes{\bm{I}}_{n})\right\|\geq
1-\gamma\|{\bm{D}}({\bm{A}}^{T}\otimes{\bm{I}}_{n})\|\geq
1-\gamma\|{\bm{A}}\|>1-\gamma_{0}>0,$ (34)

where we use $\|{\bm{A}}\|\leq M$ and $\gamma_{0}=\gamma/M<1$. Hence, the
matrix
${\bm{Q}}:=\left[{\bm{I}}_{nm}-\gamma{\bm{D}}({\bm{A}}^{T}\otimes{\bm{I}}_{n})\right]$
is invertible.

Since the fixed point ${\bm{Z}}$ is the root of $f$ and the matrix ${\bm{Q}}$
is invertible, by using the implicit function theorem, we have

$\displaystyle\frac{\partial{\bm{Z}}}{\partial{\bm{A}}}\frac{\partial
f}{\partial{\bm{Z}}}+\frac{\partial f}{\partial{\bm{A}}}=0,$

which implies

$\displaystyle\frac{\partial{\bm{Z}}}{\partial{\bm{A}}}=-\left(\frac{\partial
f}{\partial{\bm{A}}}\right)\left(\frac{\partial
f}{\partial{\bm{Z}}}\right)^{-1}=\gamma\left[{\bm{D}}({\bm{I}}_{m}\otimes{\bm{Z}})\right]^{T}{\bm{Q}}^{-T}.$

Next, we derive the derivative of ${\bm{J}}^{T}$ with respect to ${\bm{A}}$.
The differential of ${\bm{J}}^{T}$ is given by

$\displaystyle d{\bm{J}}^{T}=$ $\displaystyle d{\bm{D}}{\bm{U}}^{-1}{\bm{B}}$
$\displaystyle=$
$\displaystyle(d{\bm{D}})U^{-1}{\bm{B}}-{\bm{D}}{\bm{U}}^{-1}(d{\bm{U}}){\bm{U}}^{-1}{\bm{B}}$
$\displaystyle=$
$\displaystyle\gamma{\bm{D}}{\bm{U}}^{-1}(d{\bm{A}}){\bm{D}}{\bm{U}}^{-1}{\bm{B}}$

where we use the fact $\sigma^{\prime\prime}(z)=0$ and we denote
${\bm{U}}:={\bm{Q}}^{T}$ and ${\bm{B}}:={\bm{b}}\otimes{\bm{I}}_{n}$ to
simplify derivation. Take vectorization on both sides and we can obtain the
derivatives of ${\bm{J}}^{T}$ with respective to ${\bm{A}}$ as follows

$\displaystyle\frac{\partial{\bm{J}}^{T}}{\partial{\bm{A}}}=\left[({\bm{D}}{\bm{U}}^{-1}{\bm{B}})^{T}\otimes\gamma{\bm{D}}{\bm{U}}^{-1}\right]^{T}$

##  Appendix C Proof of Theorem 3.1

Note that at the initialization, we have

$\displaystyle\mathbb{E}\left|f_{\theta}({\bm{x}})\right|^{2}=$
$\displaystyle\mathbb{E}({\bm{z}}^{T}{\bm{b}})^{2}=\mathbb{E}\left[\sum_{r=0}^{m}\sum_{s=0}^{m}z_{r}z_{s}b_{r}b_{s}\right]$
$\displaystyle=$
$\displaystyle\frac{1}{m}\mathbb{E}\left[\sum_{r=0}^{m}z_{r}^{2}\right],\quad
b_{r}\sim\mathcal{U}\\{-1/\sqrt{m},+1\sqrt{m}\\}$ $\displaystyle=$
$\displaystyle\frac{1}{m}\mathbb{E}\|{\bm{z}}\|^{2}$ $\displaystyle\leq$
$\displaystyle\frac{1}{m}\frac{1}{(1-\gamma_{0})^{2}}\mathbb{E}\|{\bm{\phi}}\|^{2},\quad\text{Lemma~{}\ref{lemma:
forward}}$ $\displaystyle\leq$
$\displaystyle\frac{1}{m}\frac{1}{(1-\gamma_{0})^{2}}\mathbb{E}\|{\bm{W}}^{T}{\bm{x}}\|^{2},\quad\text{$\phi(\cdot)$
is $1$-Lipschitz continuous}$ $\displaystyle\leq$
$\displaystyle\frac{1}{m}\frac{1}{(1-\gamma_{0})^{2}}{\bm{x}}^{T}\left[\sum_{r=0}^{m}\mathbb{E}({\bm{w}}_{r}{\bm{w}}_{r}^{T})\right]{\bm{x}}$
$\displaystyle=$ $\displaystyle\frac{1}{(1-\gamma_{0})^{2}}$

By using Markov’s inequality, we have

$\displaystyle{\mathbb{P}}\left[\|{\bm{u}}(0)\|^{2}\geq\epsilon\right]\leq\epsilon^{-1}\mathbb{E}\|{\bm{u}}(0)\|^{2}\leq\epsilon^{-1}N(1-\gamma_{0})^{-2}=\delta.$

Thus, we obtain $\|{\bm{u}}(0)\|=\mathcal{O}\left(\sqrt{N/\delta}\right)$ with
probability at least $1-\delta$.

Now we are ready to present the convergence proof. By (Vershynin, 2018,
Theorem 4.4.5), we know $\|{\bm{A}}(0)\|\leq c\sqrt{m}$ and
$\|{\bm{W}}(0)\|\leq\sqrt{m}$ for some constant $c>0$ with probability at
least $1-\delta$. Without loss generalization, we assume $c=1$. Since we
choose $\gamma:=\gamma_{0}/\sqrt{m}$, the equilibrium point ${\bm{Z}}(0)$ is
well defined. In the following, we can show
$\|{\bm{A}}(t)\|=\mathcal{O}\left(\sqrt{m}\right)$ for all $t\geq 0$ and so
${\bm{Z}}(t)$ is well defined throughout training. Assume the results hold for
all $0\leq s\leq t$:

  * •

${\bm{A}}(s)=\mathcal{O}\left(\sqrt{m}\right)$

  * •

$\|{\bm{u}}(s)-{\bm{y}}\|^{2}\leq
e^{-\lambda_{0}t}\|{\bm{u}}(0)-{\bm{y}}\|^{2}$

We can bound $\partial L/\partial{\bm{A}}$ as follows

$\displaystyle\left\|\partial L(s)/\partial{\bm{A}}\right\|=$
$\displaystyle\left\|\frac{\gamma}{\sqrt{m}}\left[{\bm{D}}(s)({\bm{I}}_{m}\otimes{\bm{Z}}(s))\right]^{T}{\bm{Q}}(s)^{-T}\left({\bm{b}}^{T}\otimes{\bm{I}}_{n}\right)^{T}({\bm{u}}-{\bm{y}})\right\|$
$\displaystyle\leq$
$\displaystyle\frac{c}{m}\left\|{\bm{Z}}(s)\right\|_{F}\|{\bm{b}}\|\|{\bm{u}}(s)-{\bm{y}}\|$
$\displaystyle\leq$
$\displaystyle\frac{c}{m}\|{\bm{W}}\|\left\|{\bm{X}}\right\|_{F}\|{\bm{b}}\|\|{\bm{u}}(s)-{\bm{y}}\|,\quad\text{By
Lemma~{}\ref{lemma: forward}}$ $\displaystyle\leq$ $\displaystyle
c\|{\bm{X}}\|_{F}\|{\bm{u}}(s)-{\bm{y}}\|.$

Since $\|{\bm{W}}\|\leq\sqrt{m}$ and $\|{\bm{b}}\|=\sqrt{m}$, we have

$\displaystyle\|\partial L(s)/\partial{\bm{A}}\|\leq
c\|{\bm{X}}\|_{F}\|{\bm{u}}(s)-{\bm{y}}\|.$ (35)

Furthermore, we have

$\displaystyle\|{\bm{A}}(t)-{\bm{A}}(0)\|_{F}\leq$
$\displaystyle\int_{0}^{t}\left\|\partial L/\partial
A(s)\right\|ds\leq\frac{c}{\lambda_{0}}\|{\bm{X}}\|_{F}\|{\bm{u}}(0)-{\bm{y}}\|\leq\sqrt{m}$

where the last inequality is because
$m=\Omega\left(\frac{N^{2}}{\lambda_{0}^{2}\delta}\right)$. As a result, we
have

$\displaystyle\|{\bm{A}}(t)\|\leq\|{\bm{A}}(t)-{\bm{A}}(0)\|_{F}+\|{\bm{A}}(0)\|=\mathcal{O}\left(\sqrt{m}\right).$

Note that

$\displaystyle\left\|{\bm{H}}(t)-{\bm{H}}(0)\right\|$ $\displaystyle=$
$\displaystyle\frac{\gamma^{2}}{m}\|{\bm{J}}(t)({\bm{I}}_{m}\otimes{\bm{Z}}(t){\bm{Z}}(t)^{T}){\bm{J}}(t)^{T}-{\bm{J}}(0)({\bm{I}}_{m}\otimes{\bm{Z}}(0){\bm{Z}}(0)^{T}){\bm{J}}(0)^{T}\|$
$\displaystyle\leq$
$\displaystyle\frac{\gamma^{2}}{m}\|{\bm{Z}}(t)\|_{F}^{2}\|{\bm{J}}(t)-{\bm{J}}(0)\|\left\|{\bm{J}}(t)\right\|+\frac{\gamma^{2}}{m}\|{\bm{Z}}(t){\bm{Z}}(t)^{T}-{\bm{Z}}(0){\bm{Z}}(0)^{T}\|\left\|{\bm{J}}(0)\right\|\|{\bm{J}}(t)\|$
$\displaystyle+\frac{\gamma^{2}}{m}\|{\bm{Z}}(0)\|_{F}^{2}\|{\bm{J}}(t)-{\bm{J}}(0)\|\|{\bm{J}}(0)\|$

In the following, we will bound each terms.

By Lemma 3.2, we can bound $d\text{vec}\left({\bm{Z}}\right)/dt$ as follows

$\displaystyle\left\|\frac{d\text{vec}\left({\bm{Z}}\right)}{dt}\right\|=\left\|\left(\frac{\partial{\bm{Z}}}{\partial{\bm{A}}}\right)^{T}\left(-\frac{\partial
L}{\partial{\bm{A}}}\right)\right\|=\mathcal{O}\left(\|{\bm{X}}\|_{F}^{2}\|{\bm{u}}(t)-{\bm{y}}\|\right)$

Thus, we can further bound ${\bm{Z}}(t)-{\bm{Z}}(0)$ as follows

$\displaystyle\left\|{\bm{Z}}(t)-{\bm{Z}}(0)\right\|_{F}\leq$
$\displaystyle\int_{0}^{t}\left\|\frac{d\text{vec}\left({\bm{Z}}\right)}{ds}\right\|ds\leq\frac{c}{\lambda_{0}}\|{\bm{X}}\|_{F}^{2}\|{\bm{u}}(0)-{\bm{y}}\|.$

Similarly, we can bound the dynamics of ${\bm{J}}^{T}$ as follows

$\displaystyle\frac{d\text{vec}\left({\bm{J}}^{T}\right)}{dt}\leq\left\|\left[\frac{\partial{\bm{J}}^{T}}{\partial{\bm{A}}}\right]^{T}\right\|\left\|\left(\frac{\partial
L}{\partial{\bm{A}}}\right)\right\|\leq
c\|{\bm{X}}\|_{F}\|{\bm{u}}(t)-{\bm{y}}\|.$

Therefore, we can bound ${\bm{J}}(t)-{\bm{J}}(0)$ as follows

$\displaystyle\|{\bm{J}}(t)-{\bm{J}}(0)\|\leq\int_{0}^{t}\left\|\frac{d\text{vec}\left({\bm{J}}^{T}\right)}{ds}\right\|ds\leq\frac{c}{\lambda_{0}}\|{\bm{X}}\|_{F}\|{\bm{u}}(0)-{\bm{y}}\|,$

Combining equation 12, equation 13, equation 14, and Lemma A.5, we have

$\displaystyle\left\|{\bm{H}}(t)-{\bm{H}}(0)\right\|\leq\frac{c\|{\bm{X}}\|_{F}^{3}\|{\bm{u}}(0)-{\bm{y}}\|}{\lambda_{0}\sqrt{m}}\leq\frac{cN^{2}}{\lambda_{0}\sqrt{\delta}\sqrt{m}}\leq\lambda_{0}/2,$

where the last inequality follows from
$m=\Omega\left(\frac{N^{4}}{\lambda_{0}^{4}\delta}\right)$. By Weyl’s
inequality, we have

$\displaystyle\lambda_{\min}\left[{\bm{H}}(t)\right]\geq\lambda_{\min}\left[{\bm{H}}(0)\right]-\left\|{\bm{H}}(t)-{\bm{H}}(0)\right\|\geq\lambda_{0}/2.$

Therefore, we have

$\displaystyle\frac{d}{dt}\|{\bm{u}}-{\bm{y}}\|^{2}=$ $\displaystyle
2({\bm{u}}-{\bm{y}})^{T}\frac{d{\bm{u}}}{dt}$ $\displaystyle=$
$\displaystyle-2({\bm{u}}-{\bm{y}})^{T}{\bm{H}}(t)({\bm{u}}-{\bm{y}})$
$\displaystyle\geq$ $\displaystyle-\lambda_{0}\|{\bm{u}}-{\bm{y}}\|^{2}.$

By solving the ordinary differential equation above, we have

$\displaystyle\|{\bm{u}}(t)-{\bm{y}}\|^{2}\leq
e^{-\lambda_{0}t}\|{\bm{u}}(0)-{\bm{y}}\|^{2}.$

##  Appendix D Proof of Lemma 4.2

The dynamics of ${\bm{y}}-\hat{{\bm{u}}}(t)$ is given by

$\displaystyle\frac{d\left({\bm{y}}-\hat{{\bm{u}}}\right)}{dt}=-{\bm{H}}(0)({\bm{y}}-\hat{{\bm{u}}}(t))$

Solving the ordinary differential equation above yields

$\displaystyle{\bm{y}}-\hat{{\bm{u}}}(t)=e^{\int_{0}^{t}-{\bm{H}}(0)ds}({\bm{y}}-\hat{{\bm{u}}}(0))=e^{-{\bm{H}}(0)t}({\bm{y}}-\hat{{\bm{u}}}(0)).$

Let ${\bm{H}}(0)={\bm{Q}}{\bm{\Lambda}}{\bm{Q}}^{T}$ be the eigenvalue
decomposition of ${\bm{H}}(0)$, then we have

$\displaystyle\hat{f}_{t}({\bm{x}}_{0})=$ $\displaystyle
f_{0}({\bm{x}}_{0})+\int_{0}^{t}k_{0}({\bm{x}}_{0},{\bm{X}})({\bm{y}}-\hat{{\bm{u}}}(s))ds$
$\displaystyle=$ $\displaystyle
f_{0}({\bm{x}}_{0})+k_{0}({\bm{x}}_{0},{\bm{X}})\int_{0}^{t}e^{-{\bm{H}}(0)s}({\bm{y}}-{\bm{u}}(0))ds$
$\displaystyle=$ $\displaystyle
f_{0}({\bm{x}}_{0})+k_{0}({\bm{x}}_{0},{\bm{X}}){\bm{Q}}\left(\int_{0}^{t}e^{-{\bm{\Lambda}}s}ds\right){\bm{Q}}^{T}({\bm{y}}-{\bm{u}}(0))$

Let $t\rightarrow\infty$, then we have

$\displaystyle\hat{f}_{\infty}({\bm{x}}_{0})=f_{0}({\bm{x}}_{0})+k_{0}({\bm{x}}_{0},{\bm{X}}){\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0)).$

Let ${\bm{\alpha}}:={\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))$, then we have

$\displaystyle
B^{2}:={\bm{\alpha}}^{T}{\bm{H}}(0){\bm{\alpha}}=({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0)).$

Note that

$\displaystyle k_{0}({\bm{x}},{\bm{x}})=\left\langle\frac{\partial
f_{0}({\bm{x}})}{\partial{\bm{A}}},\frac{\partial
f_{0}({\bm{x}})}{\partial{\bm{A}}}\right\rangle\leq\left\|\frac{\partial
f_{0}({\bm{x}})}{\partial{\bm{A}}}\right\|^{2}=\mathcal{O}\left(\|{\bm{x}}\|^{2}\right).$
(36)

Therefore, the Rademacher complexity of $\mathcal{F}_{B}$ is given by

$\displaystyle\mathfrak{R}_{S}(\mathcal{F}_{B})\leq\frac{B}{n}\sqrt{\mathbf{tr}({\bm{H}}(0))}\leq
c\sqrt{\frac{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))}{n}}.$

##  Appendix E Proof of Lemma 4.3

The dynamics of ${\bm{u}}(t)-\hat{{\bm{u}}}(t)$ is given by

$\displaystyle\frac{d({\bm{u}}-\hat{{\bm{u}}})}{dt}=$
$\displaystyle{\bm{H}}(t)({\bm{y}}-{\bm{u}})-{\bm{H}}(0)({\bm{y}}-\hat{{\bm{u}}})$
$\displaystyle=$
$\displaystyle\left[{\bm{H}}(t)-{\bm{H}}(0)\right]({\bm{y}}-{\bm{u}})-{\bm{H}}(0)({\bm{u}}-\hat{{\bm{u}}})$

Then we have

$\displaystyle\frac{d}{dt}\|{\bm{u}}-\hat{{\bm{u}}}\|^{2}=$ $\displaystyle
2({\bm{u}}-\hat{{\bm{u}}})\frac{d({\bm{u}}-\hat{{\bm{u}}})}{dt}$
$\displaystyle=$ $\displaystyle
2({\bm{u}}-\hat{{\bm{u}}})^{T}\left[{\bm{H}}(t)-{\bm{H}}(0)\right]({\bm{y}}-{\bm{u}})-2({\bm{u}}-\hat{{\bm{u}}})^{T}{\bm{H}}(0)({\bm{u}}-\hat{{\bm{u}}})$
$\displaystyle\leq$ $\displaystyle
2\|{\bm{u}}-\hat{{\bm{u}}}\|\|{\bm{H}}(t)-{\bm{H}}(0)\|\|{\bm{y}}-{\bm{u}}\|-2\lambda_{0}\|{\bm{u}}-\hat{{\bm{u}}}\|^{2}$

where we use $\lambda_{\min}[{\bm{H}}(0)]\geq\lambda_{0}>0$. By using
Gronwall’s inequality, we have

$\displaystyle\|{\bm{u}}(t)-\hat{{\bm{u}}}(t)\|^{2}\leq$
$\displaystyle\int_{0}^{t}2\|{\bm{u}}(s)-\hat{{\bm{u}}}(s)\|\|{\bm{H}}(s)-{\bm{H}}(0)\|\|{\bm{y}}-{\bm{u}}(s)\|ds+\int_{0}^{t}(-2\lambda_{0})\|{\bm{u}}(s)-\hat{{\bm{u}}}(s)\|^{2}ds$
$\displaystyle\leq$
$\displaystyle\int_{0}^{t}2\|{\bm{u}}(s)-\hat{{\bm{u}}}(s)\|\|{\bm{H}}(s)-{\bm{H}}(0)\|\|{\bm{y}}-{\bm{u}}(s)\|ds\cdot\exp\left(\int_{0}^{t}-(2\lambda_{0})ds\right)$
$\displaystyle=$
$\displaystyle\int_{0}^{t}2\|{\bm{u}}(s)-\hat{{\bm{u}}}(s)\|\|{\bm{H}}(s)-{\bm{H}}(0)\|\|{\bm{y}}-{\bm{u}}(s)\|ds\cdot
e^{-2\lambda_{0}t}$ $\displaystyle\leq$
$\displaystyle\frac{cN^{2}}{\lambda_{0}\sqrt{\delta}\sqrt{m}}\int_{0}^{t}\|{\bm{u}}(s)-\hat{{\bm{u}}}(s)\|\|{\bm{y}}-{\bm{u}}(s)\|ds\cdot
e^{-2\lambda_{0}t}$ $\displaystyle\leq$
$\displaystyle\frac{cN^{2}}{\lambda_{0}\sqrt{\delta}\sqrt{m}}\int_{0}^{t}\|{\bm{y}}-\hat{{\bm{u}}}(s)\|\|{\bm{y}}-{\bm{u}}(s)\|^{2}ds\cdot
e^{-2\lambda_{0}t}$ $\displaystyle\leq$
$\displaystyle\frac{cN^{2}}{\lambda_{0}\sqrt{\delta}\sqrt{m}}\left[\int_{0}^{t}e^{-2\lambda_{0}s}ds\right]\|{\bm{y}}-{\bm{u}}(0)\|^{2}\cdot
e^{-2\lambda_{0}t}$ $\displaystyle\leq$
$\displaystyle\frac{cN^{3}}{\lambda_{0}^{2}\delta^{2}\sqrt{m}}\cdot
e^{-2\lambda_{0}t}$

Thus, we have

$\displaystyle\|{\bm{u}}(t)-\hat{{\bm{u}}}(t)\|=\mathcal{O}\left(\frac{N^{3/2}}{\lambda_{0}\delta
m^{1/4}}\cdot e^{-\lambda_{0}t}\right).$

##  Appendix F Proof of Lemma 4.4

Note that

$\displaystyle\frac{d(f_{t}({\bm{x}}_{0})-\hat{f}_{t}({\bm{x}}_{0}))}{dt}=$
$\displaystyle
k_{t}({\bm{x}}_{0},{\bm{X}})({\bm{y}}-{\bm{u}}(t))-k_{0}({\bm{x}}_{0},{\bm{X}})({\bm{y}}-\hat{{\bm{u}}}(t))$
$\displaystyle=$
$\displaystyle(k_{t}({\bm{x}}_{0},{\bm{X}})-k_{0}({\bm{x}}_{0},{\bm{X}}))({\bm{y}}-{\bm{u}}(t))+k_{0}({\bm{x}}_{0},{\bm{X}})(\hat{{\bm{u}}}(t)-{\bm{u}}(t))$

In the following, we will bound each terms. Note that

$\displaystyle
k_{t}({\bm{x}}_{i},{\bm{x}}_{j})-k_{0}({\bm{x}}_{i},{\bm{x}}_{j})$
$\displaystyle=$
$\displaystyle\frac{\gamma^{2}}{m}\left[({\bm{J}}_{i}(t){\bm{J}}_{j}(t)^{T})({\bm{z}}_{i}(t)^{T}{\bm{z}}_{j}(t))-({\bm{J}}_{i}(){\bm{J}}_{j}()^{T})({\bm{z}}_{i}(0)^{T}{\bm{z}}_{j}(0))\right]$
$\displaystyle=$
$\displaystyle\frac{\gamma^{2}}{m}\left[{\bm{J}}_{i}(t)-{\bm{J}}_{i}(0)\right]{\bm{J}}_{j}(t)^{T}({\bm{z}}_{i}(t)^{T}{\bm{z}}_{j}(t))+\frac{\gamma^{2}}{m}{\bm{J}}_{i}(0)\left[{\bm{J}}_{j}(t)-{\bm{J}}_{j}(0)\right]^{T}({\bm{z}}_{i}(t)^{T}{\bm{z}}_{j}(t))$
$\displaystyle+$
$\displaystyle\frac{\gamma^{2}}{m}{\bm{J}}_{i}(0){\bm{J}}_{j}(0)^{T}({\bm{z}}_{i}(t)-{\bm{z}}_{i}(0))^{T}{\bm{z}}_{j}(t)+\frac{\gamma^{2}}{m}{\bm{J}}_{i}(0){\bm{J}}_{j}(0)^{T}{\bm{z}}_{i}(0)^{T}[{\bm{z}}_{j}(t)-{\bm{z}}_{j}(0)]$

where

$\displaystyle{\bm{J}}_{i}(t):=$
$\displaystyle{\bm{b}}^{T}{\bm{Q}}_{i}(t)^{-1}{\bm{D}}_{i}(t),$
$\displaystyle{\bm{Q}}_{i}(t):=$
$\displaystyle{\bm{I}}_{m}-\gamma{\bm{D}}_{i}(t){\bm{A}}(t)^{T},$
$\displaystyle{\bm{D}}_{i}(t):=$ $\displaystyle\mathop{\bf
diag}\left[\sigma^{\prime}\left(\gamma{\bm{A}}^{T}{\bm{z}}_{i}(t)+{\bm{\phi}}_{i}\right)\right].$

By equation 13, equation 14, equation 12, and Lemma A.5, we have

$\displaystyle\left|k_{t}({\bm{x}}_{i},{\bm{x}}_{j})-k_{0}({\bm{x}}_{i},{\bm{x}}_{j})\right|=\mathcal{O}\left(\frac{\sqrt{N}}{\lambda_{0}\sqrt{\delta}\sqrt{m}}\right)$
(37)

where we also use the data assumption $\|{\bm{x}}_{i}\|=\|{\bm{x}}_{j}\|=1$
and $\|{\bm{u}}(0)-{\bm{y}}\|=\mathcal{O}\left(\sqrt{N/\delta}\right)$. Thus,
we obtain

$\displaystyle\left\|k_{t}({\bm{x}}_{0},{\bm{X}})-k_{0}({\bm{x}}_{0},{\bm{X}})\right\|=\mathcal{O}\left(\frac{N}{\lambda_{0}\sqrt{\delta}\sqrt{m}}\right).$
(38)

Combining equation 38, equation 36, and equation 23 together, we have

$\displaystyle\left|\frac{d(f_{t}({\bm{x}}_{0})-\hat{f}_{t}({\bm{x}}_{0}))}{dt}\right|=\mathcal{O}\left(\frac{N^{3/2}}{\lambda_{0}\delta\sqrt{m}}e^{-\lambda_{0}t}\right)+\mathcal{O}\left(\frac{N^{2}}{\lambda_{0}\delta
m^{1/4}}e^{-\lambda_{0}t}\right)=\mathcal{O}\left(\frac{N^{2}}{\lambda_{0}\delta
m^{1/4}}e^{-\lambda_{0}t}\right)$

Therefore, we have

$\displaystyle\left|f_{t}({\bm{x}}_{0})-\hat{f}_{t}({\bm{x}}_{0})\right|\leq\int_{0}^{t}\left|\frac{d(f_{s}({\bm{x}}_{0})-\hat{f}_{s}({\bm{x}}_{0}))}{ds}\right|ds=\mathcal{O}\left(\frac{n^{2}}{\lambda_{0}^{2}\delta
m^{1/4}}\right)$ (39)

Since $m=\Omega(\lambda_{0}^{-8}\delta^{-1}N^{10})$, we have

$\displaystyle\left|f_{t}({\bm{x}}_{0})-\hat{f}_{t}({\bm{x}}_{0})\right|=\mathcal{O}\left(\sqrt{\frac{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))}{N}}\right)$

##  Appendix G Proof of Theorem 4.1

Denote
$C:=\sqrt{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))/N}$.
Let $\ell$ be a $1$-Lipschitz-continuous loss function, we have

$\displaystyle\left|\ell(y_{0},f_{\infty}({\bm{x}}_{0}))-\ell(y_{0},\tilde{f}_{\infty}({\bm{x}}_{0}))\right|\leq\left|f_{\infty}({\bm{x}}_{0})-\tilde{f}_{\infty}({\bm{x}}_{0})\right|\leq
C.$

This implies that

$\displaystyle\ell(y_{0},f_{\infty}({\bm{x}}_{0}))\leq\ell(y_{0},\tilde{f}_{\infty}({\bm{x}}_{0}))+C.$
(40)

Since $({\bm{x}}_{0},y_{0})$ is an arbitrary data point, taking expectation
the inequality still holds:

$\displaystyle R(f_{\infty})\leq R(\tilde{f}_{\infty})+C.$

As a consequence, it follows from the Rademacher complexity theorem A.4 that
we have

$\displaystyle R(f_{\infty})\leq$ $\displaystyle R(\tilde{f}_{\infty})+C$
$\displaystyle\leq$
$\displaystyle\hat{R}_{S}(\tilde{f}_{\infty})+\mathfrak{R}_{S}(\mathcal{F}_{B})+\sqrt{\frac{\log(1/\delta)}{N}}+C$
$\displaystyle=$
$\displaystyle\mathfrak{R}_{S}(\mathcal{F}_{B})+\sqrt{\frac{\log(1/\delta)}{N}}+C,\quad\text{(i)}$
$\displaystyle\leq$
$\displaystyle\sqrt{\frac{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))}{N}}+\sqrt{\frac{\log(1/\delta)}{N}}+C$
$\displaystyle\leq$
$\displaystyle\mathcal{O}\left(\sqrt{\frac{({\bm{y}}-{\bm{u}}(0))^{T}{\bm{H}}(0)^{-1}({\bm{y}}-{\bm{u}}(0))}{N}}\right)+\sqrt{\frac{\log(1/\delta)}{N}},$

where $(i)$ follows from $\tilde{f}_{\infty}({\bm{X}})={\bm{y}}$ and we obtain
$\hat{R}_{S}(\tilde{f}_{\infty})=0$.


# High-efficiency quantum rectification via nonlinear Hall effect

Yugo Onishi  Department of Physics, Massachusetts Institute of Technology,
Cambridge, MA 02139, USA  Liang Fu  Department of Physics, Massachusetts
Institute of Technology, Cambridge, MA 02139, USA

###### Abstract

Noncentrosymmetric quantum materials can convert AC input current into DC
transverse current at zero magnetic field through the nonlinear Hall effect.
We analyze the AC-DC power conversion efficiency of such “Hall rectifier” and
suggest its application in wireless charging and energy harvesting. Our key
observation is that the development of Hall field results in a change of
longitudinal conductivity in the Hall rectifier. This feedback mechanism
balances the input power and the output power and hence is crucial to
understanding the power transfer from source to load. Since the nonlinear Hall
current is always perpendicular to the electric field and therefore does not
by itself generate Joule heating, the efficiency can in principle approach
$100\text{\,}\mathrm{\char 37\relax}$. We demonstrate high power conversion
efficiency at large nonlinear Hall angle and optimized load resistance.

Technological applications of electromagnetic waves is essential to modern
society. The development and commercialization of 5G wireless network has
created a lot of interest in exploring emerging applications in millimeter
wave (mmWave) and terahertz spectrum [1, 2]. Owing to the shorter wavelength
of millimeter waves, compact-size antennas can receive power with a higher
efficiency, promising wireless powered Internet of Things (IoT) devices [1,
3]. Conversion of THz waves into usable power is important for energy
harvesting, since the solar energy reaching the earth is re-radiated in the
form of infrared light (20-40 THz) which is wasted.

A key ingredient for mm-Wave and THz technology is the conversion of
electromagnetic waves into direct current (DC) electricity. This can be
achieved with a diode combined with an antenna, called rectenna. Using its
nonlinear $I-V$ characteristic, the diode rectifies the alternating current
induced in the antenna by electromagnetic fields (EMFs) into direct current,
to produce DC power. Rectenna have been extensively investigated for wireless
power transmission via microwave radiation [4, 5]. While the power conversion
efficiency reaches as high as $80\text{\,}\mathrm{\char 37\relax}$ at
frequencies $\lesssim$10\text{\,}\mathrm{G}\mathrm{H}\mathrm{z}$$ [4], the
realization of mmWave and terahertz rectenna is highly challenging. A main
difficulty is to design diodes operating efficiently at such high frequencies,
as conventional diodes have a cutoff frequency determined by RC time constant
of semiconductor junctions. The RF-DC conversion efficiency of a rectenna with
GaAs Schottky diode is only around $2\text{\,}\mathrm{\char 37\relax}$ at
$300$GHz [6]. Another major limitation is the reduced performance of
conventional diodes at low input power for ambient energy harvesting [7, 8].

Recently, a new physical principle for current rectification has been proposed
based on the intrinsic, nonreciprocal response of noncentrosymmetric quantum
materials at zero magnetic field [9]. In particular, a transverse current can
be induced in a noncentrosymmetric conductor by an applied electric field to
the second order, known as the nonlinear Hall effect (NLHE) [10, 11, 12, 13].
When an AC electric field is applied at frequency $\omega$, NLHE
simultaneously gives rise to an AC current at frequency $2\omega$ and a DC
current. NLHE enables the generation of DC electricity from AC input by
utilizing quantum electronic properties of noncentrosymmetric materials,
without invoking p-n or metal-semiconductor junction. This quantum
rectification process can also be viewed as a special type of linear
photogalvanic effect due to Drude absorption rather than interband transition
[14, 15, 16, 17, 18, 19, 20, 21, 22, 23].

In this work, we study the power conversion efficiency of noncentrosymmetric
conductors that convert AC input power into DC power through NLHE. To this
end, we consider a circuit that consists of an antenna, a noncentrosymmetric
conductor and an external load. The electromagnetic radiation received by the
antenna induces an AC current, which flows into a noncentrosymmetric
conductor. The conductor (“Hall rectifier”) induces a DC transverse current
through NLHE, and the current flows into an external load to do work. The
rectification based on NLHE can realize high efficiency in broad frequency
range up to the terahertz regime. The high conversion efficiency is made
possible by the non-dissipative nature of NLHE, which by itself is free from
Joule heating. Importantly, in the power conversion through NLHE, the
development of Hall field necessarily results in a change of longitudinal
conductivity in the Hall rectifier. This feedback mechanism balances the input
power and the output power, and hence is crucial in understanding the energy
flow from the power source to the external load. Remarkably, we show that the
maximum power conversion efficiency reaches nearly $100\text{\,}\mathrm{\char
37\relax}$ when the nonlinear Hall angle is sufficiently large and the load
resistance is optimized in proportion to the internal resistance.

A key feature of NLHE is that the Hall current is always transverse to the
electric field ${\bf\it E}$, and therefore does not by itself produce Joule
heating. When an electric field ${\bf\it E}$ is applied to a
noncentrosymmetric conductor possessing Berry curvature in ${\bf\it k}$ space,
the anomalous velocity leads to a Hall current ${\bf\it j}^{H}$ transverse to
${\bf\it E}$:

$\displaystyle{{\bf\it j}}^{H}$
$\displaystyle=\frac{e^{2}}{\hbar}\sum_{{\bf\it k}}\left({\bf\it
E}\times{\bf\it\Omega}_{{\bf\it E}}({\bf\it k})\right)f_{{\bf\it E}}({{\bf\it
k}}).$ (1)

Here $f_{{\bf\it E}}({\bf\it k})$ is the distribution function in the current
carrying state driven by the electric field, and ${\bf\it\Omega}_{{\bf\it
E}}({\bf\it k})$ is the Berry curvature in the presence of the electric field
${\bf\it E}$. The anomalous velocity due to Berry curvature results in NLHE in
several ways. In the presence of time reversal symmetry, the leading
contribution to the Hall current ${{\bf\it j}}^{H}$ is second order in
${\bf\it E}$, as it comes from the product of electric field induced change in
distribution function and the anomalous velocity, both of which are at first
order in ${\bf\it E}$. This contribution can be described with the Berry
curvature dipole [10]. Another contribution can arise from the change of Berry
curvature due to external electric field. The change of
${\bf\it\Omega}_{{\bf\it E}}({\bf\it k})$ at first order in ${\bf\it E}$
contributes to second-order NLHE through Eq. (1) with unperturbed distribution
function. This contribution is present in time-reversal and inversion breaking
systems and comes from the quantum metric [13]. Since its theoretical
prediction [10, 13], the NLHE has been intensively studied and experimentally
observed in various quantum materials [24, 11, 25, 12, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45]. It is the dominant
mechanism for second-order nonreciprocal transport at the leading order in the
relaxation time $\tau$.

To make the discussion concrete, let us consider a noncentrosymmetric crystal
which is invariant under reflection $x\rightarrow-x$ and $z\rightarrow-z$, and
has a polar axis along $y$ direction. Based on symmetry consideration, the
current density ${{\bf\it j}}=(j_{x},j_{y})$ in this polar crystal induced by
an electric field ${\bm{E}}=(E_{x},E_{y})$ can be expanded in powers of
$\bm{E}$:

$\displaystyle j_{x}$ $\displaystyle=$
$\displaystyle\sigma_{xx}E_{x}+\sigma_{xyx}E_{x}E_{y}+\sigma_{xxxx}E_{x}^{3}+...,$
(2) $\displaystyle j_{y}$ $\displaystyle=$
$\displaystyle\sigma_{yy}E_{y}+\sigma_{yxx}E_{x}^{2}+...$ (3)

Here $\sigma_{xx}$ and $\sigma_{yy}$ are the linear conductivities in $x$ and
$y$ direction respectively. $\sigma_{yxx}$ and $\sigma_{xyx}$ are symmetry-
allowed second-order conductivities. $\sigma_{xxxx}$ is a third-order
conductivity which describes the change of conductivity proportional to
$E_{x}^{2}$. Throughout this work, we consider the case that the externally
applied electric field $E_{x}$ is along $x$ direction. Then, Eq. (2) and (3)
are exact up to the third order in $E_{x}$ as we will show below. All other
symmetry allowed terms, such as the $\sigma_{yyy}E_{y}^{2}$ term in $j_{y}$
[9], are at least fourth order in $E_{x}$. We consider the case that $E_{x}$
is not too large, so that additional terms not shown explicitly in Eq (2) and
(3) can be neglected.

Note that the second-order current from the nonlinear Hall effect is always
transverse to the electric field regardless of its direction, as shown in Eq
(1). It then follows from ${\bf\it j}^{H}\cdot{\bf\it
E}=\sigma_{xyx}E_{x}^{2}E_{y}+\sigma_{yxx}E_{x}^{2}E_{y}=0$ that

$\displaystyle\sigma_{yxx}=-\sigma_{xyx}.$ (4)

Importantly, we note that $\sigma_{xyx}$ also represents the change of
conductivity along the $x$ direction by the transverse electric field $E_{y}$,
which may be called the “bulk electric field effect”. It is thus remarkable
that NLHE also provides a direct mechanism for the bulk electric field effect.

The power dissipated in the nonlinear Hall conductor is given by

$\displaystyle
Q={\bm{j}}\cdot{\bm{E}}=\sigma_{xx}E_{x}^{2}+\sigma_{yy}E_{y}^{2}+\sigma_{xxxx}E_{x}^{4}.$
(5)

$Q$ must be positive for any $\bm{E}$. When $\sigma_{xxxx}\geq 0$, the
requirement is automatically satisfied. When $\sigma_{xxxx}<0$, in order to
satisfy $Q>0$, it becomes essential to include additional terms in the
current-voltage relation for $E_{x}^{2}>\sigma_{xx}/(-\sigma_{xxxx})$. For
simplicity, our analysis below will mainly focus on the case
$\sigma_{xxxx}\geq 0$.

Figure 1: The longitudinal and Hall voltage $V_{x},V_{y}$ as a function of the
applied electric current $I_{x}$ determined by Eqs. (4), (6) and (7) with
$V_{x0}=L_{x}E_{0}$, $V_{y0}=L_{y}\eta E_{0}^{2}$, and
$I_{x0}=(L_{y}L_{z}/L_{x})\sigma_{xx}V_{x0}$. Due to the bulk electric field
effect ($\sigma_{xyx}$), $I_{x}$-$V_{x}$ deviates from linear relation (blue
dashed line), and hence the $I_{x}$-$V_{y}$ curve deviates from the quadratic
relation (orange dashed curve) as well. Note that the $V_{x}$-$V_{y}$ curve is
still quadratic in the same regime, as shown in the bottom-right inset.

When an applied voltage $V_{x}=E_{x}L_{x}$ drives a longitudinal current
$I_{x}=j_{x}L_{y}L_{z}$ through a nonlinear Hall conductor of dimension
$L_{x}\times L_{y}\times L_{z}$, a transverse open-circuit voltage
$V_{y}=-E_{y}L_{y}$ is induced by charge accumulation on the sides of the
conductor due to NLHE. In order to obey the open-circuit condition with zero
net current in $y$ direction, the drift current $j_{d}=\sigma_{yy}E_{y}$
driven by $E_{y}$ must exactly oppose and cancel the nonlinear Hall current
$j_{H}=\sigma_{yxx}E_{x}^{2}$ driven by $E_{x}$. Solving Eq. (3) under the
condition $j_{y}=0$ yields a transverse electric field:

$\displaystyle E_{y}=-\eta E_{x}^{2},\;{\rm
with}\;\eta=\frac{\sigma_{yxx}}{\sigma_{yy}}.$ (6)

The quadratic dependence of the induced transverse voltage $V_{y}$ on the
applied voltage $V_{x}$ is the direct manifestation of second-order nonlinear
Hall effect. At $E_{x}=\eta^{-1}$, we have $E_{y}=E_{x}$ so that the nonlinear
Hall angle is $45\text{\,}\mathrm{\SIUnitSymbolDegree}$.

Interestingly, the induced transverse electric field $E_{y}$ in turn modifies
the conductivity along $x$ direction through the $\sigma_{xyx}$ term. In
addition to the $\sigma_{xxxx}$ term, this feedback effect causes the
deviation of the longitudinal current-voltage relation from Ohm’s law.
Substituting Eqs. (6) into (2) yields the longitudinal current-voltage
relation:

$\displaystyle j_{x}$ $\displaystyle=$
$\displaystyle\sigma_{xx}E_{x}+\quantity(\sigma_{xxxx}+\frac{-\sigma_{xyx}\sigma_{yxx}}{\sigma_{yy}})E_{x}^{3}.$
(7) $\displaystyle\equiv$
$\displaystyle\sigma_{xx}E_{x}\quantity(1+\frac{E^{2}_{x}}{E^{2}_{0}}).$

Note that NLHE featuring $\sigma_{xyx}=-\sigma_{yxx}$ always leads to a
decrease of $E_{x}$ induced by the applied current $j_{x}$, i.e., effectively
decreases the longitudinal resistivity, which is opposite to Joule heating
effect in metals. We defined a characteristic field strength
$E_{0}=(\sigma_{xxxx}/\sigma_{xx}-\sigma_{yxx}\sigma_{xyx}/\sigma_{xx}\sigma_{yy})^{-\frac{1}{2}}$,
at which the resistivity operationally defined as $E_{x}/j_{x}$ is half of the
linear response resistivity. Fig. 1 shows the longitudinal current and
transverse voltage as a function of input voltage, as determined by Eq. (6)
and Eq. (7).

The nonlinear current-voltage characteristic enables frequency conversion.
When an AC electric field $E_{x}(t)=E_{x}\cos(\omega t)$ is applied to the
conductor, a time-dependent transverse voltage is generated. For $\omega\ll
1/\tau$ where $\tau$ is the scattering time of the conductor, $E_{y}(t)$ is
simply determined by the instantaneous $E_{x}(t)$ through the quadratic
relation Eq. (6):

$\displaystyle E_{y}(t)=-\eta E^{2}_{x}(t)=-\frac{\eta
E_{x}^{2}}{2}(1+\cos(2\omega t)),$ (8)

which consists of a $2\omega$ component and a DC component. A
noncentrosymmetric conductor that generates a transverse DC voltage from the
applied AC current or electric field through NLHE can be called “Hall
rectifier”. The coefficient $\eta$, which measures the ratio of DC output
voltage to AC input voltage, is an intrinsic material property. We call $\eta$
the “Hall power”, in analogy to thermopower that measures the voltage
generated by temperature gradient.

Figure 2: Schematic figure of rectenna based on nonlinear Hall effect. Antenna
receives electromagnetic fields converting them into an AC electric current.
The produced AC electric current flows into the conductor, where the AC
current $I_{x}(t)$ is rectified into a DC current $I_{y}$ via NLHE along with
an AC current with frequency $2\omega$. The induced DC current and AC current
in $y$ direction flow into the external load $R_{L}$ and the power
$W_{L}^{\mathrm{dc}}$ and $W_{L}^{\mathrm{ac}}$ are delivered.

The Hall rectifier can be utilized to deliver electric power to external load.
Consider a setup shown in Fig. 2, where the Hall rectifier is connected to a
load resistance $R_{L}$. Input power $P_{\mathrm{in}}$ flows into the device
through the applied current $I_{x}$ or electric field $E_{x}$ in
$x$-direction, and some of the input energy is transferred to the external
load via the current $I_{y}$ to do useful work. The output power delivered to
the load depends on both the nonlinear current-voltage characteristic of the
Hall rectifier and the load resistance. To calculate the output power, we note
that the voltage across the load $V_{y}=-L_{y}E_{y}$ is related to $R_{L}$ and
$I_{y}$ as $V_{y}=R_{L}I_{y}$, or equivalently, $j_{y}=-\sigma_{L}E_{y}$ with
$\sigma_{L}\equiv L_{y}/(L_{x}L_{z}R_{L})$. In the case of DC input, one
obtains an expression for $E_{y}$ by solving this equation together with Eq.
(3) as

$\displaystyle E_{y}$
$\displaystyle=-\frac{\sigma_{yxx}}{\sigma_{yy}+\sigma_{L}}E_{x}^{2}.$ (9)

Substituting Eq (9) into (2), the current in $x$-direction is obtained:

$\displaystyle j_{x}$
$\displaystyle=\sigma_{xx}E_{x}+\quantity(\sigma_{xxxx}-\frac{\sigma_{xyx}\sigma_{yxx}}{\sigma_{yy}+\sigma_{L}})E_{x}^{3}.$
(10)

The relation between $j_{x}$ and $E_{x}$ reduces to the open circuit case (7)
at $\sigma_{L}\to 0$ $(R_{L}\to\infty)$.

When the applied electric field is an AC field $E_{x}(t)=E_{x}\cos(\omega t)$
with $\omega\tau\ll 1$, the induced longitudinal current $j_{x}(t)$ and
transverse voltage $E_{y}(t)$ are also time-dependent and given by Eq (9) and
Eq (10) with the instantaneous field $E_{x}(t)$. The transverse voltage
$V_{y}(t)\propto\cos^{2}(\omega t)$ now contains both a DC component and an AC
component at frequency $2\omega$. Thus, the electric power transferred to the
load consists of both the DC power and the AC power:
$W_{L}=W_{L}^{\mathrm{dc}}+W_{L}^{\mathrm{ac}}$ with

$\displaystyle W_{L}^{\mathrm{dc}}$
$\displaystyle=V_{y}^{\mathrm{dc}}I_{y}^{\mathrm{dc}}=-\Omega
E_{y}^{\mathrm{dc}}j_{y}^{\mathrm{dc}},$ (11) $\displaystyle
W_{L}^{\mathrm{ac}}$
$\displaystyle=\overline{V_{y}^{\mathrm{ac}}I_{y}^{\mathrm{ac}}}=-\Omega\overline{E_{y}^{\mathrm{ac}}j_{y}^{\mathrm{ac}}},$
(12) $\displaystyle P_{\mathrm{in}}$
$\displaystyle=\overline{V_{x}I_{x}}=\Omega\overline{E_{x}j_{x}},$ (13)

where $\Omega=L_{x}L_{y}L_{z}$ is the volume of the device. $\overline{O(t)}$
represents the time average of $O(t)$. $E_{y}^{\mathrm{dc}}$ and
$j_{y}^{\mathrm{dc}}$ are the DC components of $E_{y}$ and $j_{y}$, and
similar for $E_{y}^{\mathrm{ac}}$ and $j_{y}^{\mathrm{ac}}$. Then one can
calculate $W_{L}^{\mathrm{dc}}$, $W_{L}^{\mathrm{ac}}$, and $P_{\mathrm{in}}$
as

$\displaystyle W_{L}^{\mathrm{dc}}$
$\displaystyle=\Omega\frac{\sigma_{L}\sigma_{yxx}^{2}}{4(\sigma_{yy}+\sigma_{L})^{2}}E_{x}^{4},$
(14) $\displaystyle W_{L}^{\mathrm{ac}}$
$\displaystyle=\frac{1}{2}W_{L}^{\mathrm{dc}},$ (15) $\displaystyle
P_{\mathrm{in}}$
$\displaystyle=\Omega\quantity[\frac{1}{2}\sigma_{xx}E_{x}^{2}+\frac{3}{8}\quantity(\sigma_{xxxx}+\frac{-\sigma_{xyx}\sigma_{yxx}}{\sigma_{yy}+\sigma_{L}})E_{x}^{4}].$
(16)

These expressions for input and output power, derived from general current-
voltage relation Eqs. (2) and (3) for polar conductors, are exact up to the
fourth order in the applied electric field $E_{x}$. The inclusion of
additional terms in the current voltage relation will only add terms of sixth
or higher order in $E_{x}$. Our analysis is applicable when $E_{x}$ is not too
large so that higher-order corrections to the input and output powers are
small.

The bulk electric field effect $\sigma_{xyx}$ plays an essential role in the
transfer of input energy through the Hall rectifier to the load. It represents
the additional input energy from the applied electric field $E_{x}$ to do
external work on $R_{L}$ through the Hall voltage $-E_{y}L_{y}$. This can be
seen when one considers changing the load resistance $R_{L}$ from $0$ (short-
circuit) to a small finite value, while holding the applied electric field
$E_{x}$ fixed. When $R_{L}=0$, the output power $I_{y}^{2}R_{L}$ is zero
despite the presence of a short-circuit transverse current
$j_{y}=\sigma_{yxx}E^{2}_{x}$. Once $R_{L}$ becomes nonzero, an electric field
$\delta E_{y}$ develops in the conductor, resulting in a voltage $-\delta
E_{y}L_{y}$ across the load which does useful work

$\displaystyle\delta W_{L}/\Omega=-\delta
E_{y}j_{y}=-\sigma_{yxx}E^{2}_{x}\delta E_{y}.$ (17)

At the same time, due to the bulk electric field effect $\sigma_{xyx}$,
$\delta E_{y}$ results in an increase in the input current $\delta
j_{x}=\sigma_{xyx}E_{x}\delta E_{y}$, therefore increasing the input power by

$\displaystyle\delta P_{\mathrm{in}}/\Omega=\delta
j_{x}E_{x}=\sigma_{xyx}E^{2}_{x}\delta E_{y}.$ (18)

Remarkably, thanks to the relation $\sigma_{xyx}=-\sigma_{yxx}$ in NLHE, we
find $\delta W_{L}=\delta P_{\mathrm{in}}$, i.e., the output power produced by
small $\delta E_{y}$ is exactly compensated by the increase of input power.
Similar consideration shows $\delta W_{L}=\delta P_{\mathrm{in}}$ when $R_{L}$
is changed from infinite (open-circuit) to a large finite value while the
applied electric current $I_{x}$ is kept constant.

In general, a part of the input power $P_{\mathrm{in}}$, $W_{L}$, is
transferred to the load, while the remaining part is dissipated as Joule heat
$Q=P_{\mathrm{in}}-W_{L}$ in the Hall rectifier. We define the power transfer
efficiency as $\vartheta_{t}\equiv W_{L}/P_{\mathrm{in}}$. Since the total
power transferred to the load is a sum of DC and AC powers with
$W_{L}^{\mathrm{dc}}=2W_{L}^{\mathrm{ac}}$, the efficiency of AC-DC power
conversion or the rectification efficiency is given by $\vartheta_{c}\equiv
W_{L}^{\mathrm{dc}}/P_{\mathrm{in}}=\frac{2}{3}\vartheta_{t}$.

It is instructive to first consider the idealized limit of a nonlinear Hall
conductor without dissipative responses, i.e., $\sigma_{xx},\sigma_{yy}$ and
$\sigma_{xxxx}$ are all zero, while $\sigma_{yxx}=-\sigma_{xyx}$ is finite. In
this case, one can readily see that
$P_{\mathrm{in}}=W_{L}^{\mathrm{dc}}+W_{L}^{\mathrm{ac}}$ holds for any load
resistance $R_{L}$ and input field $E_{x}$, i.e., the power transfer
efficiency $\vartheta_{t}$ is $100\text{\,}\mathrm{\char 37\relax}$. This is
because the Hall current ${\bf\it j}^{H}$ is transverse to the electric field
and does not induce any Joule heating.

In reality, there is certainly dissipation in the nonlinear Hall conductor,
hence $\vartheta_{t}$ is in general less than unity and depends on $R_{L}$ and
$E_{x}$. Nonetheless, $\vartheta_{t}$ approaches $100\%$ in a certain regime
where NLHE dominates over the dissipative responses, as we now show
explicitly. From Eqs. (14), (16), one can readily see that $\vartheta_{t}$
increases monotonically with $E_{x}$, and at large $E_{x}$ compared to a
certain value $E_{1}$, reaches

$\displaystyle\vartheta_{t}(E_{x}\gg E_{1})$
$\displaystyle=\frac{r_{L}}{(1+r_{L})^{2}}\quantity(\alpha+\frac{r_{L}}{1+r_{L}})^{-1},$
(19)

with $E_{1}$ given by

$\displaystyle E_{1}$
$\displaystyle=\frac{2}{\sqrt{3}}E_{0}\sqrt{\frac{1+\alpha}{\alpha+r_{L}/(1+r_{L})}}$
(20)

where $r_{L}=R_{L}/R_{yy}$ is the ratio of load resistance to the internal
resistance $R_{yy}$. $\alpha=\sigma_{xxxx}\sigma_{yy}/\sigma^{2}_{yxx}\geq 0$
is a dimensionless parameter that characterizes the strength of dissipative
responses (both linear and higher-order) relative to NLHE, which is inherently
dissipationless (${\bm{j}}^{H}\cdot{\bm{E}}=0$). A small $\alpha$ indicates a
wide range of electric fields over which the current-voltage relation of the
noncentrosymmetric crystal is dominated by NLHE. As an intrinsic material
property, $\alpha$ can be experimentally determined by extracting $E_{0}$ and
$\eta$ from the current-voltage relation in the open circuit ($R_{L}=\infty$)
and using the relation $\alpha=(\sigma_{xx}/\sigma_{yy})(E_{0}\eta)^{-2}-1$.

By further maximizing $\vartheta_{c}$ with respect to $r_{L}$, we find the
maximum power conversion efficiency

$\displaystyle\max\theta_{t}$
$\displaystyle=\frac{1}{\quantity(\sqrt{\alpha+1}+\sqrt{\alpha})^{2}},$ (21)

which is attained at an optimum external resistance:

$\displaystyle r^{*}_{L}=\sqrt{\frac{\alpha}{1+\alpha}}.$ (22)

It should be noted that one can optimize $r_{L}$ by varying the length-to-
width ratio of the nonlinear Hall conductor. For example, by increasing
$L_{y}$ and decreasing $L_{x}$, one can increase $R_{yy}$ without changing the
volume. Then, for the same input power $P_{\mathrm{in}}$ and external
resistance $R_{L}$, $r_{L}$ is reduced so that the power conversion efficiency
is increased for large $R_{L}$.

The maximum power transfer efficiency $\vartheta_{t}$ is a monotonically
decreasing function of $\alpha$. As $\alpha\to 0$ or $\sigma_{xxxx}\rightarrow
0$, the maximum $\vartheta_{t}$ approaches $100\%$ at small $r_{L}$, hence the
rectification efficiency $\vartheta_{c}$ approaches $2/3$. More realistically,
for $r_{L}=1$ and assuming $\alpha=0$, we find
$\vartheta_{t}=$13.6\text{\,}\mathrm{\char 37\relax}$$ at $E_{x}=E_{0}$,
$45.2\text{\,}\mathrm{\char 37\relax}$ at $E_{x}=5E_{0}$, and
$48.7\text{\,}\mathrm{\char 37\relax}$ at $E_{x}=10E_{0}$. If $\alpha=1$, it
will be $\vartheta_{t}=$6.0\text{\,}\mathrm{\char 37\relax}$$ at
$E_{x}=E_{0}$, $15.6\text{\,}\mathrm{\char 37\relax}$ at $E_{x}=5E_{0}$, and
$16.4\text{\,}\mathrm{\char 37\relax}$ at $E_{x}=10E_{0}$. Fig. 3 shows the
dependence of the efficiency $\vartheta_{\mathrm{ac}}$ on the applied electric
field $E_{x}$ or input power $P_{\mathrm{in}}$ and on the external resistance
$R_{L}$ for the case $\alpha=0$. In general, high efficiency is achieved when
$\alpha$ is small and $E_{x}\gg E_{1}$, i.e., when NLHE
($\sigma_{y}xx,\sigma_{xyx}$) dominates over dissipative processes
($\sigma_{xx},\sigma_{yy},\sigma_{xxxx}$). Experimentally, this corresponds to
large nonlinear Hall angle.

Figure 3: The efficiency for NLHE based rectification when $\alpha=0$. (a,b)
$\vartheta_{c}$ for NLHE over $E_{x}$-$R_{L}$ plane (a) and
$P_{\mathrm{in}}$-$R_{L}$ plane (b). (c,d) $\tilde{\vartheta}_{c}$ for NLHE
when the large capacitance $C\gg(2\omega R_{yy})^{-1}$ is connected in
parallel to the load resistance over $E_{x}$-$R_{L}$ plane (c) and
$P_{\mathrm{in}}$-$R_{L}$ plane (d). Here, $P_{0}\equiv\sigma_{xx}E_{0}^{2}$.
Because the capacitor acts as the low-pass filter, the energy loss due to AC
current flowing into the load is suppressed and the efficiency can reach of
order $1$. In both cases, high rectification efficiency is achieved when
$R_{L}$ is small compared to $R_{yy}$ and $P_{\mathrm{in}}\gg P_{0}$ or
$E_{x}\gg E_{0}$.

Note that $1/3$ of the energy transferred to the load is in the form of AC
power. This sets the upper limit of $2/3$ for AC-DC power conversion
efficiency, when the Hall rectifier is directly connected to an external
resistance. We can further increase the efficiency by suppressing the high-
frequency $2\omega$ components of the output current or voltage with a low-
pass filter. This can be realized by connecting a capacitor to the load
resistance in parallel. If the capacitance $C$ satisfies $(2\omega C)^{-1}\ll
R_{L},R_{yy}$, the capacitance approximately behaves as an open circuit for DC
component but a short circuit for the $2\omega$ AC current. Hence, it is
expected that most of the $2\omega$ component of $j_{y}$ does not flow into
the load and does not consume energy, resulting in high AC-DC power conversion
efficiency. Indeed, under this condition, we find the rectification efficiency
is equal to the power transfer efficiency, given by

$\displaystyle\tilde{\vartheta}_{c}$
$\displaystyle\simeq\frac{\frac{r_{L}}{2(1+r_{L})^{2}}E_{x}^{2}}{E_{0}^{2}+\left(\frac{r_{L}}{2(1+r_{L})}+\frac{3}{4}\alpha\right)E_{x}^{2}}\quad\textrm{for
}2\omega C\gg R_{L}^{-1},R_{yy}^{-1}.$ (23)

For a given $r_{L}$, it reaches maximum at large $E_{x}$

$\displaystyle\max\tilde{\vartheta}_{c}=\frac{2r_{L}}{(1+r_{L})(2r_{L}+3\alpha(1+r_{L}))}.$
(24)

Fig. 3(c,d) shows the rectification efficiency when a large capacitance is
connected to the Hall rectifier in parallel with the load resistance for
$\alpha=0$. Now, the efficiency is much improved: it exceeds $2/3$ in broad
region and reaches nearly $100\%$ at small $r_{L}$.

Instead of utilizing the low-pass filter, one can also extract work from the
$2\omega$ components of $j_{y}$ by using another rectification device. If one
utilizes NLHE again to rectify the $2\omega$ component, the maximum
rectification efficiency will be $8/9\simeq$89\text{\,}\mathrm{\char
37\relax}$$.

To achieve efficient energy harvesting, impedance matching between the source
for AC electricity and the rectifier is important [46, 47]. On this point, the
Hall rectifier also has an advantage compared to the conventional diodes: one
can choose its length and width $L_{x}$ and $L_{y}$ properly to match $R_{x}$
with the resistance of the antenna and at the same time optimize $R_{y}$
relative to the external resistance for high conversion efficiency.

Importantly, our Hall rectifier can work efficiently over a wide range of
frequencies up to the scale of the scattering rate $1/\tau$ of the
noncentrosymmetric crystal. Our analysis assumes that the linear and nonlinear
conductivities do not depend on the frequency. This assumption is justified
when the frequency $\omega$ is sufficiently small compared to the scattering
rate $1/\tau$. Typically, $1/\tau$ of conductors at room temperature is on the
order of $10\text{\,}$-$100\text{\,}\mathrm{T}\mathrm{Hz}$, hence our theory
of power conversion efficiency of Hall rectifiers is applicable over an
ultrabroad frequency range from quasi-DC to the terahertz regime. In contrast,
conventional diodes have an upper cutoff frequency $\sim 1$THz set by $RC$
time constant of the semiconductor junction, while solar cells have a lower
cutoff frequency $\sim 20$THz set by interband absorption threshold.
Therefore, quantum rectification based on nonlinear Hall conductors may be a
promising technology to harness RF, millimeter and THz waves for wireless
charging and energy harvesting.

###### Acknowledgements.

It is our pleasure to thank Zhiqiang Mao, Lujin Min and Suyang Xu for
stimulating discussions and productive collaborations on related projects.
This work is supported by the U.S. Army Research Laboratory and the U.S. Army
Research Office through the Institute for Soldier Nanotechnologies, under
Collaborative Agreement Number W911NF-18-2-0048. Y.O. is supported by JSPS
KAKENHI Grant No. JP22J22111. L.F. is partly supported by the David and Lucile
Packard Foundation.

## References

  * Tripathi  _et al._ [2021] S. Tripathi, N. V. Sabu, A. K. Gupta, and H. S. Dhillon, Millimeter-wave and Terahertz Spectrum for 6G Wireless (2021), arXiv:2102.10267 [cs, math] .
  * Tonouchi [2007] M. Tonouchi, Cutting-edge terahertz technology, Nature Photonics 1, 97 (2007).
  * Akyildiz and Jornet [2010] I. F. Akyildiz and J. M. Jornet, The Internet of nano-things, IEEE Wireless Communications 17, 58 (2010).
  * Donchev  _et al._ [2014] E. Donchev, J. S. Pang, P. M. Gammon, A. Centeno, F. Xie, P. K. Petrov, J. D. Breeze, M. P. Ryan, D. J. Riley, and N. McN, The rectenna device: From theory to practice (a review), MRS Energy & Sustainability 1, 1 (2014).
  * Moddel and Grover [2013] G. Moddel and S. Grover,  _Rectenna Solar Cells_ , 1st ed. (Springer New York, NY, 2013).
  * Mizojiri and Shimamura [2019] S. Mizojiri and K. Shimamura, Recent progress of Wireless Power Transfer via Sub-THz wave, in  _2019 IEEE Asia-Pacific Microwave Conference (APMC)_ (2019) pp. 705–707.
  * Lu  _et al._ [2015] X. Lu, P. Wang, D. Niyato, D. I. Kim, and Z. Han, Wireless Networks With RF Energy Harvesting: A Contemporary Survey, IEEE Communications Surveys & Tutorials 17, 757 (2015).
  * Okba  _et al._ [2019] A. Okba, A. Takacs, and H. Aubert, Compact Rectennas for Ultra-Low-Power Wireless Transmission Applications, IEEE Transactions on Microwave Theory and Techniques 67, 1697 (2019).
  * Isobe  _et al._ [2020] H. Isobe, S. Y. Xu, and L. Fu, High-frequency rectification via chiral Bloch electrons, Science Advances 6, 1 (2020).
  * Sodemann and Fu [2015] I. Sodemann and L. Fu, Quantum Nonlinear Hall Effect Induced by Berry Curvature Dipole in Time-Reversal Invariant Materials, Physical Review Letters 115, 216806 (2015).
  * Ma  _et al._ [2019] Q. Ma, S.-Y. Xu, H. Shen, D. MacNeill, V. Fatemi, T.-R. Chang, A. M. Mier Valdivia, S. Wu, Z. Du, C.-H. Hsu, S. Fang, Q. D. Gibson, K. Watanabe, T. Taniguchi, R. J. Cava, E. Kaxiras, H.-Z. Lu, H. Lin, L. Fu, N. Gedik, and P. Jarillo-Herrero, Observation of the nonlinear Hall effect under time-reversal-symmetric conditions, Nature 565, 337 (2019).
  * Kang  _et al._ [2019] K. Kang, T. Li, E. Sohn, J. Shan, and K. F. Mak, Nonlinear anomalous Hall effect in few-layer WTe2, Nature Materials 18, 324 (2019).
  * Gao  _et al._ [2014] Y. Gao, S. A. Yang, and Q. Niu, Field Induced Positional Shift of Bloch Electrons and Its Dynamical Implications, Physical Review Letters 112, 166601 (2014).
  * Sipe and Shkrebtii [2000] J. E. Sipe and A. I. Shkrebtii, Second-order optical response in semiconductors, Physical Review B 61, 5337 (2000).
  * Sturman and Fridkin [2021] B. I. Sturman and V. M. Fridkin,  _The Photovoltaic and Photorefractive Effects in Noncentrosymmetric Materials_ (Routledge, 2021).
  * Belinicher and Sturman [1980] V. I. Belinicher and B. I. Sturman, The photogalvanic effect in media lacking a center of symmetry, Soviet Physics Uspekhi 23, 199 (1980).
  * Belinicher  _et al._ [1982] V. Belinicher, E. Ivchenko, and B. Sturman, Kinetic theory of the displacement photovoltaic effect in piezoelectrics, Zh. Eksp. Teor. Fiz 83, 649 (1982).
  * von Baltz and Kraut [1981] R. von Baltz and W. Kraut, Theory of the bulk photovoltaic effect in pure crystals, Physical Review B 23, 5590 (1981).
  * Deyo  _et al._ [2009] E. Deyo, L. E. Golub, E. L. Ivchenko, and B. Spivak, Semiclassical theory of the photogalvanic effect in non-centrosymmetric systems (2009).
  * Moore and Orenstein [2010] J. E. Moore and J. Orenstein, Confinement-Induced Berry Phase and Helicity-Dependent Photocurrents, Physical Review Letters 105, 026805 (2010).
  * Matsyshyn and Sodemann [2019] O. Matsyshyn and I. Sodemann, Nonlinear Hall Acceleration and the Quantum Rectification Sum Rule, Physical Review Letters 123, 246602 (2019).
  * Shi  _et al._ [2022] L.-k. Shi, O. Matsyshyn, J. C. W. Song, and I. S. Villadiego, The Berry dipole photovoltaic demon and the thermodynamics of photo-current generation within the optical gap of metals (2022).
  * Watanabe and Yanase [2021] H. Watanabe and Y. Yanase, Chiral Photocurrent in Parity-Violating Magnet and Enhanced Response in Topological Antiferromagnet, Physical Review X 11, 011001 (2021).
  * Du  _et al._ [2021] Z. Z. Du, H.-Z. Lu, and X. C. Xie, Nonlinear Hall effects, Nature Reviews Physics 3, 744 (2021).
  * He and Weng [2021] Z. He and H. Weng, Giant nonlinear Hall effect in twisted bilayer WTe2, npj Quantum Materials 6, 1 (2021).
  * Shvetsov  _et al._ [2019] O. O. Shvetsov, V. D. Esin, A. V. Timonina, N. N. Kolesnikov, and E. V. Deviatov, Nonlinear Hall Effect in Three-Dimensional Weyl and Dirac Semimetals, JETP Letters 109, 715 (2019).
  * [27] A. Gao, Y.-F. Liu, B. Ghosh, T. V. Trevisan, Y. Onishi, C. Hu, T. Qian, H.-J. Tien, S.-W. Chen, D. Bérubé, J.-X. Qiu, H. Li, C. Tzschaschel, T. Dinh, Z. Sun, S.-C. Ho, S.-W. Lien, B. Singh, K. Watanabe, T. Taniguchi, D. C. Bell, H. Lin, T.-R. Chang, A. Bansil, L. Fu, N. Ni, P. P. Orth, Q. Ma, and S.-Y. Xu, Quantum metric nonlinear hall effect in a topological antiferromagnetic heterostructure, submitted.
  * Zhang and Fu [2021] Y. Zhang and L. Fu, Terahertz detection based on nonlinear Hall effect without magnetic field, Proceedings of the National Academy of Sciences 118, e2100736118 (2021).
  * Kumar  _et al._ [2021] D. Kumar, C.-H. Hsu, R. Sharma, T.-R. Chang, P. Yu, J. Wang, G. Eda, G. Liang, and H. Yang, Room-temperature nonlinear Hall effect and wireless radiofrequency rectification in Weyl semimetal TaIrTe4, Nature Nanotechnology 16, 421 (2021).
  * Shi and Song [2019] L.-k. Shi and J. C. W. Song, Symmetry, spin-texture, and tunable quantum geometry in a ${\mathrm{}}WTe{}{}_{2}$ monolayer, Physical Review B 99, 035403 (2019).
  * Ma  _et al._ [2022] D. Ma, A. Arora, G. Vignale, and J. C. W. Song, Anomalous skew-scattering nonlinear Hall effect in $PT$-symmetric antiferromagnets (2022), arXiv:2210.14932 [cond-mat] .
  * Tiwari  _et al._ [2021] A. Tiwari, F. Chen, S. Zhong, E. Drueke, J. Koo, A. Kaczmarek, C. Xiao, J. Gao, X. Luo, Q. Niu, Y. Sun, B. Yan, L. Zhao, and A. W. Tsen, [SI] Giant c-axis nonlinear anomalous Hall effect in Td-MoTe2 and WTe2, Nature Communications 12, 2049 (2021).
  * Zhang  _et al._ [2022a] C.-P. Zhang, J. Xiao, B. T. Zhou, J.-X. Hu, Y.-M. Xie, B. Yan, and K. T. Law, Giant nonlinear Hall effect in strained twisted bilayer graphene, Physical Review B 106, L041111 (2022a).
  * Zhang  _et al._ [2018] Y. Zhang, J. van den Brink, C. Felser, and B. Yan, Electrically tuneable nonlinear anomalous Hall effect in two-dimensional transition-metal dichalcogenides WTe2 and MoTe2, 2D Materials 5, 044001 (2018).
  * Zhang  _et al._ [2022b] C.-L. Zhang, T. Liang, Y. Kaneko, N. Nagaosa, and Y. Tokura, Giant Berry curvature dipole density in a ferroelectric Weyl semimetal, npj Quantum Materials 7, 1 (2022b).
  * Tokura and Nagaosa [2018] Y. Tokura and N. Nagaosa, Nonreciprocal responses from non-centrosymmetric quantum materials, Nature Communications 9, 3740 (2018).
  * Michishita and Nagaosa [2022] Y. Michishita and N. Nagaosa, Dissipation and geometry in nonlinear quantum transports of multiband electronic systems, Physical Review B 106, 125114 (2022).
  * Ideue and Iwasa [2021] T. Ideue and Y. Iwasa, Symmetry Breaking and Nonlinear Electric Transport in van der Waals Nanostructures, Annual Review of Condensed Matter Physics 12, 201 (2021).
  * Kang  _et al._ [2018] K. Kang, T. Li, E. Sohn, J. Shan, and K. F. Mak, Observation of the nonlinear anomalous Hall effect in 2D WTe2 (2018), arXiv:1809.08744 [cond-mat] .
  * Kang  _et al._ [2022] K. Kang, W. Zhao, Y. Zeng, K. Watanabe, T. Taniguchi, J. Shan, and K. F. Mak, Switchable moir\’e potentials in ferroelectric WTe2/WSe2 superlattices (2022), arXiv:2209.04981 [cond-mat] .
  * Wang  _et al._ [2021] C. Wang, Y. Gao, and D. Xiao, Intrinsic Nonlinear Hall Effect in Antiferromagnetic Tetragonal CuMnAs, Physical Review Letters 127, 277201 (2021).
  * Xiao  _et al._ [2020] R.-C. Xiao, D.-F. Shao, W. Huang, and H. Jiang, Electrical detection of ferroelectriclike metals through the nonlinear Hall effect, Physical Review B 102, 024109 (2020).
  * Toshio and Kawakami [2022] R. Toshio and N. Kawakami, Plasmonic quantum nonlinear Hall effect in noncentrosymmetric 2D materials (2022).
  * Toshio  _et al._ [2020] R. Toshio, K. Takasan, and N. Kawakami, Anomalous hydrodynamic transport in interacting noncentrosymmetric metals, Physical Review Research 2, 032021 (2020).
  * Qin  _et al._ [2021] M.-S. Qin, P.-F. Zhu, X.-G. Ye, W.-Z. Xu, Z.-H. Song, J. Liang, K. Liu, and Z.-M. Liao, Strain Tunable Berry Curvature Dipole, Orbital Magnetization and Nonlinear Hall Effect in WSe 2 Monolayer*, Chinese Physics Letters 38, 017301 (2021).
  * Wagih  _et al._ [2020] M. Wagih, A. S. Weddell, and S. Beeby, Millimeter-Wave Power Harvesting: A Review, IEEE Open Journal of Antennas and Propagation 1, 560 (2020).
  * Brown  _et al._ [2007] E. Brown, A. Young, J. Zimmerman, H. Kazerni, and A. Gossard, Advances in schottky rectifier performance, IEEE Microwave Magazine 8, 54 (2007).