1.4pt

#  Mean-field Analysis of Generalization Errors

Gholamali Aminian∗ 1 Samuel N. Cohen∗ 1 2 Łukasz Szpruch∗ 1 3

###### Abstract

We propose a novel framework for exploring weak and $L_{2}$ generalization
errors of algorithms through the lens of differential calculus on the space of
probability measures. Specifically, we consider the KL-regularized empirical
risk minimization problem and establish generic conditions under which the
generalization error convergence rate, when training on a sample of size $n$,
is $\mathcal{O}(1/n)$. In the context of supervised learning with a one-hidden
layer neural network in the mean-field regime, these conditions are reflected
in suitable integrability and regularity assumptions on the loss and
activation functions.

$*$$*$footnotetext: Equal contribution.${}^{1}$${}^{1}$footnotetext: The Alan
Turing Institute.${}^{2}$${}^{2}$footnotetext: Mathematical Institute,
University of Oxford.${}^{3}$${}^{3}$footnotetext: School of Mathematics,
University of Edinburgh.

##  1 Introduction

A crucial problem in learning theory is understanding the performance of a
learning algorithm on unseen data. This is described by the generalization
error, which quantifies the deviation between the algorithm’s performance on
the training data and its performance on unseen data sampled from the same
underlying distribution. In the under-parameterized regime, where the number
of model parameters is significantly less than the number of training data
points, the theory of the generalization error is well established [Vapnik and
Chervonenkis, 2015, Bartlett and Mendelson, 2002]. However, in the
overparameterized regime, where the number of parameters may greatly exceed
the number of training data points, this theory becomes inadequate. For
example, deep neural network (NN) models can exhibit nearly zero training
loss, which implies that they interpolate the training data, and yet still
perform well on out-of-sample data [Belkin et al., 2019a, Spigler et al.,
2019, Bartlett et al., 2021]. Mathematical models, such as the Neural Tangent
Kernel [Jacot et al., 2018], Mean-Field [Mei et al., 2018], and Random Feature
models [Rahimi and Recht, 2008], have been proposed to understand the behavior
of overparameterized NNs. However, our current understanding of generalization
in these contexts still needs to be completed. Our approach is partially
motivated by the recently developed mean-field perspective on training NNs. In
a series of works [Mei et al., 2018, Chizat and Bach, 2018, Mei et al., 2019,
Sirignano and Spiliopoulos, 2022, Hu et al., 2020, Tzen and Raginsky, 2020],
the task of learning the optimal weights in deep NNs is viewed as a sampling
problem. The picture that emerges from the mean-field regime is that the aim
of a learning algorithm is to find an optimal distribution over the parameter
space (rather than optimal values of the parameters). This work investigates
how the mean-field perspective can shed light on the generalization
performance of overparameterized NNs.

The contributions of this work are as follows:

  * •

We introduce and develop a general approach to study generalization error (cf.
(8) for the definition) that utilizes calculus on the space of measures,
encompassing a wide range of learning models and risk functions.

  * •

In the context of Kullback–Leibler (KL) regularized empirical risk
minimization, our analysis provides if-and-only-if conditions for achieving a
convergence rate of $\mathcal{O}({1}/{n})$ for weak generalization errors and
sharp conditions for achieving a convergence rate of $\mathcal{O}({1}/{n})$
for $L_{2}$. These bounds are not asymptotic, i.e. they are valid for all $n$.

  * •

For one-hidden layer neural networks in the mean-field regime, our analysis
reveals precise conditions on model inputs that guarantee a convergence rate
of $\mathcal{O}(1/n)$ of both weak and $L_{2}$ generalization errors. This
improves the generalization error bounds which can be derived from the proof
techniques in Nishikawa et al. [2022] and Nitanda et al. [2021].

##  2 Preliminaries

##### Notation:

We adopt the following convention for random variables and their
distributions. A random variable is denoted by an upper-case letter (e.g.,
$Z$), its space of possible values is denoted with the corresponding
calligraphic letter (e.g. $\mathcal{Z}$), and an arbitrary value of this
variable is denoted with the lower-case letter (e.g., $z$). We will write
$\mathbb{E}_{Z}[\cdot]$ for the expectation taken over $Z$, all other random
variables being left constant333Formally, this corresponds to the conditional
expectation over all variables in our setting excluding $Z$. As we will only
have countably many variables in our problem, this does not cause any
technical difficulties.. We will further write $\mathbb{E}_{Z\sim m}[\cdot]$
for the expectation over a random variable $Z$ independent of all others,
which is distributed according to $m$, and similarly $\mathbb{V}_{Z\sim m}$
for the variance. We write $\delta_{z}$ for a Dirac measure supported at $z$.

If $A$ is a normed space, then $\mathcal{P}(A)$ denotes the space of
probability measures on $A$ and $\mathcal{P}_{p}(A)$ the probability measures
with finite $p$-th moment. We endow spaces of probability measures with the
weak topology and corresponding Borel $\sigma$-algebra.

We will take the metric over spaces of interest (and finite products of these
spaces) to be defined a priori, and will assume that the metric over a product
space is equivalent to the product (Euclidean) metric.

We now introduce the functional linear derivative [Cardaliaguet et al., 2019]
for functionals on measure spaces. For simplicity333This can be relaxed, along
with most integrability assumptions in this paper, at a cost of complexity.,
we will restrict our attention to the finite variance
($\mathcal{P}_{2}(\mathbb{R}^{n})$ case).

######  Definition 2.1.

Extending [Carmona and Delarue, 2018, Definition 5.43] Consider
$U:\mathcal{P}_{2}(\mathbb{R}^{n})\times\mathbb{R}^{k}\to\mathbb{R}$. We say
$m\mapsto U(m,x)$ is of class $\mathcal{C}^{1}$ if there exists a map
$\frac{\delta U}{\delta
m}:\mathcal{P}_{2}(\mathbb{R}^{n})\times\mathbb{R}^{k}\times\mathbb{R}^{n}\to\mathbb{R}$,
such that

  1. (i)

$\frac{\delta U}{\delta m}$ is measurable with respect to $x,a$, and
continuous with respect to $m$;

  2. (ii)

for every bounded set
$B\subset\mathcal{P}_{2}(\mathbb{R}^{n})\times\mathbb{R}^{k}$, there exists a
constant $C>0$ such that $|\frac{\delta U}{\delta m}(m,x,a)|\leq C(1+|a|^{2})$
for all $(m,x)\in B$;

  3. (iii)

for all $m,m^{\prime}\in\mathcal{P}_{2}(\mathbb{R}^{n})$,

$U(m^{\prime},x)-U(m,x)=\int_{0}^{1}\int_{\mathbb{R}^{n}}\frac{\delta
U}{\delta
m}(m+\lambda(m^{\prime}-m),x,a)\,(m^{\prime}-m)(da)\,\mathrm{d}\lambda\,.$

Since $\frac{\delta U}{\delta m}$ is only defined up to a constant we demand
$\int\frac{\delta U}{\delta m}(m,x,a)\,m(da)=0$. By extension, we say $U$ is
of class $\mathcal{C}^{2}$ if both $U$ and $\frac{\delta U}{\delta m}$ are of
class $\mathcal{C}^{1}$.

Similarly, we define derivatives for measure-valued functionals. See Appendix
B for proofs.

######  Definition 2.2.

A functional
$\mathfrak{m}:\mathcal{P}_{2}(\mathbb{R}^{n})\to\mathcal{P}_{2}(\mathbb{R}^{k})$
is of class $\mathcal{C}^{1}$ if for all $A\in\mathcal{B}(\mathbb{R}^{k})$,
the functional $\mathfrak{m}|_{A}:=m(\cdots)(A)$ is $\mathcal{C}^{1}$ in the
sense of Definition 2.1, and its derivative
$\frac{\delta\mathfrak{m}}{\delta\nu}(A):=\frac{\delta\mathfrak{m}|_{A}}{\delta\nu}$
is a (Borel) signed measure on $\mathbb{R}^{k}$, where the constant $C$ in
Definition 2.1(ii) does not depend on the set $A$.

######  Lemma 2.3.

Let $\mathfrak{m}$ be a functional as in Definition 2.2, with the density
representation $\mathfrak{m}(\nu)(A)=\int_{A}U(\nu,x)\mathrm{d}x$ for all
Borel sets $A$. Suppose $\mathfrak{m}$ and $U(\cdot,x)$, for each $x$, admit
linear derivatives. Then the linear derivatives of $\mathfrak{m}$ and $U$
satisfy $\frac{\delta\mathfrak{m}}{\delta\nu}(\nu,a)(A)=\int_{A}\frac{\delta
U}{\delta\nu}(\nu,a,x)\mathrm{d}x,$ or, more succinctly,
$\frac{\delta\mathfrak{m}}{\delta\nu}(\nu,a)(\mathrm{d}x)=\frac{\delta
U}{\delta\nu}(\nu,a,x)\mathrm{d}x$.

###  2.1 Problem formulation

Consider a supervised learning scenario, with input space $\mathcal{X}$ and
target space $\mathcal{Y}$. Define
$\mathcal{Z}:=\mathcal{X}\times\mathcal{Y}$. We are often interested in
learning a function $f:\mathcal{X}\to\mathcal{Y}$ parameterized using a
(large) number of parameters from $\Theta\subseteq\mathbb{R}^{k}$, for some
$k>0$.

Let $\nu_{\mathrm{pop}}\in\mathcal{P}_{2}(\mathcal{Z})$ be the (unknown)
distribution, describing the joint values of $(x,y)$ in the population.
Intuitively, we suppose that the mass of $\nu_{\mathrm{pop}}$ is near the
graph of $f$ that is, $y\approx f(x)$ when $z=(x,y)$ is sampled from
$\nu_{\mathrm{pop}}$. Since $\nu_{\mathrm{pop}}$ is unknown, we approximate
$f$ based on a finite dataset, $\mathbf{Z}_{n}=\\{Z_{i}\\}_{i=1}^{n}$. We make
the following basic assumption on our data, which will be taken to hold
throughout the paper.

######  Assumption 1.

Our observations are given by $\mathbf{Z}_{n}=\\{Z_{i}\\}_{i=1}^{n}$, where
$Z_{i}=(X_{i},Y_{i})\sim\nu_{\mathrm{pop}}$ are i.i.d. We write
$\nu_{n}:=\frac{1}{n}\sum_{i=1}^{n}\delta_{Z_{i}}$ for the corresponding
empirical measure.

We are interested in quantifying the performance of our model on unseen data.
To this end, let
$\mathbf{\,\widetilde{\\!{Z}}}_{n}=\\{\,\widetilde{\\!{Z}}_{i}\\}_{i=1}^{n}$
be a second i.i.d sample set with law $\nu_{\mathrm{pop}}$, independent of
$\mathbf{Z}_{n}$. We define the perturbations of $\nu_{n}$ obtained by
‘resampling’ one or two data points, by

$\nu_{n,(1)}=\nu_{n}+\frac{1}{n}(\delta_{\,\widetilde{\\!{Z}}_{1}}-\delta_{Z_{1}}),\qquad\nu_{n,(1,2)}=\nu_{n}+\frac{1}{n}(\delta_{\,\widetilde{\\!{Z}}_{1}}-\delta_{Z_{1}}+\delta_{\,\widetilde{\\!{Z}}_{2}}-\delta_{Z_{2}})\,.$
(1)

##### Learning algorithm:

We consider learning algorithms which can be represented as (measurable) maps
from the empirical measure $\nu_{n}$ to a measure over parameters,
$\nu_{n}\mapsto\mathfrak{m}(\nu_{n})\in\mathcal{P}(\Theta)$. In classical
statistics, learning algorithms are (deterministic) maps from observed data to
parameter estimates. By generalizing to measures over parameters, we allow for
randomness in training algorithms (e.g. stochastic gradient descent), for
Bayesian interpretations (where $\mathfrak{m}(\nu_{n})$ is the posterior
distribution), and for a formalism for overparameterized models; furthermore,
we gain access to the tools of calculus on probability spaces.

To investigate resampling the training data, we will make use of the convex
perturbations, for $\lambda\in[0,1]$ and $\mathcal{J}\in\\{(1),(1,2)\\}$,

$\displaystyle\begin{cases}\mathfrak{m}_{\mathcal{J}}(\lambda)=\mathfrak{m}(\nu_{n,\mathcal{J}})+\lambda(\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,\mathcal{J}}))\\\
\nu_{(1)}(\lambda)=\nu_{n,\mathcal{J}}+\lambda(\nu_{n}-\nu_{n,\mathcal{J}}),\\\
\end{cases}$ (2)

####  2.1.1 Loss function

We focus on training methods where $\mathfrak{m}$ is the minimizer of a loss
function. In order to motivate our generic formulation of this problem, we
present the concrete example of overparameterized one-hidden layer neural
networks. An application to expected parametric loss can be found in Appendix
F.

1.4pt

$x_{1.0}$$x_{2.0}$$x_{0.0}$$h_{0.0}$$h_{3.0}$$h_{2.0}$$h_{1.0}$white$w_{1.0,1}$white$w_{2.0,1}$white$w_{0.0,1}$$O$white$a_{1.0}$white$a_{2.0}$white$a_{3.0}$white$a_{0.0}$$\vdots$$\vdots$$\begin{aligned}
\,=\varphi\left(\sum_{i=1}^{q}w_{i,1}x_{i}\right)\end{aligned}$$\begin{aligned}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\,=\frac{1}{r}\sum_{j=1}^{r}a_{j}\varphi\left(\sum_{i=1}^{q}w_{i,j}x_{i}\right)=\hat{f}(x)\end{aligned}$

Figure 1: One Hidden-layer NN

Overparameterized one-hidden layer neural network (NN): We consider a one-
hidden layer NN, e.g. as shown in Figure 1. Let $x\in\mathbb{R}^{q}$ be a
feature vector. For each of $r$ hidden neurons, we denote the parameters of
the hidden layer by $w\in\mathbb{R}^{q}$ and the outer layer by
$a\in\mathbb{R}$. The parameter space is
$\Theta=\mathbb{R}^{(q+1)}=\\{a\in\mathbb{R},w\in\mathbb{R}^{q}\\}$. To ease
notation, we write $\phi(\theta_{i},x)=a_{i}\varphi(w_{i}\cdot x)$, where
$\varphi(\cdot)$ is the activation function. The output of the NN is

$\hat{f}(x)=\frac{1}{r}\sum_{i=1}^{r}a_{i}\varphi(w_{i}\cdot
x)=\frac{1}{r}\sum_{i=1}^{r}\phi(\theta_{i},x)=\int\phi(\theta,x)\,m_{r}\\!(\mathrm{d}\theta)=\mathbb{E}_{\theta\sim
m_{r}}[\phi(\theta,x)],$ (3)

where $m_{r}:=\frac{1}{r}\sum_{i=1}^{r}\delta_{(a_{i},w_{i})}.$ Observe that
any hidden nodes can be switched (along with their parameters) without
changing the output of the network; this symmetry implies that it is the
(joint) distribution $m_{r}$ which is important. As $r$ increases, $m_{r}$ can
converge (weakly) to a continuous distribution over the parameter space; the
weights for individual neurons can be viewed as samples from this measure.
This is the mean-field model studied in [Mei et al., 2018, 2019, Hu et al.,
2020].

Training neural networks chooses parameters $\theta$ to minimize a loss
$\ell_{o}$ between $\hat{f}(x)$ and the observed value $y$. With
$\hat{y}:=\hat{f}(x)=\mathbb{E}_{\theta\sim m_{r}}[\phi(\theta,x)]$, we can
write our single-observation loss function as

$\ell(m,z):=\ell_{o}\big{(}\mathbb{E}_{\theta\sim
m}[\phi(\theta,x)],y\big{)}.$ (4)

For example, consider $\ell_{o}\big{(}\hat{y},y\big{)}=c(y\hat{y})$ for binary
classification, where $c(\cdot)$ is a margin-based loss function [Bartlett et
al., 2006], or absolute loss $\ell_{o}(\hat{y},y)=|y-\hat{y}|$ or quadratic
loss $\ell_{o}(\hat{y},y)=(y-\hat{y})^{2}$ for classification and regression
problems.

Generic loss function based on parameter measure: Inspired by the scenario
above, we generically333As before, we assume that $\ell$ is measurable as a
map from $\mathcal{P}(\Theta)\times\mathcal{Z}\to\mathbb{R}$. represent the
single-observation loss function as

$(m,z)\mapsto\ell(m,z)\in\mathbb{R}^{+}\,.$ (5)

This represents the loss, for the parameter distribution $m$ and a data point
$z$. The loss is typically non-linear in $m$ for the overparameterized one-
hidden layer NN (4).

####  2.1.2 Risks

The risk function333The functional $\mathrm{R}$ is (weakly) continuous in
$\nu$, and is measurable in $m$ (as a consequence of the measurability of
$\ell$). As $\ell(m,z)$ is nonnegative, the integral (6) can always be
defined, without integrability assumptions on $\ell$, but may take the value
$+\infty$. takes distributions $m$ of parameters and $\nu$ of data points and
evaluates the loss:

$\displaystyle\mathrm{R}(m,\nu):=\int_{\mathcal{Z}}\ell\big{(}m,z\big{)}\nu(\mathrm{d}z)\,.$
(6)

The population risk is $\mathrm{R}(m,\nu_{\mathrm{pop}})$, and the empirical
risk is $\mathrm{R}(m,\nu_{n})$, which simplifies as

$\displaystyle\mathrm{R}(m,\nu_{n})=\int_{\mathcal{Z}}\ell\big{(}m,z\big{)}\nu_{n}(\mathrm{d}z)=\frac{1}{n}\sum_{i=1}^{n}\ell\big{(}m,z_{i}\big{)}.$
(7)

##### Generalization error:

We aim to study the performance of the model trained with the empirical data
measure $\nu_{n}$, and evaluated against the population measure
$\nu_{\mathrm{pop}}$, that is,
$\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})$. Following Bousquet and
Elisseeff [2002] and Xu and Raginsky [2017], the risk can be decomposed:

$\displaystyle\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})=\underbrace{\Big{(}\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})\Big{)}}_{\text{generalization
error}}+\underbrace{\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})}_{\text{training
error}}.$ (8)

Due to the randomness of the training data $\mathbf{Z_{n}}$ (and hence
$\nu_{n}$), the generalization error is not deterministic, so we seek to
estimate its statistics.

######  Assumption 2.

The training map $\mathfrak{m}$, loss function $\ell$, and population measure
$\nu$ are such that the training error satisfies
$\mathbb{E}_{\mathbf{Z}_{n}}[\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})]=\mathbb{E}_{\mathbf{Z}_{n}}[\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)\nu_{n}(\mathrm{d}z)]<\infty$.

For a training map $\mathfrak{m}$, we will consider both the Weak
Generalization Error (WGE),

$\displaystyle\mathrm{gen}(\mathfrak{m},\nu_{\mathrm{pop}})\triangleq\mathbb{E}_{\mathbf{Z}_{n}}\big{[}\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})\big{]}.$
(9)

and the $L_{2}$-Generalization Error (LGE)

$\displaystyle\mathrm{gen}_{L_{2}}(\mathfrak{m},\nu_{\mathrm{pop}})\triangleq\mathbb{E}_{\mathbf{Z}_{n}}\big{[}|\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})|^{2}\big{]}.$
(10)

Our main results will be conditions under which these errors are
$\mathcal{O}(1/n)$. Markov’s inequality shows that, for all $\epsilon>0$,

$\mathbb{P}\big{(}|\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})|>\epsilon\big{)}\leq\mathrm{gen}_{L_{2}}(\mathfrak{m},\nu_{\mathrm{pop}})\big{/}\epsilon^{2},$

giving probabilistic bounds on the (random) generalization error.

###  2.2 Related works

##### Generalization error and overparameterization:

There are three main approaches to analyzing learning problems in an
overparameterized regime: the neural tangent kernel (NTK), random feature and
mean-field approaches. Under some assumptions, the NTK approach (a.k.a. lazy
training) shows that an overparameterized one-hidden layer NN converges to an
(infinite dimensional) linear model. The neural tangent random feature
approach is similar to NTK, where the model is defined based on the network
gradients at the initialization. The random feature model is similar to NTK,
with an extra assumption of constant weights in the single hidden layer of the
NN. The mean-field approach uses the exchangeability of neurons to work with
distributions of parameters. The study of these methods’ generalization
performance allows us to extend our understanding of the overparameterized
regime. See, for example, for NTK [Li and Liang, 2018, Cao and Gu, 2019, Arora
et al., 2019b, a, Allen-Zhu et al., 2019, Ji and Telgarsky, 2020], neural
tangent random feature [Cao and Gu, 2019, Chen et al., 2019], random feature
[Mei and Montanari, 2022, Ma et al., 2019] and mean-field [Nitanda et al.,
2021, Nishikawa et al., 2022] settings. The neural tangent kernel and random
feature results do not precisely reflect practice due to their constraints,
such as the solution’s inability to deviate too far from the weights’
initialization [Fang et al., 2021]. A generalized NTK approach is considered
in [Chen et al., 2020], inspired by the mean-field approach, they derive a
high-probability bound on the generalization error of one-hidden layer NNs
with convergence rate $\mathcal{O}(1/\sqrt{n})$. A high probability
generalization error bound with rate $\mathcal{O}(1/\sqrt{n})$ in one-hidden
layer NN and with 0–1 and quadratic loss is studied in [Nitanda et al., 2021,
Nishikawa et al., 2022] assuming a mean-field regime. See Appendix A for
further discussion of related work.

##  3 Generalization errors

In this section, we apply functional calculus to study generalization error.
Our goal is to give abstract conditions under which bounds can be established,
which can be applied in a range of contexts, as will be explored in subsequent
sections. All proofs are deferred to Appendix C.

###  3.1 WGE via functional derivatives

We begin this section by providing a general representation of generalization
error, inspired by the approach of [Bousquet and Elisseeff, 2002, Lemma 7].

######  Lemma 3.1.

Consider a generic loss function $(m,z)\mapsto\ell(m,z)$, and $\nu_{n,(1)}$ as
defined in (1). The WGE (9) is given by:

$\displaystyle\mathrm{gen}(\mathfrak{m},\nu_{\mathrm{pop}})=\mathbb{E}_{\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1}}\Big{[}\ell\big{(}\mathfrak{m}(\nu_{n}),\,\widetilde{\\!{Z}}_{1}\big{)}-\ell\big{(}\mathfrak{m}(\nu_{n,(1)}),\,\widetilde{\\!{Z}}_{1}\big{)}\Big{]}\,.$
(11)

The right hand side of (11) measures the expected change in the loss function
subject to resampling one training data point, which connects generalization
error to stability of $\mathfrak{m}$ with respect to the training data. We
will next quantify this stability precisely, in terms of functional
derivatives.

######  Assumption 3.

In addition to Assumption 2,

  1. (i)

The loss function $\ell$ is $\mathcal{C}^{1}$ (cf. Definition 2.1).

  2. (ii)

When restricted to $\nu\in\mathcal{P}_{2}(\mathcal{Z})$, the training map
$\nu\mapsto\mathfrak{m}(\nu)$ is $\mathcal{C}^{1}$, in the sense of Definition
2.2. (In particular, this requires
$\mathfrak{m}(\nu)\in\mathcal{P}_{2}(\Theta)$ for all
$\nu\in\mathcal{P}_{2}(\mathcal{Z})$.)

######  Theorem 3.2.

Given Assumption 3, the weak generalization error has the representation

$\mathrm{gen}(\mathfrak{m},\nu_{\mathrm{pop}})=\mathbb{E}_{\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1}}\Big{[}\int_{0}^{1}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}_{(1)}(\lambda),\,\widetilde{\\!{Z}}_{1},\theta\big{)}\Big{)}\big{(}\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1)})\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda\Big{]}.$

Note that Theorem 3.2 holds for all differentiable loss functions, including
non-convex cases. Assuming convexity of the loss function with respect to the
parameter measure, we can also apply the functional derivative to provide a
lower bound on the WGE.

######  Proposition 3.3.

Given Assumption 3, suppose the loss function is convex with respect to $m$,
i.e. for all $m_{1},m_{2}\in\mathcal{P}_{2}(\Theta)$, and $\lambda\in[0,1]$,
$\ell\big{(}\lambda
m_{1}+(1-\lambda)m_{2},z\big{)}\leq\lambda\ell(m_{1},z)+(1-\lambda)\ell(m_{2},z)$.
Then

$\displaystyle\mathrm{gen}(\mathfrak{m},\nu_{\mathrm{pop}})\geq\mathbb{E}_{\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1}}\Big{[}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}(\mathfrak{m}(\nu_{n,(1)}),\,\widetilde{\\!{Z}}_{1},\theta)\Big{)}\mathfrak{m}(\nu_{n})(\mathrm{d}\theta)\Big{]}.$

By applying the functional linear derivative to the term
$\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1)})$ in Theorem 3.2, we can
provide yet another representation of the WGE.

######  Theorem 3.4.

The WGE can be written

$\mathrm{gen}(\mathfrak{m},\nu_{\mathrm{pop}})=\frac{1}{n}\mathbb{E}_{\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1}}\big{[}h(\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1})\big{]},$
(12)

where

$\begin{split}h(\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1})=\int_{0}^{1}\int_{0}^{1}\int_{\mathcal{Z}}\Big{(}\int_{\Theta}\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}_{(1)}(\lambda),\,\widetilde{\\!{Z}}_{1},\theta\big{)}\frac{\delta\mathfrak{m}}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),z\big{)}(\mathrm{d}\theta)\Big{)}\,\big{(}\delta_{Z_{1}}-\delta_{\tilde{Z}_{1}}\big{)}(\mathrm{d}z)\mathrm{d}\tilde{\lambda}\,\mathrm{d}\lambda.\end{split}$

######  Remark 3.5.

The representation in (12) reveals that the convergence rate of the WGE is at
worst $\mathcal{O}({1}/{n})$ provided that
$\mathbb{E}_{\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1}}[h(\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1})]\leq\mathcal{O}(1)$
with respect to $n\to\infty$.

###  3.2 LGE via functional derivatives

We now consider the $L_{2}$-generalization error
$\mathrm{gen}_{L_{2}}(\mathfrak{m},\nu)$. Our first result shows that,
similarly to the WGE, the LGE can be controlled in terms of the functional
derivatives of $\ell$.

######  Theorem 3.6.

Given Assumption 3, the $L_{2}$-generalization error satisfies

$\mathrm{gen}_{L_{2}}(\mathfrak{m},\nu_{\mathrm{pop}})<\frac{4K}{n}+2K^{1/2}\mathbb{E}_{\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2}}\big{[}h_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})^{2}\big{]}^{1/2}+\mathbb{E}_{\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2}}\big{[}h_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})^{2}\big{]},$

where

$\begin{split}h_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})&=\int_{0}^{1}\int_{\Theta}\int_{\mathcal{Z}}\Big{(}\frac{\delta\ell}{\delta
m}(\mathfrak{m}_{(1,2)}(\lambda),z,\theta)\Big{)}\big{(}\delta_{Z_{1}}-\nu_{\mathrm{pop}}\big{)}(\mathrm{d}z)\big{(}\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1,2)})\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda,\\\
K&=\max_{Z\in\\{Z_{1},\tilde{Z}_{1}\\}}\mathbb{E}\big{[}\big{(}\ell(\mathfrak{m}(\nu_{n}),Z)\big{)}^{2}\big{]}.\end{split}$

Similarly to Theorem 3.4, by applying the functional derivative to
$\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1,2)})$, we can derive a further
upper bound on LGE.

######  Theorem 3.7.

Given Assumption 3, with $K$ as in Theorem 3.6, we have the upper bound on LGE

$\mathrm{gen}_{L_{2}}(\mathfrak{m},\nu_{\mathrm{pop}})\leq\frac{1}{n}\bigg{(}4K+2K^{1/2}\mathbb{E}_{\mathbf{Z}_{2},\mathbf{\,\widetilde{\\!{Z}}}_{2}}\Big{[}\tilde{h}_{2}(\mathbf{Z}_{2},\mathbf{\,\widetilde{\\!{Z}}}_{2})^{2}\Big{]}^{1/2}+\frac{1}{n}\mathbb{E}_{\mathbf{Z}_{2},\mathbf{\,\widetilde{\\!{Z}}}_{2}}\Big{[}\tilde{h}_{2}(\mathbf{Z}_{2},\mathbf{\,\widetilde{\\!{Z}}}_{2})^{2}\Big{]}\bigg{)},$

where, with a slight abuse of notation,

$\displaystyle\tilde{h}_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})$
$\displaystyle=\int_{0}^{1}\int_{0}^{1}\int_{\Theta}\int_{\mathcal{Z}}\Big{(}\frac{\delta\ell}{\delta
m}(\mathfrak{m}_{(1,2)}(\lambda),z,\theta)\Big{)}\big{(}\delta_{Z_{1}}-\nu_{\mathrm{pop}}\big{)}(\mathrm{d}z)\Delta_{\mathfrak{m}}(\mathbf{Z}_{2},\mathbf{\,\widetilde{\\!{Z}}}_{2};\mathrm{d}\theta)\,\mathrm{d}\lambda\mathrm{d}\tilde{\lambda},$
$\displaystyle\Delta_{\mathfrak{m}}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2};\mathrm{d}\theta)$
$\displaystyle=\int_{\mathcal{Z}}\Big{(}\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1,2)}(\tilde{\lambda}),z^{\prime})(\mathrm{d}\theta)\Big{)}\big{(}\delta_{Z_{1}}-\delta_{\tilde{Z}_{1}}+\delta_{Z_{2}}-\delta_{\tilde{Z}_{2}}\big{)}(\mathrm{d}z^{\prime}).$

######  Remark 3.8.

The term $4K/n$ in the upper bound on LGE in Theorem 3.6 and Proposition 5.6,
corresponds loosely to the estimator’s bias and is also observed in [Bousquet
and Elisseeff, 2002, Aminian et al., 2021b, Harutyunyan et al., 2021].

######  Remark 3.9.

Theorem 3.7 yields an $\mathcal{O}(1/n)$ bound on LGE, provided
$\mathbb{E}_{\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}_{2}}}[\tilde{h}^{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})]\leq\mathcal{O}(1)$
as $n\to\infty$.

##  4 KL-regularized risk minimization

In the last section, we gave general conditions under which we can establish
convergence rates for generalization error, in terms of the functional
derivatives of the loss function and training map. Practically, these
conditions are only useful if one can verify that they hold in specific
examples. In this section, we will provide an intermediate step in this
direction, where we assume the training map $\mathfrak{m}$ is chosen to
minimize a regularized version of the empirical risk. This will provide us
with criteria which can be easily verified in practical examples; we will see
this fully in Section 5.

We will frequently use $\mathfrak{m}_{\theta}$ to represent the density of a
measure or measure-valued functional, that is,
$\mathfrak{m}(\nu)(\mathrm{d}\theta)=\mathfrak{m}_{\theta}(\nu;\theta)d\theta$.
For probability distributions $m$ and $m^{\prime}$ with positive densities
$m_{\theta}(\theta)$ and $m^{\prime}_{\theta}(\theta)$ we define the
Kullback–Leibler divergence
$\mathrm{KL}(m^{\prime}\|m)=\int\log\big(\frac{m^{\prime}_{\theta}(\theta)}{m_{\theta}(\theta)}\big{missing})m_{\theta}(\theta)\mathrm{d}\theta$,
and $\mathrm{KL}(m^{\prime}\|m)=\infty$ otherwise333Extending to the case
where $m$ and $m^{\prime}$ are equivalent but do not have (Lebesgue) densities
will not be needed here.. We will also write the Kullback–Leibler divergence
in terms of the densities, with the abuse of notation
$\mathrm{KL}(m^{\prime}\|m)=\mathrm{KL}(m_{\theta}^{\prime}\|m_{\theta})=\mathrm{KL}(m^{\prime}\|m_{\theta})$
as convenient.

We will consider training maps which minimize the regularized objective
$\mathcal{V}^{\beta,\sigma}$ given by the sum of a risk function and KL-
divergence, that is,

$\displaystyle\mathcal{V}^{\beta,\sigma}(m,\nu)=\mathrm{R}(m,\nu)+\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(m\|\gamma^{\sigma}),$
(13)

where $R(m,\nu)=\mathbb{E}_{Z\sim\nu}[\ell(m,z)]$ and
$\gamma^{\sigma}(\theta)=\frac{1}{F^{\sigma}}\exp\\{-\frac{1}{\sigma^{2}}U(\theta)\big{\\}}$
is a Gibbs measure which serves as prior to the parameter measure; here
$F^{\sigma}$ is simply a normalizing constant to ensure
$\int\gamma^{\sigma}(\theta)\mathrm{d}\theta=1$, and we call
$U:\Theta\to\mathbb{R}$ the ‘regularizing potential’.

For notational convenience, we similarly define
$\tilde{\gamma}^{\sigma}_{p}=\frac{1}{\tilde{F}^{\sigma}}\exp\\{-\frac{1}{\sigma^{2}}U(\theta)+\|\theta\|^{p}\big{\\}}$,
this density will play a particularly useful role in obtaining our bounds. We
know that $F^{\sigma},\tilde{F}^{\sigma}_{p}<\infty$ under the following
assumption, which we will use for our results in this section.

######  Assumption 4.

For a fixed $p\geq 2$, there exists $g:\mathcal{P}_{2}(\Theta)\to(0,\infty)$
and $g_{1}:\mathcal{P}_{2}(\Theta)\times\Theta\to(0,\infty)$ such that,

  1. (i)

The loss function is $\ell$ is $\mathcal{C}^{2}$ (Definition 2.1),
nonnegative, and convex with respect to $m$;

  2. (ii)

For all $z\in\mathcal{Z}$ and $m\in\mathcal{P}_{p}(\Theta)$, the loss
satisfies

$\big{|}\ell(m,z)\big{|}\leq g(m)\big{(}1+\|z\|^{2}\big{)}$

and the derivative of the loss satisfies

$\Big{|}\frac{\delta\ell}{\delta m}(m,z,\theta)\Big{|}\leq
g_{1}(m,\theta)\big{(}1+\|z\|^{2}\big{)};$

  3. (iii)

For all $m\in\mathcal{P}_{p}(\Theta)$, the regularizing potential $U$
satisfies
$\lim_{\|\theta\|\to\infty}\frac{U(\theta)}{\|\theta\|^{p}+g_{1}(m,\theta)}=\infty$;

  4. (iv)

There exists $C_{\theta}>0$ such that, for all
$m,m^{\prime}\in\mathcal{P}_{p}(\Theta)$,

$\mathbb{E}_{\theta\sim
m^{\prime}}\Big{[}\big{(}g_{1}(m,\theta)\big{)}^{2}\Big{]}^{1/2}\leq
C_{\theta}\Big{(}1+\mathbb{E}_{\theta\sim
m^{\prime}}\big{[}\|\theta\|^{p}\big{]}+\mathbb{E}_{\theta\sim
m}\big{[}\|\theta\|^{p}\big{]}\Big{)},$ (14)

  5. (v)

We have the pointwise integrability conditions
$g(\tilde{\gamma}_{p}^{\sigma})<\infty$ and, for all
$\nu\in\mathcal{P}_{2}(\mathcal{Z})$ and
$m,m^{\prime}\in\mathcal{P}_{p}(\Theta)$;

$\mathbb{E}_{\theta,\theta^{\prime}\sim
m}\mathbb{E}_{z\sim\nu}\Big{[}\Big{(}\frac{\delta^{2}\mathrm{\ell}}{\delta
m^{2}}(m,z,\theta,\theta^{\prime})\Big{)}^{2}\Big{]}^{1/2}<\infty.$ (15)

We begin by identifying the minimizer of $\mathcal{V}^{\beta,\sigma}$ in the
following lemma; the proof, together with others in this section, is deferred
to Appendix D.

######  Lemma 4.1.

Under Assumption 4, the unique minimizer of
$m\mapsto\mathcal{V}^{\beta,\sigma}(m,\nu)$, denoted
$\mathfrak{m}^{\beta,\sigma}(\nu)$ exists, and will be called the _Gibbs
measure_ associated with $\mathcal{V}^{\beta,\sigma}$. It is the fixed point
of $m\mapsto M(m,\nu)$, where
$M:\mathcal{P}_{p}(\Theta)\times\mathcal{P}_{2}(\mathcal{Z})\to\mathcal{P}_{p}(\Theta)$
is defined in terms of the density of its output

$\displaystyle
M_{\theta}(m,\nu;\theta)=\frac{1}{F_{\beta,\sigma}}\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\Big{[}\frac{\delta\mathrm{R}}{\delta
m}(m,\nu,\theta)+\frac{1}{2\beta^{2}}U(\theta)\Big{]}\Bigg{\\}},$ (16)

for $F_{\beta,\sigma}$ is a normalization constant. Furthermore, the map
$\nu\mapsto\mathfrak{m}^{\beta,\sigma}(\nu)$ is $\mathcal{C}^{1}$.

We observe that, unless $\frac{\delta\mathrm{R}}{\delta m}$ does not depend on
$m$ (i.e. unless $\mathrm{R}$ is linear in $m$), (16) does not provide an
explicit representation of $\mathfrak{m}^{\beta,\sigma}(\nu_{n})$, but instead
describes it implicitly in terms of a fixed point.

Applying Theorems 3.4 and 3.7, we can drive upper bounds on the generalization
errors.

######  Theorem 4.2 (WGE of the Gibbs measure).

Given Assumption 4 and
$\mathbb{E}_{Z\sim\nu_{\mathrm{pop}}}\big{[}\|Z\|^{8}\big{]}<\infty$, the weak
generalization error satisfies

$|\mathrm{gen}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})|\leq\frac{c}{n}\frac{2\beta^{2}}{\sigma^{2}}\mathbb{E}_{Z_{1}}\Big{[}\Big{(}1+\|Z_{1}\|^{2}\Big{)}^{4}\Big{]}$

where
$c=\frac{9\sqrt{2}}{2}C_{\theta}^{2}\Big{[}1+2\int_{\Theta}\|\theta\|^{p}\tilde{\gamma}_{p}^{\sigma}(\mathrm{d}\theta)+4\frac{g(\tilde{\gamma}^{\sigma}_{p})}{n}\Big{]}^{2}>0$.

######  Theorem 4.3 (LGE of the Gibbs measure).

Given Assumption 4 and
$\mathbb{E}_{Z\sim\nu_{\mathrm{pop}}}\big{[}\|Z\|^{16}\big{]}<\infty$, the
$L_{2}$ generalization error satisfies

$\begin{split}&\mathrm{gen}_{L_{2}}(\mathfrak{m}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})\leq\frac{1}{n}\Big{(}4K+2K^{1/2}A^{1/2}+\frac{A}{n}\Big{)},\end{split}$
(17)

where

$\begin{split}&A=(2\times
5^{4})\Big{(}\frac{2\beta^{2}}{\sigma^{2}}\Big{)}^{2}C_{\theta}^{4}\Big{(}1+2\int_{\Theta}\|\theta\|^{p}\tilde{\gamma}_{p}^{\sigma}(\mathrm{d}\theta)+8g(\tilde{\gamma}^{\sigma}_{p})\Big{)}^{4}\mathbb{E}_{Z_{1}}[(1+\|Z_{1}\|^{2})^{8}]<\infty,\end{split}$

and
$K=\max_{Z\in\\{Z_{1},\tilde{Z}_{1}\\}}\mathbb{E}\big{[}\big{(}\ell(\mathfrak{m}^{\beta,\sigma}(\nu_{n}),Z)\big{)}^{2}\big{]}$.

Expected population risk upper bound: Using Theorem 4.2, we can also provide
an upper bound on the expected population risk. Suppose that there exists
$\bar{m}\in\mathcal{P}_{2}(\Theta)$ such that $\mathrm{R}(\bar{m},\nu_{n})=0$,
cf. [Nishikawa et al., 2022, Appendix A]. From the optimization defining
$\mathfrak{m}^{\beta,\sigma}$, we know that

$\mathrm{R}(\mathfrak{m}^{\beta,\sigma}(\nu_{n}),\nu_{n})\leq\mathcal{V}^{\beta,\sigma}(\mathfrak{m}^{\beta,\sigma}(\nu_{n}),\nu_{n})\leq\mathcal{V}^{\beta,\sigma}(\bar{m},\nu_{n})=\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(\bar{m}\|\gamma^{\sigma})\,.$
(18)

Hence

$\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})\leq\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{n})+\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(\bar{m}\|\gamma^{\sigma})\,.$
(19)

Theorem 4.2 implies that

$\begin{split}\Big{|}\mathbb{E}_{\mathbf{Z}_{n}}\big{[}\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})\big{]}\Big{|}&\leq\Big{|}\mathbb{E}_{\mathbf{Z}_{n}}\Big{[}\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{n})\Big{]}\Big{|}+\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(\bar{m}\|\gamma^{\sigma})\\\
&\leq\frac{c}{n}\frac{2\beta^{2}}{\sigma^{2}}\mathbb{E}_{Z_{1}}\Big{[}\big{(}1+\|Z_{1}\|^{2}\big{)}^{4}\Big{]}+\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(\bar{m}\|\gamma^{\sigma})\,.\end{split}$

Consequently, setting $\beta$ to be of order $n^{1/4}$ and using Theorem 4.2,
we deduce that expected population risk,
$\big{|}\mathbb{E}_{\mathbf{Z}_{n}}\big{[}\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})\big{]}\big{|}$,
is of order $n^{-1/2}$. This agrees in principle with the order of error
computed in [Nitanda et al., 2021] for specific cases.

$L_{2}$ population risk upper bound: Similarly, squaring and taking the
expected value of (19), and using Theorem 4.3, indicates that taking $\beta$
to be of order $n^{1/6}$ we have that
$\mathbb{E}_{\mathbf{Z}_{n}}\big{[}\big{|}\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})\big{|}^{2}\big{]}$,
is of order $n^{-2/3}$.

##  5 Application: Overparameterized one-hidden layer NN

We now highlight the application of our model to an overparameterized one-
hidden layer neural network in the mean-field regime [Hu et al., 2020, Mei et
al., 2018]. The loss function for an overparameterized one-hidden layer NN is
represented in (4). For simplicity, we also write
$(\hat{y},y)\mapsto\ell_{o}(\hat{y},y)$ for the loss function, where
$\hat{y}=\mathbb{E}_{\theta\sim m}[\phi(\theta,x)]$ is the predicted output of
the overparameterized one-hidden layer NN. We assume that the risk function
$\mathrm{R}$ is given by
$\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})=\mathbb{E}_{(X,Y)\sim\nu_{\mathrm{pop}}}[\ell_{o}(\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{n})}[\phi(\theta,X)],Y)]$,
as is done in mean-field models of one-hidden layer neural networks [Hu et
al., 2020, Mei et al., 2019, Tzen and Raginsky, 2020], and is KL-regularized
as seen previously. We make the following assumptions on an overparameterized
one-hidden layer NN to investigate generalization performance:

######  Assumption 5.

For a constant $p\geq 2$, the regularizing potential $U$ satisfies
$\lim_{\|\theta\|\to\infty}\frac{U(\theta)}{\|\theta\|^{p}}=\infty,$ the loss
$\ell_{o}$ is convex and nonnegative, and there exist finite constants
$L_{\ell},L_{\ell,1},L_{\ell,2}$, and $L_{\phi}$ such that, for all
$(x,y)\in\mathcal{X}\times\mathcal{Y}$ and $\theta\in\Theta$,

$\begin{split}&\big{|}\ell_{o}(\hat{y},y)\big{|}\leq
L_{\ell}(1+\|\hat{y}\|^{2}+\|y\|^{2}),\qquad|\partial_{\hat{y}}\ell_{o}(\hat{y},y)\big{|}\leq
L_{\ell,1}(1+\|\hat{y}\|+\|y\|),\qquad\big{|}\partial_{\hat{y}\hat{y}}\ell_{o}(\hat{y},y)\big{|}\leq
L_{\ell,2},\\\ \end{split}$

and

$\big{|}\phi(\theta,x)\big{|}\leq L_{\phi}(1+\|x\|)(1+\|\theta\|^{2}).$

######  Remark 5.1 (Activation function examples).

Recall that $\phi(\theta,x)=a\varphi(w\cdot x)$, where $a$ is the parameter of
the outer layer and $\varphi(\cdot)$ is the activation function. If the
activation function is of linear growth, then Assumption 5 is satisfied. Note
that the ReLU, Heaviside unit-step, tanh and sigmoid activation functions are
of this type, and no smoothness assumption is needed.

######  Remark 5.2 (Loss function examples).

Twice differentiable loss functions which are of quadratic growth with
derivatives of linear growth satisfy Assumption 5. For example, the quadratic,
product-type (as in [Bartlett et al., 2006]), and logcosh loss functions [Wang
et al., 2022] are of this type.

We begin by verifying Assumption 4, for one-hidden layer neural network model
under Assumption 5, in the following lemma; the proof, together with others in
this section, is deferred to Appendix E. While the constants are not given
precisely here, they can be explicitly determined with only moderate effort,
if so desired.

######  Lemma 5.3.

Under Assumption 5, there exists a constant $c>0$ independent of $n$ such
that, for all $z\in\mathcal{Z}$, $\theta\in\Theta$,
$m\in\mathcal{P}_{p}(\Theta)$, and $\nu\in\mathcal{P}_{2}(\mathcal{Z})$,

$\displaystyle\big{|}\ell(m,z)\big{|}$ $\displaystyle\leq
c\Big{[}\mathbb{E}_{\theta\sim
m}[1+\|\theta\|^{2}]^{2}\Big{]}\big{(}1+\|z\|^{2}\big{)},$
$\displaystyle\Big{|}\frac{\delta\ell}{\delta m}(m,z,\theta)\Big{|}$
$\displaystyle\leq c\Big{[}1+\|\theta\|^{4}+\mathbb{E}_{\theta^{\prime}\sim
m}[\|\theta^{\prime}\|^{4}]\Big{]}\big{(}1+\|z\|^{2}\big{)},$
$\displaystyle\mathbb{E}_{\theta,\theta^{\prime}\sim
m}\mathbb{E}_{z\sim\nu}\Big{[}\Big{(}\frac{\delta^{2}\mathrm{\ell}}{\delta
m^{2}}(m,z,\theta,\theta^{\prime})\Big{)}^{2}\Big{]}^{1/2}$ $\displaystyle\leq
c\Big{[}\mathbb{E}_{\theta\sim
m}[1+\|\theta\|^{4}]\Big{]}\mathbb{E}_{Z\sim\nu}[1+\|Z\|^{2}],$ $\displaystyle
K=\max_{Z\in\\{Z_{1},\,\widetilde{\\!{Z}}_{1}\\}}\mathbb{E}\big{[}\big{(}\ell(\mathfrak{m}(\nu_{n}),Z)\big{)}^{2}\big{]}$
$\displaystyle\leq
c\,\mathbb{E}_{Z\sim\nu_{\mathrm{pop}}}\Big{[}(1+\|Z\|^{2})^{6}\Big{]}.$

In particular, if $p\geq 8$ in Assumption 5, then Assumption 4 is satisfied
for some choice of $C_{\theta}<\infty$, and if
$\mathbb{E}_{Z\sim\nu_{\mathrm{pop}}}[\|Z\|^{12}]<\infty$ then $K<\infty.$

######  Remark 5.4.

With stronger assumptions on the loss and activation function, it is possible
(using the same techniques as in Lemma 5.3) to reduce the order of the moment
assumption which needs to be made on $Z$.

We now derive upper bounds on WGE and LGE of the overparameterized one hidden-
layer NN by directly combining Lemma 5.3 with Theorems 4.2 and 4.3,
respectively.

######  Proposition 5.5 (WGE of overparameterized one-hidden layer NN).

Given Assumption 5 with $p\geq 8$, and
$\mathbb{E}_{Z\sim\nu_{\mathrm{pop}}}\big{[}\|Z\|^{8}\big{]}<\infty$, there
exists a constant $c>0$ such that

$|\mathrm{gen}(\mathfrak{m}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})|\leq\frac{c\beta^{2}}{n}.$

######  Proposition 5.6 (LGE of overparameterized one-hidden layer NN).

Given Assumption 5 with $p\geq 8$, and
$\mathbb{E}_{Z\sim\nu_{\mathrm{pop}}}\big{[}\|Z\|^{16}\big{]}<\infty$, there
exists a constant $c>0$ such that

$\begin{split}&\mathrm{gen}_{L_{2}}(\mathfrak{m}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})\leq\frac{c\beta^{2}}{n}.\end{split}$

It is instructive to compare our results to those of Nishikawa et al. [2022],
Nitanda et al. [2021]. There, building on [Chen et al., 2020, Lemma 5.5] and
Rademacher complexity analysis, the authors study generalization error for
one-hidden layer NN for binary classification and square loss regression
problems. The authors make the key assumption that there exists
$\bar{m}\in\mathcal{P}(\Theta)$ such that $\mathrm{R}(\bar{m},\nu_{n})=0$. We
see in [Nishikawa et al., 2022, Appendix A], that there is a constant $c>0$
such that, for $\delta\in(0,1)$, with probability $1-\delta$ we have

$\displaystyle\Big{(}\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu_{n}),\nu_{n})\Big{)}\leq\frac{c}{\sqrt{n}}\Big{(}\sqrt{\mathrm{KL}(\bar{m}\|\gamma^{\sigma})}+\sqrt{\log(1/\delta)}\Big{)}\,.$
(20)

In contrast, our bounds are non-asymptotic and hold under verifiable
assumptions on model inputs.

##  6 Conclusions and future works

Our study introduces a novel framework for analyzing the generalization error
of risk functions. The framework utilizes calculus on the space of probability
measures, which allows us to gain a deeper understanding of the factors that
influence the generalization of machine learning models. We demonstrate the
efficacy of our framework by applying it to learning problems within the mean-
field regime. This application results in the derivation of new bounds for
overparameterized models that are particularly relevant to one-hidden layer
neural networks.

In future work, it is promising to extend our methods to establish upper
bounds for weak and $L_{2}$ generalization errors of deep neural networks, via
the mean-field analysis outlined in [Sirignano and Spiliopoulos, 2022], Chizat
et al. [2022], Jabir et al. [2019] and Geshkovski et al. [2023]. Additionally,
we intend to apply our framework to a broader range of examples, beyond mean-
field models, while relaxing the regularization requirements. Our goal is to
develop comparable generalization and stability bounds for specific
algorithms.

## Acknowledgements

Authors acknowledge the support of the UKRI Prosperity Partnership Scheme
(FAIR) under EPSRC Grant EP/V056883/1 and the Alan Turing Institute. Samuel N.
Cohen and Łukasz Szpruch also acknowledge the support of the Oxford–Man
Institute for Quantitative Finance.

We also thank Joe Colvin (former PhD student at MAC-MIGS - EPSRC-funded
doctoral school in Mathematical Modelling, Analysis and Computation based in
Edinburgh) for his contributions to the initial draft of this manuscript.

## References

  * Allen-Zhu et al. [2019] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang.  Learning and generalization in overparameterized neural networks, going beyond two layers.  _Advances in neural information processing systems_ , 32, 2019.
  * Alquier et al. [2016] Pierre Alquier, James Ridgway, and Nicolas Chopin.  On the properties of variational approximations of Gibbs posteriors.  _The Journal of Machine Learning Research_ , 17(1):8374–8414, 2016.
  * Aminian et al. [2021a] Gholamali Aminian, Yuheng Bu, Laura Toni, Miguel Rodrigues, and Gregory Wornell.  An exact characterization of the generalization error for the gibbs algorithm.  _Advances in Neural Information Processing Systems_ , 34:8106–8118, 2021a.
  * Aminian et al. [2021b] Gholamali Aminian, Laura Toni, and Miguel RD Rodrigues.  Information-theoretic bounds on the moments of the generalization error of learning algorithms.  In _2021 IEEE International Symposium on Information Theory (ISIT)_ , pages 682–687. IEEE, 2021b.
  * Aminian et al. [2022a] Gholamali Aminian, Yuheng Bu, Gregory Wornell, and Miguel Rodrigues.  Tighter expected generalization error bounds via convexity of information measures.  In _IEEE International Symposium on Information Theory (ISIT)_ , 2022a.
  * Aminian et al. [2022b] Gholamali Aminian, Saeed Masiha, Laura Toni, and Miguel RD Rodrigues.  Learning algorithm generalization error bounds via auxiliary distributions.  _arXiv preprint arXiv:2210.00483_ , 2022b.
  * Amit et al. [2022] Ron Amit, Baruch Epstein, Shay Moran, and Ron Meir.  Integral probability metrics PAC-Bayes bounds.  _arXiv preprint arXiv:2207.00614_ , 2022.
  * Arora et al. [2018] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang.  Stronger generalization bounds for deep nets via a compression approach.  In _International Conference on Machine Learning_ , pages 254–263, 2018.
  * Arora et al. [2019a] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang.  Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks.  In _International Conference on Machine Learning_ , pages 322–332. PMLR, 2019a.
  * Arora et al. [2019b] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang.  On exact computation with an infinitely wide neural net.  In _Advances in Neural Information Processing Systems_ , 2019b.
  * Asadi et al. [2018] Amir Asadi, Emmanuel Abbe, and Sergio Verdú.  Chaining mutual information and tightening generalization bounds.  In _Advances in Neural Information Processing Systems_ , pages 7234–7243, 2018.
  * Bartlett and Mendelson [2002] Peter L Bartlett and Shahar Mendelson.  Rademacher and Gaussian complexities: Risk bounds and structural results.  _Journal of Machine Learning Research_ , 3(Nov):463–482, 2002.
  * Bartlett et al. [2006] Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe.  Convexity, classification, and risk bounds.  _Journal of the American Statistical Association_ , 101(473):138–156, 2006.
  * Bartlett et al. [2017] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky.  Spectrally-normalized margin bounds for neural networks.  In _Advances in Neural Information Processing Systems_ , pages 6240–6249, 2017.
  * Bartlett et al. [2020] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler.  Benign overfitting in linear regression.  _Proceedings of the National Academy of Sciences_ , 117(48):30063–30070, 2020.
  * Bartlett et al. [2021] Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin.  Deep learning: a statistical viewpoint, 2021.
  * Belkin et al. [2018] Mikhail Belkin, Siyuan Ma, and Soumik Mandal.  To understand deep learning we need to understand kernel learning.  In _International Conference on Machine Learning_ , pages 540–548, 2018.
  * Belkin et al. [2019a] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.  Reconciling modern machine-learning practice and the classical bias–variance trade-off.  _Proceedings of the National Academy of Sciences_ , 116(32):15849–15854, 2019a.
  * Belkin et al. [2019b] Mikhail Belkin, Daniel Hsu, and Ji Xu.  Two models of double descent for weak features.  _arXiv preprint arXiv:1903.07571_ , 2019b.
  * Bousquet and Elisseeff [2002] Olivier Bousquet and André Elisseeff.  Stability and generalization.  _J. Mach. Learn. Res._ , 2:499–526, March 2002.  ISSN 1532-4435.  doi: 10.1162/153244302760200704.  URL https://doi.org/10.1162/153244302760200704.
  * Bu et al. [2020] Yuheng Bu, Shaofeng Zou, and Venugopal V Veeravalli.  Tightening mutual information-based bounds on generalization error.  _IEEE Journal on Selected Areas in Information Theory_ , 1(1):121–130, 2020.
  * Bubeck and Sellke [2023] Sébastien Bubeck and Mark Sellke.  A universal law of robustness via isoperimetry.  _Journal of the ACM_ , 70(2):1–18, 2023.
  * Cao and Gu [2019] Yuan Cao and Quanquan Gu.  Generalization bounds of stochastic gradient descent for wide and deep neural networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Cao et al. [2021] Yuan Cao, Quanquan Gu, and Mikhail Belkin.  Risk bounds for over-parameterized maximum margin classification on sub-gaussian mixtures.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Cao et al. [2022] Yuan Cao, Zixiang Chen, Mikhail Belkin, and Quanquan Gu.  Benign overfitting in two-layer convolutional neural networks.  _arXiv preprint arXiv:2202.06526_ , 2022.
  * Cardaliaguet et al. [2019] Pierre Cardaliaguet, François Delarue, Jean-Michel Lasry, and Pierre-Louis Lions.  _The master equation and the convergence problem in mean field games_.  Princeton University Press, 2019.
  * Carmona and Delarue [2018] René Carmona and François Delarue.  _Probabilistic Theory of Mean Field Games with Applications I_.  Springer, 2018.
  * Catoni [2007] Olivier Catoni.  PAC-Bayesian supervised classification: the thermodynamics of statistical learning.  _arXiv preprint arXiv:0712.0248_ , 2007.
  * Chen et al. [2018] Yuansi Chen, Chi Jin, and Bin Yu.  Stability and convergence trade-off of iterative optimization algorithms.  _arXiv preprint arXiv:1804.01619_ , 2018.
  * Chen et al. [2019] Zixiang Chen, Yuan Cao, Difan Zou, and Quanquan Gu.  How much over-parameterization is sufficient to learn deep relu networks?  In _International Conference on Learning Representations_ , 2019.
  * Chen et al. [2020] Zixiang Chen, Yuan Cao, Quanquan Gu, and Tong Zhang.  A generalized neural tangent kernel analysis for two-layer neural networks.  _Advances in Neural Information Processing Systems_ , 33:13363–13373, 2020.
  * Chizat and Bach [2018] Lenaic Chizat and Francis Bach.  On the global convergence of gradient descent for over-parameterized models using optimal transport.  _Advances in neural information processing systems_ , 31, 2018.
  * Chizat et al. [2022] Lénaïc Chizat, Maria Colombo, Xavier Fernández-Real, and Alessio Figalli.  Infinite-width limit of deep linear neural networks.  _arXiv preprint arXiv:2211.16980_ , 2022.
  * Dziugaite and Roy [2017] Gintare Karolina Dziugaite and Daniel M Roy.  Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data.  In _Uncertainty in Artificial Intelligence_ , 2017.
  * Esposito et al. [2021] Amedeo Roberto Esposito, Michael Gastpar, and Ibrahim Issa.  Generalization error bounds via Rényi-, f-divergences and maximal leakage.  _IEEE Transactions on Information Theory_ , 2021.
  * Fang et al. [2019] Cong Fang, Hanze Dong, and Tong Zhang.  Over parameterized two-level neural networks can learn near optimal feature representations.  _arXiv preprint arXiv:1910.11508_ , 2019.
  * Fang et al. [2021] Cong Fang, Hanze Dong, and Tong Zhang.  Mathematical models of overparameterized neural networks.  _Proceedings of the IEEE_ , 109(5):683–703, 2021\.
  * Gálvez et al. [2021] Borja Rodríguez Gálvez, Germán Bassi, Ragnar Thobaben, and Mikael Skoglund.  Tighter expected generalization error bounds via Wasserstein distance.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Geshkovski et al. [2023] Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet.  The emergence of clusters in self-attention dynamics.  _arXiv preprint arXiv:2305.05465_ , 2023.
  * Gibbs [1902] Josiah Willard Gibbs.  Elementary principles of statistical mechanics.  _Compare_ , 289:314, 1902.
  * Golowich et al. [2018] Noah Golowich, Alexander Rakhlin, and Ohad Shamir.  Size-independent sample complexity of neural networks.  In _Conference On Learning Theory_ , pages 297–299, 2018.
  * Hafez-Kolahi et al. [2020] Hassan Hafez-Kolahi, Zeinab Golgooni, Shohreh Kasaei, and Mahdieh Soleymani.  Conditioning and processing: Techniques to improve information-theoretic generalization bounds.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Hardt et al. [2016] Moritz Hardt, Benjamin Recht, and Yoram Singer.  Train faster, generalize better: Stability of stochastic gradient descent, 2016.
  * Harutyunyan et al. [2021] Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan.  Information-theoretic generalization bounds for black-box learning algorithms.  _Advances in Neural Information Processing Systems_ , 34:24670–24682, 2021.
  * Hastie et al. [2019] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani.  Surprises in high-dimensional ridgeless least squares interpolation.  _arXiv preprint arXiv:1903.08560_ , 2019.
  * Hu et al. [2020] Kaitong Hu, Zhenjie Ren, David Šiška, and Łukasz Szpruch.  Mean-field langevin dynamics and energy landscape of neural networks, 2020\.
  * Jabir et al. [2019] Jean-François Jabir, David Šiška, and Łukasz Szpruch.  Mean-field neural odes via relaxed optimal control.  _arXiv preprint arXiv:1912.05475_ , 2019.
  * Jacot et al. [2018] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  _Advances in neural information processing systems_ , 31, 2018.
  * Jaynes [1957] Edwin T Jaynes.  Information theory and statistical mechanics.  _Physical review_ , 106(4):620, 1957.
  * Ji and Telgarsky [2020] Ziwei Ji and Matus Telgarsky.  Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks.  In _International Conference on Learning Representations_ , 2020.
  * Kuzborskij et al. [2019] Ilja Kuzborskij, Nicolò Cesa-Bianchi, and Csaba Szepesvári.  Distribution-dependent analysis of Gibbs-ERM principle.  In _Conference on Learning Theory_ , pages 2028–2054. PMLR, 2019\.
  * Li and Liang [2018] Yuanzhi Li and Yingyu Liang.  Learning overparameterized neural networks via stochastic gradient descent on structured data.  _Advances in neural information processing systems_ , 31, 2018.
  * Li et al. [2021] Zhu Li, Zhi-Hua Zhou, and Arthur Gretton.  Towards an understanding of benign overfitting in neural networks.  _arXiv preprint arXiv:2106.03212_ , 2021.
  * Liang and Rakhlin [2020] Tengyuan Liang and Alexander Rakhlin.  Just interpolate: Kernel “ridgeless” regression can generalize.  _The Annals of Statistics_ , 48(3):1329–1347, 2020.
  * Lopez and Jog [2018] Adrian Tovar Lopez and Varun Jog.  Generalization error bounds using Wasserstein distances.  In _2018 IEEE Information Theory Workshop (ITW)_ , pages 1–5. IEEE, 2018.
  * Ma et al. [2019] Chao Ma, Lei Wu, et al.  The generalization error of the minimum-norm solutions for over-parameterized neural networks.  _arXiv preprint arXiv:1912.06987_ , 2019.
  * Masiha et al. [2023] Saeed Masiha, Amin Gohari, and Mohammad Hossein Yassaee.  f-divergences and their applications in lossy compression and bounding generalization error.  _IEEE Transactions on Information Theory_ , 2023.
  * McAllester [2003] David A McAllester.  PAC-Bayesian stochastic model selection.  _Machine Learning_ , 51(1):5–21, 2003.
  * Mei and Montanari [2022] Song Mei and Andrea Montanari.  The generalization error of random features regression: Precise asymptotics and the double descent curve.  _Communications on Pure and Applied Mathematics_ , 75(4):667–766, 2022.
  * Mei et al. [2018] Song Mei, Andrea Montanari, and Phan-Minh Nguyen.  A mean field view of the landscape of two-layer neural networks.  _Proceedings of the National Academy of Sciences_ , 115(33):E7665–E7671, 2018.
  * Mei et al. [2019] Song Mei, Theodor Misiakiewicz, and Andrea Montanari.  Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit, 2019.
  * Montanari and Zhong [2020] Andrea Montanari and Yiqiao Zhong.  The interpolation phase transition in neural networks: Memorization and generalization under lazy training.  _arXiv preprint arXiv:2007.12826_ , 2020.
  * Mou et al. [2017] Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng.  Generalization bounds of SGLD for non-convex learning: Two theoretical viewpoints.  _arXiv preprint arXiv:1707.05947_ , 2017.
  * Neyshabur et al. [2015] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro.  Norm-based capacity control in neural networks.  In _Conference on Learning Theory_ , pages 1376–1401, 2015.
  * Neyshabur et al. [2018] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro.  A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks.  In _International Conference on Learning Representation_ , 2018.
  * Nishikawa et al. [2022] Naoki Nishikawa, Taiji Suzuki, Atsushi Nitanda, and Denny Wu.  Two-layer neural network on infinite dimensional data: global optimization guarantee in the mean-field regime.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Nitanda et al. [2021] Atsushi Nitanda, Denny Wu, and Taiji Suzuki.  Particle dual averaging: Optimization of mean field neural network with global convergence rate analysis.  _Advances in Neural Information Processing Systems_ , 34:19608–19621, 2021.
  * Perlaza et al. [2022] Samir M Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, and Stefano Rini.  Empirical risk minimization with relative entropy regularization: Optimality and sensitivity analysis.  In _2022 IEEE International Symposium on Information Theory (ISIT)_ , pages 684–689. IEEE, 2022.
  * Raginsky et al. [2016] Maxim Raginsky, Alexander Rakhlin, Matthew Tsao, Yihong Wu, and Aolin Xu.  Information-theoretic analysis of stability and bias of learning algorithms.  In _2016 IEEE Information Theory Workshop (ITW)_ , pages 26–30. IEEE, 2016.
  * Rahimi and Recht [2008] Ali Rahimi and Benjamin Recht.  Uniform approximation of functions with random bases.  In _2008 46th Annual Allerton Conference on Communication, Control, and Computing_ , pages 555–561. IEEE, 2008.
  * Rivasplata et al. [2020] Omar Rivasplata, Ilja Kuzborskij, Csaba Szepesvári, and John Shawe-Taylor.  PAC-Bayes analysis beyond the usual bounds.  _Advances in Neural Information Processing Systems_ , 33:16833–16845, 2020.
  * Russo and Zou [2019] Daniel Russo and James Zou.  How much does your data exploration overfit? controlling bias via information usage.  _IEEE Transactions on Information Theory_ , 66(1):302–323, 2019.
  * Sefidgaran et al. [2022] Milad Sefidgaran, Amin Gohari, Gael Richard, and Umut Simsekli.  Rate-distortion theoretic generalization bounds for stochastic learning algorithms.  In _Conference on Learning Theory_ , pages 4416–4463. PMLR, 2022\.
  * Sirignano and Spiliopoulos [2022] Justin Sirignano and Konstantinos Spiliopoulos.  Mean field analysis of deep neural networks.  _Mathematics of Operations Research_ , 47(1):120–152, 2022.
  * Spigler et al. [2019] Stefano Spigler, Mario Geiger, Stéphane d’Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart.  A jamming transition from under-to over-parametrization affects generalization in deep learning.  _Journal of Physics A: Mathematical and Theoretical_ , 52(47):474001, 2019.
  * Steinke and Zakynthinou [2020] Thomas Steinke and Lydia Zakynthinou.  Reasoning about generalization via conditional mutual information.  In _Conference on Learning Theory_ , pages 3437–3452. PMLR, 2020\.
  * Tolstikhin and Seldin [2013] Ilya O Tolstikhin and Yevgeny Seldin.  PAC-Bayes-empirical-Bernstein inequality.  _Advances in Neural Information Processing Systems_ , 26, 2013.
  * Tzen and Raginsky [2020] Belinda Tzen and Maxim Raginsky.  A mean-field theory of lazy training in two-layer neural nets: entropic regularization and controlled McKean–Vlasov dynamics.  _arXiv preprint arXiv:2002.01987_ , 2020.
  * Vapnik and Chervonenkis [2015] Vladimir N Vapnik and A Ya Chervonenkis.  On the uniform convergence of relative frequencies of events to their probabilities.  In _Measures of complexity_ , pages 11–30. Springer, 2015.
  * Wang et al. [2019] Hao Wang, Mario Diaz, José Cândido S Santos Filho, and Flavio P Calmon.  An information-theoretic view of generalization via Wasserstein distance.  In _2019 IEEE International Symposium on Information Theory (ISIT)_ , pages 577–581. IEEE, 2019.
  * Wang et al. [2022] Qi Wang, Yue Ma, Kun Zhao, and Yingjie Tian.  A comprehensive survey of loss functions in machine learning.  _Annals of Data Science_ , 9(2):187–212, 2022\.
  * Wu and Xu [2020] Denny Wu and Ji Xu.  On the optimal weighted $\ell_{2}$ regularization in overparameterized linear regression.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Xu and Raginsky [2017] Aolin Xu and Maxim Raginsky.  Information-theoretic analysis of generalization capability of learning algorithms.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Yin et al. [2019] Dong Yin, Ramchandran Kannan, and Peter Bartlett.  Rademacher complexity for adversarially robust generalization.  In _International conference on machine learning_ , pages 7085–7094. PMLR, 2019.

##  Appendix A Other related works

##### Expected generalization error:

The pioneering work in establishing upper bounds on the expected
generalization error through an information-theoretic approach was done by
Russo and Zou [2019] and Xu and Raginsky [2017]. They proposed using the
mutual information between the input training set and the output parameters.
Building upon their work, Bu et al. [2020] introduced tighter upper bounds
that consider the mutual information of individual samples. Asadi et al.
[2018] suggested the use of chaining mutual information, while Hafez-Kolahi et
al. [2020] and Steinke and Zakynthinou [2020] advocated conditioning and
processing techniques. In addition to mutual information, other researchers
have explored alternative information measures for establishing information-
theoretic bounds on generalization error. These measures include
$f$-divergences, $\alpha$-Rényi divergence, and generalized Jensen-Shannon
divergence, as demonstrated in the works of Esposito et al. [2021], Aminian et
al. [2022b], and Aminian et al. [2021b]. Wasserstein distance-based upper
bounds, which consider the discrepancy between the training sample data and
the output of a randomized learning algorithm, were proposed by Lopez and Jog
[2018], Wang et al. [2019], Gálvez et al. [2021], and Aminian et al. [2022a].
Furthermore, some studies have explored the application of rate-distortion
theory to analyze generalization error, as seen in the works of Masiha et al.
[2023] and Sefidgaran et al. [2022]. In our approach, we employ calculus on
probability spaces, allowing us to apply it to both the overparameterized
regime and the expected parametric loss scenarios, to derive an upper bound on
the weak generalization error and $L_{2}$ generalization error.

##### Generalization error and the parametric Gibbs measure:

The mean-field perspective on training gives close connections with ideas from
statistical mechanics. The parametric Gibbs measure (a.k.a. Gibbs Algorithm or
Gibbs Posterior) first introduced by [Gibbs, 1902] in statistical mechanics
and further studied by [Jaynes, 1957] describes the stable behaviour of an
ensemble of particles evolving subject to a potential, as is given by our
(parametric) loss function. The weak generalization error of the parametric
Gibbs measure with expected parametric loss has been studied from information-
theoretic and PAC-Bayesian perspectives. An information-theoretic upper bound
on weak (expected) generalization error, with a convergence rate of
$\mathcal{O}(1/n)$ is provided in [Raginsky et al., 2016] and [Aminian et al.,
2021a, Kuzborskij et al., 2019] for the parametric Gibbs measure, assuming a
bounded or sub-Gaussian parametric loss function, respectively. PAC-Bayesian
upper bounds for parametric Gibbs measures are also derived in [Alquier et
al., 2016, Aminian et al., 2021a]. The sensitivity of the parametric Gibbs
measure is studied in [Perlaza et al., 2022]. Our work differs from this body
of research by reparameterization of the loss function in terms of a parameter
measure, which results in a general Gibbs measure and can be applied to
overparameterized one-hidden layer NN under the mean-field regime. We also
provide an upper bounds on the expected generalization error of the parametric
Gibbs measure with weaker assumptions.

##### Other approaches to generalization error:

There are other approaches applied to study generalization error of general
learning problems, including VC dimension-based bounds, Rademacher complexity,
PAC-Bayesian bounds, and algorithmic stability-based bounds. A group of
studies [Neyshabur et al., 2015, 2018, Golowich et al., 2018, Arora et al.,
2018] has investigated the generalization error of neural networks via a
uniform convergence approach. However, these results are algorithm
independent. Rademacher complexity of functions classes like neural networks
have been studied by [Bartlett et al., 2017, Yin et al., 2019] to provide
upper bound on generalization error. These results based on VC dimension and
Rademacher complexity are often vacuous in NN scenarios [Dziugaite and Roy,
2017]. A data dependent Rademacher complexity bound is studied for
overparameterized NNs by Bubeck and Sellke [2023] which has a convergence rate
of $\mathcal{O}(1/\sqrt{n})$. Another approach is based on PAC-Bayes learning
[McAllester, 2003, Catoni, 2007]. In this approach high probability bounds are
derived on generalization error in terms of different information measures
[Rivasplata et al., 2020, Amit et al., 2022, Tolstikhin and Seldin, 2013].
Some works also studied the stability-based bounds for generalization error
[Bousquet and Elisseeff, 2002, Hardt et al., 2016, Mou et al., 2017, Chen et
al., 2018]. These approaches are not applied to derive the generalization
error upper bound in overparameterized neural networks by considering the
regularization.

##### Benign overfitting:

A body of literature has emerged to investigate the occurrence of benign
overfitting in overparameterized models, wherein accurate predictions are made
on unseen test data despite overfitting to the training data [Bartlett et al.,
2020]. Although this phenomenon has been explored in various studies, most of
them have been confined to linear models [Belkin et al., 2019b, Bartlett et
al., 2020, Hastie et al., 2019, Wu and Xu, 2020, Cao et al., 2021] and
kernel/random feature models [Belkin et al., 2018, Liang and Rakhlin, 2020,
Montanari and Zhong, 2020]. Recently, Li et al. [2021] have examined the
benign overfitting phenomenon in random feature models, which are defined as
two-layer neural networks with fixed random initialization parameters in the
first layer. Furthermore, the study by Cao et al. [2022] demonstrates that
benign overfitting can also occur in two-layer convolutional neural networks.
In contrast, our research focuses on the generalization error in the mean-
field regime, whereas these works assume that the input data dimension is
large enough with respect to the number of parameters, maintaining a constant
ratio to observe benign overfitting.

##### Mean-field:

Our study employs the mean-field framework utilized in a recent line of
research [Chizat and Bach, 2018, Mei et al., 2018, 2019, Fang et al., 2019,
2021, Sirignano and Spiliopoulos, 2022, Hu et al., 2020]. Chizat and Bach
[2018] establishes the convergence of gradient descent for training one-hidden
layer NNs with infinite width under certain structural assumptions. The study
of Mei et al. [2018] proves the global convergence of noisy stochastic
gradient descent and establishes approximation bounds between finite and
infinite neural networks. Furthermore, Mei et al. [2019] demonstrates that
this approximation error can be independent of the input dimension in certain
cases, and establishes that the residual dynamics of noiseless gradient
descent are close to the dynamics of NTK-based kernel regression under some
conditions. The mean-field approach is mostly restricted to one-hidden layer
NNs and the extension to Deep NNs is not trivial [Sirignano and Spiliopoulos,
2022, Chizat et al., 2022]. Fang et al. [2019] proposes a new concept known as
neural feature repopulation inspired by the mean field view. Lastly, the mean-
field approach’s performance with respect to other methods in feature learning
[Fang et al., 2021] suggests it is a viable option for analyzing one-hidden
layer NNs. The focus of our study is based on the work conducted by Hu et al.
[2020] in the area of non-linear functional minimization with KL
regularization. Their investigation demonstrated the linear convergence (in
continuous time) of the resulting mean-field Langevin dynamics under the
condition of sufficiently robust regularization.

##  Appendix B Preliminaries

##### Properties of functional derivatives

###### Proof of Lemma 2.3.

We know $\frac{\delta\mathfrak{m}}{\delta\nu}$ is a Borel signed measure. For
any Lebesgue null set $A$, we know $\mathfrak{m}(\nu)(A)=0$ for all $\nu$ and,
by uniqueness of the functional derivative, conclude
$\frac{\delta\mathfrak{m}}{\delta\nu}(a,\nu)(A)=0$. Therefore,
$\frac{\delta\mathfrak{m}}{\delta\nu}(a,\nu)(A)$ is absolutely continuous with
respect to Lebesgue measure, and by the Radon–Nikodym theorem we have

$\displaystyle\frac{\delta\mathfrak{m}}{\delta\nu}(a,\nu)(A)=\int_{A}G(a,\nu,x)dx,$
(21)

for some function $G$ integrable in $x$ and measurable in $a,\nu$. We now need
to show that $G(a,\nu,x)$ is a version of $\frac{\delta U}{\delta\nu}$ for
almost all $x$. For all $A$, we know

$\displaystyle\frac{1}{\text{Leb}(A)}\int_{A}G(a,\nu,x)dx$
$\displaystyle=\frac{1}{\text{Leb}(A)}\frac{\delta\mathfrak{m}}{\delta\nu}(A)=\frac{1}{\text{Leb}(A)}\frac{\delta}{\delta\nu}\Big{[}\int_{A}U(a,\nu,x)dx\Big{]}$
(22)
$\displaystyle=\frac{\delta}{\delta\nu}\Big{[}\frac{1}{\text{Leb}(A)}\int_{A}U(a,\nu,x)dx\Big{]},$
(23)

where $\text{Leb}(A)=\int_{A}1dx$. Taking $A$ small, we conclude (by linearity
of the functional derivative) that $G$ is a version of $\frac{\delta
U}{\delta\nu}$ for every $x$ which is a Lebesgue point of $G$ and $U$, that
is, for almost all $x$. ∎

##  Appendix C Proofs and details from Section 3

###### Proof of Lemma 3.1.

Recall that $\,\widetilde{\\!{Z}}_{1}\sim\nu_{\mathrm{pop}}$ is independent of
$\\{Z_{i}\\}_{i=1}^{n}$. Since elements of $\\{Z_{i}\\}_{i=1}^{n}$ are i.i.d.,
the WGE (9) can be written

$\begin{split}\mathrm{gen}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})&=\mathbb{E}_{\mathbf{Z_{n}}}\big{[}\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})\big{]}\\\
&=\mathbb{E}_{\mathbf{Z_{n}}}\Big{[}\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)\nu_{\mathrm{pop}}(\mathrm{d}z)-\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)\nu_{n}(\mathrm{d}z)\Big{]}\\\
&=\mathbb{E}_{\mathbf{Z_{n}}}\Big{[}\mathbb{E}_{\,\widetilde{\\!{Z}}_{1}}[\ell(\mathfrak{m}(\nu_{n}),\,\widetilde{\\!{Z}}_{1})]-\frac{1}{n}\sum_{i=1}^{n}\ell(\mathfrak{m}(\nu_{n}),Z_{i})\Big{]}\\\
&=\mathbb{E}_{\mathbf{Z_{n}}}\Big{[}\mathbb{E}_{\,\widetilde{\\!{Z}}_{1}}[\ell(\mathfrak{m}(\nu_{n}),\,\widetilde{\\!{Z}}_{1})]-\ell(\mathfrak{m}(\nu_{n}),Z_{1})\Big{]}.\end{split}$
(24)

Recall that in (1) we defined the perturbation
$\nu_{n,(1)}:=\nu_{n}+\frac{1}{n}\big{(}\delta_{\,\widetilde{\\!{Z}}_{1}}-\delta_{Z_{1}}\big{)},$
which corresponds to a data measure with one different data point, and that,
as $Z_{1}$ and $\,\widetilde{\\!{Z}}_{1}$ are i.i.d., we have

$\mathbb{E}_{\mathbf{Z_{n}}}\big{[}\ell(\mathfrak{m}(\nu_{n}),Z_{1})\big{]}=\mathbb{E}_{\mathbf{Z_{n}}}\mathbb{E}_{\,\widetilde{\\!{Z}}_{1}}\big{[}\ell(\mathfrak{m}(\nu_{n,(1)}),\,\widetilde{\\!{Z}}_{1})\big{]}\,.$
(25)

Combining this observation with (24) yields the representation (11). ∎

###### Proof of Theorem 3.2.

Recall that from (2),
$\nu_{n,(1)}=\nu_{n}+\frac{1}{n}(\delta_{\,\widetilde{\\!{Z}}_{1}}-\delta_{Z_{1}})$
and
$\mathfrak{m}_{(1)}(\lambda)=\mathfrak{m}(\nu_{n})+\lambda\left(\mathfrak{m}(\nu_{n,(1)})-\mathfrak{m}(\nu_{n})\right).$
Using the definition of the linear functional derivative (Definition 2.1) and
Lemma 3.1 we have

$\begin{split}\mathrm{gen}(\mathfrak{m},\nu)&=\mathbb{E}_{\mathbf{Z_{n}}}\mathbb{E}_{\,\widetilde{\\!{Z}}_{1}}\Big{[}\ell(\mathfrak{m}(\nu_{n}),\,\widetilde{\\!{Z}}_{1})-\ell(\mathfrak{m}(\nu_{n,(1)}),\,\widetilde{\\!{Z}}_{1})\Big{]}\,\\\
&=\mathbb{E}_{\mathbf{Z_{n}}}\mathbb{E}_{\,\widetilde{\\!{Z}}_{1}}\Big{[}\int_{0}^{1}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}(\mathfrak{m}_{(1)}(\lambda),\,\widetilde{\\!{Z}}_{1},\theta)\Big{)}\big{(}\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1)})\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda\Big{]}.\end{split}$

∎

###### Proof of Proposition 3.3.

As $\ell$ is convex and $\mathcal{C}^{1}$, we know that , for any
$m_{1},m_{2}\in\mathcal{P}_{2}(\Theta)$,

$\displaystyle\ell\big{(}\lambda m_{1}+(1-\lambda)m_{2},z\big{)}$
$\displaystyle\leq\lambda\ell(m_{1},z)+(1-\lambda)\ell(m_{2},z)$
$\displaystyle=\ell(m_{2},z)+\lambda\Big{(}\ell(m_{1},z)-\ell(m_{2},z)\Big{)}$
$\displaystyle=\ell(m_{2},z)+\lambda\int_{0}^{1}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\lambda^{\prime}m_{1}+(1-\lambda^{\prime})m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda^{\prime}.$

We also know that

$\displaystyle\ell\big{(}\lambda m_{1}+(1-\lambda)m_{2},z\big{)}$
$\displaystyle=\ell(m_{2},z)+\lambda\int_{0}^{1}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\lambda^{\prime}(\lambda
m_{1}+(1-\lambda)m_{2})+(1-\lambda^{\prime})m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda^{\prime},$

from which we conclude

$\displaystyle\int_{0}^{1}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\lambda^{\prime}m_{1}+(1-\lambda^{\prime})m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda^{\prime}$
$\displaystyle\geq\int_{0}^{1}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\lambda^{\prime}(\lambda
m_{1}+(1-\lambda)m_{2})+(1-\lambda^{\prime})m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda^{\prime}$
$\displaystyle=\frac{1}{1-\lambda}\int_{0}^{1-\lambda}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\lambda^{\prime}m_{1}+(1-\lambda^{\prime})m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda^{\prime}$

for all $\lambda\in[0,1)$. Observe that the RHS of this inequality is
essentially the same as the left hand side, but averaging over a smaller set
of $\lambda^{\prime}$ values. Repeating this argument and using continuity of
$\frac{\delta\ell}{\delta m}$ with respect to $m$, we conclude that, for every
$k>0$,

$\displaystyle\int_{0}^{1}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\lambda^{\prime}m_{1}+(1-\lambda^{\prime})m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda^{\prime}$
$\displaystyle\geq\frac{1}{(1-\lambda)^{k}}\int_{0}^{(1-\lambda)^{k}}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\lambda^{\prime}m_{1}+(1-\lambda^{\prime})m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta)\,\mathrm{d}\lambda^{\prime}$
$\displaystyle\to\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta)\quad\text{as
}k\to\infty.$

Substituting back, we obtain

$\ell\big{(}\lambda
m_{1}+(1-\lambda)m_{2},z\big{)}-\ell\big{(}m_{2},z\big{)}\geq\lambda\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta).$

In particular, taking $\lambda\to 1$ and using continuity,

$\ell\big{(}m_{1},z\big{)}-\ell\big{(}m_{2},z\big{)}\geq\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}m_{2},z,\theta\big{)}\Big{)}\big{(}m_{1}-m_{2}\big{)}(\mathrm{d}\theta).$

By definition of the derivative, we know

$\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}m_{2},z,\theta\big{)}\Big{)}m_{2}(\mathrm{d}\theta)=0,$

so this term can be omitted. The result now follows from Theorem 3.2, with
$m_{1}=\mathfrak{m}(\nu_{n})$ and $m_{2}=\mathfrak{m}(\nu_{n,(1)})$. ∎

###### Proof of Theorem 3.4.

We know from (1) that

$\nu_{n}-\nu_{n,(1)}=\frac{1}{n}\big{(}\delta_{Z_{1}}-\delta_{\,\widetilde{\\!{Z}}_{1}}\big{)}.$

Applying the functional derivative to $\mathfrak{m}$ and recalling the
definition of $\nu_{(1)}(\tilde{\lambda})$ in (2), for any Borel set
$B\subset\Theta$ we have

$\begin{split}\big{(}\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1)})\big{)}(B)&=\int_{0}^{1}\int_{\mathcal{Z}}\Big{(}\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1)}(\tilde{\lambda}),z)(B)\Big{)}(\nu_{n}-\nu_{n,(1,2)})(\mathrm{d}z)\mathrm{d}\tilde{\lambda}\\\
&=\int_{0}^{1}\Big{(}\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1)}(\tilde{\lambda}),Z_{1})(B)-\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1)}(\tilde{\lambda}),\,\widetilde{\\!{Z}}_{1})(B)\Big{)}\mathrm{d}\tilde{\lambda}.\end{split}$

We now subsitute this expression for
$\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1)})$ in Theorem 3.2. The result
follows immediately. ∎

###### Proof of Theorem 3.6.

We have

$\begin{split}\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})=\frac{1}{n}\sum_{i=1}^{n}\Big{(}\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)\nu_{\mathrm{pop}}(\mathrm{d}z)-\ell(\mathfrak{m}(\nu_{n}),Z_{i})\Big{)}\,.\end{split}$
(26)

Define

$\alpha_{i}:=\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)\nu_{\mathrm{pop}}(\mathrm{d}z)-\ell(\mathfrak{m}(\nu_{n}),Z_{i}),$
(27)

which gives

$\Big{(}\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})\Big{)}^{2}\,=\,\frac{1}{n^{2}}\Big{(}\sum_{i=1}^{n}(\alpha_{i})^{2}+2\sum_{i_{1}<i_{2}}\alpha_{i_{1}}\alpha_{i_{2}}\Big{)}\,.$
(28)

For indices $i_{1},i_{2}\in[n]$, we define

$\begin{split}\nu_{n,(i_{1},i_{2})}&:=\nu_{n}+\frac{1}{n}\Big{(}(\delta_{\,\widetilde{\\!{Z}}_{i_{1}}}-\delta_{Z_{i_{1}}})+(\delta_{\,\widetilde{\\!{Z}}_{i_{2}}}-\delta_{Z_{i_{2}}})\Big{)},\\\
\alpha_{i,(i_{1},i_{2})}&:=\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n,(i_{1},i_{2})}),z)\nu_{\mathrm{pop}}(\mathrm{d}z)-\ell(\mathfrak{m}(\nu_{n,(i_{1},i_{2})}),Z_{i})\,.\end{split}$
(29)

One can then write

$\begin{split}\sum_{i_{1}<i_{2}}\alpha_{i_{1}}\alpha_{i_{2}}&=\sum_{i_{1}<i_{2}}\Big{\\{}(\alpha_{i_{1}}-\alpha_{i_{1},(i_{1},i_{2})})(\alpha_{i_{2}}-\alpha_{i_{2},(i_{1},i_{2})})+(\alpha_{i_{1}}-\alpha_{i_{1},(i_{1},i_{2})})\alpha_{i_{2},(i_{1},i_{2})}\\\
&\quad+(\alpha_{i_{2}}-\alpha_{i_{2},(i_{1},i_{2})})\alpha_{i_{1},(i_{1},i_{2})}+\alpha_{i_{1},(i_{1},i_{2})}\alpha_{i_{2},(i_{1},i_{2})}\Big{\\}}\,.\end{split}$
(30)

Note that $\alpha_{i_{1},(i_{1},i_{2})}$ and $\alpha_{i_{2},(i_{1},i_{2})}$
are conditionally independent given the $\sigma$-algebra
$\mathcal{F}^{(i_{1},i_{2})}=\sigma\big{(}\\{Z_{i}\\}_{i=1}^{n}/\\{Z_{i_{1}},Z_{i_{2}}\\}\big{)}$.
Hence, by the tower property of conditional expectation,

$\mathbb{E}\big{[}\alpha_{i_{1},(i_{1},i_{2})}\alpha_{i_{2},(i_{1},i_{2})}\big{]}=\mathbb{E}\Big{[}\mathbb{E}\big{[}\alpha_{i_{1},(i_{1},i_{2})}|\mathcal{F}^{(i_{1},i_{2})}\big{]}\mathbb{E}\big{[}\alpha_{i_{2},(i_{1},i_{2})}|\mathcal{F}^{(i_{1},i_{2})}\big{]}\Big{]}=0\,.$
(31)

Using the fact that $\\{Z_{i}\\}_{i=1}^{n}$ are i.i.d. and applying the
Cauchy–Schwarz inequality,

$\begin{split}&\mathbb{E}\Big{[}\Big{(}\mathrm{R}\big{(}\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})-\mathrm{R}(\mathfrak{m}(\nu_{n}),\nu_{n})\Big{)}^{2}\Big{]}\\\
&\leq\frac{1}{n}\mathbb{E}[(\alpha_{1})^{2}]+\frac{2}{n^{2}}\sum_{i_{1}<i_{2}}\bigg{[}\mathbb{E}\big{[}(\alpha_{i_{1}}-\alpha_{i_{1},(i_{1},i_{2})})^{2}\big{]}^{1/2}\mathbb{E}\big{[}(\alpha_{i_{2}}-\alpha_{i_{2},(i_{1},i_{2})})^{2}\big{]}^{1/2}\\\
&\qquad+\mathbb{E}\big{[}(\alpha_{i_{1}}-\alpha_{i_{1},(i_{1},i_{2})})^{2}\big{]}^{1/2}\mathbb{E}\big{[}(\alpha_{i_{2},(i_{1},i_{2})})^{2}\big{]}^{1/2}\\\
&\qquad+\mathbb{E}\big{[}(\alpha_{i_{2}}-\alpha_{i_{2},(i_{1},i_{2})})^{2}\big{]}^{1/2}\mathbb{E}\big{[}(\alpha_{i_{1},(i_{1},i_{2})})^{2}\big{]}^{1/2}\bigg{]}.\end{split}$
(32)

It remains to estimate all the terms on the right hand side. By Assumption 3
(and the elementary inequality $(a-b)^{2}\leq 2(a^{2}+b^{2})$)

$\begin{split}\mathbb{E}\big{[}(\alpha_{1})^{2}\big{]}&=\mathbb{E}\Big{[}\Big{(}\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)\nu(\mathrm{d}z)-\ell(\mathfrak{m}(\nu_{n}),Z_{1})\Big{)}^{2}\Big{]}\\\
&\leq
2\mathbb{E}\Big{[}\Big{(}\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)\nu_{\mathrm{pop}}(\mathrm{d}z)\Big{)}^{2}\Big{]}+2\mathbb{E}\big{[}\big{(}\ell(\mathfrak{m}(\nu_{n}),Z_{1})\big{)}^{2}\big{]}\\\
&\leq
2\mathbb{E}\Big{[}\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)^{2}\nu_{\mathrm{pop}}(\mathrm{d}z)\Big{]}+2\mathbb{E}\big{[}\big{(}\ell(\mathfrak{m}(\nu_{n}),Z_{1})\big{)}^{2}\big{]}\\\
&\leq
2\mathbb{E}\Big{[}\big{(}\ell(\mathfrak{m}(\nu_{n}),\tilde{Z}_{1})\big{)}^{2}\Big{]}+2\mathbb{E}\big{[}\big{(}\ell(\mathfrak{m}(\nu_{n}),Z_{1})\big{)}^{2}\big{]}\leq
4K\,.\end{split}$ (33)

Observe that $\nu_{n}$ and $\nu_{n,(i_{1},i_{2})}$ are both empirical measures
from iid samples.

Therefore
$\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)\nu_{\mathrm{pop}}(\mathrm{d}z)$
and
$\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n,(i_{1},i_{2})}),z)\nu_{\mathrm{pop}}(\mathrm{d}z)$
are equal in law. Similarly, as $Z_{i}$ does not appear in the construction of
$\nu_{n,(i_{1},i_{2})}$, we know
$\ell(\mathfrak{m}(\nu_{n,(i_{1},i_{2})}),Z_{i})$ and
$\ell(\mathfrak{m}(\nu_{n}),\tilde{Z}_{1})$ are equal in law. Therefore, by
essentially the same calculation as above,
$\mathbb{E}\big{[}(\alpha_{i_{1},(i_{1},i_{2})})^{2}\big{]}\leq 4K$.

To study
$\mathbb{E}\big{[}\big{(}\alpha_{i}-\alpha_{i,(i_{1},i_{2})}\big{)}^{2}\big{]}$
for $i\in\\{i_{1},i_{2}\\}$, first notice that, given our i.i.d. assumption
and symmetry, we just need to study
$\mathbb{E}\big{[}\big{(}\alpha_{1}-\alpha_{1,(1,2)}\big{)}^{2}\big{]}$. We
first express the integrals in terms of expectations over the independent
value $\,\widetilde{\\!{Z}}_{3}$,

$\begin{split}&\mathbb{E}[(\alpha_{1}-\alpha_{1,(1,2)})^{2}]\\\
&=\mathbb{E}_{\mathbf{Z_{n}},\mathbf{\,\widetilde{\\!{Z}}_{2}}}\bigg{[}\bigg{(}\int_{\mathcal{Z}}\ell(\mathfrak{m}(\nu_{n}),z)-\ell(\mathfrak{m}(\nu_{n,(1,2)}),z)\nu_{\mathrm{pop}}(\mathrm{d}z)-\Big{(}\ell(\mathfrak{m}(\nu_{n}),Z_{1})-\ell(\mathfrak{m}(\nu_{n,(1,2)}),Z_{1})\Big{)}\bigg{)}^{2}\bigg{]}\\\
&=\mathbb{E}_{\mathbf{Z_{n}},\mathbf{\,\widetilde{\\!{Z}}_{2}}}\bigg{[}\bigg{(}\mathbb{E}_{\,\widetilde{\\!{Z}}_{3}}\Big{[}\ell(\mathfrak{m}(\nu_{n}),\,\widetilde{\\!{Z}}_{3})-\ell(\mathfrak{m}(\nu_{n,(1,2)}),\,\widetilde{\\!{Z}}_{3})\Big{]}-\Big{(}\ell(\mathfrak{m}(\nu_{n}),Z_{1})-\ell(\mathfrak{m}(\nu_{n,(1,2)}),Z_{1})\Big{)}\bigg{)}^{2}\bigg{]}.\end{split}$

We next expand the two central terms using the functional derivative of
$\ell$, to get

$\begin{split}&\mathbb{E}[(\alpha_{1}-\alpha_{1,(1,2)})^{2}]\\\
&=\mathbb{E}_{\mathbf{Z_{n}},\mathbf{\,\widetilde{\\!{Z}}_{3}}}\bigg{[}\bigg{(}\int_{0}^{1}\int_{\Theta}\Big{(}\mathbb{E}_{\,\widetilde{\\!{Z}}_{3}}\Big{[}\frac{\delta\ell}{\delta
m}(\mathfrak{m}_{(1,2)}(\lambda_{1}),\,\widetilde{\\!{Z}}_{3},\theta)\Big{]}-\frac{\delta\ell}{\delta
m}(\mathfrak{m}_{(1,2)}(\lambda_{1}),Z_{i},\theta)\Big{)}\\\
&\qquad\qquad\qquad\times\big{(}\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1,2)})\big{)}(\mathrm{d}\theta)\mathrm{d}\lambda_{1}\bigg{)}^{2}\bigg{]}\\\
&=\mathbb{E}_{\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2}}\big{[}h_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})^{2}\big{]},\end{split}$
(34)

where $h_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})$ is as given in
the theorem statement. Returning to the estimate (32), noting that the double
sum consists of $n(n-1)/2<n^{2}/2$ terms, and combining with (34) and (33), we
have

$\mathrm{gen}_{L_{2}}(\mathfrak{m},\nu_{\mathrm{pop}})<\frac{4K}{n}+2K^{1/2}\mathbb{E}_{\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2}}\big{[}h_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})^{2}\big{]}^{1/2}+\mathbb{E}_{\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2}}\big{[}h_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})^{2}\big{]}.$

∎

###### Proof of Theorem 3.7.

We know from (1) that

$\nu_{n}-\nu_{n,(1,2)}=\frac{1}{n}\big{(}\delta_{Z_{1}}-\delta_{\,\widetilde{\\!{Z}}_{1}}+\delta_{Z_{2}}-\delta_{\,\widetilde{\\!{Z}}_{2}}\big{)}.$

Applying the functional derivative to $\mathfrak{m}$ and recalling the
definition of $\nu_{(1,2)}(\tilde{\lambda})$ in (2), for any Borel set
$B\subset\Theta$ we have

$\begin{split}\big{(}\mathfrak{m}(\nu_{n})-\mathfrak{m}(\nu_{n,(1,2)})\big{)}(B)&=\int_{0}^{1}\int_{\mathcal{Z}}\Big{(}\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1,2)}(\tilde{\lambda}),z)(B)\Big{)}(\nu_{n}-\nu_{n,(1,2)})(\mathrm{d}z)\mathrm{d}\tilde{\lambda}\\\
&=\int_{0}^{1}\Big{(}\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1,2)}(\tilde{\lambda}),Z_{1})(B)-\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1,2)}(\tilde{\lambda}),\,\widetilde{\\!{Z}}_{1})(B)\\\
&\qquad+\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1,2)}(\tilde{\lambda}),Z_{2})(B)-\frac{\delta\mathfrak{m}}{\delta\nu}(\nu_{(1,2)}(\tilde{\lambda}),\,\widetilde{\\!{Z}}_{2})(B)\Big{)}\mathrm{d}\tilde{\lambda}\\\
&=\int_{0}^{1}\Big{(}\Delta_{\mathfrak{m}}(\mathbf{Z}_{2},\mathbf{\,\widetilde{\\!{Z}}}_{2};B)\Big{)}\mathrm{d}\tilde{\lambda}.\end{split}$

Substituting into the definition of $h_{2}$ in Theorem 3.6 and applying
Fubini’s theorem,

$\begin{split}&h_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2})\\\
&=\frac{1}{n}\int_{0}^{1}\int_{0}^{1}\int_{\Theta}\Big{(}\mathbb{E}_{\,\widetilde{\\!{Z}}_{3}}\Big{[}\frac{\delta\ell}{\delta
m}(\mathfrak{m}_{(1,2)}(\lambda_{1}),\,\widetilde{\\!{Z}}_{3},\theta)\Big{]}-\frac{\delta\ell}{\delta
m}(\mathfrak{m}_{(1,2)}(\lambda),Z_{1},\theta)\Big{)}\Delta_{\mathfrak{m}}(\mathbf{Z}_{2},\mathbf{\,\widetilde{\\!{Z}}}_{2};\mathrm{d}\theta)\,\mathrm{d}\lambda\mathrm{d}\tilde{\lambda}\\\
&=\frac{1}{n}\tilde{h}_{2}(\mathbf{Z}_{n},\mathbf{\,\widetilde{\\!{Z}}}_{2}).\end{split}$

The result follows by rearrangement and Theorem 3.6. ∎

##  Appendix D Proofs from Section 4

###### Proof of Lemma 4.1.

Our first task is to verify that $m\mapsto M_{\theta}(m,\nu;\cdot)$ is a valid
map from $\mathcal{P}_{p}(\Theta)$ to (densities of measures in)
$\mathcal{\mathcal{missing}}{P}_{p}(\Theta)$, for any
$\nu\in\mathcal{P}_{2}(\mathcal{Z})$.

Given the form of (16), it is enough to verify that for fixed $m,\nu$, for
some $c>0$, and all $\theta\in\Theta$ sufficiently large, we have the
inequality

$-\frac{2\beta^{2}}{\sigma^{2}}\Big{[}\frac{\delta\mathrm{R}}{\delta
m}(m,\nu,\theta)+\frac{1}{2\beta^{2}}U(\theta)\Big{]}\leq-c\|\theta\|^{p}.$
(35)

Assumption 4(ii) guarantees that

$\begin{split}\frac{\delta\ell}{\delta m}(m,z,\theta)&\leq
g_{1}(m,\theta)\big{(}1+\|z\|^{2}\big{)},\end{split}$

and hence, as $\nu\in\mathcal{P}_{2}(\mathcal{Z})$, our definition (6)
guarantees that for some $C^{\prime}>0$ we have

$\begin{split}\frac{\delta\mathrm{R}}{\delta
m}(m,\nu,\theta)&=\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(m,z,\theta)\nu(\mathrm{d}z)\leq(1+C^{\prime})g_{1}(m,\theta).\end{split}$

On the other hand, with the same $C^{\prime}$ as above, Assumption 4(iii)
guarantees that for all $\theta$ sufficiently large, we have

$\begin{split}\frac{1}{2\beta^{2}}U(\theta)&\geq(1+C^{\prime})\big{(}\|\theta\|^{p}+g_{1}(m,\theta)\big{)}\geq(1+C^{\prime})\|\theta\|^{p}+\frac{\delta\mathrm{R}}{\delta
m}(m,\nu,\theta).\end{split}$

Substituting in (35) gives the desired bound. We therefore know that $M$ is a
well defined map
$\mathcal{P}_{p}(\Theta)\times\mathcal{P}_{2}(\mathcal{Z})\to\mathcal{P}_{p}(\Theta)$.

It is easy to check that, under Assumption 4,
$\mathcal{V}^{\beta,\sigma}(\gamma^{\sigma},\nu)$ is finite, and that
$V^{\beta,\sigma}(m,\nu)\geq 0$ for all $m\in\mathcal{P}_{2}(\Theta)$.
Consequently, no minimizer of $\mathcal{V}^{\beta,\sigma}$ can be outside the
set
$\\{m:\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(m\|\gamma^{\sigma})\leq\mathrm{R}(\gamma^{\sigma},\nu)\\}\subset\mathcal{P}_{p}(\Theta)$,
and this set is relatively compact (in the weak topology). By convexity, (as
the Kullback–Leibler divergence is strictly convex, and $\mathrm{R}$ is
convex), we conclude that $\mathcal{V}^{\beta,\sigma}(\cdot,\nu)$ has a unique
minimizer, which will have a density (due to the Kullback–Leibler divergence).
Taking the first order condition for optimality derived in [Hu et al., 2020,
Proposition 2.5] , the minimizer of
$\mathcal{V}^{\beta,\sigma}(\cdot,\nu_{n})$ is uniquely characterized, for
some normalizing constant $F$ and almost all $\theta$, by

$\begin{split}0=\frac{\delta R}{\delta
m}(m,\nu,\theta)+\frac{1}{2\beta^{2}}U(\theta)+\frac{\sigma^{2}}{2\beta^{2}}\log(m_{\theta}(\theta))+F\end{split}$

which rearranges to give $m_{\theta}(\theta)=M_{\theta}(m,\nu;\theta)$, as
desired. We defer the proof of differentiability to Lemma D.4 ∎

######  Lemma D.1.

Defining the set valued map

$\begin{split}B(\nu)&:=\Big{\\{}m\in\mathcal{P}_{p}(\Theta):\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(m\|\gamma^{\sigma})\leq\mathrm{R}(\gamma^{\sigma},\nu)\\\
&\qquad\text{ and
}\int_{\Theta}\|\theta\|^{p}m(\mathrm{d}\theta)\leq\mathrm{R}(\tilde{\gamma}^{\sigma},\nu)+\int_{\Theta}\|\theta\|^{p}\tilde{\gamma}^{\sigma}(\theta)\mathrm{d}\theta\Big{\\}}\subset\mathcal{P}_{2}(\Theta),\end{split}$

for any $\nu\in\mathcal{P}_{2}(\mathcal{Z})$, the KL-regularized risk
minimizer satisfies $\mathfrak{m}^{\beta,\sigma}(\nu)\in B(\nu)$.

###### Proof.

We have already seen in Lemma 4.1 that
$\mathfrak{m}^{\beta,\sigma}(\nu)\in\\{\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(m\|\gamma^{\sigma})\leq\mathrm{R}(\gamma^{\sigma},\nu)\\}$.
It remains to show that $\mathfrak{m}^{\beta,\sigma}(\nu)$ satisifes the
stated $p$-moment bound.

Recall that we defined the measure $\tilde{\gamma}^{\sigma}_{p}$ through its
density
$(\tilde{F}^{\alpha,\beta})^{-1}\exp(-\frac{1}{2\beta^{2}}U(\theta)+\|\theta\|^{p})$,
for $\tilde{F}^{\alpha,\beta}$ a normalization constant. Consider the problem
of minimizing
$\tilde{\mathrm{R}}(m,\nu)+\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(m\|\tilde{\gamma}^{\sigma})$,
where
$\tilde{\mathrm{R}}(m,\nu)=\mathrm{R}(m,\nu)+\int_{\Theta}\|\theta\|^{p}m(\mathrm{d}\theta)$.
As this falls in the class considered in Lemma 4.1, we know that its optimizer
exists and its density is the fixed point of

$\begin{split}m_{\theta}&\mapsto\frac{1}{\tilde{F}_{\beta,\sigma}}\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\Big{[}\frac{\delta\tilde{\mathrm{R}}}{\delta
m}(m,\nu,\theta)+\frac{1}{2\beta^{2}}U(\theta)-\|\theta\|^{p}\Big{]}\Bigg{\\}}\\\
&=\frac{1}{\tilde{F}_{\beta,\sigma}}\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\Big{[}\frac{\delta\mathrm{R}}{\delta
m}(m,\nu,\theta)+\frac{1}{2\beta^{2}}U(\theta)\Big{]}\Bigg{\\}}.\end{split}$

In other words, $m=\mathfrak{m}^{\beta,\sigma}(\nu)$ as defined in Lemma 4.1.
It follows that

$\mathrm{R}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu)+\int_{\Theta}\|\theta\|^{p}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta)+\frac{\sigma^{2}}{2\beta^{2}}\mathrm{KL}(\mathfrak{m}^{\beta,\sigma}(\nu)\|\tilde{\gamma}_{p}^{\sigma})\leq\mathrm{R}(\tilde{\gamma}_{p}^{\sigma},\nu)+\int_{\Theta}\|\theta\|^{p}\tilde{\gamma}_{p}^{\sigma}(\mathrm{d}\theta).$

As $\ell$, and hence $\mathrm{R}$, is nonnegative, we conclude that

$\int_{\Theta}\|\theta\|^{p}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta)\leq\mathrm{R}(\tilde{\gamma}_{p}^{\sigma},\nu)+\int_{\Theta}\|\theta\|^{p}\tilde{\gamma}_{p}^{\sigma}(\mathrm{d}\theta).$

∎

######  Lemma D.2.

For any choice of $m\in B(\nu)$, the linear map
$\mathcal{C}_{m}:L^{2}(m,\Theta)\to L^{2}(m,\Theta)$ defined by

$\mathcal{C}_{m}f(\theta):=\mathbb{E}_{\theta^{\prime}\sim
m}\Big{[}\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(m,\nu,\theta,\theta^{\prime})f(\theta^{\prime})\Big{]}$

is positive, in the sense that

$\langle
f,\mathcal{C}_{m}f\rangle_{L^{2}(m,\Theta)}=\int_{\Theta}f(\theta)(\mathcal{C}_{m}f)(\theta)m(\mathrm{d}\theta)\geq
0.$

In particular, $\mathcal{C}_{m}$ is a Hilbert–Schmidt operator with discrete
spectrum

$\sigma(\mathcal{C}_{m})=\\{\lambda^{\mathcal{C}}_{i}\\}_{i\geq
0}\subset[0,\infty).$

###### Proof.

We first write

$\begin{split}\int_{\Theta}f(\theta)(\mathcal{C}_{m}f)(\theta)m(\mathrm{d}\theta)&=\int_{\Theta}\int_{\Theta}f(\theta)f(\theta^{\prime})\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(m,\nu,\theta,\theta^{\prime})m(\mathrm{d}\theta)m(\mathrm{d}\theta^{\prime})\\\
&=\int_{\mathcal{Z}}\int_{\Theta}\int_{\Theta}f(\theta)f(\theta^{\prime})\frac{\delta^{2}\mathrm{\ell}}{\delta
m^{2}}(m,z,\theta,\theta^{\prime})m(\mathrm{d}\theta)m(\mathrm{d}\theta^{\prime})\nu(\mathrm{d}z).\end{split}$

As $\ell$ is convex with respect to $m$, we know that
$\frac{\delta^{2}\ell}{\delta m^{2}}$ is positive semidefinite, which gives
positivity of $\mathcal{C}_{m}$. We also have the symmetry property,
[Cardaliaguet et al., 2019, Lemma 2.4]

$\frac{\delta^{2}\mathrm{\ell}}{\delta
m^{2}}(m,z,\theta,\theta^{\prime})=\frac{\delta^{2}\mathrm{\ell}}{\delta
m^{2}}(m,z,\theta^{\prime},\theta).$

We next recall that, by Assumption 4(v),

$\int_{\mathcal{Z}}\int_{\Theta}\int_{\Theta}\Big{(}\frac{\delta^{2}\mathrm{\ell}}{\delta
m^{2}}(m,z,\theta,\theta^{\prime})\Big{)}^{2}m(\mathrm{d}\theta)m(\mathrm{d}\theta^{\prime})\nu(\mathrm{d}z)<\infty,$

which implies $\mathcal{C}_{m}$ is Hilbert–Schmidt, in particular it is a
self-adjoint compact operator. The spectral theorem then states that
$\mathcal{C}_{m}$ has a discrete spectrum made up of its eigenvalues, which
are nonnegative as $\mathcal{C}_{m}$ is positive. ∎

Our next challenge is to prove that the Gibbs measure is differentiable. We
begin by considering an auxiliary function $S$.

######  Lemma D.3.

Define

$S(\nu,\theta):=\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta)=\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),z,\theta)\nu(\mathrm{d}z).$ (36)

Under Assumption 4, we know $S$ is differentiable in $\nu$, and its derivative
satisfies the bound

$\begin{split}&\int_{\Theta}\Big{(}\frac{\delta
S}{\delta\nu}(\nu,\theta,z)\Big{)}^{2}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta)\\\
&\leq\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z)-\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z^{\prime})\nu(\mathrm{d}z^{\prime})\Big{)}^{2}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta).\end{split}$

In particular, we have the representation:

$\begin{split}\frac{\delta
S}{\delta\nu}(\nu,\theta,z)&=\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}^{\beta,\sigma}(\nu),z,\theta\big{)}-\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}^{\beta,\sigma}(\nu),z^{\prime},\theta\big{)}\nu(\mathrm{d}z^{\prime})\\\
&\quad-\frac{2\beta^{2}}{\sigma^{2}}\mathrm{Cov}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}\int_{\mathcal{Z}}\frac{\delta^{2}\ell}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}(\nu),z^{\prime},\theta,\theta^{\prime})\nu(\mathrm{d}z^{\prime}),\,\frac{\delta
S}{\delta\nu}(\nu,\theta^{\prime},z)\Big{]}.\end{split}$

###### Proof.

Fix $\nu,\nu_{1}\in\mathcal{P}_{2}(\mathcal{Z})$ and write
$\nu^{\epsilon}=\nu+\epsilon(\nu_{1}-\nu)$. Recall that

$S(\nu,\theta):=\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta)=\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),z,\theta)\nu(\mathrm{d}z)$ (37)

and $\mathfrak{m}^{\beta,\sigma}(\nu)$ satisfies

$\mathfrak{m}^{\beta,\sigma}_{\theta}(\nu;\theta)=\frac{Q(\nu,\theta)}{\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}},$
(38)

where
$Q(\nu,\theta)=\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\Big{(}S(\nu,\theta)+\frac{1}{2\beta^{2}}U(\theta)\Big{)}\Big{\\}}.$
We define an approximate directional derivative of $S$:

$\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta):=\frac{S(\nu^{\epsilon},\theta)-S(\nu,\theta)}{\epsilon},$

and similarly

$\frac{\delta^{\epsilon}\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu^{\epsilon}}(\nu^{\epsilon}):=\frac{\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu^{\epsilon})-\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu)}{\epsilon},\qquad\frac{\delta^{\epsilon}Q}{\delta\nu^{\epsilon}}:=\frac{Q(\nu^{\epsilon};\theta)-Q(\nu;\theta)}{\epsilon}.$

We note that $S(\nu^{\epsilon},\theta)\to S(\nu,\theta)$ as $\epsilon\to 0$.
Using the fact that $|S(\nu^{\prime},\theta)|<C(1+\|\theta\|^{2})$ for all
$\nu^{\prime}$ in a compact set around $\nu$, the dominated convergence
theorem shows that, as $\epsilon\to 0$,

$\epsilon\Big{\|}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}\Big{\|}_{L^{2}(\mathfrak{m}^{\sigma,\beta}(\nu))}=\|S(\nu^{\epsilon},\cdot)-S(\nu,\cdot)\|_{L^{2}(\mathfrak{m}^{\sigma,\beta}(\nu))}\to
0.$ (39)

We now give a representation of $\delta^{\epsilon}Q/\delta\nu^{\epsilon}$.
Expanding the definition of the approximate directional derivative,

$\begin{split}&\frac{\delta^{\epsilon}Q}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta,z)\\\
&=\frac{1}{\epsilon}\left(\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\Big{(}S(\nu^{\epsilon},\theta)+\frac{1}{2\beta^{2}}U(\theta)\Big{)}\Big{\\}}-\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\Big{(}S(\nu,\theta)+\frac{1}{2\beta^{2}}U(\theta)\Big{)}\Big{\\}}\right)\\\
&=\frac{1}{\epsilon}Q(\nu,\theta)\left(\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\Big{(}S(\nu^{\epsilon},\theta)-S(\nu,\theta))\Big{)}\Big{\\}}-1\right)\\\
&=\frac{1}{\epsilon}Q(\nu,\theta)\left(\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\,\epsilon\,\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)\Big{\\}}-1\right)\\\
&=-\frac{2\beta^{2}}{\sigma^{2}}Q(\nu,\theta)\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)\int_{0}^{1}\exp\\{\frac{-2\beta^{2}}{\sigma^{2}}(\epsilon\lambda\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta,z))\Big{\\}}\mathrm{d}\lambda\\\
&=-\frac{2\beta^{2}}{\sigma^{2}}Q(\nu,\theta)\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)G(\nu^{\epsilon},\theta),\end{split}$
(40)

where

$\begin{split}G(\nu^{\epsilon},\theta)&:=\int_{0}^{1}\exp\\{\frac{-2\beta^{2}}{\sigma^{2}}(\epsilon\lambda\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta))\Big{\\}}\mathrm{d}\lambda\\\
&=\int_{0}^{1}\exp\\{\frac{-2\beta^{2}}{\sigma^{2}}\lambda(S(\nu^{\epsilon},\theta)-S(\nu,\theta))\Big{\\}}\mathrm{d}\lambda.\end{split}$

Observe that (39) implies
$\|G(\nu^{\epsilon},\theta)-1\|_{L^{2}(\mathfrak{m}^{\beta,\sigma}(\nu))}\to
0$ as $\epsilon\to 0$.

Next, we reexpress $\delta^{\epsilon}S/\delta\nu^{\epsilon}$ in terms of
derivatives of $\mathrm{R}(m,\nu)=\int_{\mathcal{Z}}\ell(m,z)\mathrm{d}z$.
Note that

$\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\nu^{\epsilon},\theta)-\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\nu,\theta)=\epsilon\int_{0}^{1}\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\theta,z)(\nu_{1}-\nu)(\mathrm{d}z)\mathrm{d}\lambda,$

and

$\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\nu,\theta)-\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta)=\int_{\Theta}\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}_{\lambda}(\nu^{\epsilon}),\nu,\theta,\theta^{\prime})(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon})-\mathfrak{m}^{\beta,\sigma}(\nu))(\mathrm{d}\theta^{\prime}),$

where
$\mathfrak{m}^{\beta,\sigma}_{\lambda}(\nu^{\epsilon}):=\mathfrak{m}^{\beta,\sigma}(\nu)+\lambda(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon})-\mathfrak{m}^{\beta,\sigma}(\nu))$.
Hence

$\begin{split}&\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta,z)\\\
&=\frac{1}{\epsilon}\left(\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\nu^{\epsilon},\theta)-\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta)\right)\\\
&=\frac{1}{\epsilon}\left(\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\nu^{\epsilon},\theta)-\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\nu,\theta)+\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\nu,\theta)-\frac{\delta\mathrm{R}}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta)\right)\\\
&=\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\theta,z)(\nu_{1}-\nu)(\mathrm{d}z)+\int_{0}^{1}\int_{\Theta}\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}_{\lambda}(\nu^{\epsilon}),\nu,\theta,\theta^{\prime})\frac{\delta^{\epsilon}\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta^{\prime},z)(\mathrm{d}\theta^{\prime})\mathrm{d}\lambda.\end{split}$
(41)

Given these calculations, we can now give a characterization of
$\delta\mathfrak{m}_{\theta}^{\beta,\sigma}/\delta\nu^{\epsilon}$, making use
of (38):

$\begin{split}&\frac{\delta^{\epsilon}\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)\\\
&=\frac{1}{\epsilon}\left(\frac{Q(\nu^{\epsilon},\theta)}{\int_{\Theta}Q(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}}-\frac{Q(\nu,\theta)}{\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}}\right)\\\
&=\frac{Q(\nu^{\epsilon},\theta)\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}-Q(\nu,\theta)\int_{\Theta}Q(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}}{\epsilon\int_{\Theta}Q(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}}\\\
&=\frac{Q(\nu^{\epsilon},\theta)\Big{(}\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}-\int_{\Theta}Q(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\Big{)}+\int_{\Theta}Q(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\Big{(}Q(\nu^{\epsilon},\theta)-Q(\nu,\theta)\Big{)}}{\epsilon\int_{\Theta}Q(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}}\\\
&=\frac{Q(\nu^{\epsilon},\theta)\Big{(}\frac{1}{\epsilon}\int_{\Theta}\left(Q(\nu,\theta^{\prime})-Q(\nu^{\epsilon},\theta^{\prime})\right)\mathrm{d}\theta^{\prime}\Big{)}+\int_{\Theta}Q(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\Big{(}\frac{1}{\epsilon}\left(Q(\nu^{\epsilon},\theta)-Q(\nu,\theta)\right)\Big{)}}{\int_{\Theta}Q(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}}\\\
&=-\frac{\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu^{\epsilon};\theta)\Big{(}\int_{\Theta}\frac{\delta^{\epsilon}Q}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\Big{)}}{\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}}+\frac{\frac{\delta^{\epsilon}Q}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)}{\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}}\\\
&=-\frac{2\beta^{2}}{\sigma^{2}}\Big{(}\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu;\theta)\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)G(\nu^{\epsilon},\theta)-\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu^{\epsilon};\theta)\Big{[}\int_{\Theta}\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu;\theta^{\prime})\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta^{\prime})G(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\Big{]}\Big{)},\end{split}$

where the last equality follows from (40). For any sufficiently integrable
function $f:\Theta\to\mathbb{R}$, we can therefore compute

$\begin{split}&\int_{\Theta}f(\theta)\frac{\delta^{\epsilon}\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)\mathrm{d}\theta\\\
&=-\frac{2\beta^{2}}{\sigma^{2}}\int_{\Theta}\bigg{(}f(\theta)\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu;\theta)\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)G(\nu^{\epsilon},\theta)\\\
&\qquad-f(\theta)\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu^{\epsilon};\theta)\Big{[}\int_{\Theta}\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu;\theta^{\prime})\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta^{\prime})G(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\Big{]}\bigg{)}\mathrm{d}\theta\\\
&=-\frac{2\beta^{2}}{\sigma^{2}}\mathrm{Cov}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}f(\theta),\,\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)G(\nu^{\epsilon},\theta)\Big{]}\\\
&\quad+\frac{2\beta^{2}\epsilon}{\sigma^{2}}\int_{\Theta}f(\theta)\frac{\delta^{\epsilon}\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)\mathrm{d}\theta\Big{[}\int_{\Theta}\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu;\theta^{\prime})\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta^{\prime})G(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\theta^{\prime}\Big{]}.\end{split}$

By rearranging terms, we have

$\int_{\Theta}f(\theta)\frac{\delta^{\epsilon}\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)\mathrm{d}\theta=-\frac{2\beta^{2}}{\sigma^{2}}\mathrm{Cov}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}f(\theta),\,\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)\frac{G(\nu^{\epsilon},\theta)}{H(\nu^{\epsilon},\epsilon)}\Big{]},$
(42)

with

$H(\nu^{\epsilon},\nu)=1-\frac{2\beta^{2}\epsilon}{\sigma^{2}}\mathbb{E}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta^{\prime})G(\nu^{\epsilon},\theta^{\prime})\Big{]}\,.$

Observe that, from (39), as $\epsilon\to 0$, we know $H(\nu^{\epsilon},\nu)\to
1$ (and this convergence is uniform in $\theta$).

Given the characterization in (42), we return to the result of (41), to derive
a new representation of $\delta^{\epsilon}S/\delta\nu^{\epsilon}$:

$\begin{split}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta)&=\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\theta,z)(\nu_{1}-\nu)(\mathrm{d}z)\\\
&\quad-\frac{2\beta^{2}}{\sigma^{2}}\int_{0}^{1}\mathrm{Cov}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}_{\lambda}(\nu^{\epsilon}),\nu,\theta,\theta^{\prime})\frac{G(\nu^{\epsilon},\theta^{\prime})}{H(\nu^{\epsilon},\nu)},\,\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\theta^{\prime})\Big{]}\mathrm{d}\lambda.\end{split}$
(43)

Assumption 4(v) guarantees that $\delta^{2}\mathrm{R}/\delta m^{2}$ is
sufficiently integrable for this result to hold (as we can approximate it with
bounded functions $f$ in (42)).

In order to show $S$ is differentiable, we will now construct a candidate for
the directional derivative. For this purpose we temporarily fix $z$. Recalling
that we expect $G(\nu^{\epsilon},\theta)\to 1$ and $H(\nu^{\epsilon},\nu)\to
1$ as $\epsilon\to 0$, consider the map
$\mathcal{G}:L^{2}(\Theta;\mathfrak{m}^{\beta,\sigma}(\nu))\to
L^{2}(\Theta;\mathfrak{m}^{\beta,\sigma}(\nu))$ defined by

$\begin{split}\big{(}\mathcal{G}v\big{)}(\nu,\theta,z)&=\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z)-\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z^{\prime})\nu(\mathrm{d}z^{\prime})\\\
&\quad-\frac{2\beta^{2}}{\sigma^{2}}\mathbb{E}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta,\theta^{\prime})v(\nu,\theta^{\prime},z)\Big{]}\\\
\end{split}$ (44)

or equivalently, at the level of operators, for $z$ fixed,

$\mathcal{G}v=\mathcal{L}-\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}v$

with $\mathcal{L}=\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z)-\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z^{\prime})\nu(\mathrm{d}z^{\prime})$
and $\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}$ as in Lemma D.2. We wish
to define $v$ to be the fixed point of $\mathcal{G}v$, that is

$v=\mathcal{G}v=\mathcal{L}-\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}v$
(45)

which rearranges to give the formal expression

$v=\Big{(}\mathrm{id}+\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{)}^{-1}\mathcal{L}.$
(46)

We know from Lemma D.2 that $\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}$
has a nonnegative spectrum $\\{\lambda^{i}\\}_{i\geq 0}$, and write
$\\{e_{i}\\}_{i\geq 0}$ for a corresponding orthonormal set of eigenvectors.
Observe that the spectrum of
$\mathrm{id}+\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}$
is given by $\\{1+\frac{2\beta^{2}}{\sigma^{2}}\lambda_{i}\\}_{i\geq 0}$, with
the same set of eigenvectors. By decomposing $\mathcal{L}=\sum_{i}c_{i}e_{i}$,
as $\lambda_{i}\geq 0$ for all $i$ we compute

$\begin{split}v&=\sum_{i}\Big{(}\mathrm{id}+\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{)}^{-1}c_{i}e_{i}=\sum_{i}\frac{c_{i}}{1+\frac{2\beta^{2}}{\sigma^{2}}\lambda_{i}}e_{i}.\end{split}$

In particular, for every fixed value of $z$,

$\|v\|^{2}_{L^{2}(\mathfrak{m}^{\beta,\sigma}(\nu),\Theta)}=\sum_{i}\frac{c_{i}^{2}}{(1+\frac{2\beta^{2}}{\sigma^{2}}\lambda_{i})^{2}}\leq\sum_{i}c_{i}^{2}=\|\mathcal{L}\|^{2}_{L^{2}(\mathfrak{m}^{\beta,\sigma}(\nu),\Theta)}.$
(47)

Solving this system for all $z$, we obtain a function $v(\nu,\theta,z)$, which
we can integrate with respect to $\nu(\mathrm{d}z)$ to see (from integrating
(44)) that $\int_{\mathcal{Z}}v(\nu,\theta,z)\nu(dz)=0$.

If we integrate with respect to $(\nu_{1}-\nu)(\mathrm{d}z)$, we get our
candidate directional derivative:

$\begin{split}&\int_{\mathcal{Z}}v(\nu,\theta,z)(\nu_{1}-\nu)(\mathrm{d}z)\\\
&=\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z)(\nu_{1}-\nu)(\mathrm{d}z)\\\
&\quad-\frac{2\beta^{2}}{\sigma^{2}}\mathrm{Cov}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta,\theta^{\prime}),\,\int_{\mathcal{Z}}v(\nu,\theta^{\prime},z)(\nu_{1}-\nu)(\mathrm{d}z)\Big{]},\end{split}$

(recalling that
$\mathbb{E}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\big{[}\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta,\theta^{\prime})\big{]}=0$,
so the covariance and the expectation of the product agree). We next show that
this candidate is indeed the directional derivative of $S$.

We take (43), and write

$\begin{split}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}&=\int_{\mathcal{Z}}\mathcal{L}^{\epsilon}(z)(\nu_{1}-\nu)(\mathrm{d}z)-\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}^{\epsilon}\Big{[}\frac{1}{H(\nu^{\epsilon},\nu)}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\cdot)\Big{]}\\\
&=\int_{\mathcal{Z}}\mathcal{L}^{\epsilon}(z)(\nu_{1}-\nu)(\mathrm{d}z)-\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}^{\epsilon}\Big{[}\Big{(}\frac{1}{H(\nu^{\epsilon},\nu)}-1\Big{)}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\cdot)\Big{]}-\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}^{\epsilon}\Big{[}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\cdot)\Big{]}\\\
\end{split}$

where

$\begin{split}\mathcal{L}^{\epsilon}(z)&:=\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\theta,z)-\int_{\mathcal{Z}}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon}),\theta,z^{\prime})\nu(\mathrm{d}z^{\prime}),\\\
\mathcal{C}^{\epsilon}v&:=\mathrm{Cov}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}\int_{0}^{1}\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}_{\lambda}(\nu^{\epsilon}),\nu,\theta,\theta^{\prime})G(\nu^{\epsilon},\theta^{\prime})\mathrm{d}\lambda,\,v(\theta^{\prime})\Big{]}.\end{split}$

We formally rearrange to get

$\begin{split}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}&=\bigg{(}\mathrm{id}+\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}^{\epsilon}\bigg{)}^{-1}\bigg{(}\int_{\mathcal{Z}}\mathcal{L}^{\epsilon}(z)(\nu_{1}-\nu)(\mathrm{d}z)-\frac{2\beta^{2}}{\sigma^{2}}\Big{(}\frac{1}{H(\nu^{\epsilon},\nu)}-1\Big{)}\mathcal{C}^{\epsilon}\Big{[}\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\cdot)\Big{]}\bigg{)}.\end{split}$
(48)

Taking $\epsilon\to 0$, we know that
$\mathfrak{m}^{\beta,\sigma}(\nu^{\epsilon})\to\mathfrak{m}^{\beta,\sigma}(\nu)$
and
$\mathfrak{m}^{\beta,\sigma}_{\lambda}(\nu^{\epsilon})\to\mathfrak{m}^{\beta,\sigma}(\nu)$
for all $\lambda\in[0,1]$. Therefore,
$\mathcal{L}^{\epsilon}(z)\to\mathcal{L}(z)$ and
$\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}_{\lambda}(\nu^{\epsilon}),\nu,\theta,\theta^{\prime})\to\frac{\delta^{2}\mathrm{R}}{\delta
m^{2}}(\mathfrak{m}^{\beta,\sigma}(\nu),\nu,\theta,\theta^{\prime})$. We have
already seen $H(\nu^{\epsilon},\nu)\to 1$ and $G(\nu^{\epsilon},\theta)\to 1$,
and therefore,
$\mathcal{C}^{\epsilon}\to\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}$ in
the sense of Hilbert–Schmidt operators. In particular, as
$\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}$ has a positive spectrum, this
implies
$(\mathrm{id}+\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}^{\epsilon})^{-1}\to(\mathrm{id}+\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)})^{-1}$.

Therefore, we determine that, as $\epsilon\to 0$,

$\frac{\delta^{\epsilon}S}{\delta\nu^{\epsilon}}(\nu^{\epsilon},\cdot)\to\bigg{(}\mathrm{id}+\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}\bigg{)}^{-1}\bigg{(}\int_{\mathcal{Z}}\mathcal{L}(z)(\nu_{1}-\nu)(\mathrm{d}z)\bigg{)}=\int_{\mathcal{Z}}v(z)(\nu_{1}-\nu)(\mathrm{d}z),$

where $v$ is the fixed point constructed in (45).

From this, we conclude that
$\int_{\mathcal{Z}}v(\nu,\theta,z)(\nu_{1}-\nu)(\mathrm{d}z)$ is the
directional derivative of $S$ in the direction $(\nu_{1}-\nu)$. As the
directional derivative is given by integrating $v$, it follows that

$\frac{\delta S}{\delta\nu}(\nu,\theta,z)=v(\nu,\theta,z),$

and hence $S$ is differentiable in $\nu$, and has the desired representation.
The bound in (47) completes the argument. ∎

######  Lemma D.4 (Derivative of Gibbs Measure).

Under Assumption 4, the density of the Gibbs measure, that is, the map
$\nu\mapsto\mathfrak{m}_{\theta}^{\beta,\sigma}(\nu)$, has derivative

$\frac{\delta\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu}(\nu,\theta,z)=-\frac{2\beta^{2}}{\sigma^{2}}\mathfrak{m}^{\beta,\sigma}_{\theta}(\nu;\theta)\Big{(}\frac{\delta
S}{\delta\nu}(\nu,\theta,z)-\Big{[}\int_{\Theta}\mathfrak{m}^{\beta,\sigma}_{\theta}(\nu;\theta^{\prime})\frac{\delta
S}{\delta\nu}(\nu,\theta^{\prime},z)\mathrm{d}\theta\Big{]}\Big{)}.$

In particular, for any $f\in L^{2}(\mathrm{d}\theta)$,

$\int_{\Theta}f(\theta)\frac{\delta\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu}(\nu,\theta,z)\mathrm{d}\theta=-\frac{2\beta^{2}}{\sigma^{2}}\mathrm{Cov}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}f(\theta),\,\frac{\delta
S}{\delta\nu}(\nu,\theta,z)\Big{]}.$

As a consequence, Assumption 4 implies Assumption 3.

###### Proof.

From Lemma D.3, we know $S$ is differentiable. Recall that
$\mathfrak{m}^{\beta,\sigma}(\nu)$ satisfies

$\mathfrak{m}^{\beta,\sigma}_{\theta}(\nu;\theta)=\frac{Q(\nu,\theta)}{\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}},$

where
$Q(\nu,\theta)=\exp\\{-\frac{2\beta^{2}}{\sigma^{2}}\Big{(}S(\nu,\theta)+\frac{1}{2\beta^{2}}U(\theta)\Big{)}\Big{\\}}$.
Differentiating with respect to $\nu$,

$\frac{\delta
Q}{\delta\nu}(\nu,\theta,z)=-\frac{2\beta^{2}}{\sigma^{2}}Q(\nu,\theta)\frac{\delta
S}{\delta\nu}(\nu,\theta,z),$

and hence

$\begin{split}&\frac{\delta\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu}(\nu,\theta,z)\\\
&=\frac{\frac{\delta
Q}{\delta\nu}(\nu,\theta,z)\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}-Q(\nu,\theta)\frac{\delta}{\delta\nu}\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}}{\big{(}\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}\big{)}^{2}}\\\
&=-\frac{2\beta^{2}}{\sigma^{2}}\frac{Q(\nu,\theta)\frac{\delta
S}{\delta\nu}(\nu,\theta,z)\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}-Q(\nu,\theta)(\int_{\Theta}Q(\nu,\theta^{\prime})\frac{\delta
S}{\delta\nu}(\nu,\theta^{\prime},z)\mathrm{d}\theta^{\prime})}{\big{(}\int_{\Theta}Q(\nu,\theta^{\prime})\mathrm{d}\theta^{\prime}\big{)}^{2}}\\\
&=-\frac{2\beta^{2}}{\sigma^{2}}\mathfrak{m}^{\beta,\sigma}_{\theta}(\nu;\theta)\Big{(}\frac{\delta
S}{\delta\nu}(\nu,\theta,z)-\Big{[}\int_{\Theta}\mathfrak{m}^{\beta,\sigma}_{\theta}(\nu;\theta^{\prime})\frac{\delta
S}{\delta\nu}(\nu,\theta^{\prime},z)\mathrm{d}\theta^{\prime}\Big{]}\Big{)},\end{split}$

yielding the desired representation. The integrability of $\delta S/\delta\nu$
in Lemma D.3 also shows that
$\frac{\delta\mathfrak{m}^{\beta,\sigma}_{\theta}}{\delta\nu}(\nu,\theta,z)$
is square integrable with respect to $\mathrm{d}\theta$. The covariance
representation follows immediately, as

$\int_{\Theta}\mathfrak{m}^{\beta,\sigma}_{\theta}(\nu;\theta^{\prime})\frac{\delta
S}{\delta\nu}(\nu,\theta^{\prime},z)\mathrm{d}\theta^{\prime}=\mathbb{E}_{\theta^{\prime}\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{[}\frac{\delta
S}{\delta\nu}(\nu,\theta^{\prime},z)\Big{]}.$

Finally, we observe that Assumption 4(i) implies Assumption 3(i) and
Assumption 3(ii) is the result we have just shown. ∎

######  Proposition D.5.

Suppose Assumption 4 holds. Then

$\begin{split}&\int_{\Theta}\Big{[}\frac{\delta
S}{\delta\nu}(\nu,\theta,z)-\frac{\delta
S}{\delta\nu}(\nu,\theta,z^{\prime})\Big{]}^{2}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta)\leq
2C_{\theta}^{2}\Big{(}1+2\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu)}\big{[}\|\theta\|^{p}\big{]}\Big{)}^{2}\Big{(}2+\|z\|^{2}+\|z^{\prime}\|^{2}\Big{)}^{2}.\end{split}$

###### Proof.

We write, using the notation of (46) and linearity,

$\displaystyle\frac{\delta S}{\delta\nu}(\nu,\theta,z)-\frac{\delta
S}{\delta\nu}(\nu,\theta,z^{\prime})$
$\displaystyle=\Big{(}\mathrm{id}+\frac{2\beta^{2}}{\sigma^{2}}\mathcal{C}_{\mathfrak{m}^{\beta,\sigma}(\nu)}\Big{)}^{-1}\Big{[}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\cdot,z)-\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\cdot,z^{\prime})\Big{]}.$

We know from (47) that this is bounded in
$L^{2}(\mathfrak{m}^{\beta,\sigma}(\nu))$, in particular

$\begin{split}&\int_{\Theta}\Big{[}\frac{\delta
S}{\delta\nu}(\nu,\theta,z)-\frac{\delta
S}{\delta\nu}(\nu,\theta,z^{\prime})\Big{]}^{2}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta)\\\
&\leq\int_{\Theta}\Big{[}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z)-\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z^{\prime})\Big{]}^{2}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta)\\\
&\leq 2\int_{\Theta}\Big{[}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z)\Big{]}^{2}+\Big{[}\frac{\delta\ell}{\delta
m}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta,z^{\prime})\Big{]}^{2}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta).\end{split}$

Assumption 4(ii) yields

$\begin{split}&\int_{\Theta}\Big{[}\frac{\delta
S}{\delta\nu}(\nu,\theta,z)-\frac{\delta
S}{\delta\nu}(\nu,\theta,z^{\prime})\Big{]}^{2}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta)\\\
&\leq
2\Big{(}2+\|z\|^{2}+\|z^{\prime}\|^{2}\Big{)}^{2}\int_{\Theta}\big{[}g_{1}(\mathfrak{m}^{\beta,\sigma}(\nu),\theta)\big{]}^{2}\mathfrak{m}^{\beta,\sigma}(\nu;\mathrm{d}\theta).\end{split}$

The final result follows from Assumption 4(iv).

∎

######  Remark D.6.

In the following arguments, we will regularly make use of the basic inequality
(based on Cauchy–Schwarz), for any square integrable random variables $X,Y$,

$|\mathrm{Cov}(X,Y)|\leq\sqrt{\mathbb{V}(X)\mathbb{V}(Y)}\leq\sqrt{\mathbb{E}(X^{2})\mathbb{E}(Y^{2})}.$

A slightly more involved, but fundamentally similar, inequality is given in
the following lemma.

######  Lemma D.7.

Let $\pi:(\mathbb{R}^{+})^{2n}\to\mathbb{R}$ be a polynomial with positive
coefficients. Then

$\mathbb{E}_{\mathbf{Z}_{n},\,\widetilde{\\!{\mathbf{Z}}}_{n}}\Big{[}\pi\Big{(}\|Z_{1}\|,\|Z_{2}\|,...,\|Z_{n}\|,\|\,\widetilde{\\!{Z}}_{1}\|,\|\,\widetilde{\\!{Z}}_{2}\|,...,\|\,\widetilde{\\!{Z}}_{n}\|\Big{)}\Big{]}\leq\mathbb{E}_{Z_{1}}\Big{[}\pi\Big{(}\|Z_{1}\|,\|Z_{1}\|,...,\|Z_{1}\|\Big{)}\Big{]}.$

The same result also holds when the polynomial involves expectation of
$\mathbb{E}_{Z_{i}}[\|Z_{i}\|^{\alpha}]$ terms.

###### Proof.

First note that, by Jensen’s inequality, for any $1\leq\beta\leq\alpha$ and
any nonnegative random variable $X$,

$\mathbb{E}[X^{\alpha}]\mathbb{E}[X^{\beta}]=\mathbb{E}[X^{\alpha}]\mathbb{E}[X^{\alpha(\beta/\alpha)}]\leq\mathbb{E}[X^{\alpha}]\mathbb{E}[X^{\alpha}]^{\beta/\alpha}=\mathbb{E}[X^{\alpha}]^{1+\beta/\alpha}\leq\mathbb{E}[X^{\alpha+\beta}].$
(49)

By symmetry, this holds also for $1\leq\alpha\leq\beta$. As $\pi$ is a
polynomial, we can expand it fully as

$\pi\Big{(}\|Z_{1}\|,\|Z_{2}\|,...,\|Z_{j}\|,\|\,\widetilde{\\!{Z}}_{1}\|,\|\,\widetilde{\\!{Z}}_{2}\|,...,\|\,\widetilde{\\!{Z}}_{j}\|\Big{)}=\sum_{k=1}^{\kappa}c_{k}\prod_{j=1}^{n}\|Z_{j}\|^{\alpha_{j,k}}\|\,\widetilde{\\!{Z}}_{j}\|^{\beta_{j,k}}$

where $\kappa\in\mathbb{N}$ and $c_{k}>0$ for each $k$. As every $Z_{i}$,
$\,\widetilde{\\!{Z}}_{i}$ is iid, for each term in this expansion

$\begin{split}\mathbb{E}_{\mathbf{Z}_{n},\,\widetilde{\\!{\mathbf{Z}}}_{n}}\Big{[}\prod_{j=1}^{n}\|Z_{j}\|^{\alpha_{j,k}}\|\,\widetilde{\\!{Z}}_{j}\|^{\beta_{j,k}}\Big{]}&=\prod_{j=1}^{n}\mathbb{E}_{Z_{j}}[\|Z_{j}\|^{\alpha_{j,k}}]\mathbb{E}_{\,\widetilde{\\!{Z}}_{j}}[\|\,\widetilde{\\!{Z}}_{j}\|^{\beta_{j,k}}]=\prod_{j=1}^{n}\mathbb{E}_{Z_{1}}[\|Z_{1}\|^{\alpha_{j,k}}]\mathbb{E}_{Z_{1}}[\|Z_{1}\|^{\beta_{j,k}}]\\\
&\leq\mathbb{E}_{Z_{1}}\Big{[}\|Z_{1}\|^{\sum_{j=1}^{n}(\alpha_{j,k}+\beta_{j,k})}\Big{]},\end{split}$

where we have used (49) to simplify the expansion. Substituting into the
expansion of $\pi$, we obtain the desired inequality. If the polynomial
involves expectation terms, these can be resolved using Jensen’s inequality. ∎

###### Proof of Theorem 4.2.

Recall from (2) that
$\nu_{(1)}(\lambda)=\nu_{n,(1)}+\lambda(\nu_{n}-\nu_{n,(1)})$. From Theorem
3.4,

$\mathrm{gen}(\mathfrak{m}(\nu_{n}),\nu_{\mathrm{pop}})=\frac{1}{n}\mathbb{E}_{\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1}}\big{[}h(\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1})\big{]},$
(50)

where

$\begin{split}&h(\mathbf{Z}_{n},\,\widetilde{\\!{Z}}_{1})\\\
&=\int_{0}^{1}\int_{0}^{1}\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}_{(1)}(\lambda),Z_{1},\theta\big{)}\Big{)}\Big{(}\frac{\delta\mathfrak{m}}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),Z_{1}\big{)}-\frac{\delta\mathfrak{m}}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),\,\widetilde{\\!{Z}}_{1}\big{)}\Big{)}(\mathrm{d}\theta)\,\mathrm{d}\tilde{\lambda}\,\mathrm{d}\lambda.\end{split}$
(51)

As
$\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\big{[}\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),z,\theta\big{)}\big{]}\equiv 0$
(by definition of $S=\delta\mathrm{R}/\delta m$ and the normalization
condition and chain rule), from Lemma D.4 we know

$\begin{split}&\int_{\Theta}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}_{(1)}(\lambda),Z_{1},\theta\big{)}\Big{)}\Big{(}\frac{\delta\mathfrak{m}}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),Z_{1}\big{)}-\frac{\delta\mathfrak{m}}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),\,\widetilde{\\!{Z}}_{1}\big{)}\Big{)}(\mathrm{d}\theta)\\\
&=-\frac{2\beta^{2}}{\sigma^{2}}\mathrm{Cov}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\Big{[}\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}_{(1)}(\lambda),Z_{1},\theta\big{)},\,\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),Z_{1},\theta\big{)}-\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),\tilde{Z}_{1},\theta\big{)}\Big{]}\\\
&\leq\frac{2\beta^{2}}{\sigma^{2}}\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\Big{[}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}_{(1)}(\lambda),Z_{1},\theta\big{)}\Big{)}^{2}\Big{]}^{1/2}\\\
&\qquad\times\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\Big{[}\Big{(}\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),Z_{1},\theta\big{)}-\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),\tilde{Z}_{1},\theta\big{)}\Big{)}^{2}\Big{]}^{1/2}.\end{split}$
(52)

Using Proposition D.5 and (14),

$\begin{split}&\Big{[}\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\Big{(}\Big{|}\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),Z_{1},\theta\big{)}-\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),\tilde{Z}_{1},\theta\big{)}\Big{|}^{2}\Big{)}\Big{]}^{1/2}\\\
&\leq\sqrt{2}\Big{(}\int_{\Theta}\big{[}g_{1}(\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda})),\theta)\big{]}^{2}\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda});\mathrm{d}\theta)\Big{)}^{1/2}\Big{(}2+\|Z_{1}\|^{2}+\|\,\widetilde{\\!{Z}}_{1}\|^{2}\Big{)}\\\
&\leq\sqrt{2}C_{\theta}\Big{(}1+2\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\big{[}\|\theta\|^{p}\big{]}\Big{)}\Big{(}2+\|Z_{1}\|^{2}+\|\,\widetilde{\\!{Z}}_{1}\|^{2}\Big{)}.\end{split}$
(53)

From Lemma D.1, we know

$\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\big{[}\|\theta\|^{p}\big{]}\leq\mathrm{R}(\tilde{\gamma}_{p}^{\sigma},\nu_{(1)}(\tilde{\lambda}))+\int_{\Theta}\|\theta\|^{p}\tilde{\gamma}_{p}^{\sigma}(\mathrm{d}\theta).$

By construction,

$\nu_{(1)}(\tilde{\lambda})=\nu_{n}+\frac{1-\tilde{\lambda}}{n}\big{(}\delta_{\,\widetilde{\\!{Z}}_{1}}-\delta_{Z_{1}}\big{)},$

so

$\begin{split}\mathrm{R}(\tilde{\gamma}_{p}^{\sigma},\nu_{(1)}(\tilde{\lambda}))&=\mathrm{R}(\tilde{\gamma}_{p}^{\sigma},\nu_{n})+\frac{1-\tilde{\lambda}}{n}\Big{(}\ell(\tilde{\gamma}^{\sigma}_{p},\,\widetilde{\\!{Z}}_{1})-\ell(\tilde{\gamma}^{\sigma}_{p},Z_{1})\Big{)}\\\
&\leq\mathrm{R}(\tilde{\gamma}_{p}^{\sigma},\nu_{n})+g(\tilde{\gamma}^{\sigma}_{p})\frac{1-\tilde{\lambda}}{n}\Big{(}2+\|\,\widetilde{\\!{Z}}_{1}\|^{2}+\|Z_{1}\|^{2})\Big{)},\end{split}$
(54)

and hence

$\begin{split}&\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\Big{[}\Big{|}\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),Z_{1},\theta\big{)}-\frac{\delta
S}{\delta\nu}\big{(}\nu_{(1)}(\tilde{\lambda}),\tilde{Z}_{1},\theta\big{)}\Big{|}^{2}\Big{]}^{1/2}\\\
&\leq\sqrt{2}C_{\theta}\Big{(}1+2\mathrm{R}(\tilde{\gamma}_{p}^{\sigma},\nu_{n})+2\int_{\Theta}\|\theta\|^{p}\tilde{\gamma}_{p}^{\sigma}(\mathrm{d}\theta)+2g(\tilde{\gamma}^{\sigma}_{p})\frac{1-\tilde{\lambda}}{n}\Big{(}2+\|Z_{1}\|^{2}+\|\,\widetilde{\\!{Z}}_{1}\|^{2}\Big{)}\Big{)}\\\
&\quad\times\Big{(}2+\|Z_{1}\|^{2}+\|\,\widetilde{\\!{Z}}_{1}\|^{2}\Big{)}.\end{split}$
(55)

We also know

$\begin{split}&\mathbb{E}_{\theta\sim\mathfrak{m}^{\beta,\sigma}(\nu_{(1)}(\tilde{\lambda}))}\Big{[}\Big{(}\frac{\delta\ell}{\delta
m}\big{(}\mathfrak{m}_{(1)}(\lambda),Z_{1},\theta\big{)}\Big{)}^{2}\Big{]}^{1/2}\\\