Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Learning via Wasserstein-Based High Probability Generalisation Bounds

 Paul Viallard

Inria, CNRS, Ecole Normale Superieure, PSL Research University, Paris, France

paul.viallard@inria.fr

&Maxime Haddouche

Inria, University College London and

Universite de Lille, France

maxime.haddouche@inria.fr

&Umut Simsekli

Inria, CNRS, Ecole Normale Superieure

PSL Research University, Paris, France

umut.simsekli@inria.fr

The authors contributed equally to this work

Benjamin Guedj

Inria and University College London, France and UK

benjamin.guedj@inria.fr

###### Abstract

Minimising upper bounds on the population risk or the generalisation gap has been widely used in structural risk minimisation (SRM) - this is in particular at the core of PAC-Bayesian learning. Despite its successes and unfailing surge of interest in recent years, a limitation of the PAC-Bayesian framework is that most bounds involve a Kullback-Leibler (KL) divergence term (or its variations), which might exhibit erratic behavior and fail to capture the underlying geometric structure of the learning problem - hence restricting its use in practical applications. As a remedy, recent studies have attempted to replace the KL divergence in the PAC-Bayesian bounds with the Wasserstein distance. Even though these bounds alleviated the aforementioned issues to a certain extent, they either hold in expectation, are for bounded losses, or are nontrivial to minimize in an SRM framework. In this work, we contribute to this line of research and prove novel Wasserstein distance-based PAC-Bayesian generalisation bounds for both batch learning with independent and identically distributed (_i.i.d._) data, and online learning with potentially non-_i.i.d._ data. Contrary to previous art, our bounds are stronger in the sense that _(i)_ they hold with high probability, _(ii)_ they apply to unbounded (potentially heavy-tailed) losses, and _(iii)_ they lead to optimizable training objectives that can be used in SRM. As a result we derive novel Wasserstein-based PAC-Bayesian learning algorithms and we illustrate their empirical advantage on a variety of experiments.

## 1 Introduction

Understanding generalisation is one of the main challenges in statistical learning theory, and even more so in modern machine learning applications. Typically, a _learning problem_ is described by a tuple \((\mathcal{H},\mathcal{Z},\ell)\) consisting of a hypothesis (or predictor) space \(\mathcal{H}\), a data space \(\mathcal{Z}\), and a loss function \(\ell:\mathcal{H}\times\mathcal{Z}\rightarrow\mathbb{R}\). The goal is to estimate the _population risk_ of a given hypothesis \(h\), defined as \(\mathsf{R}_{\mu}(h)=\mathbb{E}_{\mathbf{z}\sim\mu}[\ell(h,\mathbf{z})]\), where \(\mu\) denotes the unknown _data distribution_ over \(\mathcal{Z}\).

As \(\mu\) is not known, in practice, a hypothesis \(h\) is usually built by (approximately) minimising the _empirical risk_, given by \(\hat{\mathsf{R}}_{\mathcal{S}}(h)=\frac{1}{m}\sum_{i=1}^{m}\ell(h,\mathbf{z}_{ i})\), where \(\mathcal{S}=\{\mathbf{z}_{i}\in\mathcal{Z}\}_{i=1}^{m}\) is a dataset of \(m\) data points, independent and identically distributed (_i.i.d._) from \(\mu\). We define the generalisation gap of a hypothesis \(h\) as \(\hat{\mathsf{R}}_{\mathcal{S}}(h)-\mathsf{R}_{\mu}(h)\).

Developing upper bounds on the generalisation gap, _i.e._, _generalisation bounds_ has been a long-standing topic in statistical learning. While a plethora of techniques have been introduced, the PAC-Bayesian framework has gained significant traction over the past two decades to provide non-vacuous generalisation guarantees for complex structures such as neural networks during the training phase (see DR17, PORPH\({}^{+}\)21, PRSS21, among others). In these works, the bounds are also used to derive learning algorithms by minimising the right-hand side of a given bound. Beyond neural networks, the flexibility of PAC-Bayes learning makes it a useful toolbox to derive both theoretical results and practical algorithms in various learning fields such as reinforcement learning [15], online learning [16], multi-armed bandits [17, 18, 19], meta-learning [20, 21, 22, 13] to name but a few. The PAC-Bayesian bounds focus on a _randomised_ setting where the hypothesis is drawn from a _data-dependent_ distribution \(\rho\in\mathcal{M}(\mathcal{H})\), where \(\mathcal{M}(\mathcal{H})\) denotes the set of probability distributions defined on \(\mathcal{H}\). A classical PAC-Bayesian result is [20, Theorem 5] (the so-called McAllester bound), which states that, with probability at least \(1-\delta\), for any posterior distribution \(\rho\in\mathcal{M}(\mathcal{H})\),

\[\operatorname*{\mathbb{E}}_{h\sim\rho}\left[\operatorname*{\mathbb{R}}_{\mu}(h )-\hat{\operatorname*{\mathbb{R}}}_{\mathcal{S}}(h)\right]\leq\sqrt{\frac{ \operatorname*{KL}(\rho\|\pi)+\ln\frac{2\sqrt{m}}{\delta}}{2m}},\]

where \(\pi\in\mathcal{M}(\mathcal{H})\) is any data-free distribution and \(\operatorname*{KL}\) denotes the Kullback-Leibler divergence. In analogy with Bayesian statistics, \(\pi\) is often called the _prior_, and \(\rho\) is called the _posterior_ - we refer to [14] for a discussion on these terms.

While PAC-Bayesian bounds remain nowadays of the utmost interest to explain generalisation in various learning problems [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 111, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 140, 132, 133, 134, 135, 136, 137, 138, 139, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 199, 199, 198, 199, 199, 190, 191, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 199, 1999, 199, 1999, 199, 1999, 199, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 19999, 19999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 19999, 19999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 19999, 1999, 19999, 19999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 19999, 1999, 19999, 1999, 19999, 1999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 19999, 1999, 19999, 1999, 19999, 19999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 199999, 11. Using the supermartingale toolbox introduced in [13, 14], we prove in Section 3.1, novel PAC-Bayesian bounds based on the Wasserstein distance for _i.i.d._ data. While [1] proposed a McAllester-like bound for bounded losses, we propose a Catoni-like bound (see _e.g._, [16, Theorem 4.1] valid for heavy-tailed losses with bounded order 2 moments. This assumption is less restrictive than assuming subgaussian or bounded losses, which are at the core of many PAC-Bayes results. This assumption also covers distributions beyond subgaussian or subexponential ones (_e.g._, gamma distributions with a scale smaller than 1, which have an infinite exponential moment).
2. We provide in Section 3.2 the first generalisation bounds based on Wasserstein distances for the online PAC-Bayes framework of [13]. Our results are, again, Catoni-like bounds and hold for heavy-tailed losses with bounded order 2 moments. Previous work [1] already provided online strategies mixing PAC-Bayes and Wasserstein distances. However, their contributions focus on the best deterministic strategy, regularised by a Wasserstein distance, with respect to the deterministic notion of regret. Our results differ significantly as we provide the best-regularised strategy (still in the sense of a Wasserstein term) with respect to the notion of generalisation, which is new.
3. As our bounds are linear with respect to Wasserstein terms (contrary to those of [1]), they are well suited for optimisation procedures. Thus, we propose the first PAC-Bayesian learning algorithms based on Wasserstein distances instead of KL divergences. For the first time, we design PAC-Bayes algorithms able to output deterministic predictors (instead of distributions over all \(\mathcal{H}\)) designed from deterministic priors. This is due to the ability of the Wasserstein distance to measure the discrepancy between Dirac distributions. We then instantiate those algorithms in Section 4 on various datasets, paving the way to promising practical developments of PAC-Bayes learning.

To sum up, we highlight two benefits of PAC-Bayes learning with Wasserstein distance. First, it ships with sound theoretical results exploiting the geometry of the predictor space, holding for heavy-tailed losses. Such a weak assumption on the loss extends the usefulness of PAC-Bayes with Wasserstein distances to a wide range of learning problems, encompassing bounded losses. Second, it allows us to consider deterministic algorithms (_i.e._, sampling from Dirac measures) designed with respect to the notion of generalisation: we showcase their performance in our experiments.

**Outline.** Section 2 describes our framework and background, Section 3 contains our new theoretical results and Section 4 gathers our experiments. Appendix A gathers supplementary discussion, Appendix B contains all proofs of our claims, and Appendix C provides insights into our practical results as well as additional experiments.

## 2 Our framework

**Framework.** We consider a Polish predictor space \(\mathcal{H}\) equipped with a distance \(d\) and a \(\sigma\)-algebra \(\Sigma_{\mathcal{H}}\), a data space \(\mathcal{Z}\), and a loss function \(\ell:\mathcal{H}\times\mathcal{Z}\to\mathbb{R}\). In this work, we consider Lipschitz functions with respect to \(d\). We also associate a filtration \((\mathcal{F}_{i})_{i\geq 1}\) adapted to our data \((\mathbf{z}_{i})_{i=1,\dots,m}\), and we assume that the dataset \(\mathcal{S}\) follows the distribution \(\mathcal{D}\). In PAC-Bayes learning, we construct a data-driven posterior distribution \(\rho\in\mathcal{M}(\mathcal{H})\) with respect to a prior distribution \(\pi\).

**Definitions.** For all \(i\), we denote by \(\mathbb{E}_{i}[\cdot]\) the conditional expectation \(\mathbb{E}[\,\cdot\mid\mathcal{F}_{i}]\). In this work, we consider data-dependent priors. A stochastic kernel is a mapping \(\pi:\cup_{m=1}^{\infty}\mathcal{Z}^{m}\times\Sigma_{\mathcal{H}}\to[0,1]\) where _(i)_ for any \(B\in\Sigma_{\mathcal{H}}\), the function \(\mathcal{S}\mapsto\pi(\mathcal{S},B)\) is measurable, _(ii)_ for any dataset \(\mathcal{S}\), the function \(B\mapsto\pi(\mathcal{S},B)\) is a probability measure over \(\mathcal{H}\).

In what follows, we consider two different learning paradigms: _batch learning_, where the dataset is directly available, and _online learning_, where data streams arrive sequentially.

**Batch setting.** We assume the dataset \(\mathcal{S}\) to be _i.i.d._, so there exists a distribution \(\mu\) over \(\mathcal{Z}\) such that \(\mathcal{D}=\mu^{m}\). We then define, for a given \(h\in\mathcal{H}\), the _risk_ to be \(\mathsf{R}_{\mu}:=\mathbb{E}_{\mathbf{\kappa}\sim\mu}[\ell(h,\mathbf{z})]\) and its empirical counterpart \(\hat{\mathsf{R}}_{\mathcal{S}}:=\frac{1}{m}\sum_{i=1}^{m}\ell(h,\mathbf{z}_{i})\). Our results aim to bound the _expected generalisation gap_ defined by \(\mathbb{E}_{h\sim\rho}[\mathsf{R}_{\mu}(h)-\hat{\mathsf{R}}_{\mathcal{S}}(h)]\). We assume that the dataset \(\mathcal{S}\) is split into \(K\) disjoint sets \(\mathcal{S}_{1},\dots,\mathcal{S}_{K}\). We consider \(K\) stochastic kernels \(\pi_{1},\dots,\pi_{K}\) such that for any \(\mathcal{S}\), the distribution \(\pi_{i}(\mathcal{S},\cdot)\)_does not_ depend on \(\mathcal{S}_{i}\).

**Online setting.** We adapt the online PAC-Bayes framework of [13]. We assume that we have access to a stream of data \(\mathcal{S}=(\mathbf{z}_{i})_{i=1,\dots,m}\), arriving sequentially, with no assumption on \(\mathcal{D}\). In online PAC-Bayes, the goal is to define a posterior sequence \((\rho_{i})_{i\geq 1}\) from a prior sequence \((\pi_{i})_{i\geq 1}\), which can be data-dependent. We define an _online predictive sequence_\((\pi_{i})_{i=1\dots m}\) satisfying: _(i)_ for all \(i\) and dataset \(\mathcal{S}\), the distribution \(\pi_{i}(S,.)\) is \(\mathcal{F}_{i-1}\) measurable and _(ii)_ there exists \(\pi_{0}\) such that for all \(i\geq 1\), we have \(\pi_{i}(S,.)\gg\pi_{0}\). This last condition covers, in particular, the case where \(\mathcal{H}\) is an Euclidean space and for any \(i\), the distribution \(\pi_{i,\mathcal{S}}\) is a Dirac mass. All of those measures are uniformly continuous with respect to any Gaussian distribution.

**Wasserstein distance.** In this paper, we focus on the Wasserstein distance of order 1 (_a.k.a._, Earth Mover's distance) introduced by [14] in the optimal transport literature. Given a distance \(d:\mathcal{A}\times\mathcal{A}\to\mathbb{R}\) and a Polish space \((\mathcal{A},d)\), for any probability measures \(\alpha\) and \(\beta\) on \(\mathcal{A}\), the Wasserstein distance is defined by

\[\mathrm{W}(\alpha,\beta):=\inf_{\gamma\in\Gamma(\alpha,\beta)}\left\{\mathop{ \mathbb{E}}_{(a,b)\sim\gamma}d(a,b)\right\}, \tag{1}\]

where \(\Gamma(\alpha,\beta)\) is the set of joint probability measures \(\gamma\in\mathcal{M}(\mathcal{A}^{2})\) such that the marginals are \(\alpha\) and \(\beta\). The Wasserstein distance aims to find the probability measure \(\gamma\in\mathcal{M}(\mathcal{A}^{2})\) minimising the expected cost \(\mathop{\mathbb{E}}_{(a,b)\sim\gamma}d(a,b)\). We refer the reader to [20, 13] for an introduction to optimal transport.

## 3 Wasserstein-based PAC-Bayesian generalisation bounds

We present novel high-probability PAC-Bayesian bounds involving Wasserstein distances instead of the classical Kullback-Leibler divergence. Our bounds hold for heavy-tailed losses (instead of classical subgaussian and subexponential assumptions), extending the remits of [1, Theorem 11]. We exploit the supermartingale toolbox, recently introduced in PAC-Bayes framework by [13, 14], to derive bounds for both batch learning (Theorems 1 and 2) and online learning (Theorems 3 and 4).

### PAC-Bayes for batch learning with _i.i.d._ data

In this section, we use the batch setting described in Section 2. We state our first result, holding for heavy-tailed losses admitting order 2 moments. Such an assumption is in line, for instance, with reinforcement learning with heavy-tailed reward (see, _e.g._, [1, 1, 10]).

**Theorem 1**.: _We assume the loss \(\ell\) to be \(L\)-Lipschitz. Then, for any \(\delta\in(0,1]\), for any sequence of positive scalar \((\lambda_{i})_{i\in\{1,\dots,K\}}\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), the following holds for the distributions \(\pi_{i,\mathcal{S}}:=\pi_{i}(\mathcal{S},.)\) and for any \(\rho\in\mathcal{M}(\mathcal{H})\):_

\[\mathop{\mathbb{E}}_{h\sim\rho}\Big{[}\mathcal{R}_{\mu}(h)-\hat{ \mathcal{R}}_{\mathcal{S}}(h)\Big{]}\\ \leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho, \pi_{i,\mathcal{S}})+\frac{1}{m}\sum_{i=1}^{K}\frac{\ln\left(\frac{K}{\delta} \right)}{\lambda_{i}}+\frac{\lambda_{i}}{2}\left(\mathop{\mathbb{E}}_{h\sim \pi_{i,\mathcal{S}}}\Big{[}\hat{V}_{|\mathcal{S}_{i}|}(h)+V_{|\mathcal{S}_{i} |}(h)\Big{]}\right),\]

_where \(\pi_{i,\mathcal{S}}\) does not depend on \(\mathcal{S}_{i}\). Also, for any \(i,|S_{i}|\), we have \(\hat{V}_{|\mathcal{S}_{i}|}(h)=\sum_{\mathbf{z}\in\mathcal{S}_{i}}\left(\ell(h,\mathbf{z})-R_{\mu}(h)\right)^{2}\) and \(V_{|\mathcal{S}_{i}|}(h)=\mathop{\mathbb{E}}_{\mathcal{S}_{i}}\Big{[}\hat{V}_{ |\mathcal{S}_{i}|}(h)\Big{]}\)._

The proof is deferred to Appendix B.1. While Theorem 1 holds for losses taking values in \(\mathbb{R}\), many learning problems rely in practice on more constrained losses. This loss can be bounded as in the case of, _e.g._, supervised learning or the multi-armed bandit problem [15], or simply non-negative as in regression problems involving the quadratic loss (studied, for instance, in [13, 14]). Using again the supermartingale toolbox, we prove in Theorem 2 a tighter bound holding for heavy-tailed non-negative losses.

**Theorem 2**.: _We assume our loss \(\ell\) to be non-negative and \(L\)-Lipschitz. We also assume that, for any \(1\leq i\leq K\), for any dataset\(\mathcal{S}\), we have \(\mathop{\mathbb{E}}_{h\sim\pi_{i}(.,\mathcal{S}),z\sim\mu}\left[\ell(h,z)^{2} \right]\leq 1\) (bounded order 2 moments forpriors). Then, for any \(\delta\in(0,1]\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), the following holds for the distributions \(\pi_{i,\mathcal{S}}:=\pi_{i}(\mathcal{S},.)\) and for any \(\rho\in\mathcal{M}(\mathcal{H})\):_

\[\operatorname*{\mathbb{E}}_{h\sim\rho}\left[\mathcal{R}_{\mu}(h)-\hat{\mathcal{ R}}_{\mathcal{S}}(h)\right]\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}( \rho,\pi_{i,\mathcal{S}})+\sum_{i=1}^{K}\sqrt{\frac{2|\mathcal{S}_{i}|\ln\frac {K}{\delta}}{m^{2}}},\]

_where \(\pi_{i,\mathcal{S}}\) does not depend on \(\mathcal{S}_{i}\)._

Note that when the loss function takes values in \([0,1]\), an alternative strategy allows tightening the last term of the bound by a factor \(\nicefrac{{1}}{{2}}\). This result is rigorously stated in Theorem 6 of Appendix B.3.

**High-level ideas of the proofs.** Theorems 1 and 2 are structured around two tools. First, we exploit the Kantorovich-Rubinstein duality [22, Remark 6.5] to replace the change of measure inequality [14, 23]; this allows us to consider a Wasserstein distance instead of a KL term. Then, we exploit the supermartingales used in [11, 1] alongside Ville's inequality (instead of Markov's one) to obtain a high probability bound holding for heavy-tailed losses. Combining those techniques provides our PAC-Bayesian bounds.

**Analysis of our bounds.** Our results hold for Lipschitz losses and allow us to consider heavy-tailed losses with bounded order 2 moments. While such an assumption on the loss is more restrictive than in classical PAC-Bayes, allowing heavy-tailed losses is strictly less restrictive. While Theorem 1 is our most general statement, Theorem 2 allows recovering a tighter result (without empirical variance terms) for non-negative heavy-tailed losses. An important point is that the variance terms are considered with respect to the prior distributions \(\pi_{i,\mathcal{S}}\) and not \(\rho\) as in [11, 1]. This is crucial as these papers rely on the implicit assumption of order 2 moments, holding uniformly for all \(\rho\in\mathcal{M}(\mathcal{H})\), while we only require this assumption for the prior distributions \((\pi_{i,\mathcal{S}})_{i=1,\ldots,K}\). Such an assumption is in line with the PAC-Bayesian literature, which often relies on bounding an averaged quantity with respect to the prior. This strength is a consequence of the Kantorovich-Rubinstein duality. To illustrate this, consider _i.i.d._ data with distribution \(\mu\) admitting a finite variance bounded by \(V\) and the loss \(\ell(h,z)=|h-z|\) where both \(h\) and \(z\) lie in the real axis. Notice that in this particular case, we can imagine that \(z\) is a data point and \(h\) is a hypothesis outputting the same scalar for all data. To satisfy the assumption of Theorem 2, it is enough, by Cauchy Schwarz, to satisfy \(\operatorname*{\mathbb{E}}_{h\sim\pi_{i,\mathcal{S}},z\sim\mathcal{S}}[\ell(h, z)^{2}]\leq\operatorname*{\mathbb{E}}[h^{2}]+2V\operatorname*{\mathbb{E}}[|h|]+V^{2} \leq 1\) for all \(\pi_{i,\mathcal{S}}\). On the contrary, [11, 1] would require this condition to hold for all \(\rho\), which is more restrictive. Finally, an important point is that our bound allows us to consider Dirac distributions with disjoint support as priors and posteriors. On the contrary, KL divergence forces us to consider a non-Dirac prior for our bound to be non-vacuous. This allows us to retrieve a uniform-convergence bound described in Corollary 7.

**Role of data-dependent priors.** Theorems 1 and 2 allow the use of prior distributions depending possibly on a fraction of data. Such a dependency is crucial to control our sum of Wasserstein terms as we do not have an explicit convergence rate. For instance, for a fixed \(K\), consider a compact predictor space \(\mathcal{H}\), a bounded loss and the _Gibbs posterior_ defined as \(d\rho(h)\propto\exp\left(-\lambda\hat{\mathcal{R}}_{\mathcal{S}}(h)\right)dh\) where \(\lambda>0\). Also define for any \(i\) and \(\mathcal{S}\), the distribution \(d\pi_{i,\mathcal{S}}(h)\propto\exp\left(-\lambda\mathcal{R}_{\mathcal{S}/ \mathcal{S}_{i}}(h)\right)dh\). Then, by the law of large numbers, when \(m\) goes to infinity, for any \(h\), both \(\mathcal{R}_{\mathcal{S}}(h)\) and \((\mathcal{R}_{\mathcal{S}/\mathcal{S}_{i}}(h))_{i=1,\ldots,m}\) converge to \(\operatorname*{\mathbb{R}}_{\mu}(h)\). This ensures, alongside with the dominated convergence theorem, that for any \(i\), the Wasserstein distance \(\mathrm{W}(\rho,\pi_{i,\mathcal{S}})\) goes to zero as \(m\) goes to infinity.

**Comparison with the literature.**[1, Theorem 11] establishes a PAC-Bayes bound with Wasserstein distance valid for bounded losses being Lipschitz with high probability. While we circumvent the first assumption, the second one is less restrictive than actual Lipschitzness and can also be used in our setting. Also [1, Theorem 12] proposes an explicit convergence for finite predictor classes. We show in Appendix A that we are also able to recover such a convergence.

**Towards new PAC-Bayesian algorithms.** From Theorem 2, we derive a new PAC-Bayesian algorithm for Lipschitz non-negative losses:

\[\operatorname*{argmin}_{\rho\in\mathcal{M}(\mathcal{H})}\operatorname*{ \mathbb{E}}_{h\sim\rho}\left[\hat{\mathcal{R}}_{\mathcal{S}}(h)\right]+\sum_{ i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho,\pi_{i,\mathcal{S}}). \tag{2}\]

Equation (2) uses Wasserstein distances as regularisers and allows the use of multiple priors. We compare ourselves to the classical PAC-Bayes algorithm derived from [1, Theorem 1.2.6] (whichleads to Gibbs posteriors):

\[\operatorname*{argmin}_{\rho\in\mathcal{M}(\mathcal{H})}\operatorname*{\mathbb{E}}_ {h\sim\rho}\big{[}\hat{\mathsf{R}}_{\mathcal{S}}(h)\big{]}+\frac{\operatorname {KL}(\rho,\pi)}{\lambda}. \tag{3}\]

Considering a Wasserstein distance in Equation (2) makes our algorithm more flexible than in Equation (3), the KL divergence implies absolute continuity _w.r.t._ the prior \(\pi\). Such an assumption is not required to use Equation (2) and covers the case of prior Dirac distributions. Finally, Equation (2) relies on a fixed value \(K\) whose value is discussed below.

**Role of \(K\).** We study the cases \(K=1\), \(\sqrt{m}\), and \(m\) in Theorem 2. We refer to Appendix A for a detailed treatment. First of all, when \(K=1\), we recover a classical batch learning setting where all data are collected at once. In this case, we have a single Wasserstein with no convergence rate coupled with a statistical srsatz of \(\sqrt{\nicefrac{{\ln(1/\delta)}}{{m}}}\). However, similarly to [1, Theorem 12], in the case of a finite predictor class, we are able to recover an explicit convergence rate. The case \(K=\sqrt{m}\) provides a tradeoff between the number of points required to have good data-dependent priors (which may lead to a small \(\sum_{i=1}^{\sqrt{m}}\operatorname{W}(\rho,\pi_{i})\)) and the number of sets required to have an explicit convergence rate. Finally, the case \(K=m\) leads to a vacuous bound as we have the incompressible term \(\sqrt{\ln\nicefrac{{(m/\delta)}}{{\delta}}}\), which makes the bound vacuous for large values of \(m\). This means that the batch setting is not fitted to deal with a data stream arriving sequentially. To mitigate that weakness, we propose in Section 3.2 the first online PAC-Bayes bounds with Wasserstein distances.

### Wasserstein-based generalisation bounds for online learning

Here, we use the online setting described in Section 2 and derive the first online PAC-Bayes bounds involving Wasserstein distances in Theorems 3 and 4. Online PAC-Bayes bounds are meant to derive online counterparts of classical PAC-Bayesian algorithms [1], where the KL-divergence acts as a regulariser. We show in Theorems 3 and 4 that it is possible to consider online PAC-Bayesian algorithms where the regulariser is a Wasserstein distance, which allows us to optimise on measure spaces without a restriction of absolute continuity.

**Theorem 3**.: _We assume our loss \(\ell\) to be \(L\)-Lipschitz. Then, for any \(\delta\in(0,1]\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), the following holds for the distributions \(\pi_{i,\mathcal{S}}:=\pi_{i}(\mathcal{S},.)\) and for any sequence \((\rho_{i})_{i=1\cdots m}\in\mathcal{M}(\mathcal{H})^{m}\):_

\[\sum_{i=1}^{m}\operatorname*{\mathbb{E}}_{h_{i}\sim\rho_{i}}\Big{[} \operatorname*{\mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})\mid\mathcal{F}_{i-1}]- \ell(h_{i},\mathbf{z}_{i})\Big{]} \leq 2L\sum_{i=1}^{m}\operatorname{W}(\rho_{i},\pi_{i,\mathcal{S}})\] \[+\frac{1}{2}\sum_{i=1}^{m}\operatorname*{\mathbb{E}}_{h_{i}\sim \pi_{i,\mathcal{S}}}\Big{[}\hat{V}_{i}(h_{i},\mathbf{z}_{i})+V_{i}(h_{i})\Big{]} +\frac{\ln(1/\delta)}{\lambda},\]

_where for all \(i\), \(\hat{V}_{i}(h_{i},\mathbf{z}_{i})=(\ell(h_{i},\mathbf{z}_{i})-\operatorname*{ \mathbb{E}}_{i-1}[\ell(h_{i},\mathbf{z}_{i})])^{2}\) is the conditional empirical variance at time \(i\) and \(V_{i}(h_{i})=\operatorname*{\mathbb{E}}_{i-1}[\hat{V}(h_{i},\mathbf{z}_{i})]\) is the true conditional variance._

The proof is deferred to Appendix B.4. We also provide the following bound, being an online analogous of Theorem 2, valid for non-negative heavy-tailed losses.

**Theorem 4**.: _We assume our loss \(\ell\) to be non-negative and \(L\)-Lipschitz. We also assume that, for any \(i\), \(\mathcal{S}\), \(\operatorname*{\mathbb{E}}_{h\sim\pi_{i},\mathcal{S}}\big{[}\operatorname*{ \mathbb{E}}_{i-1}[\ell(h,\mathbf{z}_{i})^{2}]\big{]}\leq 1\) (bounded conditional order \(2\) moments for priors). Then, for any \(\delta\in(0,1]\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), any online predictive sequence (used as priors) \((\pi_{i})_{i\geq 1}\), we have with probability at least \(1-\delta\) over the sample \(S\sim\mu\), the following, holding for the data-dependent measures \(\pi_{i,\mathcal{S}}:=\pi_{i}(S,.)\) and any posterior sequence \((\rho_{i})_{i\geq 1}\):_

\[\frac{1}{m}\sum_{i=1}^{m}\operatorname*{\mathbb{E}}_{h_{i}\sim \rho_{i}}\Big{[}\operatorname*{\mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})\mid \mathcal{F}_{i-1}]-\ell(h_{i},\mathbf{z}_{i})\Big{]}\leq\frac{2L}{m}\sum_{i=1} ^{m}\operatorname{W}(\rho_{i},\pi_{i,\mathcal{S}})+\sqrt{\frac{2\ln\big{(} \frac{1}{\delta}\big{)}}{m}}.\]

The proof is deferred to Appendix B.5.

**Analysis of our bounds.** Theorems 3 and 4 are, to our knowledge, the first results involving Wasserstein distances for online PAC-Bayes learning. They are the online counterpart of Theorems 1 and 2, and the discussion of Section 3.1 about the involved assumptions also apply here. The sum of Wasserstein distances involved here is a consequence of the online setting and must grow sublinearly for the bound to be tight. For instance, when \((\rho_{i}=\delta_{h_{i}})_{i\geq 1}\) is the output of an online algorithm outputting Dirac measures and \(\pi_{i,\mathcal{S}}=\rho_{i-1}\), the sum of Wasserstein is exactly \(\sum_{i=1}^{m}d(h_{i},h_{i-1})\). This sum has to be sublinear for the bound to be non-vacuous, and the tightness depends on the considered learning problem. An analogous of this sum can be found in dynamic online learning (Zin2013) where similar sums appear as _path lengths_ to evaluate the complexity of the problem.

**Comparison with literature.** We compare our results to existing PAC-Bayes bounds for martingales of [SLCB\({}^{+}\)12]. [SLCB\({}^{+}\)12, Theorem 4] is a PAC-Bayes bound for martingales, which controls an average of martingales, similar to our Theorem 1. Under a boundedness assumption, they recover a McAllester-typed bound, while Theorem 1 is more of a Catoni-typed result. Also, [SLCB\({}^{+}\)12, Theorem 7] is a Catoni-typed bound involving a conditional variance, similar to our Theorem 4. They require to bound uniformly the variance on all the predictor sets, while we only assume averaged variance with respect to priors, which is what we required to perform Theorem 4.

**A new online algorithm.**[HG22] derived from their main theorem, an online counterpart of Equation (3), proving it comes with guarantees. Similarly, we exploit Theorem 4 to derive the online counterpart of Equation (2), from the data-free initialisation \(\rho_{1}\)

\[\forall i\geq 1,\ \ \rho_{i}\in\operatorname*{argmin}_{\rho\in\mathcal{M}( \mathcal{H})}\operatorname*{\mathbb{E}}_{h\sim\rho}[\ell(h_{i},\mathbf{z}_{i}) ]+2L\mathrm{W}(\rho,\pi_{i,\mathcal{S}}). \tag{4}\]

We highlight the merits of the algorithm defined by Equation (4), alongside with the one from Equation (2), in Section 4.

## 4 Learning via Wasserstein regularisation

Theorems 2 and 4 are designed to be informative on the generalisation ability of a single hypothesis even when Dirac distributions are considered. In particular, our results involve Wasserstein distances acting as regularisers on \(\mathcal{H}\). In this section, we show that a Wasserstein regularisation of the learning objective, which comes from our theoretical bounds, helps to better generalise in practice. Inspired by Equations (2) and (4), we derive new PAC-Bayesian algorithms for both batch and online learning involving a Wasserstein distance (see Section 4.1), we describe our experimental framework in Section 4.2 and we present some of the results in Section 4.3. Additional details, experiments, and discussions are gathered in Appendix C due to space constraints. All the experiments are reproducible with the source code provided on GitHub at [https://github.com/paulviallard/NeurIPS23-PB-Wasserstein](https://github.com/paulviallard/NeurIPS23-PB-Wasserstein).

### Learning algorithms

**Classification.** In the classification setting, we assume that the data space \(\mathcal{Z}=\mathcal{X}\times\mathcal{Y}\) is composed of a \(d\)-dimensional _input space_\(\mathcal{X}=\{\mathbf{x}\in\mathbb{R}^{d}\mid\|\mathbf{x}\|_{2}\leq 1\}\) and a finite _label space_\(\mathcal{Y}=\{1,\ldots,|\mathcal{Y}|\}\) with \(|\mathcal{Y}|\) labels. We aim to learn models \(h_{\mathbf{w}}:\mathbb{R}^{d}\to\mathbb{R}^{|\mathcal{Y}|}\) parameterised by a weight vector \(\mathbf{w}\) that outputs, given an input \(\mathbf{x}\in\mathcal{X}\), a score \(h_{\mathbf{w}}(\mathbf{x})[y^{\prime}]\in\mathbb{R}\) for each label \(y^{\prime}\). This score allows us to assign a label to \(\mathbf{x}\in\mathcal{X}\); to check if \(h_{\mathbf{w}}\) classifies correctly the example \((\mathbf{x},y)\), we use the _classification loss_ defined by \(\ell^{c}(h_{\mathbf{w}},(\mathbf{x},y)):=\mathds{1}\left[h_{\mathbf{w}}( \mathbf{x})[y]-\max_{y^{\prime}\neq y}h_{\mathbf{w}}(\mathbf{x})[y^{\prime}] \leq 0\right]\), where \(\mathds{1}\) denotes the indicator function.

**Batch algorithm.** In the batch setting, we aim to learn a parametrised hypothesis \(h_{\mathbf{w}}\in\mathcal{H}\) that minimises the population classification risk \(\mathfrak{R}_{\mu}(h_{\mathbf{w}})=\operatorname*{\mathbb{E}}_{(\mathbf{x},y) \sim\mu}\ell^{c}(h_{\mathbf{w}},(\mathbf{x},y))\) that we can only estimate through the empirical classification risk \(\mathfrak{R}_{\mathcal{S}}(h_{\mathbf{w}})=\frac{1}{m}\sum_{i=1}^{m}\ell^{c}( h_{\mathbf{w}},(\mathbf{x}_{i},y_{i}))\). To learn the hypothesis, we start from Equation (2), when the distributions \(\rho\) and \(\pi_{1},\ldots,\pi_{K}\) are Dirac masses, localised at \(h_{\mathbf{w}},h_{\mathbf{w}_{1}},\ldots h_{\mathbf{w}_{K}}\in\mathcal{H}\) respectively. Indeed, in this case, \(\mathrm{W}(\rho,\pi_{i,\mathcal{S}})=d(h_{\mathbf{w}},h_{\mathbf{w}_{i}})\) for any \(i\). However, the loss \(\ell^{c}(.,\mathbf{z})\) is not Lipschitz and the derivatives are zero for all examples \(\mathbf{z}\in\mathcal{X}\times\mathcal{Y}\), which prevents its use in practice to obtain such a hypothesis \(h_{\mathbf{w}}\). Instead, for the population risk \(\mathds{R}_{\mu}(h)\) and the empirical risk \(\mathds{R}_{\mathcal{S}}(h)\) (in Theorem 2 and Equation (2)), we consider the loss \(\ell(h,(\mathbf{x},y))=\frac{1}{|\mathcal{Y}|}\sum_{y^{\prime}\neq y}\max(0,1 -\eta(h[y]-h[y^{\prime}]))\), which is \(\eta\)-Lipschitz _w.r.t._ the outputs \(h[1],\ldots,h[|\mathcal{Y}|]\). This loss has subgradients everywhere, which is convenient in practice. We go a step further by _(a)_ setting \(L=\frac{1}{2}\) and _(b)_ adding a parameter \(\varepsilon>0\) to obtain the objective

\[\operatorname*{argmin}_{h_{\mathbf{w}}\in\mathcal{H}}\left\{\hat{\mathbf{R}}_{ \mathcal{S}}(h_{\mathbf{w}})+\varepsilon\left[\sum_{i=1}^{K}\frac{|\mathcal{S} _{i}|}{m}d\left(h_{\mathbf{w}},h_{\mathbf{w}_{i}}\right)\right]\right\}. \tag{5}\]

To (approximately) solve Equation (5), we propose a two-step algorithm. First, Priors Learning learns \(K\) hypotheses \(h_{\mathbf{w}_{1}},\ldots,h_{\mathbf{w}_{K}}\in\mathcal{H}\) by minimising the empirical risk via stochastic gradient descent. Second, Posterior Learning learns the hypothesis \(h_{\mathbf{w}}\in\mathcal{H}\) by minimising the objective associated with Equation (5). More precisely, Priors Learning outputs the hypotheses \(h_{\mathbf{w}_{1}},\cdots,h_{\mathbf{w}_{K}}\), obtained by minimising the empirical risk through mini-batches. Those batches are designed such that for any \(i\), the hypothesis \(h_{\mathbf{w}_{i}}\) does not depend on \(\mathcal{S}_{i}\). Then, given \(h_{\mathbf{w}_{1}},\ldots,h_{\mathbf{w}_{K}}\in\mathcal{H}\), Posterior Learning minimises the objective in Equation (5) with mini-batches. Those algorithms are presented in Algorithm 1 of Appendix C. While \(\varepsilon\) is not suggested by Equation (2), it helps to control the impact of the regularisation in practice. Equation (5) then optimises a tradeoff between the empirical risk and the regularisation term \(\varepsilon\sum_{i=1}^{K}\frac{|\mathcal{S}_{i}|}{m}d(h_{\mathbf{w}},h_{ \mathbf{w}_{i}})\).

**Online algorithm.** Online algorithms output, at each time step \(i\in\{1,\ldots,m\}\), a new hypothesis \(h_{\mathbf{w}_{i}}\). From Equation (4), particularised to a sequence of Dirac distributions (localised in \(h_{\mathbf{w}_{1}},\cdots,h_{\mathbf{w}_{K}}\)), we design a novel online PAC-Bayesian algorithm with a Wasserstein regulariser:

\[\forall i\geq 1,\;\;h_{i}\in\operatorname*{argmin}_{h_{\mathbf{w}}\in\mathcal{H}} \ell(h_{\mathbf{w}},\mathbf{z}_{i})+d\left(h_{\mathbf{w}},h_{\mathbf{w}_{i-1}} \right)\;\;s.t.\;\;d\left(h_{\mathbf{w}},h_{\mathbf{w}_{i-1}}\right)\leq 1. \tag{6}\]

According to Theorem 4, such an algorithm aims to bound the _population cumulative classification loss_\(\mathfrak{C}_{\mu}=\sum_{i=1}^{m}\mathbb{E}[\ell^{c}(h_{\mathbf{w}_{i}}, \mathbf{z}_{i})\mid\mathcal{F}_{i-1}]\). Note that we added the constraint \(d\left(h_{\mathbf{w}},h_{\mathbf{w}_{i-1}}\right)\leq 1\) compared to Equation (4). This constraint ensures that the new hypothesis \(h_{\mathbf{w}_{i}}\) is not too far from \(h_{\mathbf{w}_{i-1}}\) (in the sense of the distance \(\|\cdot\|_{2}\)). Note that the constrained optimisation problem in Equation (6) can be rewritten in an unconstrained form (see [1]) thanks to a barrier \(B(\cdot)\) defined by \(B(a)=0\) if \(a\leq 0\) and \(B(a)=+\infty\) otherwise; we have

\[\forall i\geq 1,\;\;h_{i}\in\operatorname*{argmin}_{h_{\mathbf{w}}\in\mathcal{H}} \ell(h_{\mathbf{w}},\mathbf{z}_{i})+d\left(h_{\mathbf{w}},h_{\mathbf{w}_{i-1} }\right)+B(d\left(h_{\mathbf{w}},h_{\mathbf{w}_{i-1}}\right)-1). \tag{7}\]

When solving the problem in Equation (7) is not feasible, we approximate it with a log barrier of [1] (suitable in a stochastic gradient setting); given a parameter \(t>0\), the log barrier extension is defined by \(\hat{B}(a)=-\frac{1}{t}\ln(-a)\) if \(a\leq-\frac{1}{t^{2}}\) and \(\hat{B}(a)=ta-\frac{1}{t}\ln(\frac{1}{t^{2}})+\frac{1}{t}\) otherwise. We present in Appendix C Algorithm 2 that aims to (approximately) solve Equation (7). To do so, for each new example \((\mathbf{x}_{i},y_{i})\), the algorithm runs several gradient descent steps to optimise Equation (7).

### Experimental framework

In this part, we assimilate the predictor space \(\mathcal{H}\) to the parameter space \(\mathbb{R}^{d}\). Thus, the distance \(d\) is the Euclidean distance between two parameters: \(d\left(h_{\mathbf{w}},h_{\mathbf{w}^{\prime}}\right)=\|\mathbf{w}-\mathbf{w}^ {\prime}\|_{2}\). This implies that the Lipschitzness of \(\ell\) has to be taken _w.r.t._\(\mathbf{w}\) instead of \(h_{\mathbf{w}}\).

**Models.** We consider that the models are either linear or neural networks (NN). Linear models are defined by \(h_{\mathbf{w}}(\mathbf{x})=W\mathbf{x}+b\), where \(W\in\mathbb{R}^{|\mathcal{Y}|\times d}\) is the weight matrix, \(b\in\mathbb{R}^{|\mathcal{Y}|}\) is the bias, and \(\mathbf{w}=\operatorname{vec}(\{W,b\})\) its vectorisation; the vector \(\mathbf{w}\) with the zero vector. Thanks to the definition of \(\mathcal{X}\), we know from Lemma 8 (and the composition of Lipschitz functions) that the loss is \(\sqrt{2}\eta\)-Lipschitz _w.r.t._\(\mathbf{w}\). For neural networks, we consider fully connected ReLU neural networks with \(L\) hidden layers and \(D\) nodes, where the leaky ReLU activation function \(\operatorname{ReLU}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}\) applies elementwise \(x\mapsto\max(x,0.01x)\). More precisely, the network is defined by \(h_{\mathbf{w}}(\mathbf{x})=Wh^{L}(\cdots h^{1}(\mathbf{x}))+b\) where \(W\in\mathbb{R}^{|\mathcal{Y}|\times D}\), \(b\in\mathbb{R}^{|\mathcal{Y}|}\). Each layer \(h^{i}(\mathbf{x})=\operatorname{ReLU}(W_{i}\mathbf{x}+b_{i})\) has a weight matrix \(W_{i}\in\mathbb{R}^{D\times D}\) and bias \(b_{i}\in\mathbb{R}^{D}\) except for \(i=1\) where we have \(W_{1}\in\mathbb{R}^{D\times d}\). The weights \(\mathbf{w}\) are also the vectorisation \(\mathbf{w}=\operatorname{vec}(\{W,W_{L},\ldots,W_{1},b,b_{L},\ldots,b_{1}\})\). We have precised in Lemma 9 that our loss is Lipschitz _w.r.t._ the weights \(\mathbf{w}\). We initialise the network similarly to [1] by sampling the weights from a Gaussian distribution with zero mean and a standard deviation of \(\sigma=0.04\); the weights are further clipped between \(-2\sigma\) and \(+2\sigma\). Moreover, the values in the biases \(b_{1},\ldots,b_{L}\) are set to 0.1, while the values for \(b\) are set to 0. In the following, we consider \(D=600\) and \(L=2\); more experiments are considered in the appendix.

**Optimisation.** To perform the gradient steps, we use the COCOB-Backprop optimiser [1] (with parameter \(\alpha=10000\)).2 This optimiser is flexible as the learning rate is adaptative and, thus, does not require hyperparameter tuning. For Algorithm 1, which solves Equation (5), we fix a batch size of \(100\), _i.e._, \(|\mathcal{U}|=100\), and the number of epochs \(T\) and \(T^{\prime}\) are fixed to perform at least \(20000\) iterations. Regarding Algorithm 2, which solves Equation (7), we set \(t=100\) for the log barrier, which is enough to constrain the weights and the number of iterations to \(T=10\).

**Datasets.** We study the performance of Algorithms 1 and 2 on UCI datasets [1] along with MNIST [14] and FashionMNIST [15]. We also split all the data (from the original training/test set) in two halves; the first part of the data serves in the algorithm (and is considered as a training set), while the second part is used to approximate the population risks \(\mathfrak{R}_{\mu}(h)\) and \(\mathfrak{C}_{\mu}\) (and considered as a testing set).

### Results

We present in Table 1 the performance of Algorithms 1 and 2 compared to the Empirical Risk Minimisation (ERM) and the Online Gradient Descent (OGD) with the COCOB-Backprop optimiser. Tables 0(a) and 0(c) present the results of Algorithm 1 for the _i.i.d._ setting on linear and neural networks respectively, while Tables 0(b) and 0(d) present the results of Algorithm 2 for the online case.

**Analysis of the results.** In batch learning, we note that the regularisation term brings generalisation improvements compared to the empirical risk minimisation. Indeed, our batch algorithm (Algorithm 1) has a lower population risk \(\mathfrak{R}_{\mu}(h)\) on 11 datasets for the linear models and 9 datasets for the neural networks. In particular, notice that NNs obtained from Algorithm 1 are more efficient than the ones obtained from ERM on MNIST and FashionMNIST, which are the more challenging datasets. This suggests that the regularisation term helps to generalise well. For the online case, the performance of the linear models obtained from our algorithm (Algorithm 2) and by OGD are comparable: we have a tighter population classification risk \(\mathfrak{R}_{\mu}(h)\) on \(5\) datasets over \(13\). However, notice that the risk difference is less than \(0.05\) on \(6\) datasets. The advantage of Algorithm 2 is more pronounced for neural networks: we improve the performance in all datasets except adult and sensorless. Hence, this confirms that optimising the regularised loss \(\ell(h_{\mathbf{w}},\mathbf{z}_{i})+\|\mathbf{w}-\mathbf{w}_{i-1}\|\) brings a good advantage compared to the loss \(\ell(h_{\mathbf{w}},\mathbf{z}_{i})\) only. A possible explanation would be that OGD suffers from underfitting (with a high empirical risk \(\mathfrak{C}_{\mu}\)) while we are able to control overfitting through a regularisation term. Indeed, only one gradient descent step is done for each new datum \((\mathbf{x}_{i},y_{i})\), which might not be sufficient to decrease the loss. Instead, our method solves the problem associated with Equation (7) and constrains the descent with the norm \(\|\mathbf{w}-\mathbf{w}_{i-1}\|\).

## 5 Conclusion and Perspectives

We derived novel generalisation bounds based on the Wasserstein distance, both for batch and online learning, allowing for the use of deterministic hypotheses through PAC-Bayes. We derived new learning algorithms which are inspired by the bounds, with remarkable empirical performance on a number of datasets: we hope our work can pave the way to promising future developments (both theoretical and practical) of generalisation bounds based on the Wasserstein distance. Given the mostly theoretical nature of our work, we do not foresee an immediate negative societal impact, although we hope a better theoretical understanding of generalisation will ultimately benefit practitioners of machine learning algorithms and encourage virtuous initiatives.

## 6 Acknowledgements

We warmly thank the reviewers who provided insightful comments and suggestions which greatly helped us improve our manuscript. P.V. and U.S. are partially supported by the French government under the management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute). U.S. is also partially supported by the European Research Council Starting Grant DYNASTY - 101039676. B.G. acknowledges partial support by the U.S. Army Research Laboratory and the U.S. Army Research Office, and by the U.K. Ministry of Defence and the U.K. Engineering and Physical Sciences Research Council (EPSRC) under grant number EP/R013616/1. B.G. acknowledges partial support from the French National Agency for Research, grants ANR-18-CE40-0016-01 and ANR-18-CE23-0015-02.

## References

* [ACB17] Martin Arjovsky, Soumith Chintala, and Leon Bottou. Wasserstein Generative Adversarial Networks. In _International Conference on Machine Learning (ICML)_, 2017.
* [AEMM22] Ron Amit, Baruch Epstein, Shay Moran, and Ron Meir. Integral Probability Metrics PAC-Bayes Bounds. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2022.
* [AG18] Pierre Alquier and Benjamin Guedj. Simpler PAC-Bayesian bounds for hostile data. _Machine Learning_, 107(5), 2018.
* [AM18] Ron Amit and Ron Meir. Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory. In _International Conference on Machine Learning (ICML)_, 2018.

\begin{table}

\end{table}
Table 1: Performance of Algorithms 1 and 2 compared respectively to ERM and OGD on different datasets on linear and neural network models. For the _i.i.d._ setting, we consider \(\varepsilon=1/_{m}\) and \(\varepsilon=1/\sqrt{m}\) and with \(K=0.2\sqrt{m}\). For each method, we plot the empirical risk \(\mathfrak{R}_{\mathcal{S}}(h)\) or \(\mathcal{E}_{\mathcal{S}}\) with its associated test risk \(\mathfrak{R}_{\mu}(h)\) or \(\mathfrak{e}_{\mu}\). The risk in **bold** corresponds to the lowest one among the ones considered. For the online case, the two population risks are underlined when the absolute difference is lower than 0.05.

* [ARC16] Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of Gibbs posteriors. _Journal of Machine Learning Research_, 2016.
* [BG21] Felix Biggs and Benjamin Guedj. Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networks. _Entropy_, 23(10), 2021.
* [BG22a] Felix Biggs and Benjamin Guedj. Non-Vacuous Generalisation Bounds for Shallow Neural Networks. In _International Conference on Machine Learning (ICML)_, 2022.
* [BG22b] Felix Biggs and Benjamin Guedj. On Margins and Derandomisation in PAC-Bayes. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_, 2022.
* [BG23] Felix Biggs and Benjamin Guedj. Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_, 2023.
* [BM01] Peter Bartlett and Shahar Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. In _Conference on Computational Learning Theory (COLT)_, 2001.
* [BM02] Peter Bartlett and Shahar Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. _Journal of Machine Learning Research_, 2002.
* [BT08] Bernard Bercu and Abderrahmen Touati. Exponential inequalities for self-normalized martingales with applications. _The Annals of Applied Probability_, 2008.
* [BV04] Stephen Boyd and Lieven Vandenberghe. _Convex optimization_. Cambridge University Press, 2004.
* [BZG22] Felix Biggs, Valentina Zantedeschi, and Benjamin Guedj. On Margins and Generalisation for Voting Classifiers. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2022.
* [Cat07] Olivier Catoni. _PAC-Bayesian supervised classification: the thermodynamics of statistical learning_. Institute of Mathematical Statistics, 2007.
* [Cat16] Olivier Catoni. PAC-Bayesian bounds for the Gram matrix and least squares regression with a random design. _arXiv_, abs/1603.05229, 2016.
* [CDE\({}^{+}\)21] Alexander Camuto, George Deligiannidis, Murat A. Erdogdu, Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2021.
* [CG17] Olivier Catoni and Ilaria Giulini. Dimension-free PAC-Bayesian bounds for matrices, vectors, and linear least squares regression. _arXiv_, abs/1712.02747, 2017.
* Bregman and Optimal Transport divergences. 2021.
* [CSDG22] Badr-Eddine Cherief-Abdellatif, Yuyang Shi, Arnaud Doucet, and Benjamin Guedj. On PAC-Bayesian reconstruction guarantees for VAEs. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_, 2022.
* [Csi75] Imre Csiszar. \(I\)-Divergence Geometry of Probability Distributions and Minimization Problems. _The Annals of Probability_, 3(1), 1975.
* [CWR23] Ben Chugg, Hongjian Wang, and Aaditya Ramdas. A unified recipe for deriving (time-uniform) PAC-Bayes bounds. _arXiv_, abs/2302.03421, 2023.
* [DCL\({}^{+}\)21] Nan Ding, Xi Chen, Tomer Levinboim, Sebastian Goodman, and Radu Soricut. Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2021.
* [DG17] Dheeru Dua and Casey Graff. UCI Machine Learning Repository, 2017.
* [DR17] Gintare Karolina Dziugaite and Daniel Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. In _Conference on Uncertainty in Artificial Intelligence (UAI)_, 2017.
* [DV76] Monroe David Donsker and Srinivasa Varadhan. Asymptotic evaluation of certain Markov process expectations for large time--III. _Communications on Pure and Applied Mathematics_, 29(4), 1976.

* [FM21] Alec Farid and Anirudha Majumdar. Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2021.
* [FP10] Mahdi Milani Fard and Joelle Pineau. PAC-Bayesian Model Selection for Reinforcement Learning. In _Advances in Neural Information Processing Systems (NIPS)_, 2010.
* [FRKP22] Hamish Flynn, David Reeb, Melih Kandemir, and Jan Peters. PAC-Bayesian lifelong learning for multi-armed bandits. _Data Mining and Knowledge Discovery_, 2022.
* [GBTS21] Borja Rodriguez Galvez, German Bassi, Ragnar Thobaben, and Mikael Skoglund. Tighter Expected Generalization Error Bounds via Wasserstein Distance. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2021.
* European Conference (ECML-PKDD)_, 2017.
* [Gue19] Benjamin Guedj. A Primer on PAC-Bayesian Learning. _arXiv_, abs/1901.05353, 2019.
* [HG22] Maxime Haddouche and Benjamin Guedj. Online PAC-Bayes Learning. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2022.
* [HG23a] Maxime Haddouche and Benjamin Guedj. PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingales. _Transactions on Machine Learning Research_, 2023.
* [HG23b] Maxime Haddouche and Benjamin Guedj. Wasserstein PAC-Bayes Learning: A Bridge Between Generalisation and Optimisation. _arXiv_, abs/2304.07048, 2023.
* [HGRS21] Maxime Haddouche, Benjamin Guedj, Omar Rivasplata, and John Shawe-Taylor. PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses. _Entropy_, 23(10), 2021.
* [JJKO23] Kyoungseok Jang, Kwang-Sung Jun, Ilja Kuzborskij, and Francesco Orabona. Tighter PAC-Bayes Bounds Through Coin-Betting. In _Conference on Learning Theory (COLT)_, 2023.
* [Kan60] Leonid Vitalievitch Kantorovitch. Mathematical Methods of Organizing and Planning Production. _Management Science_, 1960.
* [KDY\({}^{+}\)22] Hoel Kervadec, Jose Dolz, Jing Yuan, Christian Desrosiers, Eric Granger, and Ismail Ben Ayed. Constrained deep networks: Lagrangian optimization via log-barrier extensions. In _European Signal Processing Conference (EUSIPCO)_, 2022.
* [KP00] Vladimir Koltchinskii and Dmitriy Panchenko. Rademacher processes and bounding the risk of function learning. In _High dimensional probability II_, 2000.
* [LeC98] Yann LeCun. The MNIST database of handwritten digits, 1998.
* [LFK\({}^{+}\)22] Sanae Lotfi, Marc Finzi, Sanyam Kapoor, Andres Potapczynski, Micah Goldblum, and Andrew Wilson. PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2022.
* [LGGL19] Gael Letarte, Pascal Germain, Benjamin Guedj, and Francois Laviolette. Dichotomize and Generalize: PAC-Bayesian Binary Activated Deep Neural Networks. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2019.
* [LN22] Gabor Lugosi and Gergely Neu. Generalization Bounds via Convex Analysis. In _Conference on Learning Theory (COLT)_, 2022.
* [LWHZ19] Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Optimal Algorithms for Lipschitz Bandits with Heavy-tailed Rewards. In _International Conference on Machine Learning (ICML)_, 2019.
* [LZ11] Keqin Liu and Qing Zhao. Multi-Armed Bandit Problems with Heavy Tail Reward Distributions. _Allerton Conference on Communication, Control, and Computing_, 2011.

* [Mau04] Andreas Maurer. A note on the PAC-Bayesian theorem. _arXiv_, cs/0411099, 2004.
* [MGG19] Zakaria Mhammedi, Peter Grunwald, and Benjamin Guedj. PAC-Bayes Un-Expected Bernstein Inequality. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2019.
* [MGW20] Zakaria Mhammedi, Benjamin Guedj, and Robert Williamson. PAC-Bayesian Bound for the Conditional Value at Risk. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2020.
* [Mon81] Gaspard Monge. Memoire sur la theorie des deblais et des remblais. _Histoire de l'Academie Royale des Sciences de Paris_, 1781.
* [Mu97] Alfred Muller. Integral Probability Metrics and Their Generating Classes of Functions. _Advances in Applied Probability_, 29(2), 1997.
* [NGG20] Kento Nozawa, Pascal Germain, and Benjamin Guedj. PAC-Bayesian Contrastive Unsupervised Representation Learning. In _Conference on Uncertainty in Artificial Intelligence (UAI)_, 2020.
* [OH21] Yuki Ohnishi and Jean Honorio. Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_, 2021.
* [OT17] Francesco Orabona and Tatiana Tommasi. Training Deep Networks without Learning Rates Through Coin Betting. In _Advances in Neural Information Processing Systems (NIPS)_, 2017.
* [PC19] Gabriel Peyre and Marco Cuturi. Computational Optimal Transport. _Foundations and Trends in Machine Learning_, 11(5-6), 2019.
* [PORPH\({}^{+}\)21] Maria Perez-Ortiz, Omar Rivasplata, Emilio Parrado-Hernandez, Benjamin Guedj, and John Shawe-Taylor. Progress in Self-Certified Neural Networks. In _NeurIPS 2021 Workshop on Bayesian Deep Learning_, 2021.
* [PRSS21] Maria Perez-Ortiz, Omar Rivasplata, John Shawe-Taylor, and Csaba Szepesvari. Tighter risk certificates for neural networks. _Journal of Machine Learning Research_, 22, 2021.
* [PWG22] Antoine Picard-Weibel and Benjamin Guedj. On change of measure inequalities for \(f\)-divergences. _arXiv_, abs/2202.05568, 2022.
* [RACA23] Charles Riou, Pierre Alquier, and Badr-Eddine Cherief-Abdellatif. Bayes meets Bernstein at the Meta Level: an Analysis of Fast Rates in Meta-Learning with PAC-Bayes. _arXiv_, abs/2302.11709, 2023.
* [RFJK21] Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: Bayes-optimal meta-learning with PAC-guarantees. In _International Conference on Machine Learning (ICML)_, 2021.
* [RJFK22] Jonas Rothfuss, Martin Josifoski, Vincent Fortuin, and Andreas Krause. PAC-Bayesian Meta-Learning: From Theory to Practice. _arXiv_, abs/2211.07206, 2022.
* [RZ20] Daniel Russo and James Zou. How Much Does Your Data Exploration Overfit? Controlling Bias via Information Usage. _IEEE Transactions on Information Theory_, 66(1), 2020.
* [SAC23] Otmane Sakhi, Pierre Alquier, and Nicolas Chopin. PAC-Bayesian Offline Contextual Bandits With Guarantees. In _International Conference on Machine Learning (ICML)_, 2023.
* [SLCB\({}^{+}\)12] Yevgeny Seldin, Francois Laviolette, Nicolo Cesa-Bianchi, John Shawe-Taylor, and Peter Auer. PAC-Bayesian Inequalities for Martingales. _IEEE Transactions on Information Theory_, 58(12), 2012.
* [Sli19] Aleksandrs Slivkins. Introduction to Multi-Armed Bandits. _Foundations and Trends in Machine Learning_, 2019.
* [SLST\({}^{+}\)11] Yevgeny Seldin, Francois Laviolette, John Shawe-Taylor, Jan Peters, and Peter Auer. PAC-Bayesian Analysis of Martingales and Multiarmed Bandits. _arXiv_, abs/1105.2416, 2011.

* [VC68] Vladimir Vapnik and Alexey Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. In _Doklady Akademii Nauk USSR_, 1968.
* [VC74] Vladimir Vapnik and Alexey Chervonenkis. Theory of pattern recognition, 1974.
* [Vil09] Cedric Villani. _Optimal transport: old and new_. Number 338 in Grundlehren der mathematischen Wissenschaften. Springer, 2009.
* [WDFC19] Hao Wang, Mario Diaz, Jose Candido Silveira Santos Filho, and Flavio P. Calmon. An Information-Theoretic View of Generalization via Wasserstein Distance. In _IEEE International Symposium on Information Theory (ISIT)_, 2019.
* [XR17] Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2017.
* [XRV17] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms, 2017.
* [Zin03] Martin Zinkevich. Online Convex Programming and Generalized Infinitesimal Gradient Ascent. In _International Conference on Machine Learning (ICML)_, 2003.
* [ZLT18] Jingwei Zhang, Tongliang Liu, and Dacheng Tao. An Optimal Transport View on Generalization. _arXiv_, abs/1811.03270, 2018.
* [ZS21] Vincent Zhuang and Yanan Sui. No-Regret Reinforcement Learning with Heavy-Tailed Rewards. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_, 2021.
* [ZVM\({}^{+}\)21] Valentina Zantedeschi, Paul Viallard, Emilie Morvant, Remi Emonet, Amaury Habrard, Pascal Germain, and Benjamin Guedj. Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Bound. In _Advances in Neural Information Processing Systems (NeurIPS)_, 2021.

The supplementary material is organized as follows:

1. We provide more discussion about Theorems 1 and 2 in Appendix A;
2. The proofs of Theorems 1 to 4 are presented in Appendix B;
3. We present in Appendix C additional information about the experiments.

## Appendix A Additional insights on Section 3.1

In Appendix A.1, we provide additional discussion about Theorem 1 while Appendix A.2 discuss about the convergence rates for Theorem 2.

### Supplementary discussion about Theorem 1

[16, Corollary 10] proposed PAC-Bayes bounds with Wasserstein distances on a Euclidean predictor space with Gaussian prior and posteriors. The bounds have an explicit convergence rate of \(\mathcal{O}(\sqrt{\nicefrac{{dW_{1}}}{{m}}(\rho,\pi)/m})\) where the predictor space is Euclidean with dimension \(d\). While our bound does not propose such an explicit convergence rate, it allows us to derive learning algorithms as described in Section 4. A broader discussion about the role of \(K\) is detailed in Theorem 2. Furthermore, our bound holds for any Polish predictor space and does not require Gaussian distributions. Furthermore, our result exploits data-dependent priors and deals with the dimension only through the Wasserstein distance, which can attenuate the impact of the dimension.

### Convergence rates for Theorem 2

In this section, we discuss more deeply the values of \(K\) in Theorem 2. This implies a tradeoff between the number of sets \(K\) and the cardinal of each \(\mathcal{S}_{i}\). The tightness of the bound depends highly on the sets \(\mathcal{S}_{1},\ldots,\mathcal{S}_{K}\).

**Full batch setting K=1.** When \(\mathcal{S}_{1}=\mathcal{S}\) with \(K=1\), the bound of Theorem 2 becomes, with probability \(1-\delta\), for any \(\rho\in\mathcal{M}(\mathcal{H})\)

\[\operatorname*{\mathbb{E}}_{h\sim\rho}\left[\operatorname{\mathbf{R}}_{\mu}(h )-\hat{\operatorname{\mathbf{R}}}_{\mathcal{S}}(h)\right]\leq 2L\mathrm{W}(\rho, \pi)+2\sqrt{\frac{\ln\frac{1}{\delta}}{m}}\;,\]

where \(\pi=\pi_{1}\) is data-free. This bound can be seen as the high-probability (PAC-Bayesian) version of the expected bound of [17]. Furthermore, in this setting, we are able, through our proof technique, to recover an explicit convergence rate similar to the one of [1, Theorem 12]. It is stated below.

**Corollary 5**.: _For any distribution \(\mu\) on \(\mathcal{Z}\), for any finite hypothesis space \(\mathcal{H}\) equipped with a distance \(d\), for any \(L\)-Lipschitz loss function \(\ell:\mathcal{H}\times\mathcal{Z}\to[0,1]\), for any \(\delta\in(0,1]\), we have, with probability \(1-\delta\) over the sample \(\mathcal{S}\), for any \(\rho\in\mathcal{M}(\mathcal{H})\):_

\[\operatorname*{\mathbb{E}}_{h\sim\rho}\left[\operatorname{\mathbf{R}}_{\mu}(h) -\hat{\operatorname{\mathbf{R}}}_{\mathcal{S}}(h)\right]\leq L\sqrt{\frac{2\ln \left(\nicefrac{{4|\mathcal{H}|^{2}}}{{\delta}}\right)}{m}}\mathrm{W}(\rho,\pi) +2\sqrt{\frac{\ln\left(\nicefrac{{2}}{{\delta}}\right)}{m}}\]

_where \(\pi\) is a data-free prior._

Proof.: We exploit [1, Equation 35] to state that with probability at least \(1-\nicefrac{{\delta}}{{2}}\), for any \((h,h^{\prime})\in\mathcal{H}^{2}\):

\[\left|\frac{1}{m}\sum_{i=1}^{m}\left[\ell\left(h^{\prime},\mathbf{z}_{i} \right)-\ell\left(h,\mathbf{z}_{i}\right)\right]-\operatorname*{\mathbb{E}}_{ \mathbf{z}\sim\mu}\left[\ell\left(h^{\prime},\mathbf{z}\right)-\ell(h, \mathbf{z})\right]\right|\leq L\sqrt{\frac{2\ln\left(\frac{4|\mathcal{H}|^{2} }{\delta}\right)}{m}}d\left(h,h^{\prime}\right).\]

So, with high probability, we can exploit the Kantorovich-Rubinstein duality with this new Lipschitz constant: with probability at least \(1-\delta/2\):

\[\operatorname*{\mathbb{E}}_{h\sim\rho}\left[\operatorname{\mathbf{R}}_{\mu}(h )-\hat{\operatorname{\mathbf{R}}}_{\mathcal{S}}(h)\right]\leq L\sqrt{\frac{2 \ln\left(\frac{4|\mathcal{H}|^{2}}{\delta}\right)}{m}}\mathrm{W}(\rho,\pi)+ \operatorname*{\mathbb{E}}_{h\sim\pi}\frac{1}{m}\left[\sum_{i=1}^{m} \operatorname{\mathbf{R}}_{\mu}(h)-\ell(h,\mathbf{z}_{i})\right],\]To conclude, we control the quantity on the right-hand side the same way as in Theorem 1 and Theorem 2. We then have, with probability at least \(1-\delta/2\), for a loss function in \([0,1]\):

\[\frac{1}{m}\sum_{i=1}^{m}\mathsf{R}_{\mu}(h)-\ell(h,\mathbf{z}_{i})\leq 2\sqrt{ \frac{\ln\frac{K}{\delta}}{m}}.\]

Taking the union bound concludes the proof.

**Mini-batch setting \(K=\sqrt{m}\).** When a tradeoff is desired between the quantity of data we want to infuse in our priors and an explicit convergence rate, a meaningful candidate is when \(K=\sqrt{m}\). Theorem 2's bound becomes, in this particular case:

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\mathsf{R}_{\mu}(h)-\hat{\mathsf{R}}_{ \mathcal{S}}(h)\right]\leq\frac{2L}{\sqrt{m}}\sum_{i=1}^{\sqrt{m}}\mathrm{W}( \rho,\pi_{i})+2\sqrt{\frac{\ln\frac{\sqrt{m}}{\delta}}{\sqrt{m}}}. \tag{8}\]

**Towards online learning:**\(K=m\). When \(K=m\), the sets \(\mathcal{S}_{i}\) contain only one example. More precisely, we have for all \(i\in\{1,\ldots,m\}\) the set \(\mathcal{S}_{i}=\{\mathbf{z}_{i}\}\). In this case, the bound becomes:

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\mathsf{R}_{\mu}(h)-\hat{\mathsf{R}}_{ \mathcal{S}}(h)\right]\leq\frac{2L}{m}\sum_{i=1}^{m}\mathrm{W}(\rho,\pi_{i})+ 2\sqrt{\ln\frac{m}{\delta}}.\]

This bound is vacuous since the last term is incompressible, hence the need for a new technique detailed in Section 3.2 to deal with it.

## Appendix B Proofs

The proof of Theorem 1 is presented in Appendix B.1. Appendices B.2 and B.3 introduce two proofs of Theorem 2. Theorem 3's proof is presented in Appendix B.4. Appendix B.5 provides the proof of Theorem 3.

### Proof of Theorem 1

**Theorem 1**.: _We assume the loss \(\ell\) to be \(L\)-Lipschitz. Then, for any \(\delta\in(0,1]\), for any sequence of positive scalar \((\lambda_{i})_{i\in\{1,\ldots,K\}}\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), the following holds for the distributions \(\pi_{i,\mathcal{S}}:=\pi_{i}(\mathcal{S},.)\) and for any \(\rho\in\mathcal{M}(\mathcal{H})\):_

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[R_{\mu}(h)-\hat{\mathsf{R}}_{ \mathcal{S}}(h)\right]\\ \leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho, \pi_{i,\mathcal{S}})+\frac{1}{m}\sum_{i=1}^{K}\frac{\ln\left(\frac{K}{\delta} \right)}{\lambda_{i}}+\frac{\lambda_{i}}{2}\left(\mathop{\mathbb{E}}_{h\sim \pi_{i,\mathcal{S}}}\left[\hat{V}_{|\mathcal{S}_{i}|}(h)+V_{|\mathcal{S}_{i}| }(h)\right]\right),\]

_where \(\pi_{i,\mathcal{S}}\) does not depend on \(\mathcal{S}_{i}\). Also, for any \(i,|S_{i}|\), we have \(\hat{V}_{|\mathcal{S}_{i}|}(h)=\sum_{\mathbf{z}\in\mathcal{S}_{i}}\left(\ell( h,\mathbf{z})-R_{\mu}(h)\right)^{2}\) and \(V_{|\mathcal{S}_{i}|}(h)=\mathop{\mathbb{E}}_{\mathcal{S}_{i}}\left[\hat{V}_{| \mathcal{S}_{i}|}(h)\right]\)._

Proof.: For the sake of readability, we identify, for any \(i\), \(\pi_{i}\) and \(\pi_{i,\mathcal{S}}\).

**Step 1: Exploit the Kantorovich duality [20, Remark 6.5].** First of all, note that for a \(L\)-Lipschitz loss function \(\ell:\mathcal{H}\times\mathcal{Z}\rightarrow[0,1]\), we have

\[\left|\left(|\mathcal{S}_{i}|\mathsf{R}_{\mu}(h_{1})-\sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left(|\mathcal{S}_{i}|\mathsf{ R}_{\mu}(h_{2})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{2},\mathbf{z}) \right)\right|\leq 2|\mathcal{S}_{i}|Ld(h_{1},h_{2}). \tag{9}\]Indeed, we can deduce Equation (9) from Jensen inequality, the triangle inequality, and by definition that we have

\[\left|\left(|\mathcal{S}_{i}|\text{R}_{\mu}(h_{1})-\sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left(|\mathcal{S}_{i}|\text{R}_{ \mu}(h_{2})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{2},\mathbf{z})\right)\right|\] \[=\left|\left(\sum_{\mathbf{z}\in\mathcal{S}_{i}}\text{R}_{\mu}(h_{1})- \sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left(\sum_{\mathbf{z }\in\mathcal{S}_{i}}\text{R}_{\mu}(h_{2})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell( h_{2},\mathbf{z})\right)\right|\] \[\leq\sum_{\mathbf{z}\in\mathcal{S}_{i}}\mathop{\mathbb{E}}_{\mathbf{z}^{ \prime}\sim\mu}\left[|\ell(h_{1},\mathbf{z}^{\prime})-\ell(h_{2},\mathbf{z}^{ \prime})|+|\ell(h_{2},\mathbf{z})-\ell(h_{1},\mathbf{z})|\right]\] \[\leq\mathop{\mathbb{E}}_{\mathbf{z}^{\prime}\sim\mu}\sum_{\mathbf{z}\in \mathcal{S}_{i}}2Ld(h_{1},h_{2})\] \[=2|\mathcal{S}_{i}|Ld(h_{1},h_{2}).\]

We are now able to upper-bound \(\mathop{\mathbb{E}}_{h\sim\rho}[\text{R}_{\mu}(h)-\hat{\text{R}}_{\mathcal{S}} (h)]\). Indeed, we have

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\text{R}_{\mu}(h)-\hat{\text {R}}_{\mathcal{S}}(h)\right] =\frac{1}{m}\sum_{i=1}^{K}\mathop{\mathbb{E}}_{h\sim\rho}\left[| \mathcal{S}_{i}|\text{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h, \mathbf{z})\right]\] \[\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\text{W}(\rho,\pi_ {i})+\sum_{i=1}^{K}\mathop{\mathbb{E}}_{h\sim\pi_{i}}\frac{1}{m}\left[| \mathcal{S}_{i}|\text{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h, \mathbf{z})\right], \tag{10}\]

where the inequality comes from the Kantorovich-Rubinstein duality theorem.

Step 2: Define an adapted supermartingale.For any \(1\leq i\leq K\), we fix \(\lambda_{i}>0\) and we provide an arbitrary order to the elements of \(\mathcal{S}_{i}:=\{\mathbf{z}_{i,1},\cdots,\mathbf{z}_{i,|\mathcal{S}_{i}|}\}\). Then we define for any \(h\):

\[M_{|\mathcal{S}_{i}|}(h):=|\mathcal{S}_{i}|\text{R}_{\mu}(h)-\sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell(h,\mathbf{z})=\sum_{j=1}^{|\mathcal{S}_{i}|}R_{\mu}(h)- \ell(h,\mathbf{z}_{i,j}).\]

Remark that, because our data are _i.i.d._, \((M_{|\mathcal{S}_{i}|})_{|\mathcal{S}_{i}|\geq 1}\) is a martingale. We then exploit the technique [1] to define a supermartingale. More precisely, we exploit a result from [1] cited in Lemma 1.3 of [1] coupled with Lemma 2.2 of [1] to ensure that the process

\[SM_{|\mathcal{S}_{i}|}:=\mathop{\mathbb{E}}_{h\sim\pi_{i}}\left[\exp\left( \lambda_{i}M_{|\mathcal{S}_{i}|}(h)-\frac{\lambda_{i}^{2}}{2}\left(\hat{V}_{| \mathcal{S}_{i}|}(h)+V_{|\mathcal{S}_{i}|}(h)\right)\right)\right],\]

is a supermartingale, where \(\hat{V}_{|\mathcal{S}_{i}|}(h)=\sum_{j=1}^{|\mathcal{S}_{i}|}\left(\ell(h, \mathbf{z}_{i,j})-R_{\mu}(h)\right)^{2}\) and \(V_{|\mathcal{S}_{i}|}(h)=\mathop{\mathbb{E}}_{\mathcal{S}_{i}}\left[\hat{V}_{| \mathcal{S}_{i}|}(h)\right]\).

Step 3. Combine steps 1 and 2.We restart from Equation (10) to exploit again the Kantorovich-Rubinstein duality.

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\text{R}_{\mu}(h)-\hat{\text {R}}_{\mathcal{S}}(h)\right] =\frac{1}{m}\sum_{i=1}^{K}\mathop{\mathbb{E}}_{h\sim\rho}\left[| \mathcal{S}_{i}|\text{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h, \mathbf{z})\right]\] \[\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\text{W}(\rho,\pi_ {i})+\sum_{i=1}^{K}\frac{1}{m\lambda_{i}}\lambda_{i}\mathop{\mathbb{E}}_{h\sim \pi_{i}}\left[|\mathcal{S}_{i}|\text{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i }}\ell(h,\mathbf{z})\right],\] \[=\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\text{W}(\rho,\pi_{i}) +\sum_{i=1}^{K}\frac{1}{m\lambda_{i}}\mathop{\mathbb{E}}_{h\sim\pi_{i}}\left[ \lambda_{i}M_{|\mathcal{S}_{i}|}|\right],\] \[\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\text{W}(\rho,\pi_ {i})+\sum_{i=1}^{K}\frac{1}{m\lambda_{i}}\ln\left(SM_{|\mathcal{S}_{i}|}\right)\] \[+\frac{1}{m}\sum_{i=1}^{K}\mathop{\mathbb{E}}_{h\sim\pi_{i}} \left[\frac{\lambda_{i}}{2}\left(\hat{V}_{|\mathcal{S}_{i}|}(h)+V_{|\mathcal{ S}_{i}|}(h)\right)\right].\]The last line holds thanks to Jensen's inequality. We now apply Ville's inequality (see _e.g._, Section 1.2 of [13]). We have for any \(i\):

\[\mathbb{P}_{\mathcal{S}_{i}\sim\mu^{|\mathcal{S}_{i}|}}\left(\forall|S_{i}|\geq 1,SM_{|S_{i}|}\leq\frac{1}{\delta}\right)\geq 1-\delta.\]

Applying an union bound and authorising \(\lambda_{i}\) to be a function of \(|S_{i}|\) (thus the inequality does not hold for all \(|\mathcal{S}_{i}|\) simultaneously) finally gives with probability at least \(1-\delta\), for all \(\rho\in\mathcal{M}(\mathcal{H})\) :

\[\mathbb{E}_{h\sim\rho}\left[\mathbf{R}_{\mu}(h)-\hat{\mathbf{R}}_{\mathcal{S}}( h)\right]\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho,\pi_{i})+ \sum_{i=1}^{K}\frac{\ln\left(\frac{K}{\delta}\right)}{\lambda_{i}m}+\frac{ \lambda_{i}}{2m}\mathbb{E}_{h\sim\pi_{i}}\left[\hat{V}_{|\mathcal{S}_{i}|}(h)+ V_{|\mathcal{S}_{i}|}(h)\right].\]

### Proof of Theorem 2

**Theorem 2**.: _We assume our loss \(\ell\) to be non-negative and \(L\)-Lipschitz. We also assume that, for any \(1\leq i\leq K\), for any dataset\(\mathcal{S}\), we have \(\mathbb{E}_{h\sim\pi_{i},(\cdot,\mathcal{S}),z\sim\mu}\left[\ell(h,z)^{2} \right]\leq 1\) (bounded order 2 moments for priors). Then, for any \(\delta\in(0,1]\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), the following holds for the distributions \(\pi_{i,\mathcal{S}}:=\pi_{i}(\mathcal{S},.)\) and for any \(\rho\in\mathcal{M}(\mathcal{H})\):_

\[\mathbb{E}_{h\sim\rho}\left[\mathbf{R}_{\mu}(h)-\hat{\mathbf{R}}_{\mathcal{S}} (h)\right]\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho,\pi_{ i,\mathcal{S}})+\sum_{i=1}^{K}\sqrt{\frac{2|\mathcal{S}_{i}|\ln\frac{K}{\delta} }{m^{2}}},\]

_where \(\pi_{i,\mathcal{S}}\) does not depend on \(\mathcal{S}_{i}\)._

Proof.: For the sake of readability, we identify, for any \(i\), \(\pi_{i}\) and \(\pi_{i,\mathcal{S}}\).

Step 1: Exploit the Kantorovich duality [14, Remark 6.5].First of all, note that for a \(L\)-Lipschitz loss function \(\ell:\mathcal{H}\times\mathcal{Z}\to[0,1]\), we have

\[\left|\left(|\mathcal{S}_{i}|\mathbf{R}_{\mu}(h_{1})-\sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left(|\mathcal{S}_{i}|\mathbf{ R}_{\mu}(h_{2})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{2},\mathbf{z}) \right)\right|\leq 2|\mathcal{S}_{i}|Ld(h_{1},h_{2}). \tag{11}\]

Indeed, we can deduce Equation (11) from Jensen inequality, the triangle inequality, and by definition that we have

\[\left|\left(|\mathcal{S}_{i}|\mathbf{R}_{\mu}(h_{1})-\sum_{\mathbf{ z}\in\mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left(|\mathcal{S}_{i}|\mathbf{R} _{\mu}(h_{2})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{2},\mathbf{z})\right)\right|\] \[=\left|\left(\sum_{\mathbf{z}\in\mathcal{S}_{i}}\mathbf{R}_{\mu}( h_{1})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left( \sum_{\mathbf{z}\in\mathcal{S}_{i}}\mathbf{R}_{\mu}(h_{2})-\sum_{\mathbf{z} \in\mathcal{S}_{i}}\ell(h_{2},\mathbf{z})\right)\right|\] \[\leq\sum_{\mathbf{z}\in\mathcal{S}_{i}}\mathbb{E}_{\mathbf{z}^{ \prime}\sim\mu}\left[\ell(h_{1},\mathbf{z}^{\prime})-\ell(h_{2},\mathbf{z}^{ \prime})|+|\ell(h_{2},\mathbf{z})-\ell(h_{1},\mathbf{z})|\right]\] \[\leq\mathbb{E}_{\mathbf{z}^{\prime}\sim\mu}\sum_{\mathbf{z}\in \mathcal{S}_{i}}2Ld(h_{1},h_{2})\] \[=2|\mathcal{S}_{i}|Ld(h_{1},h_{2}).\]

We are now able to upper-bound \(\mathbb{E}_{h\sim\rho}[\mathbf{R}_{\mu}(h)-\hat{\mathbf{R}}_{\mathcal{S}}(h)]\). Indeed, we have

\[\mathbb{E}_{h\sim\rho}\left[\mathbf{R}_{\mu}(h)-\hat{\mathbf{R }}_{\mathcal{S}}(h)\right] =\frac{1}{m}\sum_{i=1}^{K}\mathbb{E}_{h\sim\rho}\left[|\mathcal{ S}_{i}|\mathbf{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h, \mathbf{z})\right]\] \[\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho, \pi_{i})+\sum_{i=1}^{K}\mathbb{E}_{h\sim\pi_{i}}\frac{1}{m}\left[|\mathcal{S}_ {i}|\mathbf{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h,\mathbf{z}) \right], \tag{12}\]

where the inequality comes from the Kantorovich-Rubinstein duality theorem.

Step 2: Define an adapted supermartingale. For any \(1\leq i\leq K\), we fix \(\lambda_{i}>0\) and we provide an arbitrary order to the elements of \(\mathcal{S}_{i}:=\{\mathbf{z}_{i,1},\cdots,\mathbf{z}_{i,|\mathcal{S}_{i}|}\}\). Then we define for any \(h\):

\[M_{|\mathcal{S}_{i}|}(h):=|\mathcal{S}_{i}|\mathbf{R}_{\mu}(h)-\sum_{\mathbf{z }\in\mathcal{S}_{i}}\ell(h,\mathbf{z})=\sum_{j=1}^{|\mathcal{S}_{i}|}R_{\mu}(h )-\ell(h,\mathbf{z}_{i,j}).\]

Remark that, because our data are _i.i.d._, \((M_{|\mathcal{S}_{i}|})_{|\mathcal{S}_{i}|\geq 1}\) is a martingale. We then exploit the technique [12] to define a supermartingale. More precisely, we exploit [12, Lemma A.2 and Lemma B.1] to ensure that the process

\[SM_{|\mathcal{S}_{i}|}:=\mathop{\mathbb{E}}_{h\sim\pi_{i}}\left[\exp\left( \lambda_{i}M_{|\mathcal{S}_{i}|}(h)-\frac{\lambda_{i}^{2}}{2}L_{|\mathcal{S}_ {i}|}(h)\right)\right],\]

is a supermartingale, where, because our data are _i.i.d._, \(L_{|\mathcal{S}_{i}|}(h)=\mathop{\mathbb{E}}_{\mathcal{S}}\left[\sum_{j=1}^{| \mathcal{S}_{i}|}\ell(h,\mathbf{z}_{i,j})^{2}\right]=|\mathcal{S}_{i}|\mathop{ \mathbb{E}}_{z\sim\mu}[\ell(h,z)^{2}]\).

Step 3. Combine steps 1 and 2.We restart from Equation (12) to exploit the Kantorovich-Rubinstein duality again.

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\mathbf{R}_{\mu}(h)-\hat{ \mathbf{R}}_{\mathcal{S}}(h)\right] =\frac{1}{m}\sum_{i=1}^{K}\mathop{\mathbb{E}}_{h\sim\rho}\left[| \mathcal{S}_{i}|\mathbf{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h, \mathbf{z})\right]\] \[\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho, \pi_{i})+\sum_{i=1}^{K}\frac{1}{m\lambda_{i}}\lambda_{i}\mathop{\mathbb{E}}_{h \sim\pi_{i}}\left[|\mathcal{S}_{i}|\mathbf{R}_{\mu}(h)-\sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell(h,\mathbf{z})\right],\] \[=\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho,\pi_{ i})+\sum_{i=1}^{K}\frac{1}{m\lambda_{i}}\mathop{\mathbb{E}}_{h\sim\pi_{i}} \left[\lambda_{i}M_{|\mathcal{S}_{i}|}\right],\] \[\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho, \pi_{i})+\sum_{i=1}^{K}\frac{1}{m\lambda_{i}}\ln\left(SM_{|\mathcal{S}_{i}|}\right)\] \[+\frac{1}{m}\sum_{i=1}^{K}\mathop{\mathbb{E}}_{h\sim\pi_{i}}\left[ \frac{\lambda_{i}}{2}L_{|\mathcal{S}_{i}|}(h)\right].\]

The last line holds thanks to Jensen's inequality. We now apply Ville's inequality (see _e.g._, section 1.2 of [10]). We have for any \(i\):

\[\mathop{\mathbb{P}}_{\mathcal{S}_{i}\sim\mu|\mathcal{S}_{i}|}\left(\forall|S_ {i}|\geq 1,SM_{|\mathcal{S}_{i}|}\leq\frac{1}{\delta}\right)\geq 1-\delta.\]

Applying an union bound and authorising \(\lambda_{i}\) to be a function of \(|S_{i}|\) (thus the inequality does not hold for all \(|\mathcal{S}_{i}|\) simultaneously) finally gives with probability at least \(1-\delta\), for all \(\rho\in\mathcal{M}(\mathcal{H})\) :

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\mathbf{R}_{\mu}(h)-\hat{\mathbf{R}}_{ \mathcal{S}}(h)\right]\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{ W}(\rho,\pi_{i})+\sum_{i=1}^{K}\frac{\ln\left(\frac{K}{\delta}\right)}{ \lambda_{i}m}+\frac{\lambda_{i}}{2m}\mathop{\mathbb{E}}_{h\sim\pi_{i}}\left[L |\mathcal{S}_{i}|(h)\right].\]

Finally, using the assumption \(\mathop{\mathbb{E}}_{h\sim\pi_{i}}\mathop{\mathbb{E}}_{z\sim\mu}[\ell(h,z)^{2 }]\leq 1\) gives, with probability at least \(1-\delta\), for all \(\rho\in\mathcal{M}(\mathcal{H})\):

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\mathbf{R}_{\mu}(h)-\hat{\mathbf{R}}_{ \mathcal{S}}(h)\right]\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{ W}(\rho,\pi_{i})+\sum_{i=1}^{K}\frac{\ln\left(\frac{K}{\delta}\right)}{ \lambda_{i}m}+\frac{\lambda_{i}[\mathcal{S}_{i}|]}{2m}.\]

Taking for each \(i\), \(\lambda_{i}=\sqrt{\frac{2\ln(K/\delta)}{|\mathcal{S}_{i}|}}\) concludes the proof.

### Alternative proof of Theorem 2

We state here a slightly tighter version of Theorem 2 for bounded losses, which relies on an application of McDiarmid's inequality instead of supermartingale techniques. This is useful for the numerical evaluations of our bound.

**Theorem 6**.: _We assume our loss \(\ell\) to be in \([0,1]\) and \(L\)-Lipschitz. Then, for any \(\delta\in(0,1]\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), the following holds for the distributions \(\pi_{i,\mathcal{S}}:=\pi_{i}(\mathcal{S},.)\) and for any \(\rho\in\mathcal{M}(\mathcal{H})\):_

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\mathsf{R}_{\mu}(h)-\hat{\mathsf{R}}_{ \mathcal{S}}(h)\right]\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W} (\rho,\pi_{i,\mathcal{S}})+\sum_{i=1}^{K}\sqrt{\frac{|\mathcal{S}_{i}|\ln\frac{ K}{\delta}}{2m^{2}}}\]

_where \(\pi_{i}\) does not depend on \(\mathcal{S}_{i}\)._

Proof.: For the sake of readability, we identify, for any \(i\), \(\pi_{i}\) and \(\pi_{i,\mathcal{S}}\).

First of all, note that for a \(L\)-Lipschitz loss function \(\ell:\mathcal{H}\times\mathcal{Z}\to[0,1]\), we have

\[\left|\left(|\mathcal{S}_{i}|\mathsf{R}_{\mu}(h_{1})-\sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left(|\mathcal{S}_{i}|\mathsf{ R}_{\mu}(h_{2})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{2},\mathbf{z})\right) \right|\leq 2|\mathcal{S}_{i}|Ld(h_{1},h_{2}). \tag{13}\]

Indeed, we can deduce Equation (13) from Jensen's inequality, the triangle inequality, and by definition that we have

\[\left|\left(|\mathcal{S}_{i}|\mathsf{R}_{\mu}(h_{1})-\sum_{\mathbf{ z}\in\mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left(|\mathcal{S}_{i}| \mathsf{R}_{\mu}(h_{2})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{2},\mathbf{ z})\right)\right|\] \[=\left|\left(\sum_{\mathbf{z}\in\mathcal{S}_{i}}\mathsf{R}_{\mu}(h _{1})-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h_{1},\mathbf{z})\right)-\left( \sum_{\mathbf{z}\in\mathcal{S}_{i}}\mathsf{R}_{\mu}(h_{2})-\sum_{\mathbf{z}\in \mathcal{S}_{i}}\ell(h_{2},\mathbf{z})\right)\right|\] \[\leq\sum_{\mathbf{z}\in\mathcal{S}_{i}}\mathop{\mathbb{E}}_{ \mathbf{z}^{\prime}\sim\mu}\left[|\ell(h_{1},\mathbf{z}^{\prime})-\ell(h_{2}, \mathbf{z}^{\prime})|+|\ell(h_{2},\mathbf{z})-\ell(h_{1},\mathbf{z})|\right]\] \[\leq\mathop{\mathbb{E}}_{\mathbf{z}^{\prime}\sim\mu}\sum_{ \mathbf{z}\in\mathcal{S}_{i}}2Ld(h_{1},h_{2})\] \[=2|\mathcal{S}_{i}|Ld(h_{1},h_{2}).\]

We are now able to upper-bound \(\mathop{\mathbb{E}}_{h\sim\rho}[\mathsf{R}_{\mu}(h)-\hat{\mathsf{R}}_{ \mathcal{S}}(h)]\). Indeed, we have

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\mathsf{R}_{\mu}(h)-\hat{ \mathsf{R}}_{\mathcal{S}}(h)\right] =\frac{1}{m}\sum_{i=1}^{K}\mathop{\mathbb{E}}_{h\sim\rho}\left[ |\mathcal{S}_{i}|\mathsf{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell( h,\mathbf{z})\right]\] \[\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L}{m}\mathrm{W}(\rho, \pi_{i})+\sum_{i=1}^{K}\mathop{\mathbb{E}}_{h\sim\pi_{i}}\frac{1}{m}\left[| \mathcal{S}_{i}|\mathsf{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell (h,\mathbf{z})\right], \tag{14}\]

where the inequality comes from the Kantorovich-Rubinstein duality theorem. Let \(f(\mathcal{S}_{i})=\mathop{\mathbb{E}}_{h\sim\pi_{i}}\frac{1}{m}\left[| \mathcal{S}_{i}|\mathsf{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h,\mathbf{z}_{i})\right]\), the function has the bounded difference inequality, _i.e._, for two datasets \(\mathcal{S}_{i}\) and \(\mathcal{S}_{i}^{\prime}\) that differs from one example (the \(k\)-th example, without loss of generality), we have

\[|f(\mathcal{S}_{i})-f(\mathcal{S}_{i}^{\prime})| =\left|\mathop{\mathbb{E}}_{h\sim\pi_{i}}\frac{1}{m}\left[| \mathcal{S}_{i}|\mathsf{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h,\mathbf{z})\right]-\mathop{\mathbb{E}}_{h\sim\pi_{i}}\frac{1}{m}\left[| \mathcal{S}_{i}|\mathsf{R}_{\mu}(h)-\sum_{\mathbf{z}^{\prime}\in\mathcal{S}_{i }^{\prime}}\ell(h,\mathbf{z}^{\prime})\right]\right|\] \[=\left|\mathop{\mathbb{E}}_{h\sim\pi_{i}}\left[\frac{1}{m}| \mathcal{S}_{i}|\mathsf{R}_{\mu}(h)-\frac{1}{m}\sum_{\mathbf{z}\in\mathcal{S}_{i }}\ell(h,\mathbf{z})-\frac{1}{m}|\mathcal{S}_{i}|\mathsf{R}_{\mu}(h)+\frac{1}{ m}\sum_{\mathbf{z}^{\prime}\in\mathcal{S}_{i}^{\prime}}\ell(h,\mathbf{z}^{ \prime})\right]\right|\] \[=\left|\mathop{\mathbb{E}}_{h\sim\pi_{i}}\left[\frac{1}{m}\sum_ {\mathbf{z}^{\prime}\in\mathcal{S}_{i}^{\prime}}\ell(h,\mathbf{z}^{\prime})- \frac{1}{m}\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h,\mathbf{z})\right]\right|\] \[=\left|\mathop{\mathbb{E}}_{h\sim\pi_{i}}\left[\frac{1}{m}\ell(h,\mathbf{z}^{\prime}_{k})-\frac{1}{m}\ell(h,\mathbf{z}_{k})\right]\right|\] \[\leq\frac{1}{m}.\]Hence, from Mediarmid's inequality, we have with probability at least \(1-\frac{\delta}{K}\) over \(\mathcal{S}\sim\mu^{m}\)

\[\mathop{\mathbb{E}}_{h\sim\pi_{i}}\frac{1}{m}\left[|\mathcal{S}_{i} |\mathbb{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h,\mathbf{z})\right]\] \[\leq\mathop{\mathbb{E}}_{\mathcal{S}\sim\mu^{m}}\mathop{\mathbb{E} }_{h\sim\pi_{i}}\frac{1}{m}\left[|\mathcal{S}_{i}|\mathbb{R}_{\mu}(h)-\sum_{ \mathbf{z}\in\mathcal{S}_{i}}\ell(h,\mathbf{z})\right]+\sqrt{\frac{|\mathcal{S} _{i}|\ln\frac{K}{\delta}}{2m^{2}}}\] \[=\mathop{\mathbb{E}}_{\mathcal{S}_{i}^{c}\sim\mu^{m-|\mathcal{S}_ {i}|}}\mathop{\mathbb{E}}_{h\sim\pi_{i}}\frac{1}{m}\left[|\mathcal{S}_{i}| \mathbb{R}_{\mu}(h)-\sum_{\mathbf{z}\in\mathcal{S}_{i}}\ell(h,\mathbf{z}) \right]+\sqrt{\frac{|\mathcal{S}_{i}|\ln\frac{K}{\delta}}{2m^{2}}}\] \[=\mathop{\mathbb{E}}_{\mathcal{S}_{i}^{c}\sim\mu^{m-|\mathcal{S}_ {i}|}}\mathop{\mathbb{E}}_{h\sim\pi_{i}}\frac{1}{m}\left[|\mathcal{S}_{i}| \mathbb{R}_{\mu}(h)-|\mathcal{S}_{i}|\mathbb{R}_{\mu}(h)\right]+\sqrt{\frac{| \mathcal{S}_{i}|\ln\frac{K}{\delta}}{2m^{2}}}\] \[=\sqrt{\frac{|\mathcal{S}_{i}|\ln\frac{K}{\delta}}{2m^{2}}}.\]

From the union bound, we have with probability at least \(1-\delta\) over \(\mathcal{S}\sim\mu^{m}\), for any \(\rho\in\mathcal{M}(\mathcal{H})\),

\[\mathop{\mathbb{E}}_{h\sim\rho}\left[\mathbb{R}_{\mu}(h)-\hat{ \mathbb{R}}_{\mathcal{S}}(h)\right]\leq\sum_{i=1}^{K}\frac{2|\mathcal{S}_{i}|L }{m}\mathrm{W}(\rho,\pi_{i})+\sum_{i=1}^{K}\sqrt{\frac{|\mathcal{S}_{i}|\ln \frac{K}{\delta}}{2m^{2}}},\]

which is the claimed result.

We are now able to give a corollary of Theorem 6.

**Corollary 7**.: _We assume our loss \(\ell\) to be in \([0,1]\) and \(L\)-Lipschitz. Then, for any \(\delta\in(0,1]\), with probability at least \(1-\delta\) over the sample \(\hat{\mathcal{S}}\), the following holds for the hypotheses \(h_{i,\mathcal{S}}\in\mathcal{H}\) associated with the Dirac distributions \(\pi_{i,\mathcal{S}}\) and for any \(h\in\mathcal{H}\):_

\[R_{\mu}(h)\leq\hat{R}_{\mathcal{S}}(h)+\sum_{i=1}^{K}\frac{2| \mathcal{S}_{i}|L}{m}d(h,h_{i,\mathcal{S}})+\sum_{i=1}^{K}\sqrt{\frac{| \mathcal{S}_{i}|\ln\frac{K}{\delta}}{2m^{2}}}.\]

Such a bound was impossible to obtain from the PAC-Bayesian bounds based on a KL divergence. Indeed, the KL divergence is infinite for two distributions with disjoint supports. Hence, the PAC-Bayesian framework based on the Wasserstein distance allows us to provide uniform-convergence bounds from a proof technique different from the ones based on the Rademacher complexity [20, 1, 1] or the VC-dimension [21, 22]. In Section 4, we provide an algorithm minimising such a bound.

### Proof of Theorem 3

**Theorem 3**.: _We assume our loss \(\ell\) to be \(L\)-Lipschitz. Then, for any \(\delta\in(0,1]\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), the following holds for the distributions \(\pi_{i,\mathcal{S}}:=\pi_{i}(\mathcal{S},.)\) and for any sequence \((\rho_{i})_{i=1\cdots m}\in\mathcal{M}(\mathcal{H})^{m}\):_

\[\sum_{i=1}^{m}\mathop{\mathbb{E}}_{h_{i}\sim\rho_{i}}\Big{[} \mathop{\mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})\mid\mathcal{F}_{i-1}]-\ell(h_{ i},\mathbf{z}_{i})\Big{]} \leq 2L\sum_{i=1}^{m}\mathrm{W}(\rho_{i},\pi_{i,\mathcal{S}})\] \[+\frac{\lambda}{2}\sum_{i=1}^{m}\mathop{\mathbb{E}}_{h_{i}\sim \pi_{i,\mathcal{S}}}\Big{[}\hat{V}_{i}(h_{i},\mathbf{z}_{i})+V_{i}(h_{i})\Big{]} +\frac{\ln(1/\delta)}{\lambda},\]

_where for all \(i\), \(\hat{V}_{i}(h_{i},\mathbf{z}_{i})=(\ell(h_{i},\mathbf{z}_{i})-\mathop{\mathbb{ E}}_{i-1}[\ell(h_{i},\mathbf{z}_{i})])^{2}\) is the conditional empirical variance at time \(i\) and \(V_{i}(h_{i})=\mathop{\mathbb{E}}_{i-1}[\hat{V}(h_{i},\mathbf{z}_{i})]\) is the true conditional variance._Proof.: First of all, note that for a \(L\)-Lipschitz loss function \(\ell:\mathcal{H}\times\mathcal{Z}\to\mathbb{R}\), we have

\[\left|\left(\mathop{\mathbb{E}}_{i-1}[\ell(h_{i},\mathbf{z}_{i})]-\ell(h_{i}, \mathbf{z}_{i})\right)-\left(\mathop{\mathbb{E}}_{i-1}[\ell(h_{i}^{\prime}, \mathbf{z}_{i})]-\ell(h_{i}^{\prime},\mathbf{z}_{i})\right)\right|\leq 2Ld(h_{i},h_{i} ^{\prime}). \tag{15}\]

Indeed, we can deduce Equation (15) from Jensen inequality, the triangle inequality, and by definition that we have

\[\left|\left(\mathop{\mathbb{E}}_{i-1}[\ell(h_{i},\mathbf{z}_{i})]- \ell(h_{i},\mathbf{z}_{i})\right)-\left(\mathop{\mathbb{E}}_{i-1}[\ell(h_{i}^ {\prime},\mathbf{z}_{i})]-\ell(h_{i}^{\prime},\mathbf{z}_{i})\right)\right|\\ \leq\mathop{\mathbb{E}}_{i-1}\left[|\ell(h_{i},\mathbf{z}_{i}^{ \prime})-\ell(h_{i}^{\prime},\mathbf{z}_{i}^{\prime})|+|\ell(h_{i},\mathbf{z}_ {i})-\ell(h_{i}^{\prime},\mathbf{z}_{i})|\right]\\ \leq\mathop{\mathbb{E}}_{i-1}2Ld(h_{i},h_{i}^{\prime})=2Ld(h_{i},h_{i}^{\prime}).\]

From the Kantorovich-Rubinstein duality theorem [22, Remark 6.5], we have

\[\sum_{i=1}^{m}\mathop{\mathbb{E}}_{h_{i}\sim\rho_{i}}\left[\mathop{\mathbb{E}} _{i-1}[\ell(h_{i},\mathbf{z}_{i})]-\ell(h_{i},\mathbf{z}_{i})\right]\leq 2L \sum_{i=1}^{m}W_{1}(\rho_{i},\pi_{i,\mathcal{S}})+\sum_{i=1}^{m}\mathop{\mathbb{ E}}_{h\sim\pi_{i,\mathcal{S}}}\left[\mathbb{R}_{\mu}(h_{i})-\ell(h_{i},\mathbf{z}_{i}) \right].\]

Now, we define \(X_{i}(h_{i},\mathbf{z}_{i}):=\mathbb{E}_{i-1}[\ell(h_{i},\mathbf{z}_{i})]-\ell (h_{i},\mathbf{z}_{i})\). We also recall that for any \(i\), we have \(\hat{V}_{i}(h_{i},\mathbf{z}_{i})=(\ell(h_{i},\mathbf{z}_{i})-\mathbb{E}_{i-1 }[\ell(h_{i},\mathbf{z}_{i})])^{2}\) and \(V_{i}(h_{i})=\mathbb{E}_{i-1}[\hat{V}(h_{i},\mathbf{z}_{i})]\). To apply the supermartingales techniques of [1], we define the following function:

\[f_{m}(S,h_{1},...,h_{m}):=\sum_{i=1}^{m}\lambda X_{i}(h_{i},\mathbf{z}_{i})- \frac{\lambda^{2}}{2}\sum_{i=1}^{m}(\hat{V}_{i}(h_{i},\mathbf{z}_{i})+V_{i}(h _{i})).\]

Now, Lemma 3.2 of [1] state that the sequence \((SM_{m})_{m\geq 1}\) defined for any \(m\) as:

\[SM_{m}:=\mathop{\mathbb{E}}_{(h_{1},\cdots,h_{m})\sim\pi_{1,\mathcal{S}} \otimes\cdots\otimes\pi_{m,\mathcal{S}}}\left[\exp\left(f_{m}(\mathcal{S},h_{ 1},...,h_{m})\right)\right],\]

is a supermartingale. We exploit this fact as follows:

\[\sum_{i=1}^{m}\mathop{\mathbb{E}}_{h\sim\rho_{i-1}}\left[\mathop {\mathbb{E}}_{i-1}[\ell(h_{i},\mathbf{z}_{i})]-\ell(h_{i},\mathbf{z}_{i}) \right] =\mathop{\mathbb{E}}_{(h_{1},\cdots,h_{m})\sim\pi_{1,\mathcal{S}} \otimes\cdots\otimes\pi_{m,\mathcal{S}}}\left[\sum_{i=1}^{m}X_{i}(h_{i}, \mathbf{z}_{i})\right]\] \[=\frac{1}{\lambda}\mathop{\mathbb{E}}_{(h_{1},\cdots,h_{m})\sim \pi_{1,\mathcal{S}}\otimes\cdots\otimes\pi_{m,\mathcal{S}}}\left[f_{m}( \mathcal{S},h_{1},\cdots,h_{m})\right]\] \[+\frac{\lambda}{2}\sum_{i=1}^{m}\mathop{\mathbb{E}}_{h_{i}\sim\pi _{i,\mathcal{S}}}\left[\hat{V}_{i}(h_{i},\mathbf{z}_{i})+V_{i}(h_{i})\right]\] \[\leq\frac{\ln\left(SM_{m}\right)}{\lambda}+\frac{\lambda}{2}\sum _{i=1}^{m}\mathop{\mathbb{E}}_{h_{i}\sim\pi_{i,\mathcal{S}}}\left[\hat{V}_{i}(h _{i},\mathbf{z}_{i})+V_{i}(h_{i})\right]\]

The last line holds thanks to Jensen's inequality. Now using Ville's inequality ensures us that:

\[\mathop{\mathbb{P}}_{\mathcal{S}}\left(\forall m,SM_{m}\leq\frac{1}{\delta} \right)\geq\frac{1}{\delta}.\]

Thus, with probability \(1-\delta\), for any \(m\) we have \(\ln\left(SM_{m}\right)\leq\ln\left(\frac{1}{\delta}\right)\). This concludes the proof.

### Proof of Theorem 4

**Theorem 4**.: _We assume our loss \(\ell\) to be non-negative and \(L\)-Lipschitz. We also assume that, for any \(i,\mathcal{S}\), \(\mathbb{E}_{h\sim\pi_{i}(\cdot,\mathcal{S})}\left[\mathbb{E}_{i-1}[\ell(h, \mathbf{z}_{i})^{2}]\right]\leq 1\) (bounded conditional order 2 moments for priors). Then, for any \(\delta\in(0,1]\), with probability at least \(1-\delta\) over the sample \(\mathcal{S}\), any online predictive sequence (used as priors) \((\pi_{i})_{i\geq 1}\), we have with probability at least \(1-\delta\) over the sample \(\mathcal{S}\sim\mu\), the following, holding for the data-dependent measures \(\pi_{i,\mathcal{S}}:=\pi_{i}(S,.)\) and any posterior sequence \((\rho_{i})_{i\geq 1}\):_

\[\frac{1}{m}\sum_{i=1}^{m}\mathop{\mathbb{E}}_{h_{i}\sim\rho_{i}}\left[\mathbb{E} [\ell(h_{i},\mathbf{z}_{i})\mid\mathcal{F}_{i-1}]-\ell(h_{i},\mathbf{z}_{i}) \right]\leq\frac{2L}{m}\sum_{i=1}^{m}\mathrm{W}(\rho_{i},\pi_{i,\mathcal{S}})+ \sqrt{\frac{2\ln\left(\frac{1}{\delta}\right)}{m}}.\]Proof.: The proof starts similarly to the one of Theorem 3. Indeed, note that for a \(L\)-Lipschitz loss function \(\ell:\mathcal{H}\times\mathcal{Z}\to\mathbb{R}\), we have

\[\left|\left(\underset{i-1}{\mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})]-\ell(h_{i}, \mathbf{z}_{i})\right)-\left(\underset{i-1}{\mathbb{E}}[\ell(h_{i}^{\prime}, \mathbf{z}_{i})]-\ell(h_{i}^{\prime},\mathbf{z}_{i})\right)\right|\leq 2Ld(h_{i},h_{i} ^{\prime}). \tag{16}\]

Indeed, we can deduce Equation (16) from Jensen inequality, the triangle inequality, and by definition that we have

\[\left|\left(\underset{i-1}{\mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})]-\ell(h_{i},\mathbf{z}_{i})\right)-\left(\underset{i-1}{\mathbb{E}}[\ell(h_{i}^{\prime}, \mathbf{z}_{i})]-\ell(h_{i}^{\prime},\mathbf{z}_{i})\right)\right|\\ \leq\underset{i-1}{\mathbb{E}}\left[|\ell(h_{i},\mathbf{z}_{i}^{ \prime})-\ell(h_{i}^{\prime},\mathbf{z}_{i}^{\prime})|+|\ell(h_{i},\mathbf{z}_ {i})-\ell(h_{i}^{\prime},\mathbf{z}_{i})|\right]\\ \leq\underset{i-1}{\mathbb{E}}2Ld(h_{i},h_{i}^{\prime})=2Ld(h_{ i},h_{i}^{\prime}).\]

From the Kantorovich-Rubinstein duality theorem [11, Remark 6.5], we have

\[\sum_{i=1}^{m}\underset{h_{i}\sim\rho_{i}}{\mathbb{E}}\left[\underset{i-1}{ \mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})]-\ell(h_{i},\mathbf{z}_{i})\right]\leq 2 L\sum_{i=1}^{m}W_{1}(\rho_{i},\pi_{i,\mathcal{S}})+\sum_{i=1}^{m}\underset{h\sim \pi_{i,\mathcal{S}}}{\mathbb{E}}\left[\mathbb{R}_{\mu}(h_{i})-\ell(h_{i}, \mathbf{z}_{i})\right].\]

Now, we define \(X_{i}(h_{i},\mathbf{z}_{i}):=\mathbb{E}_{i-1}[\ell(h_{i},\mathbf{z}_{i})]-\ell (h_{i},\mathbf{z}_{i})\). To apply the supermartingales techniques of [10], we define the following function:

\[f_{m}(S,h_{1},...,h_{m}):=\sum_{i=1}^{m}\lambda X_{i}(h_{i},\mathbf{z}_{i})- \frac{\lambda^{2}}{2}\sum_{i=1}^{m}\underset{i-1}{\mathbb{E}}[\ell(h_{i}, \mathbf{z}_{i})^{2}].\]

Now, because our loss is nonnegative, [10, Lemma A.2 and Lemma B.1] state that the sequence \((SM_{m})_{m\geq 1}\) defined for any \(m\) as:

\[SM_{m}:=\underset{(h_{1},\cdots,h_{m})\sim\pi_{1,\mathcal{S}}\otimes\cdots \otimes\pi_{m,\mathcal{S}}}{\mathbb{E}}\left[\exp\left(f_{m}(\mathcal{S},h_{1},...,h_{m})\right)\right],\]

is a supermartingale. We exploit this fact as follows:

\[\sum_{i=1}^{m}\underset{h\sim\rho_{i-1}}{\mathbb{E}}\left[\underset {i-1}{\mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})]-\ell(h_{i},\mathbf{z}_{i})\right] =\underset{(h_{1},\cdots,h_{m})\sim\pi_{1,\mathcal{S}}\otimes \cdots\otimes\pi_{m,\mathcal{S}}}{\mathbb{E}}\left[\sum_{i=1}^{m}X_{i}(h_{i}, \mathbf{z}_{i})\right]\] \[=\frac{1}{\lambda}\left(\underset{h_{1},\cdots,h_{m})\sim\pi_{1,\mathcal{S}}\otimes\cdots\otimes\pi_{m,\mathcal{S}}}{\mathbb{E}}\left[f_{m}( \mathcal{S},h_{1},\cdots,h_{m})\right]\right.\] \[+\frac{1}{2}\sum_{i=1}^{m}\underset{h_{i}\sim\pi_{i,\mathcal{S}}} {\mathbb{E}}\left[\underset{i-1}{\mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})^{2}]\right]\] \[\leq\frac{\ln(SM_{m})}{\lambda}+\frac{\lambda}{2}\sum_{i=1}^{m} \underset{h_{i}\sim\pi_{i,\mathcal{S}}}{\mathbb{E}}\left[\underset{i-1}{ \mathbb{E}}[\ell(h_{i},\mathbf{z}_{i})^{2}]\right]\]

The last line holds thanks to Jensen's inequality. Now using Ville's inequality ensures us that:

\[\underset{\mathcal{S}}{\mathbb{P}}\left(\forall m,SM_{m}\leq\frac{1}{\delta} \right)\geq\frac{1}{\delta}\]

Thus, with probability \(1-\delta\), for any \(m\) we have \(\ln(SM_{m})\leq\ln\frac{1}{\delta}\). We conclude the proof by exploiting the boundedness assumption on conditional order 2 moments and optimising the bound in \(\lambda\).

## Appendix C Supplementary insights on experiments

In this section, Appendix C.1 presents the learning algorithm for the _i.i.d._ setting. We also introduce the online algorithm in Appendix C.2. We prove the Lipschitz constant of the loss for the linear models in Appendix C.3. Finally, we provide more experiments in Appendix C.5.

### Batch algorithm for the _i.i.d._ setting

The pseudocode of our batch algorithm is presented in Algorithm 1.

```
1:procedurePriors Learning
2:\(h_{1},\ldots,h_{K}\leftarrow\) initialize the hypotheses
3:for\(t\gets 1,\ldots,T\)do
4:for each mini-batch \(\mathcal{U}\subseteq\mathcal{S}\)do
5:for\(i\gets 1,\ldots,K\)do
6:\(\mathcal{U}_{i}\leftarrow\mathcal{U}\setminus\mathcal{S}_{i}\)
7:\(h_{i}\leftarrow\) perform a gradient descent step with \(\nabla\mathsf{R}_{\mathcal{U}_{i}}(h_{i})\)
8:return hypotheses \(h_{1},\ldots,h_{K}\)
9:
10:procedurePosterior Learning
11:\(h\leftarrow\) initialize the hypothesis
12:for\(t\gets 1,\ldots,T^{\prime}\)do
13:for each mini-batch \(\mathcal{U}\subseteq\mathcal{S}\)do
14:\(h\leftarrow\) perform a gradient descent step with \(\nabla[\mathsf{R}_{\mathcal{U}}(h)+\varepsilon\sum_{i=1}^{K}\frac{|\mathcal{S} _{i}|}{m}d(h,h_{i})]\)
15:return hypothesis \(h\)
```

**Algorithm 1** (Mini-)Batch Learning Algorithm with Wasserstein distances

Priors Learning minimises the empirical risk through mini-batches \(\mathcal{U}\subseteq\mathcal{S}\) for \(T\) epochs. More precisely, for each epoch, we _(a)_ sample a mini-batch \(\mathcal{U}\) (line 4) by excluding the set \(\mathcal{S}_{i}\) from \(\mathcal{U}\) for each \(h_{i}\in\mathcal{H}\) (line 5-6), then _(b)_ the hypotheses \(h_{1},\ldots,h_{K}\in\mathcal{H}\) are updated (line 7). In Posterior Learning, we perform a gradient descent step (line 14) on the objective function associated with Equation (5) for \(T^{\prime}\) epochs in a mini-batch fashion.

### Learning algorithm for the online setting

Algorithm 2 presents the pseudocode of our online algorithm.

```
1:Initialize the hypothesis \(h_{0}\in\mathcal{H}\)
2:for\(i\gets 1,\ldots,m\)do
3:for\(t\gets 1,\ldots,T\)do
4:\(h_{i}\leftarrow\) perform a gradient step with \(\nabla[\ell(h_{i},\mathbf{z}_{i})+\hat{B}(d(h_{i},h_{i-1}){-}1)]\) (Eq. (7) with \(\hat{B}\))
5:return hypotheses \(h_{1},\ldots,h_{m}\)
```

**Algorithm 2** Online Learning Algorithm with Wasserstein distances

For each time step \(i\), we perform \(T\) gradient descent steps on the objective associated with Equation (6) (line 4). Note that we can retrieve OGD from Algorithm 2 by _(a)_ setting \(T=1\) and _(b)_ removing the regularisation term \(\hat{B}(d(h_{i},h_{i-1}){-}1)\).

### Lipschitzness for the linear model

Recall that we use, in our experiments, the multi-margin loss function from the Pytorch module defined for any linear model with weights \(W\in\mathbb{R}^{|\mathcal{Y}|\times d}\) and biases \(b\in\mathbb{R}^{|\mathcal{Y}|}\), any data point \(\mathbf{z}\in\mathcal{X}\times\mathcal{Y}\)

\[\ell(W,b,\mathbf{z})=\frac{1}{|\mathcal{Y}|-1}\sum_{y^{\prime}\neq y}\max \left(0,f(W,b,\mathbf{z},y^{\prime})\right),\]

where \(f(W,b,\mathbf{z},y^{\prime})=1+\langle W[y^{\prime}]-W[y],\mathbf{x}\rangle+b[ y^{\prime}]-b[y]\), and \(W[y]\in\mathbb{R}^{d}\) and \(b[y]\in\mathbb{R}\) are respectively the vector and the scalar for the \(y\)-th output.

To apply our theorems, we must ensure that our loss function is Lipschitz with respect to the linear model, hence the following lemma.

**Lemma 8**.: _For any \(\mathbf{z}=(\mathbf{x},y)\in\mathcal{X}\times\mathcal{Y}\) with the norm of \(\mathbf{x}\) bounded by \(1\), the function \(W,b\mapsto\ell(W,b,\mathbf{z})\) is \(2\)-Lipschitz._

Proof.: Let \((W,b),(W^{\prime},b^{\prime})\) both in \(\mathbb{R}^{|\mathcal{Y}|\times d}\times\mathbb{R}^{|\mathcal{Y}|}\), we have

\[|\ell(W,b,\mathbf{z})-\ell(W^{\prime},b^{\prime},\mathbf{z})|\leq\frac{1}{| \mathcal{Y}|-1}\sum_{y^{\prime}\neq y}|\max\left(0,f(W,b,\mathbf{z},y^{\prime} )\right)-\max\left(0,f(W^{\prime},b^{\prime},\mathbf{z},y^{\prime})\right)|.\]

Note that because \(\alpha\mapsto\max(0,\alpha)\) is 1-Lipschitz, we have:

\[|\ell(W,b,\mathbf{z})-\ell(W^{\prime},b^{\prime},\mathbf{z})|\leq\frac{1}{| \mathcal{Y}|-1}\sum_{y^{\prime}\neq y}|f(W,b,\mathbf{z},y^{\prime})-f(W^{ \prime},b^{\prime},\mathbf{z},y^{\prime})|.\]

Finally, notice that:

\[\frac{1}{|\mathcal{Y}|-1}\sum_{y^{\prime}\neq y}|f(W,b,\mathbf{z},y^{\prime})-f(W^{\prime},b^{\prime},\mathbf{z},y^{\prime})| \leq\frac{1}{|\mathcal{Y}|-1}\sum_{y^{\prime}\neq y}|\langle(W-W ^{\prime})[y^{\prime}]-(W-W^{\prime})[y],\mathbf{x}\rangle|\] \[+\frac{1}{|\mathcal{Y}|-1}\sum_{y^{\prime}\neq y}|(b-b^{\prime})[ y^{\prime}]-(b-b^{\prime})[y]|\] \[\leq\frac{1}{|\mathcal{Y}|-1}\sum_{y^{\prime}\neq y}\|(W-W^{ \prime})[y^{\prime}]-(W-W^{\prime})[y]\|\,\|\mathbf{x}\|\] \[+\frac{1}{|\mathcal{Y}|-1}\sum_{y^{\prime}\neq y}|(b-b^{\prime})[ y^{\prime}]-(b-b^{\prime})[y]|.\]

Because we consider the Euclidean norm, we have for any \(y^{\prime}\in\mathcal{Y}\):

\[\|(W-W^{\prime})[y^{\prime}]-(W-W^{\prime})[y]\| =\sqrt{\|(W-W^{\prime})[y^{\prime}]-(W-W^{\prime})[y]\|^{2}}\] \[\leq\sqrt{2\left(\|(W-W^{\prime})[y^{\prime}]\|^{2}+\|(W-W^{ \prime})[y]\|^{2}\right)}\] \[\leq\sqrt{2}\|W-W^{\prime}\|.\]

The second line holding because for any scalars \(a,b\), we have \((a-b)^{2}\leq 2(a^{2}+b^{2})\) and the last line holding because \(\|W-W^{\prime}\|^{2}=\sum_{y\in\mathcal{Y}}\|(W-W^{\prime})[y]\|^{2}\). A similar argument gives

\[\frac{1}{|\mathcal{Y}|-1}\sum_{y^{\prime}\neq y}|(b-b^{\prime})[y^{\prime}]-( b-b^{\prime})[y]|\leq\sqrt{2}\|b-b^{\prime}\|.\]

Then, using that \(\|x\|\leq 1\) and summing on all \(y^{\prime}\) gives:

\[|\ell(W,b,\mathbf{z})-\ell(W^{\prime},b^{\prime},\mathbf{z})|\leq\sqrt{2} \left(\|W-W^{\prime}\|+\|b-b^{\prime}\|\right).\]

Finally, notice that \((\|W-W^{\prime}\|+\|b-b^{\prime}\|)^{2}\leq 2(\|W-W^{\prime}\|^{2}+\|b-b^{ \prime}\|^{2})=2\|(W,b)-(W^{\prime},b^{\prime})\|^{2}\). Thus \(\|W-W^{\prime}\|+\|b-b^{\prime}\|\leq\sqrt{2}\|(W,b)-(W^{\prime},b^{\prime})\|\). This concludes the proof.

### Lipschitzness for neural networks

Recall that we use, in our experiments, the multi-margin loss function from the Pytorch module defined we consider the loss \(\ell(h,(\mathbf{x},y))=\frac{1}{|\mathcal{Y}|}\sum_{y^{\prime}\neq y}\max(0,1- \eta(h[y]-h[y^{\prime}]))\), which is \(\eta\)-Lipschitz _w.r.t._ the outputs \(h[1],\ldots,h[|\mathcal{Y}|]\). For neural networks, \(h\) is the output of the neural network with input \(\mathbf{x}\). Note that this loss is \(\eta\)-lipschitz with respect to the outputs. To apply our theorems, we must ensure that our loss function is Lipschitz with respect to the weights of the neural networks, hence the following lemma with associated background.

We define a FCN recursively as follows: for a vector \(\mathbf{W}_{1}=\operatorname{vec}(\{W_{1},b\})\), (_i.e._, the vectorisation of a weight matrix \(W_{1}\) and a bias \(b\)) and an input datum \(\mathbf{x}\), \(\operatorname{FCN}_{1}(\mathbf{W}_{1},\mathbf{x})=\sigma_{1}\left(W_{1}\mathbf{ x}+b_{1}\right)\), where \(\sigma_{1}\) is the activation function. Also, for any \(i\geq 2\) we define for a vector \(\mathbf{W}_{i}=(W_{i},b_{i},\mathbf{W}_{i-1})\) (defined recursively as well), \(\operatorname{FCN}_{i}(\mathbf{W}_{i},\mathbf{x})=\sigma_{i}\left(W_{i} \text{FCN}_{i-1}(\mathbf{W}_{i-1},\mathbf{x})+b_{i}\right)\). Then, setting \(\mathbf{z}=(\mathbf{x},y)\) a datum and \(h_{i}(\mathbf{x}):=\operatorname{FCN}_{i}(\mathbf{W}_{i},\mathbf{x})\) we can rewrite our loss as a function of \((\mathbf{W}_{i},\mathbf{z})\).

**Lemma 9**.: _Assume that all the weight matrices of \(\mathbf{W}_{i}\) are bounded and that the activation functions are Lipschitz continuous with constant bounded by \(K_{\sigma}\). Then for any datum \(\mathbf{z}=(\mathbf{x},y)\), any \(i\), \(\mathbf{W}_{i}\to\ell(\mathbf{W}_{i},\mathbf{z})\) is Lipschitz continuous._

Proof.: We consider the Frobenius norm on matrices as \(\mathbf{W}_{2}\) is a vector as we consider the L2-norm on the vector. We prove the result for \(i=2\), assuming it is true for \(i=1\). We then explain how this proof generalises the case \(i=1\) and works recursively. Let \(\mathbf{z},\mathbf{W}_{2},\mathbf{W}_{2}^{\prime}\), for clarity we write \(\operatorname{FCN}_{2}(\mathbf{x}):=\operatorname{FCN}(\mathbf{W}_{2},\mathbf{ x})\) and \(\operatorname{FCN}_{2}^{\prime}(\mathbf{x}):=\operatorname{FCN}(\mathbf{W}_{2}^{ \prime},\mathbf{x})\). As \(\ell\) is Lipschitz on the outputs \(\operatorname{FCN}_{2}(\mathbf{x}),\operatorname{FCN}_{2}^{\prime}(\mathbf{x})\). We have

\[|\ell(\mathbf{W}_{2},\mathbf{z})-\ell(\mathbf{W}_{2}^{\prime}, \mathbf{z})|\leq\eta\left\|\operatorname{FCN}_{2}(\mathbf{x})-\operatorname{ FCN}_{2}^{\prime}(\mathbf{x})\right\|\\ \leq\eta\left\|\sigma_{2}\left(W_{2}\text{FCN}_{1}(\mathbf{x})+b_ {2}\right)-\sigma_{2}\left(W_{2}^{\prime}\text{FCN}_{1}^{\prime}(\mathbf{x})+b _{2}^{\prime}\right)\right\|\\ \leq\eta K_{\sigma}\|W_{2}\text{FCN}_{1}(\mathbf{x})+b_{2}-W_{2}^ {\prime}\text{FCN}_{1}^{\prime}(\mathbf{x})-b_{2}^{\prime}\|\\ \leq\eta K_{\sigma}\left(||(W_{2}-W_{2}^{\prime})\text{FCN}_{1}( \mathbf{x})||+||W_{2}^{\prime}(\text{FCN}_{1}(\mathbf{x})-\text{FCN}_{1}^{ \prime}(\mathbf{x}))||+\|b_{2}-b_{2}^{\prime}\|\right).\]

Then, we have \(||(W_{2}-W_{2}^{\prime})\text{FCN}_{1}(\mathbf{x})||\leq||(W_{2}-W_{2}^{\prime })||_{F}||\text{FCN}_{1}(\mathbf{x})||\leq K_{\mathbf{x}}||(W_{2}-W_{2}^{\prime })||_{F}\). The second inequality holding as \(\text{FCN}_{1}(\mathbf{x})\) is a continuous function of the weights. Indeed, as on a compact space, a continuous function reaches its maximum, then its norm is bounded by a certain \(K_{\mathbf{x}}\). Also, as the weights are bounded, any weight matrix has its norm bounded by a certain \(K_{W}\) thus \(\|W_{2}^{\prime}(\text{FCN}_{1}(\mathbf{x})-\text{FCN}_{1}^{\prime}(\mathbf{x} )\|\leq\|W_{2}^{\prime}\|_{F}\|(\text{FCN}_{1}(\mathbf{x})-\text{FCN}_{1}^{ \prime}(\mathbf{x})\|\leq K_{W}\|\text{FCN}_{1}(\mathbf{x})-\text{FCN}_{1}^{ \prime}(\mathbf{x})\|\). Finally, taking \(K_{\text{temp}}=\eta K_{\sigma}\max(K_{\mathbf{x}},K_{W},1)\) gives:

\[|\ell(\mathbf{W}_{2},\mathbf{z})-\ell(\mathbf{W}_{2}^{\prime}, \mathbf{z})|\leq K_{\text{temp}}\left(\|(W_{2}-W_{2}^{\prime})\|_{F}+\|b_{2} -b_{2}^{\prime}\|+\|\text{FCN}_{1}(\mathbf{x})-\text{FCN}_{1}^{\prime}(\mathbf{ x})\|\right).\]

Exploiting the recursive assumption that \(\text{FCN}_{1}\) is Lipschitz with respect to its weights \(\mathbf{W}_{1}\) gives \(\|\text{FCN}_{1}(\mathbf{x})-\text{FCN}_{1}^{\prime}(\mathbf{x})\|\leq K_{1} ||\mathbf{W}_{1}-\mathbf{W}_{1}^{\prime}||\).

If we denote by \((W_{2},b_{2})\) the vector of all concatenated weights, notice that \(\|(W_{2}-W_{2}^{\prime})\|_{F}+\|b_{2}-b_{2}^{\prime}\|=\sqrt{(\|(W_{2}-W_{2}^ {\prime})\|_{F}+\|b_{2}-b_{2}^{\prime}\|)^{2}}\leq\sqrt{2(\|(W_{2}-W_{2}^{ \prime})\|^{2}_{F}+\|b_{2}-b_{2}^{\prime}\|^{2})}=\sqrt{2}\|(W_{2},b_{2})-(W_{2}^ {\prime},b_{2}^{\prime})\|\) (we used that for any real numbers \(a,b,(a+b)^{2}\leq 2(a^{2}+b^{2})\)). We then have:

\[|\ell(\mathbf{W}_{2},\mathbf{z})-\ell(\mathbf{W}_{2}^{\prime}, \mathbf{z})| \leq K_{\text{temp}}\max(\sqrt{2},K_{1})\left(\|(W_{2},b_{2})-(W_{ 2}^{\prime},b_{2}^{\prime})\|+\|\mathbf{W}_{1}-\mathbf{W}_{1}^{\prime}\|\right)\] \[\leq\sqrt{2}K_{\text{temp}}\max(\sqrt{2},K_{1})||\mathbf{W}_{2}- \mathbf{W}_{2}^{\prime}||.\]

The last line holds by reusing the same calculation trick. This concludes the proof for \(i=2\). Then for \(i=1\) the same proof holds by replacing \(W_{2},b_{2},\text{FCN}_{2}\) by \(W_{1},b_{1},\text{FCN}_{1}\) and replacing \(\text{FCN}_{1}(\mathbf{x}),\text{FCN}_{1}^{\prime}(\mathbf{x})\) by \(\mathbf{x}\) (we then do not need to assume a recursive Lipschitz behaviour). Therefore the result holds for \(i=1\).

We then properly apply a recursive argument by assuming the result at rank \(i-1\) reusing the same proof at any rank \(i\) by replacing \(W_{2},b_{2},\text{FCN}_{2}\) by \(W_{i},b_{i},\text{FCN}_{i}\) and \(\text{FCN}_{1}(\mathbf{x}),\text{FCN}_{1}^{\prime}(\mathbf{x})\) by \(\text{FCN}_{i-1}(\mathbf{x}),\text{FCN}_{i-1}^{\prime}(\mathbf{x})\). This concludes the proof.

### Experiments with varying number of priors

The experiments of Section 4 rely on data-dependent priors constructed through the procedure Priors Learning. We fixed a number of priors \(K\) equal to \(0.2\sqrt{m}\). This number is an empirical tradeoff between the informativeness of our priors and time-efficient computation. However, there is no theoretical intuition for the value of this parameter (the discussion of Section 3.1 considered \(K=\sqrt{m}\) as a potential tradeoff; see Appendix A). Thus, we gather below the performance of our learning procedures for \(K=\alpha\sqrt{m}\), where \(\alpha\in\{0,0.4,0.6,0.8,1\}\) (the case \(\alpha=0\) being a convention to denote \(K=1\)). The experiments are gathered below, and all remaining hyperparameters (except \(K\)) are identical to those described in Section 4.

**Analysis of our results.** First, when considering neural networks, note that for any dataset except segmentation, letter, the performances of our methods are similar or better when considering data-dependent priors (_i.e._, when \(\alpha>0\)). A similar remark holds for the linear models for all datasets except for satimage, segmentation, and tictactoe. This illustrates the relevance of data-dependent priors. We also remark that there is no value of \(\alpha\), which provides a better performance on all datasets. For instance, considering neural networks, note that \(\alpha=1\) gives the better performance (_i.e._, the smallest \(\mathfrak{R}_{\mu}(h)\)) for Algorithm 1 (\(/\sqrt{m}\)) for the satimage dataset while, for the same algorithm, the better performance on the segmentation dataset is attained for \(\alpha=0.8\). Sometimes, the number \(K\) does not have a clear influence: on mnist with NNs, for Algorithm 1 (\(/\sqrt{m}\)), our performances are similar, whatever the value of \(K\), but still significantly better than ERM. In any case, note that for every dataset, there exists a value of \(K\) and such that our algorithm attains either similar or significantly better performances than ERM on every dataset, which shows the relevance of our learning algorithm to ensure a good generalisation ability. Moreover, there is no obvious choice for the parameters \(\varepsilon\). For instance, in Tables 2 and 3, for the segmentation dataset, the parameters \(K=1,\varepsilon=\frac{1}{m}\) are optimal (in terms of test risks) for both models. As \(K=1\) means that our single prior is data-free, this shows that the intrinsic structure of segmentation makes it less sensitive to both the information contained in the prior (\(K=1\) meaning data-free prior) and the place of the prior itself (\(\varepsilon=1/m\) meaning that we give less weight to the regularisation within our optimisation procedure). On the contrary, in Table Table 1, the yeast dataset performs significantly better when \(\varepsilon=1/\sqrt{m}(K=0.2\sqrt{m})\), exhibiting a positive impact of our data-dependent priors.

### Experiments on classical regularisation methods

We perform additional experiments to see the performance of the weight decay, _i.e._, the L2 regularisation on the weights; the results are presented in Table 4. Moreover, notice that the 'distance to initialisation' \(\|\mathbf{w}-\mathbf{w}_{0}\|\) (where \(\mathbf{w}_{0}\) is the weights initialized randomly) is a particular case of Algorithm 1 when \(K=1\) (_i.e._, we treat the data as a single batch, and the prior is the data-free initialisation); the results are in Tables 2 and 3.

**Analysis of our results.** This experiment on the weight decay demonstrates that on a few datasets (namely sensorless and yeast), when our predictors are neural nets, the weight decay regularisation fails to learn while ours succeeds, as shown in Table 1. In general, this table shows that, on most of the datasets, considering data-dependent priors leads to sharper results. This shows the efficiency of our method compared to the 'distance to initialisation' regularisation.

\begin{table}

\end{table}
Table 2: Performance of Algorithm 1 compared to ERM on different datasets for neural network models. We consider \(\varepsilon=1/m\) and \(\varepsilon=1/\sqrt{m}\), with \(K=\alpha\sqrt{m}\) and \(\alpha\in\{0,0.4,0.6,0.8,1\}\). We plot the empirical risk \(\mathfrak{R}_{\mathcal{S}}(h)\) with its associated test risk \(\mathfrak{R}_{\mu}(h)\).

\begin{table}

\end{table}
Table 3: Performance of Algorithm 1 compared to ERM on different datasets for linear models. We consider \(\varepsilon=\nicefrac{{1}}{{m}}\) and \(\varepsilon=\nicefrac{{1}}{{\sqrt{m}}}\), with \(K=\alpha\sqrt{m}\) and \(\alpha\in\{0,0.4,0.6,0.8,1\}\). We plot the empirical risk \(\mathfrak{R}_{\mathcal{S}}(h)\) with its associated test risk \(\mathfrak{R}_{\mu}(h)\).

\begin{table}

\end{table}
Table 4: Performance of ERM with weight decay (with the L2 regularisation) for linear and neural network models.

Response:
## Review

### Summary

This paper presents novel PAC-Bayesian learning methods utilizing Wasserstein distance instead of KL divergence, contributing to both batch and online learning frameworks. It derives high-probability generalization bounds applicable to heavy-tailed loss functions and proposes new learning algorithms based on these bounds. The results are validated with experiments demonstrating improved generalization performance over traditional methods like ERM and OGD. Notably, the work accommodates data-dependent priors, enhancing the applicability of the approach. The theoretical foundations are sound, and the empirical results are promising, though some aspects require further clarification.

### Strengths

- The introduction and results are well-contextualized within recent literature.
- The theorems presented have weak assumptions, making them broadly applicable.
- The proofs are clearly explained, even for complex results.
- Empirical validation demonstrates improved performance over existing methods.

### Weaknesses

- Variability in model performance based on batch size needs clarification.
- The role of the Lipschitz assumption and its implications should be better explained.
- Some sections, especially regarding algorithm comparisons and implications, require clearer discussion.
- Experimental results should be compared with standard regularization methods for a more comprehensive evaluation.

### Questions

- What justifies the use of the ℓ2 norm on weights if the loss function is only Lipschitz with respect to outputs?
- Can the authors clarify the performance disparities between algorithms based on their empirical results?
- How does the proposed approach compare with standard KL-based PAC-Bayes bounds, especially regarding variance terms?

### Soundness

**Score:** 3

**Description:** 3 = good; the theoretical contributions are solid, but some assumptions and implications lack full clarity.

### Presentation

**Score:** 3

**Description:** 3 = good; while the writing is mostly clear, some sections could benefit from improved clarity and organization.

### Contribution

**Score:** 4

**Description:** 4 = excellent; the paper introduces significant advancements in PAC-Bayesian learning with practical implications.

### Rating

**Score:** 7

**Description:** 7 = accept, but needs minor improvements; the paper is technically solid with important contributions, but certain aspects require refinement.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper demonstrates originality in applying Wasserstein distance to PAC-Bayesian learning, presenting sound theoretical foundations and promising empirical results. While some weaknesses exist, particularly in clarity and depth of discussion, the overall significance and contribution of the work warrant acceptance.