Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Boosting with Tempered Exponential Measures

 Richard Nock

Google Research

richardnock@google.com &Ehsan Amid

Google DeepMind

eamid@google.com &Manfred K. Warmuth

Google Research

manfred@google.com

###### Abstract

One of the most popular ML algorithms, AdaBoost, can be derived from the dual of a relative entropy minimization problem subject to the fact that the positive weights on the examples sum to one. Essentially, harder examples receive higher probabilities. We generalize this setup to the recently introduced _tempered exponential measures_ (tems) where normalization is enforced on a specific power of the measure and not the measure itself. tems are indexed by a parameter \(t\) and generalize exponential families (\(t=1\)). Our algorithm, \(t\)-AdaBoost, recovers AdaBoost as a special case (\(t=1\)). We show that \(t\)-AdaBoost retains AdaBoost's celebrated exponential convergence rate on margins when \(t\in[0,1)\) while allowing a slight improvement of the rate's hidden constant compared to \(t=1\). \(t\)-AdaBoost partially computes on a generalization of classical arithmetic over the reals and brings notable properties like guaranteed bounded leveraging coefficients for \(t\in[0,1)\). From the loss that \(t\)-AdaBoost minimizes (a generalization of the exponential loss), we show how to derive a new family of _tempered_ losses for the induction of domain-partitioning classifiers like decision trees. Crucially, strict properness is ensured for all while their boosting rates span the full known spectrum of boosting rates. Experiments using \(t\)-AdaBoost+trees display that significant leverage can be achieved by tuning \(t\).

## 1 Introduction

AdaBoost is one of the most popular ML algorithms [8, 30]. It efficiently aggregates weak hypotheses into a highly accurate linear combination [10]. The common motivations of boosting algorithms focus on choosing good linear weights (the leveraging coefficients) for combining the weak hypotheses. A dual view of boosting highlights the dual parameters, which are the weights on the examples. These weights define a distribution, and AdaBoost can be viewed as minimizing a relative entropy to the last distribution subject to a linear constraint introduced by the current hypothesis [12]. For this reason (more in SS 2), AdaBoost's weights define an exponential family.

**In this paper**, we go beyond weighing the examples with a discrete exponential family distribution, relaxing the constraint that the total mass be unit but instead requiring it for the measure's \(1/(2-t)\)'th power, where \(t\) is a temperature parameter. Such measures, called _tempered exponential measures_ (tems), have been recently introduced [4]. Here we apply the discrete version of these tems for deriving a novel boosting algorithm called \(t\)-AdaBoost. Again the measures are solutions to a relative entropy minimization problem, but the relative entropy is built from Tsallis entropy and "tempered" by a parameter \(t\). As \(t\to 1\)tems become standard exponential family distributions and our new algorithm merges into AdaBoost. As much as AdaBoost minimizes the exponential loss, \(t\)-AdaBoost minimizes a generalization of this loss we denote as the _tempered exponential loss_.

tems were introduced in the context of clustering, where they were shown to improve the robustness to outliers of clustering's population minimizers [4]. They have also been shown to bring low-level sparsity features to optimal transport [3]. Boosting is a high-precision machinery: AdaBoost is known to achieve near-optimal boosting rates under the weak learning assumption [1], but it haslong been known that numerical issues can derail it, in particular, because of the unbounded weight update rule [14]. So the question of what the tem setting can bring for boosting is of primordial importance. As we show, \(t\)-AdaBoost can suffer no rate setback as boosting's exponential rate of convergence on _margins_ can be preserved for all \(t\in[0,1)\). Several interesting features emerge: the weight update becomes bounded, margin optimization can be _tuned_ with \(t\) to focus on examples with very low margin and besides linear separators, the algorithm can also learn _progressively_ clipped models1. Finally, the weight update makes appear a new regime whereby weights can "switch off and on": an example's weight can become zero if too well classified by the current linear separator, and later on revert to non-zero if badly classified by a next iterate. \(t\)-AdaBoost makes use of a generalization of classical arithmetic over the reals introduced decades ago [18].

Footnote 1: Traditionally, clipping a sum is done after it has been fully computed. In our case, it is clipped after each new summand is added.

Boosting algorithms for linear models like AdaBoost bring more than just learning good linear separators: it is known that (ada)boosting linear models can be used to emulate the training of _decision trees_ (DT) [16], which are models known to lead to some of the best of-the-shelf classifiers when linearly combined [9]. Unsurprisingly, the algorithm obtained emulates the classical top-down induction of a tree found in major packages like CART [6] and C4.5 [23]. The _loss_ equivalently minimized, which is, _e.g._, Matusita's loss for AdaBoost[30, Section 4.1], is a lot more consequential. Contrary to losses for real-valued classification, losses to train DTs rely on the estimates of the posterior learned by the model; they are usually called _losses for Class Probability Estimation_ (CPE [25]). The CPE loss is crucial to elicit because (i) it is possible to check whether it is "good" from the standpoint of properness (Bayes rule is optimal for the loss [28]), and (ii) it conditions boosting rates, only a handful of them being known, for the most popular CPE losses [11; 22; 31].

**In this paper**, we show that this emulation scheme on \(t\)-AdaBoost provides a new family of CPE losses with remarkable constancy with respect to properness: losses are _strictly_ proper (Bayes rule is the _sole_ optimum) for any \(t\in(-\infty,2)\) and proper for \(t=-\infty\). Furthermore, over the range \(t\in[-\infty,1]\), the range of boosting rates spans the full spectrum of known boosting rates [11].

We provide experiments displaying the boosting ability of \(t\)-AdaBoost over a range of \(t\) encompassing potentially more than the set of values covered by our theory, and highlight the potential of using \(t\) as a parameter for efficient tuning the loss [25, Section 8]. Due to a lack of space, proofs are relegated to the appendix (APP). A primer on tems is also given in APP., Section I.

## 2 Related work

Boosting refers to the ability of an algorithm to combine the outputs of moderately accurate, "weak" hypotheses into a highly accurate, "strong" ensemble. Originally, boosting was introduced in the context of Valiant's PAC learning model as a way to circumvent the then-existing amount of related negative results [10; 34]. After the first formal proof that boosting is indeed achievable [29], AdaBoost became the first practical and proof-checked boosting algorithm [8; 30]. Boosting was thus born in a machine learning context, but later on, it also emerged in statistics as a way to learn from class residuals computed using the gradient of the loss [9; 21], resulting this time in a flurry of computationally efficient algorithms, still called boosting algorithms, but for which the connection with the original weak/strong learning framework is in general not known.

Our paper draws its boosting connections with AdaBoost's formal lineage. AdaBoost has spurred a long line of work alongside different directions, including statistical consistency [5], noise handling [15; 16], low-resource optimization [22], _etc_. The starting point of our work is a fascinating result in convex optimization establishing a duality between the algorithm and its memory of past iteration's performances, a probability distribution of so-called _weights_ over examples [12]. From this standpoint, AdaBoost solves the dual of the optimization of a Bregman divergence (constructed from the negative Shannon entropy as the generator) between weights subject to zero correlation with the last weak classifier's performance. As a consequence, weights define an exponential family. Indeed, whenever a relative entropy is minimized subject to linear constraints, then the solution is a member of an exponential family of distributions (see _e.g._[2, Section 2.8.1] for an axiomatization of exponential families). AdaBoost's distribution on the examples is a member of a discrete exponential family where the training examples are the finite support of the distribution, sufficient statistics are defined from the weak learners, and the leveraging coefficients are the natural parameters. In summary, there is an intimate relationship between boosting a-la-AdaBoost, exponential families, and Bregman divergences [7; 12; 20] and our work "elevates" these methods above exponential families.

## 3 Definitions

We define the \(t\)-logarithm and \(t\)-exponential [17; Chapter 7],

\[\log_{t}(z)\doteq\frac{1}{1-t}\cdot\left(z^{1-t}-1\right)\quad,\quad\exp_{t}(z )\doteq[1+(1-t)z]_{+}^{1/(1-t)}\quad([z]_{+}\doteq\max\{0,z\}), \tag{1}\]

where the case \(t=1\) is supposed to be the extension by continuity to the \(\log\) and \(\exp\) functions, respectively. To preserve the concavity of \(\log_{t}\) and the convexity of \(\exp_{t}\), we need \(t\geq 0\). In the general case, we also note the asymmetry of the composition: while \(\exp_{t}\log_{t}(z)=z,\forall t\in\mathbb{R}\), we have \(\log_{t}\exp_{t}(z)=z\) for \(t=1\) (\(\forall z\in\mathbb{R}\)), but

\[\log_{t}\exp_{t}(z)=\max\left\{-\frac{1}{1-t},z\right\}\quad(t<1)\quad\mathrm{ and}\quad\log_{t}\exp_{t}(z)=\min\left\{\frac{1}{t-1},z\right\}\quad(t>1).\]

Comparisons between vectors and real-valued functions written on vectors are assumed component-wise. We assume \(t\neq 2\) and define notation \(t\)* \(\doteq 1/(2-t)\). We now define the key set in which we model our weights (boldfaces denote vector notation).

**Definition 3.1**.: _The co-simplex of \(\mathbb{R}^{m}\), \(\tilde{\Delta}_{m}\) is defined as \(\tilde{\Delta}_{m}\doteq\{\mathbf{q}\in\mathbb{R}^{m}:\mathbf{q}\geq\mathbf{0}\wedge\mathbf{1}^ {\top}\mathbf{q}^{1/t\text{*}}=1\}\)._

The letters \(\mathbf{q}\) will be used to denote tems in \(\tilde{\Delta}_{m}\) while \(\mathbf{p}\) denote the co-density \(\mathbf{q}^{\frac{1}{t\text{*}}}\) or any element of the probability simplex. We define the general tempered relative entropy as

\[D_{t}(\mathbf{q}^{\prime}\|\mathbf{q}) = \sum_{i\in[m]}q^{\prime}_{i}\cdot\left(\log_{t}q^{\prime}_{i}-\log _{t}q_{i}\right)-\log_{t-1}q^{\prime}_{i}+\log_{t-1}q_{i}, \tag{2}\]

where \([m]\doteq\{1,...,m\}\). The tempered relative entropy is a Bregman divergence with convex generator \(\varphi_{t}(z)\doteq z\log_{t}z-\log_{t-1}(z)\) (for \(t\in\mathbb{R}\)) and \(\varphi_{t}(z)^{\prime}=\log_{t}(x)\). As \(t\to 1\), \(D_{t}(\mathbf{q},\mathbf{q}^{\prime})\) becomes the relative entropy with generator \(\varphi_{1}(x)=x\log(x)-x\).

## 4 Tempered boosting as tempered entropy projection

We start with a fixed sample \(\mathcal{S}=\{(\mathbf{x}_{i},y_{i}):i\in[m]\}\) where observations \(\mathbf{x}_{i}\) lie in some domain \(\mathcal{X}\) and labels \(y_{i}\) are \(\pm 1\). AdaBoost maintains a distribution \(\mathbf{p}\) over the sample. At the current iteration, this distribution is updated based on a current _weak hypothesis_\(h\in\mathbb{R}^{\mathcal{X}}\) using an exponential update:

\[p^{\prime}_{i}=\frac{p_{i}\cdot\exp(-\mu u_{i})}{\sum_{k}p_{k}\cdot\exp(-\mu u _{k})},\;\text{where}\;u_{i}\doteq y_{i}h(\mathbf{x}_{i}),\mu\in\mathbb{R}.\]

In [12] this update is motivated as minimizing a relative entropy subject to the constraint that \(\mathbf{p}^{\prime}\) is a distribution summing to 1 and \(\mathbf{p}^{\prime\top}\mathbf{u}=0\). Following this blueprint, we create a boosting algorithm maintaining a discrete tem over the sample which is motivated as a constrained minimization of the tempered relative entropy, with a normalization constraint on the co-simplex of \(\mathbb{R}^{m}\):

\[\mathbf{q}^{\prime} \doteq \arg\min_{\mathbf{\tilde{q}}\in\tilde{\Delta}_{m}}\quad D_{t}(\mathbf{ \tilde{q}}\|\mathbf{q}),\quad\text{ with }\mathbf{u}\in\mathbb{R}^{m}.\] \[\mathbf{\tilde{q}}^{\top}\mathbf{u}=0\]

We now show that the solution \(\mathbf{q}^{\prime}\) is a tempered generalization of AdaBoost's exponential update.

**Theorem 1**.: _For all \(t\in\mathbb{R}\backslash\{2\}\), all solutions to (3) have the form_

\[q^{\prime}_{i}=\frac{\exp_{t}(\log_{t}q_{i}-\mu u_{i})}{Z_{t}}\quad\left(- \frac{q_{i}\otimes_{t}\exp_{t}(-\mu u_{i})}{Z_{t}},\;\text{with}\;a\otimes_{t}b \doteq[a^{1-t}+b^{1-t}-1]_{+}^{\frac{1}{1-t}}\right), \tag{4}\]

_where \(Z_{t}\) ensures co-simplex normalization of the co-density. Furthermore, the unknown \(\mu\) satisfies_

\[\mu\in\arg\max-\log_{t}(Z_{t}(\mu))\quad(=\arg\min Z_{t}(\mu)), \tag{5}\]or equivalently is a solution to the nonlinear equation_

\[\mathbf{q}^{\prime}(\mu)^{\top}\mathbf{u} = 0. \tag{6}\]

_Finally, if either (i) \(t\in\mathbb{R}_{>0}\backslash\{2\}\) or (ii) \(t=0\) and \(\mathbf{q}\) is not collinear to \(\mathbf{u}\), then \(Z_{t}(\mu)\) is strictly convex: the solution to (3) is thus unique, and can be found from expression (4) by finding the unique minimizer of (5) or (equivalently) the unique solution to (6)._

(Proof in APP., Section II.1) The \(t\)-product \(\otimes_{t}\), which satisfies \(\exp_{t}(a+b)=\exp_{t}(a)\otimes_{t}\exp_{t}(b)\), was introduced in [18]. Collinearity never happens in our ML setting because \(\mathbf{u}\) contains the edges of a weak classifier: \(\mathbf{q}>0\) and collinearity would imply that \(\pm\) the weak classifier performs perfect classification, and thus defeats the purpose of training an ensemble. \(\forall t\in\mathbb{R}\backslash\{2\}\), we have the simplified expression for the normalization coefficient of the tem and the co-density \(\mathbf{p}^{\prime}\) of \(\mathbf{q}^{\prime}\):

\[Z_{t}=\left\|\exp_{t}\left(\log_{t}\mathbf{q}-\mu\cdot\mathbf{u}\right)\right\|_{1/t \ast}\ ;\ \ p^{\prime}_{i}=\frac{p_{i}\otimes_{t\ast}\exp_{t\ast}\left(- \frac{\mu u_{i}}{t^{\ast}}\right)}{Z^{\prime}_{t}}\ \ \Big{(}\ \text{with}\ Z^{\prime}_{t}\doteq Z^{1/t\ast}_{t}\Big{)}. \tag{7}\]

## 5 Tempered boosting for linear classifiers and clipped linear classifiers

ModelsA model (or classifier) is an element of \(\mathbb{R}^{\mathcal{X}}\). For any model \(H\), its empirical risk over \(\mathcal{S}\) is \(F_{\nicefrac{{0}}{{1}}}(H,\mathcal{S})\doteq(1/m)\cdot\sum_{i}[y_{i}\neq\mathrm{ sign}(H(\mathbf{x}_{i}))]\) where \([.]\), Iverson's bracket [13], is the Boolean value of the inner predicate. We learn linear separators and _clipped_ linear separators. Let \((v_{j})_{j\geq 1}\) be the terms of a series and \(\delta\geq 0\). The clipped sum of the series is:

\[\begin{array}{rcl}\stackrel{{(\delta)}}{{(-\delta)}}\!\!\!\! \sum_{j\in[J]}v_{j}&=&\min\left\{\delta,\max\left\{-\delta,v_{J}+\begin{array}{c} \stackrel{{(\delta)}}{{(-\delta)}}\!\!\!\!\sum_{j\in[J-1]}v_{j} \\ \end{array}\right\}\right\}&\quad(\in[-\delta,\delta]),\ \text{for}\ J>1,\end{array}\]

and we define the base case \(J=1\) by replacing the inner clipped sum with 0. Note that clipped summation is non-commutative, and so is different from clipping in \([-\delta,\delta]\) the whole sum itself2. Given a set of so-called weak hypotheses \(h_{j}\in\mathbb{R}^{\mathcal{X}}\) and leveraging coefficients \(\alpha_{j}\in\mathbb{R}\) (for \(j\in[J]\)), the corresponding linear separators and clipped linear separators are

Footnote 2: Fix for example \(a=-1,b=3,\delta=2\). For \(v_{1}=a,v_{2}=b\), the clipped sum is \(2=-1+3\), but for \(v_{1}=b,v_{2}=a\), the clipped sum becomes \(1=\mathbf{2}-1\).

\[H_{J}(\mathbf{x})\doteq\sum_{j\in[J]}\alpha_{j}h_{j}(\mathbf{x})\quad;\quad H^{(\delta )}_{J}(\mathbf{x})\doteq\begin{array}{c}\stackrel{{(\delta)}}{{(- \delta)}}\!\!\!\sum_{j\in[J]}\alpha_{j}h_{j}(\mathbf{x}).\end{array} \tag{9}\]

Tampered boosting and its general convergenceOur algorithm, \(t\)-AdaBoost, is presented in Algorithm 1, using presentation conventions from [30]. Before analyzing its convergence, several properties are to be noted for \(t\)-AdaBoost: first, it keeps the appealing property, introduced by AdaBoost, that examples receiving the wrong class by the current weak classifier are reweightedhigher (if \(\mu_{j}>0\)). Second, the leveraging coefficients for weak classifiers in the final classifier (\(\alpha_{j}\)s) are not the same as the ones used to update the weights (\(\mu_{j}\)s), unless \(t=1\). Third and last, because of the definition of \(\exp_{t}\) (1), if \(t<1\), tempered weights can switch off and on, _i.e._, become 0 if an example is "too well classified" and then revert back to being \(>0\) if the example becomes wrongly classified by the current weak classifier (if \(\mu_{j}>0\)). To take into account those zeroing weights, we denote \([m]_{j}^{\dagger}\doteq\{i:q_{ji}=0\}\) and \(m_{j}^{\dagger}\doteq\mathrm{Card}([m]_{j}^{\dagger})\) (\(\forall j\in[J]\) and \(\mathrm{Card}\) denotes the cardinal). Let \(R_{j}\doteq\max_{i\notin[m]_{j}^{\dagger}}|y_{i}h_{j}(\boldsymbol{x}_{i})|/q_{ ji}^{1-t}\) and \(q_{j}^{\dagger}\doteq\max_{i\in[m]_{j}^{\dagger}}|y_{i}h_{j}(\boldsymbol{x}_{i}) |^{1/(1-t)}/R_{j}^{1/(1-t)}\). It is worth noting that \(q_{j}^{\dagger}\) is homogeneous to a tempered weight.

**Theorem 2**.: _At iteration \(j\), define the weight function \(q_{ji}^{\prime}=q_{ji}\) if \(i\notin[m]_{j}^{\dagger}\) and \(q_{j}^{\dagger}\) otherwise; set_

\[\rho_{j} \doteq \frac{1}{(1+{m_{j}^{\dagger}}{q_{j}^{\dagger}}^{2-t})R_{j}}\cdot \sum_{i\in[m]}q_{ji}^{\prime}y_{i}h_{j}(\boldsymbol{x}_{i})\quad(\in[-1,1]). \tag{10}\]

_In algorithm \(t\)-AdaBoost, consider the choices (with the convention \(\prod_{k=1}^{0}v_{k}\doteq 1\))_

\[\mu_{j}\doteq-\frac{1}{R_{j}}\cdot\log_{t}\left(\frac{1-\rho_{j}}{M_{1-t}(1- \rho_{j},1+\rho_{j})}\right)\quad,\quad\alpha_{j}\doteq{m^{1-t}}^{*}\cdot\left( \prod_{k=1}^{j-1}Z_{k}\right)^{1-t}\cdot\mu_{j}, \tag{11}\]

_where \(M_{q}(a,b)\doteq((a^{q}+b^{q})/2)^{1/q}\) is the \(q\)-power mean. Then for any \(H\in\{H_{J},H_{J}^{(\nicefrac{{1}}{{1-t}})}\}\), its empirical risk is upperbounded as:_

\[F_{\nicefrac{{0}}{{1}}}(H,\mathcal{S})\leqslant\prod_{j=1}^{J}Z_{tj}^{2-t} \leqslant\prod_{j=1}^{J}\left(1+{m_{j}^{\dagger}}{q_{j}^{\dagger}}^{2-t} \right)\cdot K_{t}(\rho_{j})\quad\left(K_{t}(z)\doteq\frac{1-z^{2}}{M_{1-t}(1- z,1+z)}\right). \tag{12}\]

(Proof in APP., Section II.2) We jointly comment \(t\)-AdaBoost and Theorem 2 in two parts.

**Case \(t\to 1^{-}\):**\(t\)-AdaBoost converges to AdaBoost and Theorem 2 to its convergence analysis: \(t\)-AdaBoost converges to AdaBoost as presented in [30, Figure 1]: the tempered simplex becomes the probability simplex, \(\otimes_{t}\) converges to regular multiplication, weight update (8) becomes AdaBoost's, \(\alpha_{j}\rightarrow\mu_{j}\) in (11) and finally the expression of \(\mu_{j}\) converges to AdaBoost's leveraging coefficient in [30] (\(\lim_{t\to 1}M_{1-t}(a,b)=\sqrt{ab}\)). Even guarantee (12) converges to AdaBoost's popular guarantee of [30, Corollary 1] (\(\lim_{t\to 1}K_{t}(z)=\sqrt{1-z^{2}}\), \(m_{j}^{\dagger}=0\)). Also, in this case, we learn only the unclipped classifier since \(\lim_{t\to 1^{-}}H_{J}^{(\nicefrac{{1}}{{1-t}})}=H_{J}\).

**Case \(t<1\):** Let us first comment on the convergence rate. The proof of Theorem 2 shows that \(K_{t}(z)\leqslant\exp(-z^{2}/(2t^{*}))\). Suppose there is no weight switching, so \(m_{j}^{\dagger}=0,\forall j\) (see Section 7) and, as in the boosting model, suppose there exists \(\gamma>0\) such that \(|\rho_{j}|\geqslant\gamma,\forall j\). Then \(t\)-AdaBoost is guaranteed to attain empirical risk below some \(\varepsilon>0\) after a number of iterations equal to \(J=(2t^{*}/\gamma^{2})\cdot\log(1/\varepsilon)\). \(t^{*}\) being an increasing function of \(t\in[0,1]\), we see that \(t\)-AdaBoost is able to slightly improve upon AdaBoost's celebrated rate [32]. However, \(t^{*}=1/2\) for \(t=0\) so the improvement is just on the hidden constant. This analysis is suited for small values of \(|\rho_{j}|\) and does not reveal an interesting phenomenon for better weak hypotheses. Figure 1 compares \(K_{t}(z)\) curves (\(K_{1}(z)\doteq\lim_{t\to 1}K_{t}(z)=\sqrt{1-z^{2}}\) for AdaBoost, see [30, Corollary 1]), showing the case \(t<1\) can be substantially better, especially when weak hypotheses are not "too weak". If \(m_{j}^{\dagger}>0\), switching weights can impede our convergence _analysis_, though exponential convergence is always possible if \({m_{j}^{\dagger}}{q_{j}^{\dagger}}^{2-t}\) is small enough; also, when it is not, we may in fact have converged to a good model (see APP., Remark 1). A good criterion to train weak hypotheses is then the optimization of the edge \(\rho_{j}\), thus using \(\boldsymbol{q}_{j}^{\prime}\) normalized in the simplex. Other key features of \(t\)-AdaBoost are as follows. First, the weight update and leveraging coefficients of weak classifiers are bounded because \(|\mu_{j}|<1/(R_{j}(1-t))\) (APP., Lemma H) (this is not the case for \(t\to 1^{-}\)). This guarantees that new weights are bounded before normalization (unlike for \(t\to 1^{-}\)). Second, we remark that \(\mu_{j}\neq\alpha_{j}\) if \(t\neq 1\). Factor \(m^{1-t^{*}}\) is added for convergence analysis purposes; we can discard it to train the unclipped classifier: it does not change its empirical risk. This is, however, different for factor \(\prod_{k=1}^{j-1}Z_{k}\): from (12), we conclude that this is an indication of how well the past ensemble performs.

phenomenon that does not occur in boosting, where an excellent weak hypothesis on the current weights can have a leveraging coefficient so large that it wipes out the classification of the past ones. This can be useful to control numerical instabilities.

Extension to marginsA key property of boosting algorithms like AdaBoost is to be able to boost not just the empirical risk but more generally _margins_[19, 30], where a margin integrates both the accuracy of label prediction but also the confidence in prediction (say \(|H|\)). We generalize the margin notion of [19] to the tempered arithmetic and let \(\nu_{t}((\boldsymbol{x},y),H)\doteq\tanh_{t}(yH(\boldsymbol{x})/2)\) denote the margin of \(H\) on example \((\boldsymbol{x},y)\), where \(\tanh_{t}(z)\doteq(1-\exp_{t}(-2z))/(1+\exp_{t}(-2z))(\in[-1,1])\) is the tempered hyperbolic tangent. The objective of minimizing the empirical risk is generalized to minimizing the margin risk, \(F_{t,\theta}(H,\mathcal{S})\doteq(1/m)\cdot\sum_{i}[\nu_{t}((\boldsymbol{x}_{ i},y_{i}),H)\leq\theta]\), where \(\theta\in(-1,1)\). Guarantees on the empirical risk are guarantees on the margin risk for \(\theta=0\) only. In just a few steps, we can generalize Theorem 2 to _all_\(\theta\in(-1,1)\). For space reason, we state the core part of the generalization, from which extending it to a generalization of Theorem 2 is simple.

**Theorem 3**.: _For any \(\theta\in(-1,1)\) and \(t\in[0,1]\), the guarantee of algorithm \(t\)-AdaBoost in Theorem 2 extends to the margin risk, with notations from Theorem 2, via:_

\[F_{t,\theta}(H,\mathcal{S}) \leq \left(\frac{1+\theta}{1-\theta}\right)^{2-t}\prod_{j=1}^{J}Z_{tj} ^{2-t}. \tag{13}\]

(Proof in APP., Section II.3) At a high level, \(t\)-AdaBoost brings similar margin maximization properties as AdaBoost. Digging a bit in (13) reveals an interesting phenomenon for \(t\neq 1\) on how margins are optimized compared to \(t=1\). Pick \(\theta<0\), so we focus on those examples for which the classifier \(H\) has high confidence in its _wrong_ classification. In this case, factor \(((1+\theta)/(1-\theta))^{2-t}\) is increasing as a function of \(t\in[0,1]\) (and this pattern is reversed for \(\theta>0\)). In words, the smaller we pick \(t\in[0,1]\) and the better is the bound in (13), suggesting increased "focus" of \(t\)-AdaBoost on increasing the margins of examples _with low negative margin_ (_e.g._ the most difficult ones) compared to the case \(t=1\).

The tempered exponential lossIn the same way as AdaBoost introduced the now famous exponential loss, (12) recommends to minimize the normalization coefficient, following (7),

\[Z_{tj}^{2-t}(\mu) = \left\|\exp_{t}\left(\log_{t}\boldsymbol{q}_{j}-\mu\cdot \boldsymbol{u}_{j}\right)\right\|_{1/t^{\theta}}^{1/t^{\theta}}\quad\left( \text{with }u_{ji}\doteq y_{i}h_{j}(\boldsymbol{x}_{i})\right). \tag{14}\]

We cannot easily unravel the normalization coefficient to make appear an equivalent generalization of the exponential loss, unless we make several assumptions, one being \(\max_{i}|h_{j}(\boldsymbol{x}_{i})|\) is small enough for any \(j\in[J]\). In this case, we end up with an equivalent criterion to minimize which looks like

\[F_{t}(H,\mathcal{S}) = \frac{1}{m}\cdot\sum_{i}\exp_{t}^{2-t}\left(-y_{i}H(\boldsymbol{x }_{i})\right), \tag{15}\]

Figure 1: Plot of \(K_{t}(z)\) in (12), \(t\in[0,1]\) (the smaller, the better for convergence).

where we have absorbed in \(H\) the factor \(m^{1-t^{\mathsf{R}}}\) appearing in the \(\exp_{t}\) (scaling \(H\) by a positive value does not change its empirical risk). This defines a generalization of the exponential loss which we call the _tempered exponential loss_. Notice that one can choose to minimize \(F_{t}(H,\mathcal{S})\) disregarding any constraint on \(|H|\).

## 6 A broad family of boosting-compliant proper losses for decision trees

Losses for class probability estimationWhen it comes to tabular data, it has long been known that some of the best models to linearly combine with boosting are decision trees (DT, [9]). Decision trees, like other domain-partitioning classifiers, are not trained by minimizing a _surrogate loss_ defined over real-valued predictions, but defined over _class probability estimation_ (CPE, [26]), those estimators being posterior estimation computed at the leaves. Let us introduce a few definitions for those. A CPE loss \(\ell:\{-1,1\}\times[0,1]\rightarrow\mathbb{R}\) is

\[\ell(y,u) \doteq \llbracket y=1\rrbracket\cdot\ell_{1}(u)+\llbracket y=-1 \rrbracket\cdot\ell_{-1}(u). \tag{16}\]

Functions \(\ell_{1},\ell_{-1}\) are called _partial_ losses. The pointwise conditional risk of local guess \(u\in[0,1]\) with respect to a ground truth \(v\in[0,1]\) is:

\[\mathsf{L}\left(u,v\right) \doteq v\cdot\ell_{1}(u)+(1-v)\cdot\ell_{-1}(u). \tag{17}\]

A loss is _proper_ iff for any ground truth \(v\in[0,1]\), \(\mathsf{L}\left(v,v\right)=\inf_{u}\mathsf{L}\left(u,v\right)\), and strictly proper iff \(u=v\) is the sole minimizer [26]. The (pointwise) _Bayes_ risk is \(\underline{L}(v)\doteq\inf_{u}\mathsf{L}\left(u,v\right)\). The log/cross-entropy-loss, square-loss, Matusita loss are examples of CPE losses. One then trains a DT minimizing the expectation of this loss over leaves' posteriors, \(\mathbb{E}_{\lambda}[\underline{L}(p_{\lambda})]\), \(p_{\lambda}\) being the local proportion of positive examples at leaf \(\lambda\) - or equivalently, the local posterior.

Deriving CPE losses from (ada)boostingRecently, it was shown how to derive in a general way a CPE loss to train a DT from the minimization of a surrogate loss with a boosting algorithm [16]. In our case, the surrogate would be \(\tilde{Z}_{tj}\) (14) and the boosting algorithm, \(t\)-AdaBoost. The principle is simple and fits in four steps: (i) show that a DT can equivalently perform simple linear classifications, (ii) use a weak learner that designs splits and the boosting algorithm to fit the leveraging coefficient and compute those in closed form, (iii) simplify the expression of the loss using those, (iv) show that the expression simplified is, in fact, a CPE loss. To get (i), we remark that a DT contains a tree (graph). One can associate to each node a real value. To classify an observation, we sum all reals from the root to a leaf and decide on the class based on the sign of the prediction, just like for any real-valued predictor. Suppose we are at a leaf. What kind of weak hypotheses can create splits "in disguise"? Those can be of the form

\[h_{j}(\mathbf{x}) \doteq \llbracket x_{k}\geq a_{j}\rrbracket\cdot b_{j},\quad a_{j},b_{j }\in\mathbb{R},\]

where the observation variable \(x_{k}\) is assumed real valued for simplicity and the test \(\llbracket x_{k}\geq a_{j}\rrbracket\) splits the leaf's domain in two non-empty subsets. This creates half of the split. \(\overline{h}_{j}(\mathbf{x})\doteq\llbracket x_{k}<a_{j}\rrbracket\cdot-b_{j}\) creates the other half of the split. Remarkably \(h_{j}\) satisfies the weak learning assumption iff \(\overline{h}_{j}\) does [16]. So we get the split design part of (ii). We compute the leveraging coefficients at the new leaves from the surrogate's minimization / boosting algorithm, end up with new real predictions at the new leaves (instead of the original \(b_{j},-b_{j}\)), push those predictions in the surrogate loss for (iii), simplify it and, quite remarkably end up with a loss of the form \(\mathbb{E}_{\lambda}[\mathsf{L}(p_{\lambda})]\), where \(\mathsf{L}\) turns out to be the pointwise Bayes risk \(\underline{L}\) of a proper loss [16] (notation from [26]).

In the case of [16], it is, in fact, granted that we end up with such a "nice" CPE loss because of the choice of the surrogates at the start. In our case, however, nothing grants this _a priori_ if we start from the tempered exponential loss \(\tilde{Z}_{tj}\) (14) so it is legitimate to wonder whether such a chain of derivations (summarized) can happen to reverse engineer an interesting CPE loss:

\[\tilde{Z}_{tj}\stackrel{{?}}{{\mapsto}}\mathsf{L}\stackrel{{?}}{{\mapsto}}\mathsf{L}^{(t)}\stackrel{{?}}{{\mapsto}} \ell_{1}^{(t)};\ell_{-1}^{(t)}\quad(\text{proper? strictly proper? for which $t$s?,...)} \tag{18}\]

When such a complete derivation happens until the partial losses \(\ell_{1};\ell_{-1}\) and their properties, we shall write that minimizing \(\tilde{Z}_{tj}\)_elicits_ the corresponding loss and partial losses.

**Theorem 4**.: _Minimizing \(\tilde{Z}_{tj}\) elicits the CPE loss we define as the **tempered loss**, with partial losses:_

\[\ell_{1}^{(t)}(u)\doteq\left(\frac{1-u}{M_{1-t}(u,1-u)}\right)^{2-t}\quad,\quad \ell_{-1}^{(t)}(u)\doteq\ell_{1}^{(t)}(1-u),\quad(t\in[-\infty,2]). \tag{19}\]

_The tempered loss is symmetric, differentiable, strictly proper for \(t\in(-\infty,2)\) and proper for \(t=-\infty\)._Differentiability means the partial losses are differentiable, and symmetry follows from the relationship between partial losses [20] (the proof, in APP., Section II.4, derives the infinite case, \(\ell_{1}^{(-\infty)}(u)=2\cdot\left[u\leqslant 1/2\right]\)) Let us explicit the Bayes risk of the tempered loss and a key property.

**Lemma 1**.: _The Bayes risk of the tempered loss is (\(M_{q}\) defined in Theorem 2):_

\[\underline{L}^{(t)}(u) = \frac{2u(1-u)}{M_{1-t}(u,1-u)}, \tag{20}\]

_and it satisfies \(\forall u\in[0,1],\forall z\in[2\cdot\min\{u,1-u\},1]\), \(2t\in[-\infty,2]\) such that \(\underline{L}^{(t)}(u)=z\)._

Lemma 1, whose proof is trivial, allows to show a key boosting result: \(t=1\) retrieves Matusita's loss, for which a near-optimal boosting rate is known [11] while \(t=-\infty\) retrieves the empirical risk, which yields the worst possible guarantee [11]. In between, we have, for example, CART's Gini criterion for \(t=0\), which yields an intermediate boosting guarantee. Continuity with respect to \(t\) of the Bayes risks in between the empirical risk and Matusita's loss means the boosting ranges of the tempered loss cover _the full known spectrum of boosting rates_ for \(t\in[-\infty,1]\). We know of no (differentiable and) proper CPE loss with such coverage. Note that (i) this is a non-constructive result as we do not associate a specific \(t\) for a specific rate, and (ii) the state-of-the-art boosting rates for DT induction do not seem to cover the case \(t\in(1,2)\), thus left as an open question.

## 7 Experiments

We have performed experiments on a testbed of 10 UCI domains, whose details are given in APP. (Section A3). Experiments were carried out using a 10-fold stratified cross-validation procedure.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & \multicolumn{1}{c}{perr (not clipped)} & \multicolumn{1}{c}{perr (clipped)} & \multicolumn{1}{c}{min weight} & \multicolumn{1}{c}{max weight} \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 1: Experiments on \(t\)-AdaBoost comparing with AdaBoost (\(t=1\), bullets) on three domains (rows), displaying from left to right the estimated true error of non-clipped and clipped models, and the min and max codensity weights. These domains were chosen to give an example of three different situations: small values of \(t\) perform well (abalone), the best performance is achieved by the largest \(t\) (_e.g._ AdaBoost, qsar), and the worst performance is achieved by the largest \(t\) (adult). Topmost row is without noise (\(\eta=0\)) while the others are with \(10\%\) training noise; \(t\) scale displayed with varying color and width (colormap indicated on each plot). Averages shown for readability: see Table 2 for exhaustive statistical tests.

To compare \(t\)-AdaBoost with AdaBoost, we ran \(t\)-AdaBoost with a first range of values of \(t\in\{0.0,0.2,0.4,0.6,0.8,0.9\}\). This is in the range of values covered by our convergence result for linear separators in Theorem 2. Our results on decision tree induction cover a much wider range, in particular for \(t\in(1,2)\). To assess whether this can be an interesting range to study, we added \(t=1.1\) to the set of tested \(t\) values. When \(t>1\), some extra care is to be put into computations because the weight update becomes unbounded, in a way that is worse than AdaBoost. Indeed, as can be seen from (8), if \(\mu_{j}y_{i}h_{j}(\mathbf{x}_{i})\leq-1/(t-1)\) (the example is badly classified by the current weak hypothesis, assuming wlog \(\mu_{j}>0\)), the weight becomes infinity before renormalization. In our experiments, picking a value of \(t\) close to \(2\) clearly shows this problem, so to be able to still explore whether \(t>1\) can be useful, we picked a value close to \(1\), namely \(t=1.1\), and checked that in our experiments this produced no such numerical issue. We also considered training clipped and not clipped models.

All boosting models were trained for a number of \(J=20\) decision trees (The appendix provides experiments on training bigger sets). Each decision tree is induced using the tempered loss with the corresponding value of \(t\) (see Theorem 4) following the classical top-down template, which consists in growing the current heaviest leaf in the tree and picking the best split for the leaf chosen. We implemented \(t\)-AdaBoost exactly as in Section 5, including computing leveraging coefficients as suggested. Thus, we do not scale models. More details are provided in the appendix. In our experiments, we also included experiments on a phenomenon highlighted more than a decade ago [15] and fine-tuned more recently [16], the fact that a convex booster's model is the weakest link when it has to deal with noise in training data. This is an important task because while the tempered exponential loss is convex, it does not fit into the blueprint loss of [15, Definition 1] because it is not \(C^{1}\) if \(t\neq 1\). One might thus wonder how \(t\)-AdaBoost behaves when training data is affected by noise. Letting \(\eta\) denote the proportion of noisy data in the training sample, we tried \(\eta\in\{0.0,0.1\}\) (The appendix provides experiments on more noise levels). We follow the noise model of [15] and thus independently flip the true label of each example with probability \(\eta\).

For each run, we recorded the average test error and the average maximum and minimum co-density weight. Table 1 presents a subset of the results obtained on three domains. Table 2 presents a more synthetic view in terms of statistical significance of the results for \(t\neq 1\) vs. \(t=1\) (AdaBoost). The table reports only results for \(t\geq 0.6\) for synthesis purposes. Values \(t<0.6\) performed on average slightly worse than the others _but_ on some domains, as the example of abalone suggests in Table 2 (the plots include all values of \(t\) tested in \([0,1.1]\)), we clearly got above-par results for such small values of \(t\), both in terms of final test error but also fast early convergence to low test error. This comment can be generalized to all values of \(t\).

The weights reveal interesting patterns as well. First, perhaps surprisingly, we never encountered the case where weights switch off, regardless of the value of \(t\). The average minimum weight curves of Table 1 generalize to all our tests (see the appendix). This does not rule out the fact that boosting for a much longer number of iterations might lead to weights switching off/on, but the fact that this does not happen at least early during boosting probably comes from the fact that the leveraging coefficients for weights (\(\mu\).) are bounded. Furthermore, their maximal absolute value is all the smaller as \(t\) decreases to 0. Second, there is a pattern that also repeats on the maximum weights, not on all domains but on a large majority of them and can be seen in abalone and adult in Table 1: the maximum weight of AdaBoost tends to increase much more rapidly compared to \(t\)-AdaBoost with \(t<1\). In the latter case, we almost systematically observe that the maximum weight tends to be upperbounded, which is not the case for AdaBoost (the growth of the maximal weight looks almost linear). Having bounded weights could be of help to handle numerical issues of (ada)boosting [14].

Our experiments certainly confirm the boosting nature of \(t\)-AdaBoost if we compare its convergence to that of AdaBoost: more often than not, it is in fact comparable to that of AdaBoost. While this applies broadly for \(t\geq 0.6\), we observed examples where much smaller values (even \(t=0.0\)) could yield such fast convergence. Importantly, this applies to clipped models as well and it is important to notice because it means attaining a low "boosted" error does not come at the price of learning models with large range. This is an interesting property: for \(t=0.0\), we would be guaranteed that the computation of the clipped prediction is always in \([-1,1]\). Generalizing our comment on small values of \(t\) above, we observed that an efficient tuning algorithm for \(t\) could be able to get very substantial leverage over AdaBoost. Table 2 was crafted for a standard limit \(p\)-val of 0.1 and "blurs" the best results that can be obtained. On several domains (winered, abalone, eeg, creditcard, adult), applicable \(p\)-values for which we would conclude that some \(t\neq 1\) performs better than \(t=1\) drop in between \(7E-4\) and \(0.05\). Unsurprisingly, AdaBoost also manages to beat significantly alternative values of \(t\) in several cases. Our experiments with training noise (\(\eta=0.1\)) go in the same direction. Looking at Table 1, one could eventually be tempted to conclude that \(t\) slightly smaller than 1.0 may be a better choice than adaboosting (\(t=1\)), as suggested by our results for \(t=0.9\), but we do not think this produces a general "rule-of-thumb". There is also no apparent "noise-dependent" pattern that would obviously separate the cases \(t<1\) from \(t=1\) even when the tempered exponential loss does not fit to [15]'s theory. Finally, looking at the results for \(t>1\) also yields the same basic conclusions, which suggests that boosting can be attainable outside the range covered by our theory (in particular Theorem 2).

All this brings us to the experimental conclusion that the question does not reside on opposing the case \(t\neq 1\) to the case \(t=1\). Rather, our experiments suggest - pretty much like our theory does - that the actual question resides in how to efficiently _learn_\(t\) on a domain-dependent basis. Our experiments indeed demonstrate that substantial gains could be obtained, to handle overfitting or noise.

## 8 Discussion, limitations and conclusion

AdaBoost is one of the original and simplest Boosting algorithms. In this paper we generalized AdaBoost to maintaining a tempered measure over the examples by minimizing a tempered relative entropy. We kept the setup as simple as possible and therefore focused on generalizing AdaBoost. However more advanced boosting algorithms have been designed based on relative entropy minimization subject to linear constraints. There are versions that constrain the edges of all past hypotheses to be zero [36]. Also, when the maximum margin of the game is larger than zero, then AdaBoost cycles over non-optimal solutions [27]. Later Boosting algorithms provably optimize the margin of the solution by adjusting the constraint value on the dual edge away from zero (see e.g. [24]). Finally, the ELRP-Boost algorithm optimizes a trade off between relative entropy and the edge [35]. We conjecture that all of these orthogonal direction have generalizations to the tempered case as well and are worth exploring.

These are theoretical directions that, if successful, would contribute to bring more tools to the design of rigorous boosting algorithms. This is important because boosting suffers several impediments, not all of which we have mentioned: for example, to get statistical consistency for AdaBoost, it is known that early stopping is mandatory [5]. More generally, non-Lipschitz losses like the exponential loss seem to be harder to handle compared to Lipschitz losses [33] (but they yield in general better convergence rates). The validity of the weak learning assumption of boosting can also be discussed, in particular regarding the negative result of [15] which advocates, beyond just better (ada)boosting, for boosting for _more_ classes of models / architectures [16]. Alongside this direction, we feel that our experiments on noise handling give a preliminary account of the fact that there is no "one \(t\) fits all" case, but a much more in depth analysis is required to elicit / tune a "good" \(t\). This is a crucial issue for noise handling [16], but as we explain in Section 7, this could bring benefits in much wider contexts as well.

\begin{table}
\begin{tabular}{r|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|} \hline \(\eta\) & \multicolumn{8}{c|}{\(0.0\)} & \multicolumn{8}{c|}{\(0.1\)} \\ \(t\) & \(0.6\) & \(0.8\) & \(0.9\) & \(1.1\) & \(0.6\) & \(0.8\) & \(0.9\) & \(1.1\) & \(0.6\) & \(0.8\) & \(0.9\) & \(1.1\) \\
[clipped] & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 \\ \hline \hline \(\#\)better & 2 & 3 & 1 & 2 & 1 & 3 & & & 1 & 1 & 1 & 2 & 2 & 1 & & & \\ \hline \(\#\)equivalent & 5 & 5 & 6 & 6 & 7 & 7 & 6 & 7 & 4 & 8 & 8 & 7 & 8 & 9 & 8 & 10 \\ \hline \(\#\)worse & 3 & 2 & 3 & 2 & 2 & & 4 & 3 & 5 & 1 & 1 & 1 & & & 2 & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Outcomes of student paired \(t\)-tests over 10 UCI domains, with training noise \(\eta\in\{0.0,0.1\}\), for \(t\in\{0.6,0.8,0.9,1.0,1.1\}\) and with / without clipped models. For each triple (\(\eta\), \(t\), [clipped]), we give the number of domains for which the corresponding setting of \(t\)-AdaBoost is statistically better than AdaBoost(\(\#\)better), the number for which it is statistically worse (\(\#\)worse) and the number for which we cannot reject the assumption of identical performances. Threshold \(p-\)val = 0.1.

## Acknowledgments

The authors thank the reviewers for numerous comments that helped improving the paper's content.

## References

* [1] N. Alon, A. Gonen, E. Hazan, and S. Moran. Boosting simple learners. In _STOC'21_, 2021.
* [2] S.-I. Amari. _Information Geometry and Its Applications_. Springer-Verlag, Berlin, 2016.
* [3] E. Amid, F. Nielsen, R. Nock, and M.-K. Warmuth. Optimal transport with tempered exponential measures. _CoRR_, abs/2309.04015, 2023.
* [4] E. Amid, R. Nock, and M.-K. Warmuth. Clustering above exponential families with tempered exponential measures. In _26\({}^{th}\) AISTATS_, 2023.
* [5] P. Bartlett and M. Traskin. Adaboost is consistent. In _NIPS*19_, 2006.
* [6] L. Breiman, J. H. Freidman, R. A. Olshen, and C. J. Stone. _Classification and regression trees_. Wadsworth, 1984.
* [7] M. Collins, R. Schapire, and Y. Singer. Logistic regression, adaboost and Bregman distances. In _Proc. of the 13 \({}^{th}\) International Conference on Computational Learning Theory_, pages 158-169, 2000.
* [8] Y. Freund and R. E. Schapire. A Decision-Theoretic generalization of on-line learning and an application to Boosting. _J. Comp. Syst. Sc._, 55:119-139, 1997.
* [9] J. Friedman, T. Hastie, and R. Tibshirani. Additive Logistic Regression : a Statistical View of Boosting. _Ann. of Stat._, 28:337-374, 2000.
* [10] M.J. Kearns. Thoughts on hypothesis boosting, 1988. ML class project.
* [11] M.J. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. In _Proc. of the 28 \({}^{th}\) ACM STOC_, pages 459-468, 1996.
* [12] J. Kivinen and M.-K. Warmuth. Boosting as entropy projection. In _COLT'99_, pages 134-144, 1999.
* [13] D.-E. Knuth. Two notes on notation. _The American Mathematical Monthly_, 99(5):403-422, 1992.
* [14] R. Kohavi. Improving accuracy by voting classification algorithms: Boosting, bagging, and variants. In _Workshop on Computation-Intensive Machine Learning Techniques_, 1998.
* [15] P.-M. Long and R.-A. Servedio. Random classification noise defeats all convex potential boosters. _MLJ_, 78(3):287-304, 2010.
* [16] Y. Mansour, R. Nock, and R.-C. Williamson. Random classification noise does not defeat all convex potential boosters irrespective of model choice. In _40\({}^{th}\) ICML_, 2023.
* [17] J. Naudts. _Generalized thermostatistics_. Springer, 2011.
* [18] L. Nivanen, A. Le Mehaute, and Q.-A. Wang. Generalized algebra within a nonextensive statistics. _Reports on Mathematical Physics_, 52:437-444, 2003.
* [19] R. Nock and F. Nielsen. A Real Generalization of discrete AdaBoost. _Artificial Intelligence_, 171:25-41, 2007.
* [20] R. Nock and F. Nielsen. On the efficient minimization of classification-calibrated surrogates. In _NIPS*21_, pages 1201-1208, 2008.
* [21] R. Nock and F. Nielsen. The phylogenetic tree of Boosting has a bushy carriage but a single trunk. _PNAS_, 117:8692-8693, 2020.
* [22] R. Nock and R.-C. Williamson. Lossless or quantized boosting with integer arithmetic. In _36\({}^{th}\) ICML_, pages 4829-4838, 2019.
* [23] J. R. Quinlan. _C4.5 : programs for machine learning_. Morgan Kaufmann, 1993.
* [24] G. Ratsch and M.-K. Warmuth. Efficient margin maximizing with boosting. _JMLR_, pages 2131-2152, december 2005.
* [25] M.-D. Reid and R.-C. Williamson. Composite binary losses. _JMLR_, 11:2387-2422, 2010.

* [26] M.-D. Reid and R.-C. Williamson. Information, divergence and risk for binary experiments. _JMLR_, 12:731-817, 2011.
* [27] C. Rudin, I. Daubechies, and R.-E. Schapire. Dynamics of adaboost: cyclic behavior and convergence of margins. _JMLR_, pages 1557-1595, December 2004.
* [28] L.-J. Savage. Elicitation of personal probabilities and expectations. _J. of the Am. Stat. Assoc._, pages 783-801, 1971.
* [29] R. E. Schapire. The strength of weak learnability. _MLJ_, pages 197-227, 1990.
* [30] R. E. Schapire and Y. Singer. Improved boosting algorithms using confidence-rated predictions. _MLJ_, 37:297-336, 1999.
* [31] T. Sypherd, R. Nock, and L. Sankar. Being properly improper. In _39\({}^{th}\) ICML_, 2022.
* [32] M. Telgarsky. A primal-dual convergence analysis of boosting. _JMLR_, 13:561-606, 2012.
* [33] M. Telgarsky. Boosting with the logistic loss is consistent. In _26 \({}^{th}\) COLT_, pages 911-965, 2013.
* [34] L. G. Valiant. A theory of the learnable. _Communications of the ACM_, 27:1134-1142, 1984.
* [35] M.-K. Warmuth, K.-A. Glocer, and S.-V.-N. Vishwanathan. Entropy regularized LPBoost. In _Algorithmic Learning Theory_, pages 256-271. Springer Berlin Heidelberg, 2008.
* [36] M.-K. Warmuth, J. Liao, and G. Ratsch. Totally corrective boosting algorithms that maximize the margin. In _icml '06: proceedings of the 23rd international conference on machine learning_, pages 1001-1008, 2006.

**Appendix**

**Abstract**

This is the Appendix to Paper "Boosting with Tempered Exponential Measures". To differentiate with the numberings in the main file, the numbering of Theorems, Lemmata, Definitions is letter-based (A, B,...).

## Table of contents

**A short primer on Tempered Exponential Measures**

**Supplementary material on proofs**

\(\leftrightarrow\) Proof of Theorem 1

\(\leftrightarrow\) Proof of Theorem 2

\(\leftrightarrow\) Proof of Theorem 3

\(\leftrightarrow\) Proof of Theorem 4

\(\leftrightarrow\)

**Supplementary material on experiments**

Pg 15

\(\leftrightarrow\) Proof of Theorem 1

\(\leftrightarrow\) Proof of Theorem 4

Pg 32A short primer on Tempered Exponential Measures

We describe here the minimal amount of material necessary to understand how our approach to boosting connects to these measures. We refer to [4] for more details. With a slight abuse of notation, we define the perspective transforms \((\log_{t})^{*}(z)\doteq t^{*}.\log_{t*}(z/t^{*})\) and \((\exp_{t})^{*}(z)=t^{*}.\exp_{t*}(z/t^{*})\). Recall that \(t^{*}\doteq 1/(2-t)\).

**Definition A**.: _[_4_]_ _A tempered exponential measure (tem) family is a set of unnormalized densities in which each element admits the following canonical expression:_

\[q_{t|\boldsymbol{\theta}}(\boldsymbol{x})\doteq\frac{\exp_{t}( \boldsymbol{\theta}^{\top}\boldsymbol{\varphi}(\boldsymbol{x}))}{\exp_{t}(G_{t }(\boldsymbol{\theta}))}=\exp_{t}(\boldsymbol{\theta}^{\top}\boldsymbol{ \varphi}(\boldsymbol{x})\ominus_{t}G_{t}(\boldsymbol{\theta}))\quad\left(a \ominus_{t}b\doteq\frac{a-b}{1+(1-t)b}\right), \tag{21}\]

_where \(\boldsymbol{\theta}\) is the element's natural parameter, \(\boldsymbol{\varphi}(\boldsymbol{x})\) is the sufficient statistics and_

\[G_{t}(\boldsymbol{\theta}) = (\log_{t})^{*}\int(\exp_{t})^{*}(\boldsymbol{\theta}^{\top} \boldsymbol{\varphi}(\boldsymbol{x}))\mathrm{d}\xi\]

_is the (convex) cumulant, \(\xi\) being a base measure (implicit)._

Except for \(t=1\) (which reduces a tem family to a classical exponential family), the total mass of a tem is not 1 (but it has an elegant closed form expression [4]). However, the exponentiated \(q_{t|\boldsymbol{\theta}}^{1/t^{*}}\) does sum to 1. In the discrete case, this justifies extending the classical simplex to what we denote as the co-simplex.

**Definition B**.: _The co-simplex of \(\mathbb{R}^{m}\), \(\tilde{\Delta}_{m}\) is defined as \(\tilde{\Delta}_{m}\doteq\{\boldsymbol{q}\in\mathbb{R}^{m}:\boldsymbol{q}\geq \boldsymbol{0}\wedge\boldsymbol{1}^{\top}\boldsymbol{q}^{1/t^{*}}=1\}\)._

The connection between \(t\)-AdaBoost's update and tem's is immediate from the equation's update ((4) in mf). We can show that \(\tilde{\Delta}_{m}\) can also be represented as tems.

**Lemma A**.: \(\tilde{\Delta}_{m}\) _is a (discrete) family of tempered exponential measures._

Proof.: We proceed as in [2, Section 2.2.2] for exponential families: let \(\boldsymbol{q}\in\tilde{\Delta}_{m}\), which we write

\[q(n) \doteq \sum_{i\in[m]}q_{i}\cdot\llbracket i=n\rrbracket,n\in[m]. \tag{22}\]

\(\llbracket\pi\rrbracket\), the Iverson bracket [13], takes value 1 if Boolean predicate \(\pi\) is true (and 0 otherwise). We create \(m-1\) natural parameters and the cumulant,

\[\theta_{i}\doteq\log_{t}\frac{q_{i}}{q_{m}},i\in[m-1]\quad;\quad G_{t}( \boldsymbol{\theta})\doteq\log_{t}\frac{1}{q_{m}},\]

and end up with (22) also matching the atom mass function

\[q(n) = \frac{\exp_{t}\left(\sum_{i\in[m-1]}\theta_{i}\cdot\llbracket i=n \rrbracket\right)}{\exp_{t}G_{t}(\boldsymbol{\theta})},\]

which clearly defines a tempered exponential measure over \([m]\). This ends the proof of Lemma A.

Supplementary material on proofs

### Proof of Theorem 1

To improve readability, we remove dependency in \(t\) in normalization coefficient \(Z\). We use notations from [4, proof of Theorem 3.2] and denote the Lagrangian

\[\mathcal{L} = \Delta(\tilde{\mathbf{q}}\|\mathbf{q})+\lambda\left(\sum_{i}\tilde{q}_{i}^ {1/t^{\mathbf{*}}}-1\right)-\sum_{i}\nu_{i}\tilde{q}_{i}+\mu\sum_{i}\tilde{q}_{i}u _{i}, \tag{23}\]

which yields \(\partial\mathcal{L}/\partial\tilde{q}_{i}=\log_{t}\tilde{q}_{i}-\log_{t}q_{i} +\lambda\tilde{q}_{i}^{1-t}-\nu_{i}+\mu u_{i}\) (\(\lambda\) absorbs factor \(2-t\)), and, rearranging (absorbing factor \(1-t\) in \(\nu_{i}\)),

\[(1+(1-t)\lambda)\tilde{q}_{i}^{1-t} = \nu_{i}+1+(1-t)(\log_{t}q_{i}-\mu u_{i}),\forall i\in[m]. \tag{24}\]

We see that \(\lambda\neq-1/(1-t)\) otherwise the Lagrangian drops its dependence in the unknown. In fact, the solution necessarily has \(1+(1-t)\lambda>0\). To see this, we distinguish two cases: (i) if some \(u_{k}=0\), then since \(\log_{t}q_{k}\geq-1/(1-t)\) there would be no solution to (24) if \(1+(1-t)\lambda<0\) because of the KKT conditions \(\nu_{i}\geq 0,\forall i\in[m]\); (ii) otherwise, if all \(u_{k}\neq 0,\forall k\in[m]\), then there must be two coordinates of different signs otherwise there is no solution to our problem (3) (main file, we must have indeed \(\tilde{\mathbf{q}}\geq 0\) because of the co-simplex constraint). Thus, there exists at least one coordinate \(k\in[m]\) for which \(-(1-t)\mu u_{k}>0\) and since \(\log_{t}q_{k}\geq-1/(1-t)\) (definition of \(\log_{t}\)) and \(\nu_{k}\geq 0\) (KKT conditions), the RHS of (24) for \(i=k\) is \(>0\), preventing \(1+(1-t)\lambda<0\) in the LHS.

We thus have \(1+(1-t)\lambda>0\). The KKT conditions \((\nu_{i}\geq 0,\nu_{i}\tilde{q}_{i}=0,\forall i\in[m])\) yield the following: \(1+(1-t)(\log_{t}q_{i}-\mu u_{i})>0\) imply \(\nu_{i}=0\) and \(1+(1-t)(\log_{t}q_{i}-\mu u_{i})\leq 0\) imply \(\tilde{q}_{i}^{1-t}=0\) so we get the necessary form for the optimum:

\[\tilde{q}_{i} = \frac{\exp_{t}\left(\log_{t}q_{i}-\mu u_{i}\right)}{\exp_{t} \lambda} \tag{25}\] \[= \frac{q_{i}\otimes_{t}\exp_{t}(-\mu u_{i})}{Z_{t}},\]

where \(\lambda\) or \(Z_{t}\doteq\exp_{t}\lambda\) ensures normalisation for the co-density. Note that we have a simplified expression for the co-density:

\[p_{i} = \frac{p_{ji}\otimes_{t}\ast\,\exp_{t}\ast\,(-\mu u_{i}/t^{\ast}) }{Z_{t}^{\infty}}, \tag{26}\]

with \(Z_{t}^{\infty}\doteq Z_{t}^{1/t^{\mathbf{*}}}=\sum_{i}p_{ji}\otimes_{t}\ast\,\exp _{t}\ast\,(-\mu u_{i}/t^{\ast})\). For the analytic form in (25), we can simplify the Lagrangian to a dual form that depends on \(\mu\) solely:

\[\mathcal{D}(\mu) = \Delta(\tilde{\mathbf{q}}(\mu)\|\mathbf{q})+\mu\sum_{i}\tilde{q}_{i}(\mu )u_{i}. \tag{27}\]

The proof of (5) (main file) is based on a key Lemma.

**Lemma B**.: _For any \(\tilde{\mathbf{q}}\) having form (25) such that \(\tilde{\mathbf{q}}^{\top}\mathbf{u}=0\), \(\mathcal{D}(\mu)=-\log_{t}Z_{t}(\mu)\)._

Proof.: For any \(\tilde{\mathbf{q}}\) having form (25), denote

\[[m]_{\ast} \doteq \{i:\tilde{q}_{i}\neq 0\}. \tag{28}\]We first compute (still using \(\lambda\doteq\log_{t}Z_{t}(\mu)\) for short):

\[A \doteq \sum_{i}\tilde{q}_{i}\cdot\log_{t}\tilde{q}_{i} \tag{29}\] \[= \sum_{i\in[m]_{\bullet}}\tilde{q}_{i}\cdot\log_{t}\left(\frac{ \exp_{t}\left(\log_{t}q_{i}-\mu u_{i}\right)}{\exp_{t}\lambda}\right)\] \[= \sum_{i\in[m]_{\bullet}}\tilde{q}_{i}\cdot\left(\frac{1}{1-t} \cdot\left[\frac{1+(1-t)(\log_{t}q_{i}-\mu u_{i})}{1+(1-t)\lambda}-1\right]\right)\] \[= \frac{1}{1-t}\cdot\sum_{i\in[m]_{\bullet}}\tilde{q}_{i}\cdot \left(\frac{q_{i}^{1-t}-(1-t)\mu u_{i}}{1+(1-t)\lambda}\right)-\frac{1}{1-t} \cdot\sum_{i\in[m]_{\bullet}}\tilde{q}_{i}\] \[= -\frac{\mu}{1+(1-t)\lambda}\cdot\sum_{i\in[m]_{\bullet}}\tilde{q }_{i}u_{i}+\frac{1}{(1-t)(1+(1-t)\lambda)}\cdot\sum_{i\in[m]_{\bullet}}\tilde {q}_{i}q_{i}^{1-t}-\frac{1}{1-t}\cdot\sum_{i\in[m]_{\bullet}}\tilde{q}_{i}\] \[= \underbrace{-\frac{\mu}{1+(1-t)\lambda}\cdot\tilde{\boldsymbol{q }}^{\top}\boldsymbol{u}}_{\doteq B}+\underbrace{\frac{1}{(1-t)(1+(1-t)\lambda )}\cdot\sum_{i\in[m]}\tilde{q}_{i}q_{i}^{1-t}}_{\doteq C}-\underbrace{\frac{1} {1-t}\cdot\sum_{i\in[m]}\tilde{q}_{i}}_{\doteq D}.\]

Remark that in the last identity, we have put back summations over the complete set \([m]\) of indices. We note that \(B=0\) because \(\tilde{\boldsymbol{q}}^{\top}\boldsymbol{u}=0\). We then remark that without replacing the expression of \(\tilde{\boldsymbol{q}}\), we have in general for any \(\tilde{\boldsymbol{q}}\in\bar{\Delta}_{m}\):

\[E \doteq \sum_{i\in[m]}\tilde{q}_{i}\cdot\left(\log_{t}\tilde{q}_{i}-\log _{t}q_{i}\right)\] \[= \sum_{i\in[m]}\tilde{q}_{i}\cdot\left(\frac{1}{1-t}\cdot\left( \tilde{q}_{i}^{1-t}-1\right)-\frac{1}{1-t}\cdot\left(q_{i}^{1-t}-1\right)\right)\] \[= \frac{1}{1-t}\cdot\sum_{i\in[m]}\tilde{q}_{i}^{2-t}-\frac{1}{1-t }\cdot\sum_{i\in[m]}\tilde{q}_{i}q_{i}^{1-t}\] \[= \frac{1}{1-t}\cdot\left(1-\sum_{i\in[m]}\tilde{q}_{i}q_{i}^{1-t} \right)\!,\]

and we can check that for any \(\tilde{\boldsymbol{q}},\boldsymbol{q}\in\bar{\Delta}_{m}\), \(E=\Delta(\tilde{\boldsymbol{q}}\|\boldsymbol{q})\). We then develop \(\Delta(\tilde{\boldsymbol{q}}\|\boldsymbol{q})\) with a partial replacement of \(\tilde{\boldsymbol{q}}\) by its expression:

\[\Delta(\tilde{\boldsymbol{q}}\|\boldsymbol{q}) = A-\sum_{i}\tilde{q}_{i}\log_{t}q_{i}\] \[= A-\frac{1}{1-t}\cdot\sum_{i}\tilde{q}_{i}q_{i}^{1-t}+\frac{1}{1 -t}\cdot\sum_{i}\tilde{q}_{i}\] \[= C-\frac{1}{1-t}\cdot\sum_{i}\tilde{q}_{i}q_{i}^{1-t}\] \[= \frac{1}{1-t}\cdot\left(\frac{1}{1+(1-t)\lambda}-1\right)\cdot \sum_{i}\tilde{q}_{i}q_{i}^{1-t}\] \[= -\frac{\lambda}{1+(1-t)\lambda}\cdot\sum_{i}\tilde{q}_{i}q_{i}^{1 -t}\] \[= -\frac{\lambda}{1+(1-t)\lambda}\cdot\left(1-(1-t)\cdot\Delta( \tilde{\boldsymbol{q}}\|\boldsymbol{q})\right).\]

Rearranging gives that for any \(\tilde{\boldsymbol{q}},\boldsymbol{q}\in\bar{\Delta}_{m}\) such that (i) \(\tilde{\boldsymbol{q}}\) has the form (25) for some \(\mu\in\mathbb{R}\) and (ii) \(\tilde{\boldsymbol{q}}^{\top}\boldsymbol{u}=0\),

\[\Delta(\tilde{\boldsymbol{q}}\|\boldsymbol{q}) = -\lambda\] \[= -\log_{t}(Z_{t}),\]as claimed. This ends the proof of Lemma B.

We thus get from the definition of the dual that \(\mu=\arg\max-\log_{t}Z_{t}(\mu)=\arg\min Z_{t}(\mu)\). We have the explicit form for \(Z_{t}\):

\[Z_{t}(\mu) = \left(\sum_{i}\exp_{t}^{2-t}\left(\log_{t}q_{i}-\mu u_{i}\right) \right)^{\frac{1}{2-t}}\] \[= \left(\sum_{i\in[m]_{\bullet}}\exp_{t}^{2-t}\left(\log_{t}q_{i}- \mu u_{i}\right)\right)^{\frac{1}{2-t}},\]

where \([m]_{\bullet}\) is defined in (28). We remark that the last expression is differentiable in \(\mu\), and get

\[Z_{t}^{\prime}(\mu) = \frac{1}{2-t}\cdot\left(\sum_{i\in[m]_{\bullet}}\exp_{t}^{2-t} \left(\log_{t}q_{i}-\mu u_{i}\right)\right)^{-\frac{1-t}{2-t}} \tag{30}\] \[\cdot(2-t)\sum_{i\in[m]_{\bullet}}\exp_{t}^{1-t}\left(\log_{t}q_{ i}-\mu u_{i}\right)\cdot\exp_{t}^{t}\left(\log_{t}q_{i}-\mu u_{i}\right)\cdot- u_{i}\] \[= -Z_{t}^{t-1}\cdot\sum_{i\in[m]_{\bullet}}\exp_{t}\left(\log_{t}q_ {i}-\mu u_{i}\right)\cdot u_{i}\] \[= -Z_{t}^{t}\cdot\sum_{i\in[m]_{\bullet}}\tilde{q}_{i}u_{i}\] \[= -Z_{t}^{t}\cdot\tilde{\boldsymbol{q}}^{\top}\boldsymbol{u},\]

so

\[\frac{\partial-\log_{t}(Z_{t})}{\partial\mu} = -Z_{t}^{-t}Z_{t}^{\prime}\] \[= \tilde{\boldsymbol{q}}(\mu)^{\top}\boldsymbol{u},\]

and we get that any critical point of \(Z_{t}(\mu)\) satisfies \(\tilde{\boldsymbol{q}}(\mu)^{\top}\boldsymbol{u}=0\). A sufficient condition to have just one critical point, being the minimum sought is the strict convexity of \(Z_{t}(\mu)\). The next Lemma provides the proof that it is for all \(t>0\).

**Lemma C**.: \(Z_{t}^{\prime\prime}(\mu)\geqslant t\cdot Z_{t}(\mu)^{2t-1}(\tilde{\boldsymbol {q}}(\mu)^{\top}\boldsymbol{u})^{2}\)_._

Proof.: After simplifications, we have

\[Z_{t}^{3-2t}\cdot Z_{t}^{\prime\prime} = (t-1)\cdot\left(\sum_{i\in[m]}\exp_{t}\left(\log_{t}q_{i}-\mu u_{ i}\right)\cdot u_{i}\right)^{2}\] \[+\left(\sum_{i\in[m]}\exp_{t}^{2-t}\left(\log_{t}q_{i}-\mu u_{i} \right)\right)\cdot\left(\sum_{i\in[m]}\exp_{t}^{t}\left(\log_{t}q_{i}-\mu u_ {i}\right)\cdot u_{i}^{2}\right)\] \[= (t-1)\cdot\sum_{i,k\in[m]}Q_{i}Q_{k}u_{i}u_{k}+\sum_{i,k\in[m]}Q_ {i}^{2-t}Q_{k}^{t}u_{k}^{2}, \tag{33}\]

where we have let \(Q_{i}\doteq\exp_{t}\left(\log_{t}q_{i}-\mu u_{i}\right)\geqslant 0\). Since \(a^{2}+b^{2}\geqslant 2ab\), we note that for any \(i\neq k\),

\[Q_{i}^{2-t}Q_{k}^{t}u_{k}^{2}+Q_{k}^{2-t}Q_{i}^{t}u_{i}^{2} \geqslant 2\sqrt{Q_{i}^{2-t}Q_{k}^{t}Q_{k}^{2-t}Q_{i}^{t}}u_{i}u_{k} \tag{34}\] \[=2Q_{i}Q_{k}u_{i}u_{k},\]so we split (33) in two terms and get

\[Z_{t}^{3-2t}\cdot Z_{t}^{\prime\prime} = (t-1)\cdot\sum_{i\in[m]}Q_{i}^{2}u_{i}^{2}+\sum_{i\in[m]}Q_{i}^{2-t} Q_{i}^{t}u_{i}^{2} \tag{35}\] \[+\sum_{i,k\in[m],i<k}2(t-1)Q_{i}Q_{k}u_{i}u_{k}+\sum_{i,k\in[m],i< k}Q_{i}^{2-t}Q_{k}^{t}u_{k}^{2}+Q_{k}^{2-t}Q_{i}^{t}u_{i}^{2}\] \[= t\cdot\sum_{i\in[m]}Q_{i}^{2}u_{i}^{2}\] \[+\sum_{i,k\in[m],i<k}2(t-1)Q_{i}Q_{k}u_{i}u_{k}+\sum_{i,k\in[m],i< k}Q_{i}^{2-t}Q_{k}^{t}u_{k}^{2}+Q_{k}^{2-t}Q_{i}^{t}u_{i}^{2}\] \[\geqslant t\cdot\sum_{i\in[m]}Q_{i}^{2}u_{i}^{2}+2t\cdot\sum_{i,k\in[m],i <k}Q_{i}Q_{k}u_{i}u_{k}\] \[=t\cdot\left(\sum_{i\in[m]}\exp_{t}\left(\log_{t}q_{i}-\mu u_{i} \right)\cdot u_{i}\right)^{2}\] \[= tZ_{t}^{2}\cdot(\tilde{\mathbf{q}}^{\top}\mathbf{u})^{2},\]

where we have used (34) in (35). Since \(Z_{t}(\mu)>0\), we get the statement of Lemma C after reorganising (36).

Lemma C shows the strict convexity of \(Z_{t}(\mu)\) for any \(t>0\). The case \(t=0\) follows by direct differentiation: we get after simplification

\[Z_{t}^{\prime\prime}(\mu) = \frac{\left(\sum_{i\in[m]}u_{i}^{2}\right)\cdot\left(\sum_{i\in[ m]}(q_{i}-\mu u_{i})^{2}\right)-\left(\sum_{i\in[m]}(q_{i}-\mu u_{i})u_{i} \right)^{2}}{\left(\sum_{i\in[m]}(q_{i}-\mu u_{i})^{2}\right)^{\frac{3}{2}}}.\]

Cauchy-Schwartz inequality allows to conclude that \(Z_{t}^{\prime\prime}(\mu)\geqslant 0\) and is in fact \(>0\)_unless_\(\tilde{\mathbf{q}}\) is collinear to \(\mathbf{u}\). This completes the proof of Theorem 1.

### Proof of Theorem 2

The proof involves several arguments, organized into several subsections. Some are more general than what is strictly needed for the proof of the Theorem, on purpose.

#### ii.i.2.1 Clipped summations

For any \(\delta\geqslant 0\), we define clipped summations of the sequence of ordered elements \(v_{1},v_{2},...,v_{J}\): if \(J>1\),

\[{}^{(\delta)}\!\sum_{j=1}^{J}v_{j}\doteq\min\left\{v_{J}+\sum_{j=1}^{(\delta)} \!\sum_{j=1}^{J-1}v_{j},\delta\right\}\quad,\quad{}_{(-\delta)}\!\sum_{j=1}^{ J}v_{j}=\max\left\{v_{J}+\sum_{(-\delta)}\!\sum_{j=1}^{J-1}v_{j},-\delta\right\}, \tag{37}\]

and the base case (\(J=1\)) is obtained by replacing the inner sum by 0. We also define the doubly clipped summation:

\[{}^{(\delta)}\!\sum_{(-\delta)}\!\sum_{j=1}^{J}v_{j}=\max\left\{\min\left\{v_{ J}+\sum_{(-\delta)}^{(\delta)}\!\sum_{j=1}^{J-1}v_{j},\delta\right\},-\delta \right\},\]

with the same convention for the base case. We prove a series of simple but useful properties of the clipped summation.

**Lemma D**.: _The following properties hold true for clipped summation:_

1. _(doubly) clipped summations are noncommutative;_2. _(doubly) clipped summations are ordinary summation in the limit: for any_ \(J\geqslant 1\) _and any sequence_ \(v_{1},v_{2},...,v_{J}\)_,_ \[\lim_{\delta\rightarrow+\infty}\,^{(\delta)}\sum_{j=1}^{J}v_{j}=\lim_{\delta \rightarrow+\infty}\,^{J}\sum_{(-\delta)}^{J}v_{j}=\lim_{\delta\rightarrow+ \infty}\,^{(\delta)}\sum_{(-\delta)}^{J}v_{j}=\sum_{j=1}^{J}v_{j}\]
3. _clipped summations sandwich ordinary summation and the doubly clipped summation: for any_ \(\delta\geqslant 0\)_, any_ \(J\geqslant 1\) _and any sequence_ \(v_{1},v_{2},...,v_{J}\)_,_ \[\,^{(\delta)}\!\!\sum_{j=1}^{J}v_{j}\leqslant\sum_{j=1}^{J}v_{j}\leqslant\,_{ (-\delta)}\!\!\sum_{j=1}^{J}v_{j}\quad;\quad\sum_{j=1}^{(\delta)}\!\!\sum_{(- \delta)}^{J}\sum_{j=1}^{J}v_{j}\leqslant\,_{(-\delta)}\!\!\sum_{j=1}^{J}v_{j}\]

Proof.: Noncommutativity follows from simple counterexamples: for example, for \(v\doteq-1\) and \(w\doteq 2\), if we fix \(v_{1}\doteq v,v_{2}\doteq w\), then \(\,^{(0)}\!\!\sum_{j=1}^{2}v_{j}=1\) while \(\,^{(0)}\!\!\sum_{j=1}^{2}v_{3-j}=-1\). Property [2.] is trivial. The set of leftmost inequalities of property [3.] can be shown by induction, noting the base case is trivial and otherwise, using the induction hypothesis in the leftmost inequality,

\[\,^{(\delta)}\!\!\sum_{j=1}^{J+1}v_{j}\doteq\min\left\{v_{J+1}+\,\,\,^{(\delta) }\!\!\sum_{j=1}^{J}v_{j},\delta\right\}\leqslant\min\left\{v_{J+1}+\sum_{j=1}^ {J}v_{j},\delta\right\}\leqslant v_{J+1}+\sum_{j=1}^{J}v_{j}=\sum_{j=1}^{J+1}v _{j},\]

and similarly

\[\,^{(-\delta)}\!\!\sum_{j=1}^{J+1}v_{j} \doteq \max\left\{v_{J+1}+\,\,\,^{J}\!\!\sum_{(-\delta)}^{J}\!\!\sum_{j} ^{J},-\delta\right\}\] \[\geqslant \max\left\{v_{J+1}+\sum_{j=1}^{J}v_{j},-\delta\right\}\geqslant v _{J+1}+\sum_{j=1}^{J}v_{j}=\sum_{j=1}^{J+1}v_{j}.\]

A similar argument holds for the set of rightmost inequalities: for example, the induction's general case holds

\[\,^{(\delta)}\!\!\sum_{j=1}^{J+1}v_{j} \doteq \min\left\{v_{J+1}+\,\,\,^{(\delta)}\!\!\sum_{j=1}^{J}v_{j},\delta\right\}\] \[\leqslant \min\left\{v_{J+1}+\,\,\,^{(\delta)}\!\!\sum_{(-\delta)}^{J}\!\! \sum_{j}^{J}v_{j},\delta\right\}\] \[\leqslant \max\left\{\min\left\{v_{J+1}+\,\,\,^{(\delta)}\!\!\sum_{j=1}^{J} v_{j},\delta\right\},-\delta\right\}=\,\,^{(\delta)}\!\!\sum_{(-\delta)}^{J}\!\! \sum_{j=1}^{J}v_{j}.\]

for the leftmost inequality. This ends the proof of Lemma D.

#### ii.ii.2.2 Unravelling weights

**Lemma E**.: _Define_

\[v_{j} \doteq m^{1-t^{\ast}}\cdot\left(\prod_{k=1}^{j-1}Z_{tk}\right)^{1-t} \cdot\mu_{j}\quad\left(\text{convention}:\prod_{k=1}^{0}u_{k}\doteq 1\right). \tag{38}\]

_Then \(\forall J\geqslant 1\), weights unravel as:_

\[q_{(J+1)i} = \left\{\begin{array}{cc}\frac{1}{m^{t^{\ast}}\prod_{j=1}^{J}Z _{tj}}\cdot\exp_{t}\left(-\,^{(\nicefrac{{1}}{{1}}-\,^{J})}\!\sum_{j=1}^{J}v_{ j}u_{ji}\right)&\text{ if }\quad t<1\\ \frac{1}{m^{t^{\ast}}\prod_{j=1}^{J}Z_{tj}}\cdot\exp_{t}\left(-\,^{(-\nicefrac {{1}}{{1}}-\,^{J})}\!\sum_{j=1}^{J}v_{j}u_{ji}\right)&\text{ if }\quad t>1\end{array}\right..\]Proof.: We start for the case \(t<1\). We proceed by induction, noting first that the normalization constraint for the initial weights imposes \(q_{1i}=1/m^{1/(2-t)}=1/m^{t^{\bullet}}\) and so (using \((1-t)t^{\bullet}=1-t^{\bullet}\))

\[q_{2i} = \frac{\exp_{t}(\log_{t}q_{1i}-\mu_{1}u_{1i})}{Z_{1}}\] \[= \frac{1}{Z_{1}}\cdot\left[1+(1-t)\cdot\left(\frac{1}{1-t}\cdot \left(\frac{1}{m^{\frac{1-t}{2-t}}}-1\right)-\mu_{1}u_{1i}\right)\right]_{+}^{ \frac{1}{1-t}}\] \[= \frac{1}{Z_{1}}\cdot\left[\frac{1}{m^{1-t^{\bullet}}}-(1-t)\mu_{ 1}u_{1i}\right]_{+}^{\frac{1}{1-t}}\] \[= \frac{1}{m^{t^{\bullet}}Z_{1}}\cdot\left[1-(1-t)m^{1-t^{\bullet} }\mu_{1}u_{1i}\right]_{+}^{\frac{1}{1-t}}\] \[= \frac{1}{m^{t^{\bullet}}Z_{1}}\cdot\exp_{t}\left(-\sum_{j=1}^{1} v_{j}u_{ji}\right),\]

completing the base case. Using the induction hypothesis, we unravel at iteration \(J+1\):

\[q_{(J+1)i}\] \[= \frac{\exp_{t}(\log_{t}q_{Ji}-\mu_{J}u_{Ji})}{Z_{J}}\] \[= \frac{\exp_{t}\left(\log_{t}\left(\frac{1}{m^{t^{\bullet}}\prod_{ j=1}^{J-1}Z_{ij}}\cdot\exp_{t}\left(-\sum_{j=1}^{J-1}v_{j}u_{ji}\right)\right)- \mu_{J}u_{Ji}\right)}{Z_{J}}\] \[= \frac{1}{Z_{J}}\cdot\exp_{t}\left(\frac{\max\left\{-\frac{1}{1-t },-\frac{(\nicefrac{{1}}{{1-t}})}{\sum_{j=1}^{J-1}v_{j}u_{ji}}\right\}-\log_{t }\left(m^{t^{\bullet}}\prod_{j=1}^{J-1}Z_{tj}\right)}{1+(1-t)\log_{t}\left(m^{ t^{\bullet}}\prod_{j=1}^{J-1}Z_{tj}\right)}-\mu_{J}u_{Ji}\right)\] \[= \frac{1}{Z_{J}}\cdot\left[\begin{array}{c}1+\frac{(1-t)\cdot \max\left\{-\frac{1}{1-t},-\frac{(\nicefrac{{1}}{{1-t}})}{\sum_{j=1}^{J-1}v_{j }u_{ji}}\right\}-(1-t)\log_{t}\left(m^{t^{\bullet}}\prod_{j=1}^{J-1}Z_{tj} \right)}{1+(1-t)\log_{t}\left(m^{t^{\bullet}}\prod_{j=1}^{J-1}Z_{tj}\right)} \\ -(1-t)\mu_{J}u_{Ji}\end{array}\right]\] \[= \frac{1}{Z_{J}}\cdot\left[\begin{array}{c}1+\frac{(1-t)\cdot \max\left\{-\frac{1}{1-t},-\frac{(\nicefrac{{1}}{{1-t}})}{\sum_{j=1}^{J-1}v_{j }u_{ji}}\right\}-\left(\left(m^{t^{\bullet}}\prod_{j=1}^{J-1}Z_{tj}\right)^{1- t}-1\right)}{\left(m^{t^{\bullet}}\prod_{j=1}^{J-1}Z_{tj}\right)^{1-t}}\end{array} \right]_{+}^{\frac{1}{1-t}},\]

which simplifies into (using \((1-t)t^{\bullet}=1-t^{\bullet}\))

\[q_{(J+1)i}\] \[= \frac{1}{m^{t^{\bullet}}\prod_{j=1}^{J}Z_{tj}}\cdot\exp_{t}\left( -S_{J}\right),\]with

\[S_{J} \doteq \min\left\{-\max\left\{-\frac{1}{1-t},-\sum_{j=1}^{J-1}v_{j}u_{ji} \right\}+v_{J}u_{Ji},\frac{1}{1-t}\right\}\] \[= \min\left\{v_{J}u_{Ji}+\min\left\{\frac{1}{1-t},\sum_{j=1}^{J-1}v _{j}u_{ji}\right\},\frac{1}{1-t}\right\}\] \[= \min\left\{v_{J}u_{Ji}+\sqrt[(\nicefrac{{1}}{{1-t}})\sum_{j=1}^{J -1}v_{j}u_{ji},\frac{1}{1-t}\right\}\] \[\doteq \sqrt[(\nicefrac{{1}}{{1-t}})\sum_{j=1}^{J}v_{j}u_{ji}\]

(we used twice the definition of clipped summation), which completes the proof of Lemma E for \(t<1\).

We now treat the case \(t>1\). The base induction is equivalent, while unraveling gives, instead of (39):

\[q_{(J+1)i}\] \[= \frac{1}{m^{t*}\prod_{j=1}^{J}Z_{tj}}\cdot\left[1+(1-t)\cdot \left(\min\left\{-\frac{1}{1-t},-\sum_{-(\nicefrac{{1}}{{t-1}})\sum_{j=1}^{J -1}v_{j}u_{ji}\right\}-v_{J}u_{Ji}\right)\right]_{+}^{\frac{1}{1-t}}\] \[= \frac{1}{m^{t*}\prod_{j=1}^{J}Z_{tj}}\cdot\exp_{t}\left(-S_{J} \right),\]

and, this time,

\[S_{J} \doteq \max\left\{-\min\left\{-\frac{1}{1-t},-\sum_{-(\nicefrac{{1}}{{t -1}})\sum_{j=1}^{J-1}v_{j}u_{ji}\right\}+v_{J}u_{Ji},-\frac{1}{t-1}\right\} \tag{40}\] \[= \max\left\{v_{J}u_{Ji}+\max\left\{-\frac{1}{t-1},\ -\sum_{-( \nicefrac{{1}}{{t-1}})\sum_{j=1}^{J-1}v_{j}u_{ji}\right\},-\frac{1}{t-1}\right\}\] (41) \[= \max\left\{v_{J}u_{Ji}+\sum_{-(\nicefrac{{1}}{{t-1}})\sum_{j=1}^{ J-1}v_{j}u_{ji},-\frac{1}{t-1}\right\}\] (42) \[\doteq \sqrt[(\nicefrac{{1}}{{t-1}})\sum_{j=1}^{J}v_{j}u_{ji}, \tag{43}\]

which completes the proof of Lemma E.

#### ii.ii.2.3 Introducing classifiers

Ordinary linear separatorsSuppose we have a classifier

\[H_{J}(\boldsymbol{x}) \doteq \sum_{j=1}^{J}\beta_{j}^{1-t}\mu_{j}\cdot h_{j}(\boldsymbol{x}), \quad\beta_{j}\doteq m^{t*}\prod_{k=1}^{j-1}Z_{tk},\]

where \(\mu_{j}\in\mathbb{R},\forall j\in[J]\). We remark that \([\![z\neq r]\!]\leq\exp_{t}^{2-t}(-zr)\) for any \(t\leq 2\) and \(z,r\in\mathbb{R}\), and \(z\mapsto\exp_{t}^{2-t}(-z)\) is decreasing for any \(t\leq 2\), so using [3.] in Lemma D, we get for our training sample \(\mathcal{S}\doteq\{(\mathbf{x}_{i},y_{i}),i\in[m]\}\) and any \(t<1\) (from Lemma E),

\[\frac{1}{m}\cdot\sum_{i\in[m]}\llbracket\mathrm{sign}(H_{J}(\mathbf{x} _{i}))\neq y_{i}\rrbracket\] \[\leq \sum_{i\in[m]}\frac{\exp_{t}^{2-t}\left(-\sum_{j=1}^{J}m^{1-t^{ \mathfrak{k}}}\left(\prod_{k=1}^{j-1}Z_{tk}\right)^{1-t}\mu_{j}\cdot y_{i}h_{j }(\mathbf{x}_{i})\right)}{m}\] \[\leq \sum_{i\in[m]}\frac{\exp_{t}^{2-t}\left(-\sfrac{\sfrac{\sfrac{ \sfrac{\sfrac{\sfrac{\sfrac{\sfrac{\sfrac{\sfrac{\sfrac{\sfrac{\sfrac{\sfrac{\sfrac{ \sfrac{\sfrac{\s{\s{\s{\s{\s{\s{\s{\s{\s{\s{\s{\s}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}} {\right.\right\right\right\}} \right\right\right\right\} \right\right\}{\]\]\]\]\]\]\]\]\]\]\]\] \] \] \] \[\begin{\begin{}{}\begin{}\begin{}{}\{}\{}\{}\}{} \{}\{}\{}\{}}{}\{}}{}\{}}{}{}}{}{}}{}{}}{}{}}{}{}{}}{}{}{}{}}{}{}{}{}{}}{}{}{}{}{}}{}{}{}{}{}}{}{}{}{}{}{}}{}{}{}{}{}}{}{}{}{}{}}{}{}{}{}}{}{}{}{}{}}{}{}{}{}{}{}}{}{}{}}{}{}{}{}{}}{}{}{}{}}{}{}{}{}}{}{}{}{}{}}{}{}{}{}{}}{}{}{}{}{}{}}{}{}{}{}{}{}{}{}}{}{}{}{}{}{}{}}{}{}{}{}{}{}{}{}}{}{}{}{}{}{}{}{}{}{}}{}{}{}{}{}{}{}{}{}{}{}{}}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}}{}{}{}{}{}{}{}{}{}{}{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}{{}}}{{{}}}{{{}}}{{}}{{}}{{}}{{}}{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}}{{{{}}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{}}}{{{}}}{{{}}}{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{}}}{{{}}}{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{{}}}}{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{}}}}{{{{{}}}

We can now replace (44) by

\[\frac{1}{m}\cdot\sum_{i\in[m]}\big{[}\mathrm{sign}(H^{(\nicefrac{{ 1}}{{1-i}})}_{J}(\mathbf{x}_{i}))\neq y_{i}\big{]}\] \[\leq \sum_{i\in[m]}\frac{\exp_{t}^{2-t}\left(-y_{i}\cdot\frac{\binom{ \nicefrac{{ 1}}{{1-i}}}{{-i}}}{\sum_{j=1}^{J}m^{1-t}\mathfrak{*}\left(\prod_{k=1}^{j-1}Z_{ tk}\right)^{1-t}\mu_{j}\cdot h_{j}(\mathbf{x}_{i})\right)}}{m}\] \[=\sum_{i\in[m]}\frac{\exp_{t}^{2-t}\left(-\frac{\binom{ \nicefrac{{ 1}}{{1-i}}}{{-(\nicefrac{{ 1}}{{1-i}})}}}{\sum_{j=1}^{J}m^{1-t}\mathfrak{*}\left(\prod_{k=1}^{j-1}Z_{tk} \right)^{1-t}\mu_{j}\cdot y_{i}h_{j}(\mathbf{x}_{i})\right)}}{m}\] \[\leq \sum_{i\in[m]}\frac{\exp_{t}^{2-t}\left(-\frac{\binom{ \nicefrac{{ 1}}{{1-i}}}{{-1}}}{\sum_{j=1}^{J}m^{1-t}\mathfrak{*}\left(\prod_{k=1}^{j-1}Z_{ tk}\right)^{1-t}\mu_{j}\cdot y_{i}h_{j}(\mathbf{x}_{i})\right)}}{m}\] \[=\sum_{i\in[m]}\frac{\exp_{t}^{2-t}\left(-\frac{\binom{ \nicefrac{{ 1}}{{1-i}}}{{-1}}}{\sum_{j=1}^{J}v_{j}u_{ji}}\right)}{m}.\]

The first identity has used the fact that \(y_{i}\in\{-1,1\}\), so it can be folded in the doubly clipped summation without changing its value, and the second inequality used [3.] in Lemma D. This directly leads us to the following Lemma.

**Lemma G**.: _For any \(t<1\) and any clipped linear separator_

\[H^{(\nicefrac{{ 1}}{{1-i}})}_{J}(\mathbf{x}) \doteq \stackrel{{\binom{\nicefrac{{ 1}}{{1-i}}}{{-i}}}}{\sum_{j=1}^{J}\beta_{j}^{1-t}\mu_{j}\cdot h_{j}(\mathbf{x})}, \quad\left(\beta_{j}=m^{t\ast}\prod_{k=1}^{j-1}Z_{tk},\mu_{j}\in\mathbb{R},h_{ j}\in\mathbb{R}^{\mathcal{X}},\forall j\in[J]\right),\]

_where \(Z_{tk}\) is the normalization coefficient of \(\mathbf{q}\) in (25) with \(u_{ji}\doteq y_{i}h_{j}(\mathbf{x}_{i})\),_

\[\frac{1}{m}\cdot\sum_{i\in[m]}\big{[}\mathrm{sign}(H^{(\nicefrac{ { 1}}{{1-i}})}_{J}(\mathbf{x}_{i}))\neq y_{i}\big{]} \leq \prod_{j=1}^{J}Z_{tj}^{2-t}. \tag{49}\]

#### ii.ii.2.4 Geometric convergence of the empirical risk

To get the right-hand side of (47) and (49) as small as possible, we can independently compute each \(\mu_{j}\) so as to minimize

\[Z_{tj}^{2-t}(\mu) \doteq \sum_{i\in[m]}\exp_{t}^{2-t}\left(\log_{t}q_{ji}-\mu u_{ji}\right). \tag{50}\]

We proceed in two steps, first computing a convenient upperbound for (50), and then finding the \(\mu\) that minimizes this upperbound.

**Step 1**: We distinguish two cases depending on weight \(q_{ji}\). Let \([m]_{j}^{+}\doteq\{i:q_{ji}>0\}\) and \([m]_{j}^{+}\doteq\{i:q_{ji}=0\}\):

**Case 1**: \(i\in[m]_{j}^{+}\). Let \(r_{ji}=u_{ji}/q_{ji}^{1-t}\) and suppose \(R_{j}>0\) is a real that satisfies

\[|r_{ji}| \leq R_{j},\forall i\in[m]_{j}^{+}. \tag{51}\]

For any convex function \(f\) defined on \([-1,1]\), we have \(f(z)\leq((1+z)/2)\cdot f(1)+((1-z)/2)\cdot f(-1),\forall z\in[-1,1]\) (the straight line is the chord crossing \(f\) at \(z=-1,1\)). Because\[z\mapsto[1-z]_{+}^{\frac{2-t}{t}}\text{ is convex for }t\leq 2,\text{ for any }i\in[m]_{j}^{+}\] \[\exp_{t}^{2-t}\left(\log_{t}q_{ji}-\mu u_{ji}\right)\] \[= \left[q_{ji}^{1-t}-(1-t)\mu u_{ji}\right]_{+}^{\frac{2-t}{t}}\] \[= q_{ji}^{2-t}\cdot\left[1-(1-t)\mu R_{j}\cdot\frac{r_{ji}}{R_{j} }\right]_{+}^{\frac{2-t}{t}}\] \[\leq q_{ji}^{2-t}\cdot\frac{R_{j}+r_{ji}}{2R_{j}}\left[1-(1-t)\mu R_{ j}\right]_{+}^{\frac{2-t}{t}}+q_{ji}^{2-t}\cdot\frac{R_{j}-r_{ji}}{2R_{j}} \left[1+(1-t)\mu R_{j}\right]_{+}^{\frac{2-t}{t}}\] \[=\frac{q_{ji}^{2-t}R_{j}+q_{ji}u_{ji}}{2R_{j}}\left[1-(1-t)\mu R_{ j}\right]_{+}^{\frac{2-t}{t}}+\frac{q_{ji}^{2-t}R_{j}-q_{ji}u_{ji}}{2R_{j}} \left[1+(1-t)\mu R_{j}\right]_{+}^{\frac{2-t}{t}}.\]

**Case 2**: \(i\in[m]_{j}^{\dagger}\). Let \(q_{j}^{\dagger}>0\) be a real that satisfies

\[\frac{|u_{ji}|}{q_{j}^{\dagger 1-t}}<R_{j},\forall i\in[m]_{j}^{\dagger}. \tag{52}\]

Using the same technique as in case 1, we find for any \(i\in[m]_{j}^{\dagger}\)

\[\exp_{t}^{2-t}\left(\log_{t}q_{ji}-\mu u_{ji}\right)\] \[= \exp_{t}^{2-t}\left(-\frac{1}{1-t}-\mu u_{ji}\right)\] \[= \left[-(1-t)\mu u_{ji}\right]_{+}^{\frac{2-t}{t-t}}\] \[\leq \left[q_{j}^{\dagger 1-t}-(1-t)\mu u_{ji}\right]_{+}^{\frac{2-t}{t -t}}\] \[\leq \frac{q_{j}^{\dagger 2-t}R_{j}+q_{ji}^{\dagger}u_{ji}}{2R_{j}} \left[1-(1-t)\mu R_{j}\right]_{+}^{\frac{2-t}{t-t}}+\frac{q_{j}^{\dagger 2-t}R_{j}-q_{j}^{ \dagger}u_{ji}}{2R_{j}}\left[1+(1-t)\mu R_{j}\right]_{+}^{\frac{2-t}{t-t}}.\]

Folding both cases into one and letting

\[q^{\prime}{}_{ji} \doteq \left\{\begin{array}{ccc}q_{ji}&\text{ if }&i\in[m]_{j}^{+}\\ q_{j}^{\dagger}&\text{ if }&i\in[m]_{j}^{\dagger}\end{array}\right., \tag{53}\]

we get after summation, using \(m_{j}^{\dagger}\doteq\mathrm{Card}([m]_{j}^{\dagger})\) and

\[\rho_{j} \doteq \frac{1}{(1+m_{j}^{\dagger}q_{j}^{\dagger 2-t})R_{j}}\cdot\sum_{i \in[m]}q^{\prime}{}_{ji}u_{ji}\quad(\in[-1,1]), \tag{54}\]

that

\[Z_{tj}^{2-t}(\mu)\] \[\leq \frac{(1+m_{j}^{\dagger}q_{j}^{\dagger 2-t})R_{j}}{2R_{j}} \cdot\left((1+\rho_{j})\left[1-(1-t)\mu R_{j}\right]_{+}^{\frac{2-t}{t}}+(1- \rho_{j})\left[1+(1-t)\mu R_{j}\right]_{+}^{\frac{2-t}{t}}\right) \tag{55}\] \[=\frac{1+m_{j}^{\dagger}q_{j}^{\dagger 2-t}}{2}\cdot\left((1+ \rho_{j})\cdot\exp_{t}^{2-t}(-\mu R_{j})+(1-\rho_{j})\cdot\exp_{t}^{2-t}\left( \mu R_{j}\right)\right).\]

**Step 2**: we have our upperbound for (50). We now compute the minimizer \(\mu^{\ast}\) of (55). If this minimizer satisfies

\[|\mu^{\ast}| < \frac{1}{R_{j}|1-t|}, \tag{56}\]

then it can be found by ordinary differentiation, as the solution to

\[(1-\rho_{j})\cdot\exp_{t}\left(\mu^{\ast}R_{j}\right)-(1+\rho_{j}) \cdot\exp_{t}\left(-\mu^{\ast}R_{j}\right) = 0,\]which is equivalently

\[\frac{\exp_{t}\left(-\mu^{\mbox{*}}R_{j}\right)}{\exp_{t}\left(\mu^{ \mbox{*}}R_{j}\right)} = \exp_{t}\left(-\mu^{\mbox{*}}R_{j}\ominus_{t}\mu^{\mbox{*}}R_{j}\right)\] \[= \frac{1-\rho_{j}}{1+\rho_{j}},\]

where we recall \(a\ominus_{t}b\doteq(a-b)/(1+(1-t)b)\). Solving it yields

\[\mu^{\mbox{*}} = \frac{1}{R_{j}}\cdot-\frac{1}{1-t}\cdot\left(\frac{(1-\rho_{j})^{ 1-t}-(1+\rho_{j})^{1-t}}{(1-\rho_{j})^{1-t}+(1+\rho_{j})^{1-t}}\right)\] \[= \frac{1}{R_{j}}\cdot-\frac{1}{1-t}\cdot\left(\frac{2(1-\rho_{j}) ^{1-t}}{(1-\rho_{j})^{1-t}+(1+\rho_{j})^{1-t}}-1\right)\] \[= -\frac{1}{R_{j}}\cdot\log_{t}\left(\frac{1-\rho_{j}}{M_{1-t}(1- \rho_{j},1+\rho_{j})}\right),\]

where \(M_{q}(a,b)\doteq((a^{q}+b^{q})/2)^{1/q}\) is the power mean with exponent \(q\). We now check (56).

**Lemma H**.: _For any \(t\in\mathbb{R}\), let_

\[\mu_{j} \doteq -\frac{1}{R_{j}}\cdot\log_{t}\left(\frac{1-\rho_{j}}{M_{1-t}(1- \rho_{j},1+\rho_{j})}\right). \tag{57}\]

_Then \(|\mu_{j}|\leqslant 1/(R_{j}|1-t|)\)._

Proof.: Equivalently, we must show

\[\left|\log_{t}\left(\frac{1-z}{M_{1-t}(1-z,1+z)}\right)\right| \leqslant \frac{1}{|1-t|},\forall z\in[-1,1],\]

which is equivalent to showing

\[\left|\frac{2(1-z)^{1-t}}{(1-z)^{1-t}+(1+z)^{1-t}}-1\right|\left( =\left|\frac{1-\left(\frac{1+z}{1-z}\right)^{1-t}}{1+\left(\frac{1+z}{1-z} \right)^{1-t}}\right|\right) \leqslant 1,\forall z\in[-1,1].\]

Define function \(f(z,t)\doteq(1-z^{1-t})/(1+z^{1-t})\) over \(\mathbb{R}_{\geqslant 0}\times\mathbb{R}\): it is easy to check that for \(t\leqslant 1,f(z,t)\in[-1,1]\), and the symmetry \(f(z,t)=-f(z,2-t)\) also allows to conclude that for \(t\geqslant 1,f(z,t)\in[-1,1]\). This ends the proof of Lemma H.

For the expression of \(\mu_{j}\) in (57), we get from (55) the upperbound on \(Z_{tj}^{2-t}(\mu_{j})\):

\[Z_{tj}^{2-t}(\mu_{j}) \leqslant \frac{1+m_{j}^{\dagger}q_{j}^{\dagger^{2-t}}}{2}\cdot\left((1+ \rho_{j})\cdot\exp_{t}^{2-t}\left(-\mu_{j}R_{j}\right)+(1-\rho_{j})\cdot\exp _{t}^{2-t}\left(\mu_{j}R_{j}\right)\right)\] \[=\frac{1+m_{j}^{\dagger}q_{j}^{\dagger^{2-t}}}{2}\cdot\left(\frac {(1+\rho_{j})(1-\rho_{j})^{2-t}}{M_{1-t}^{2-t}(1-\rho_{j},1+\rho_{j})}+\frac{( 1-\rho_{j})(1+\rho_{j})^{2-t}}{M_{1-t}^{2-t}(1-\rho_{j},1+\rho_{j})}\right)\] \[= \left(1+m_{j}^{\dagger}q_{j}^{\dagger^{2-t}}\right)\cdot\frac{(1- \rho_{j}^{2})M_{1-t}^{1-t}(1-\rho_{j},1+\rho_{j})}{M_{1-t}^{2-t}(1-\rho_{j},1 +\rho_{j})}\] \[= \left(1+m_{j}^{\dagger}q_{j}^{\dagger^{2-t}}\right)\cdot\frac{1- \rho_{j}^{2}}{M_{1-t}(1-\rho_{j},1+\rho_{j})}.\]

We conclude that for both sets of classifiers defined in Lemmata F and G, with the choice of \(\mu_{j}\) in (57), we get

\[\frac{1}{m}\cdot\sum_{i\in[m]}[\mbox{sign}(H(\mathbf{x}_ {i}))\neq y_{i}] \leqslant \prod_{j=1}^{J}\left(1+m_{j}^{\dagger}q_{j}^{\dagger^{2-t}}\right) \cdot\frac{1-\rho_{j}^{2}}{M_{1-t}(1-\rho_{j},1+\rho_{j})},\forall H\in\{H_{J}, H_{J}^{\left(j_{1-t}\right)}\}.\]To complete the proof of Theorem 2, we just need to elicit the best \(R_{j}\) (51) and \(q_{j}^{\dagger}\) (52); looking at their constraints suggests

\[R_{j} \doteq \max_{i\notin[m]_{j}^{\dagger}}\frac{|y_{i}h_{j}(\boldsymbol{x}_{i })|}{q_{ji}^{1-t}},\] \[q_{j}^{\dagger} \doteq \frac{\max_{i\in[m]_{j}^{\dagger}}|y_{i}h_{j}(\boldsymbol{x}_{i}) |^{1/(1-t)}}{R_{j}^{1/(1-t)}}.\]

This completes the proof of Theorem 2. We complete the proof by two Lemmata of additional useful results in the context of Algorithm \(t\)-AdaBoost, and finally an important remark on the interpretation of Theorem 2.

**Lemma I**.: _The following holds true: (i) \(\rho_{j}\in[-1,1]\); (ii) if, among indexes not in \([m]_{j}^{\dagger}\), there exists at least one index with \(u_{ji}>0\) and one index with \(u_{ji}<0\), then for any \(\mu\neq 0\), \(Z_{ij}^{2-t}(\mu)>0\) in (50) (in words, the new weigh vector \(\boldsymbol{q}_{j+1}\) cannot be the null vector before normalization)._

Proof.: To show (i) for \(\rho_{j}\leqslant 1\), we write (using \(u_{ji}\doteq y_{i}h_{j}(\boldsymbol{x}_{i}),\forall i\in[m]\) for short),

\[(1+m_{j}^{\dagger}q_{j}^{\dagger^{2-t}})R_{j}\cdot\rho_{j} = \sum_{i\in[m]}q_{j}^{\prime}u_{ji}u_{ji}\] \[\leqslant \sum_{i\in[m]}q_{j}^{\prime}{}_{ji}|u_{ji}|\] \[=\sum_{i\in[m]_{j}^{+}}q_{ji}^{2-t}\cdot\frac{|u_{ji}|}{q_{ji}^{1 -t}}+q_{j}^{\dagger^{2-t}}\cdot\frac{\sum_{i\in[m]_{j}^{\dagger}}|u_{ji}|}{q_ {j}^{\dagger^{1-t}}}\] \[\leqslant R_{j}\cdot\underbrace{\sum_{i\in[m]_{j}^{+}}q_{ji}^{2-t}}_{=1}+ q_{j}^{\dagger^{2-t}}\cdot\frac{R_{j}\sum_{i\in[m]_{j}^{\dagger}}|u_{ji}|}{ \max_{i\in[m]_{j}^{\dagger}}|u_{ji}|}\] \[\leqslant R_{j}+q_{j}^{\dagger^{2-t}}m_{j}^{\dagger}R_{j}=(1+m_{j}^{ \dagger}q_{j}^{\dagger^{2-t}})R_{j},\]

showing \(\rho_{j}\leqslant 1\). Showing \(\rho_{j}\geqslant-1\) proceeds in the same way. Property (ii) is trivial.

**Lemma J**.: \[K_{t}(z) \leqslant \exp\left(-\left(1-\frac{t}{2}\right)\cdot z^{2}\right).\]

Proof.: We remark that for \(t\in[0,1),z\geqslant 0\), \(K_{t}^{\prime}(z)\) is concave and \(K_{t}^{\prime\prime}(0)=-(2-t)\), so \(K_{t}^{\prime}(z)\leqslant-(2-t)z,\forall z\geqslant 0\), from which it follows by integration

\[K_{t}(z) \leqslant 1-\left(1-\frac{t}{2}\right)\cdot z^{2},\]

and since \(1-z\leqslant\exp(-z)\), we get the statement of the Lemma.

**Remark 1**.: _The interpretation of Theorem 2 for \(t<1\) are simplified to the case where there is no weight switching, i.e. \(m_{j}^{\dagger}=0,\forall j\). While we have never observed weight switching in our experiments - perhaps owing to the fact that we did never boost for a very long number of iterations or just because our weak classifiers, decision trees, were in fact not so weak -, it is interesting, from a theoretical standpoint, to comment on convergence when this happens. Let \(Q_{j}=1+m_{j}^{\dagger}(q_{j}^{\dagger})^{2-t}\) and \(\tilde{\rho}_{j}=Q_{j}\rho_{j}\) (Notations from Theorem 2). We note that \(\tilde{\rho}_{j}\approx\beta\cdot\mathbb{E}_{\boldsymbol{p}_{j}}[y_{i}h_{j}( \boldsymbol{x}_{i})]\), where \(\boldsymbol{p}_{j}\) lives on the simplex and \(|yh|\leqslant 1,\beta\leqslant 1\). Using Lemma J and (12) (main file), to keep geometric convergence, it is roughly sufficient that \(Q_{j}\log Q_{j}\leqslant(\tilde{\rho}_{j})^{2}/(2t^{*})\). Since \(q_{j}^{\dagger}\) is homogeneous to a tempered weight, one would expect in general \(m_{j}^{\dagger}(q_{j}^{\dagger})^{2-t}\leqslant 1\), so using the Taylor approximation \(Q_{j}\log Q_{j}\approx-1+Q_{j}\), one gets the refined sufficient condition for geometric convergence_

\[m_{j}^{\dagger}(q_{j}^{\dagger})^{2-t} \leqslant (\tilde{\rho}_{j})^{2}/(2t^{*})=O((\tilde{\rho}_{j})^{2}).\]

_What does that imply? We have two cases:_* _If this holds, then we have geometric convergence;_
* _if it does not hold, then for a "large" number of training examples, we must have_ \(q_{ji}=0\) _which, because of the formula for_ \(\boldsymbol{q}\)__(_8_) implies that all these examples receive the right class with a sufficiently large margin. Breaking geometric convergence in this case is not an issue: we already have a good ensemble._

### Proof of Theorem 3

Starting from the proof of Theorem 2, we indicate the additional steps to get to the proof of Theorem 3. The key is to remark that our margin formulation has the following logical convenience:

\[\llbracket\nu_{t}((\boldsymbol{x}_{i},y_{i}),H)\leq\theta\rrbracket = \llbracket-yH(\boldsymbol{x})+\log_{t}\left(\frac{1+\theta}{1- \theta}\right)-(1-t)yH(\boldsymbol{x})\log_{t}\left(\frac{1+\theta}{1-\theta} \right)\geq 0\rrbracket\] \[= \llbracket(-yH(\boldsymbol{x}))\oplus_{t}\log_{t}\left(\frac{1+ \theta}{1-\theta}\right)\geq 0\rrbracket.\]

We then remark that since \(\llbracket z\geq 0\rrbracket\leq\exp_{t}^{2-t}(z)\), we get

\[\llbracket\nu_{t}((\boldsymbol{x}_{i},y_{i}),H)\leq\theta\rrbracket \leq \exp_{t}^{2-t}\left(\left(-yH(\boldsymbol{x})\right)\oplus_{t} \log_{t}\left(\frac{1+\theta}{1-\theta}\right)\right)\] \[=\exp_{t}^{2-t}\left(\log_{t}\left(\frac{1+\theta}{1-\theta} \right)\right)\cdot\exp_{t}^{2-t}(-yH(\boldsymbol{x}))\] \[= \left(\frac{1+\theta}{1-\theta}\right)^{2-t}\cdot\exp_{t}^{2-t}( -yH(\boldsymbol{x})).\]

We then just have to branch to (44), replacing the \(\llbracket\text{sign}(H_{J}(\boldsymbol{x}_{i}))\neq y_{i}\rrbracket\)s by \(\llbracket\nu_{t}((\boldsymbol{x}_{i},y_{i}),H)\leq\theta\rrbracket\), which yields in lieu of (46) the sought inequality:

\[F_{t,\theta}(H,\mathcal{S}) \leq \left(\frac{1+\theta}{1-\theta}\right)^{2-t}\prod_{j=1}^{J}Z_{tj }^{2-t}. \tag{58}\]

### Proof of Theorem 4

The proof proceeds in three parts. Part **(A)** makes a brief recall on encoding linear classifiers with decision trees. Part **(B)** solves (6) in mf, _i.e._ finds boosting's leveraging coefficients as solution of:

\[\boldsymbol{q}(\mu)^{\top}\boldsymbol{u} = 0. \tag{59}\]

we then simplify the loss obtained and elicit the conditional Bayes risk of the tempered loss, _i.e._ (20) in mf. Part **(C)** elicits the partial losses and shows properness and related properties.

Part **(A): encoding linear models with a tree architecture**We use the reduction trick of **(author?)**[16] to design a decision tree (DT) boosting procedure, find out the (concave) loss equivalently minimized, just like in classical top-down DT induction algorithms [6]. The trick is simple: a DT can be thought of as a set of constant linear classifiers. The prediction is the sum of predictions put at all nodes. Boosting fits those predictions at the nodes and percolating those to leaves gets a standard DT with real predictions at the leaves. Figure 2 provides a detailed description of the procedure. Let \(\lambda\) denote a leaf node of the current tree \(H\), with \(H_{\lambda}\in\mathbb{R}\) the function it implements for leaf \(\lambda\). If \(\mathrm{parent}(\lambda)\) denotes its parent node (assuming wlog it is not the root node), we have

\[H_{\lambda} \doteq H_{\mathrm{parent}(\lambda)}+\mu_{\lambda}h_{\lambda}, \tag{60}\]

Part **(B): eliciting the Bayes risk of the tempered loss**With our simple classifiers at hand, the tempered exponential loss \(Z_{tj}^{2-t}\) in (14) (mf) can be simplified to loss

\[L(H) \doteq \sum_{i}\exp_{t}^{2-t}\left(\log_{t}q_{1i}-y_{i}H_{\lambda( \boldsymbol{x}_{i})}\right) \tag{61}\] \[= \sum_{\lambda\in\Lambda(H)}m_{\lambda}^{+}\exp_{t}^{2-t}\left( \log_{t}q_{1i}-H_{\lambda}\right)+m_{\lambda}^{-}\exp_{t}^{2-t}\left(\log_{t} q_{1i}+H_{\lambda}\right),\]where \(\lambda(\mathbf{x})\) is the leaf reached by observation \(\mathbf{x}\) and \(\lambda(H)\) its set of leaf nodes of \(H\), and \(H_{\lambda}\) sums all relevant values in (60). Also, \(m_{\lambda}^{+},m_{\lambda}^{-}\) denote the cardinal of positive and negative examples at \(\lambda\) and \(p_{\lambda}\doteq m_{\lambda}^{+}/(m_{\lambda}^{+}+m_{\lambda}^{-})\) the local proportion of positive examples at \(\lambda\), and finally \(r_{\lambda}\doteq(m_{\lambda}^{+}+m_{\lambda}^{-})/m\) the total proportion of examples reaching \(\lambda\).

**Theorem A**.: _If we compute \(\mu_{\lambda}\) the solution of (59), we end up with the prediction \(H_{\lambda}\):_

\[H_{\lambda} = \frac{q_{1i}^{1-t}}{1-t}\cdot\frac{\left(\frac{m_{\lambda}^{+}}{ m_{\lambda}^{-}}\right)^{1-t}-1}{\left(\frac{m_{\lambda}^{+}}{m_{\lambda}^{-}} \right)^{1-t}+1} \tag{62}\] \[= \frac{q_{1i}^{1-t}}{1-t}\cdot\frac{p_{\lambda}^{1-t}-(1-p_{ \lambda})^{1-t}}{p_{\lambda}^{1-t}+(1-p_{\lambda})^{1-t}}, \tag{63}\]

_and the loss of the decision tree equals:_

\[L(H) = \sum_{\lambda\in\Lambda(H)}r_{\lambda}\cdot\frac{2p_{\lambda}(1-p _{\lambda})}{M_{1-t}(p_{\lambda},1-p_{\lambda})}, \tag{64}\] \[= \mathbb{E}_{\lambda}[\underline{L}^{(t)}(p_{\lambda})]. \tag{65}\]

Proof.: To compute \(\mu_{\lambda}\), (6) is reduced to the examples reaching \(\lambda\), that is, it simplifies to

\[m_{\lambda}^{+}\exp_{t}\left(\log_{t}q_{1i}-H_{\mathrm{parent}( \lambda)}-R_{\lambda}\mu_{\lambda}h_{\lambda}\right) = m_{\lambda}^{-}\exp_{t}\left(\log_{t}q_{1i}+H_{\mathrm{parent}( \lambda)}+R_{\lambda}\mu_{\lambda}h_{\lambda}\right), \tag{66}\]

that we solve for \(\mu_{\lambda}\). Equivalently,

\[\frac{\exp_{t}\left(\log_{t}q_{1i}+H_{\mathrm{parent}(\lambda)}+ R_{\lambda}\mu_{\lambda}h_{\lambda}\right)}{\exp_{t}\left(\log_{t}q_{1i}-H_{ \mathrm{parent}(\lambda)}-R_{\lambda}\mu_{\lambda}h_{\lambda}\right)} = \frac{m_{\lambda}^{+}}{m_{\lambda}^{-}},\]

or, using \(\exp_{t}(u)/\exp_{t}(v)=\exp_{t}(u\ominus_{t}v)\),

\[\frac{2H_{\mathrm{parent}(\lambda)}+2R_{\lambda}\mu_{\lambda}h_ {\lambda}}{1+(1-t)(\log_{t}q_{1i}-H_{\mathrm{parent}(\lambda)}-R_{\lambda}\mu_ {\lambda}h_{\lambda})} = \log_{t}\left(\frac{m_{\lambda}^{+}}{m_{\lambda}^{-}}\right),\]

after reorganizing:

\[R_{\lambda}\mu_{\lambda}h_{\lambda} = \frac{(1+(1-t)(\log_{t}q_{1i}-H_{\mathrm{parent}(\lambda)}))\cdot \log_{t}\left(\frac{m_{\lambda}^{+}}{m_{\lambda}^{-}}\right)-2H_{\mathrm{ parent}(\lambda)}}{2+(1-t)\log_{t}\left(\frac{m_{\lambda}^{+}}{m_{\lambda}^{-}} \right)},\]

Figure 2: The weak learner provides weak hypotheses of the form \([\![x_{k}\geq a_{j}]\!]\cdot b_{j}\). From the boosting standpoint, this weak hypothesis is "as good as" the weak hypothesis \(\overline{h}_{j}(\mathbf{x})\doteq[x_{k}<a_{j}]\!]\cdot-b_{j}\). The predicates of both are used to craft a split, _e.g._ for the root (in our depiction, \(b_{3}=-b_{2}\)) and then solving (59) provides the leveraging coefficients \(\mu_{\cdot}\). We then repeat this for as many splits as necessary. At the end, we can "percolate" nodes reals towards the leaves below and get an equivalent classifier that resembles a decision tree (right). See [16] for further details.

which yields the prediction at \(\lambda\):

\[H_{\lambda} = H_{\rm parent(\lambda)}+\frac{(1+(1-t)(\log_{t}q_{1i}-H_{\rm parent( \lambda)}))\cdot\log_{t}\left(\frac{m_{\lambda}^{+}}{m_{\lambda}^{-}}\right)-2H_ {\rm parent(\lambda)}}{2+(1-t)\log_{t}\left(\frac{m_{\lambda}^{+}}{m_{\lambda} ^{-}}\right)} \tag{67}\] \[= \frac{(1+(1-t)\log_{t}q_{1i})\cdot\log_{t}\left(\frac{m_{\lambda}^ {+}}{m_{\lambda}^{-}}\right)}{2+(1-t)\log_{t}\left(\frac{m_{\lambda}^{+}}{m_{ \lambda}^{-}}\right)}\] \[= q_{1i}^{1-t}\cdot\frac{\log_{t}\left(\frac{m_{\lambda}^{+}}{m_{ \lambda}^{-}}\right)}{2+(1-t)\log_{t}\left(\frac{m_{\lambda}^{+}}{m_{\lambda} ^{-}}\right)}\] (69) \[= \frac{q_{1i}^{1-t}}{1-t}\cdot\frac{\left(\frac{m_{\lambda}^{+}}{m _{\lambda}^{-}}\right)^{1-t}-1}{\left(\frac{m_{\lambda}^{+}}{m_{\lambda}^{+}} \right)^{1-t}+1}\] (70) \[= \frac{q_{1i}^{1-t}}{1-t}\cdot\frac{p_{\lambda}^{1-t}-(1-p_{ \lambda})^{1-t}}{p_{\lambda}^{1-t}+(1-p_{\lambda})^{1-t}}. \tag{71}\]

We plug \(H_{\lambda}\) back in the loss for all leaves and get, using \(q_{1i}=1/m^{1/(2-t)}\):

\[L(H) = \sum_{\lambda\in\Lambda(H)}\left\{\begin{array}{c}m_{\lambda}^{ +}\exp_{t}^{2-t}\left(\log_{t}q_{1i}-\frac{q_{1i}^{1-t}}{1-t}\cdot\frac{p_{ \lambda}^{1-t}-(1-p_{\lambda})^{1-t}}{p_{\lambda}^{1-t}+(1-p_{\lambda})^{1-t} }\right)\\ +m_{\lambda}^{-}\exp_{t}^{2-t}\left(\log_{t}q_{1i}+\frac{q_{1i}^{1-t}}{1-t} \cdot\frac{p_{\lambda}^{1-t}-(1-p_{\lambda})^{1-t}}{p_{\lambda}^{1-t}+(1-p_{ \lambda})^{1-t}}\right)\end{array}\right.. \tag{72}\]

We simplify. First,

\[m_{\lambda}^{+}\exp_{t}^{2-t}\left(\log_{t}q_{1i}-\frac{q_{1i}^{ 1-t}}{1-t}\cdot\frac{p_{\lambda}^{1-t}-(1-p_{\lambda})^{1-t}}{p_{\lambda}^{1- t}+(1-p_{\lambda})^{1-t}}\right) \tag{73}\] \[= m_{\lambda}^{+}\left[q_{1i}^{1-t}\cdot\left(1-\frac{p_{\lambda} ^{1-t}-(1-p_{\lambda})^{1-t}}{p_{\lambda}^{1-t}+(1-p_{\lambda})^{1-t}}\right) \right]_{+}^{\frac{2-t}{1-t}}\] \[= \frac{m_{\lambda}^{+}}{m}\cdot\left[\frac{2(1-p_{\lambda})^{1-t} }{p_{\lambda}^{1-t}+(1-p_{\lambda})^{1-t}}\right]_{+}^{\frac{2-t}{1-t}}\] \[= \frac{m_{\lambda}^{+}}{m}\cdot\left(\frac{1-p_{\lambda}}{M_{1-t} (p_{\lambda},1-p_{\lambda})}\right)^{2-t}, \tag{74}\]

and then

\[m_{\lambda}^{-}\exp_{t}^{2-t}\left(\log_{t}q_{1i}+\frac{q_{1i}^{ 1-t}}{1-t}\cdot\frac{p_{\lambda}^{1-t}-(1-p_{\lambda})^{1-t}}{p_{\lambda}^{1- t}+(1-p_{\lambda})^{1-t}}\right) \tag{75}\] \[= m_{\lambda}^{-}\left[q_{1i}^{1-t}\cdot\left(1+\frac{p_{\lambda} ^{1-t}-(1-p_{\lambda})^{1-t}}{p_{\lambda}^{1-t}+(1-p_{\lambda})^{1-t}}\right) \right]_{+}^{\frac{2-t}{1-t}}\] \[= \frac{m_{\lambda}^{-}}{m}\cdot\left[\frac{2p_{\lambda}^{1-t}}{p_ {\lambda}^{1-t}+(1-p_{\lambda})^{1-t}}\right]_{+}^{\frac{2-t}{1-t}}\] \[= \frac{m_{\lambda}^{-}}{m}\cdot\left(\frac{p_{\lambda}}{M_{1-t}(p_ {\lambda},1-p_{\lambda})}\right)^{2-t}, \tag{76}\]and we can simplify the loss,

\[L(H) = \sum_{\lambda\in\Lambda(H)}r_{\lambda}p_{\lambda}\left(\frac{1-p_{ \lambda}}{M_{1-t}(p_{\lambda},1-p_{\lambda})}\right)^{2-t}+r_{\lambda}(1-p_{ \lambda})\left(\frac{p_{\lambda}}{M_{1-t}(p_{\lambda},1-p_{\lambda})}\right)^{2 -t} \tag{77}\] \[= \sum_{\lambda\in\Lambda(H)}r_{\lambda}\cdot\frac{p_{\lambda}(1-p_{ \lambda})^{2-t}+(1-p_{\lambda})p_{\lambda}^{2-t}}{M_{1-t}^{2-t}(p_{\lambda},1- p_{\lambda})}\] (78) \[= \sum_{\lambda\in\Lambda(H)}r_{\lambda}\cdot\frac{p_{\lambda}(1-p_{ \lambda})\cdot(p_{\lambda}^{1-t}+(1-p_{\lambda})^{1-t})}{M_{1-t}^{2-t}(p_{ \lambda},1-p_{\lambda})}\] (79) \[= \sum_{\lambda\in\Lambda(H)}r_{\lambda}\cdot\frac{2p_{\lambda}(1-p _{\lambda})\cdot M_{1-t}^{1-t}(p_{\lambda},1-p_{\lambda})}{M_{1-t}^{2-t}(p_{ \lambda},1-p_{\lambda})}\] (80) \[= \sum_{\lambda\in\Lambda(H)}r_{\lambda}\cdot\frac{2p_{\lambda}(1-p _{\lambda})}{M_{1-t}(p_{\lambda},1-p_{\lambda})}, \tag{81}\]

as claimed. This ends the proof of Theorem A.

Part (C): partial losses and their propertiesThe proof relies on the following Theorem. We recall that a loss is symmetric iff its partial losses satisfy \(\ell_{1}(u)=\ell_{-1}(1-u),\forall u\in[0,1]\)[20] and differentiable iff its partial losses are differentiable.

**Theorem B**.: _Suppose \(t<2\). A set of partial losses having the conditional Bayes risk \(\underline{L}^{(t)}\) in (65) are_

\[\ell_{1}^{(t)}(u)\doteq\left(\frac{1-u}{M_{1-t}(u,1-u)}\right)^{2- t}\quad,\quad\ell_{-1}^{(t)}(u)\doteq\ell_{1}^{(t)}(1-u). \tag{82}\]

_The tempered loss is then symmetric and differentiable. It is strictly proper for any \(t\in(-\infty,2)\) and proper for \(t=-\infty\)._

Proof.: Symmetry and differentiability are straightforward. To check strict properness, we analyze the cases for \(t\neq 1\) (otherwise, it is Matusita's loss, and thus strictly proper), we compute the solution \(u\) to

\[\frac{\partial}{\partial u}L\left(u,v\right) = 0. \tag{83}\]

To this end, let \(N(u)\doteq v(1-u)^{2-t}+(1-v)u^{2-t}\) and the \(q\)-sum

\[S_{q}(a,b) \doteq (a^{q}+b^{q})^{1/q}=2^{1/q}\cdot M_{q}(a,b). \tag{84}\]

We also let \(D(u)\doteq S_{1-t}^{2-t}(u,1-u)\). Noting \(L^{(t)}(u,v)=2^{\frac{2-t}{1-t}}\cdot N(u)/D(u)\) and \(D(u)\neq 0,\forall u\in[0,1]\), the set of solutions of (83) are the set of solutions to \(N^{\prime}(u)D(u)=N(u)D^{\prime}(u)\), which boils down, after simplification, to

\[((1-v)u^{1-t}-v(1-u)^{1-t})S_{1-t}^{2-t}(u,1-u)\] \[= (v(1-u)^{2-t}+(1-v)u^{2-t})(u^{-t}-(1-u)^{-t})S_{1-t}(u,1-u),\]

developing and simplifying yields a first simplified expression \((1-2v)(u(1-u))^{1-t}=vu^{-1}(1-u)^{2-t}-(1-v)(1-u)^{-t}u^{2-t}\), which, after reorganising to isolate expressions depending on \(v\), yields

\[(u(1-u))^{1-t}+(1-u)^{-t}u^{2-t} = v\cdot\left(u^{-t}(1-u)^{2-t}+(1-u)^{-t}u^{2-t}+2(u(1-u))^{1-t}\right) \tag{85}\]

Assuming \(v\in(0,1)\), we multiply by \((u(1-u))^{1-t}\) (we shall check \(u\in(0,1)\)) and simplify, which yields \(u(1-u)+u^{2}=v((1-u)^{2}+u^{2}+2u(1-u))\), and indeed yields

\[u = v, \tag{86}\]

and we check from (85) that if \(v=0\) (resp. \(v=1\)), then necessarily \(u=0\) (resp. \(u=1\)). To complete the proof, using the previous derivations, we can then simplify

\[\frac{\partial}{\partial u}L\left(u,v\right) = (2-t)\cdot 2^{\frac{2-t}{1-t}}\cdot\frac{u-v}{(u(1-u))^{t}\cdot S_{1-t}^{3-2t}(u,1-u)}, \tag{87}\]which shows that if \(2-t>0\) but \(t\neq-\infty\), \(u=v\) is a strict minimum of the pointwise conditional risk, completing the proof for strict properness. Strict properness is sufficient to show by a simple computation that \(\underline{L}^{(t)}\) is (65). For \(t=-\infty\), we pass to the limit and use the fact that we can also write

\[\ell_{1}^{(t)}(u) = \frac{1}{M_{1-t^{\mathfrak{k}}}\left(1,\left(\frac{u}{1-u}\right) ^{\frac{1}{t^{\mathfrak{k}}}}\right)}\quad(\mbox{we recall }t^{\mathfrak{k}} \doteq 1/(2-t)) \tag{88}\]

\(t\rightarrow-\infty\) is equivalent to \(t^{\mathfrak{s}}\to 0^{+}\). If \(u<1/2\), \(u/(1-u)<1\) and so we see that

\[\lim_{t^{\mathfrak{s}}\to 0^{+}}M_{1-t^{\mathfrak{s}}}\left(1, \left(\frac{u}{1-u}\right)^{\frac{1}{t^{\mathfrak{k}}}}\right) = \frac{1}{2},\]

because \(M_{1}\) is the arithmetic mean. When \(u>1/2\), \(u/(1-u)>1\) and so this time

\[\lim_{t^{\mathfrak{s}}\to 0^{+}}M_{1-t^{\mathfrak{k}}}\left(1, \left(\frac{u}{1-u}\right)^{\frac{1}{t^{\mathfrak{k}}}}\right) = +\infty.\]

Hence,

\[\ell_{1}^{(-\infty)}(u) = 2\cdot[\![u\leqslant 1/2]\!], \tag{89}\]

which is (twice) the partial loss of the 0/1 loss [26].

This ends the proof of Theorem 4.

## III Supplementary material on experiments

### Domains

Table A3 presents the 10 domains we used for our experiments.

### Implementation details and full set of experiments on linear combinations of decision trees

SummaryThis Section depicts the full set of experiments summarized in Table 2 (mf), from Table A4 to Table A15. Tables are ordered in increasing size of the domain (Table A3). In all cases, up to \(J=20\) trees have been trained, of size 15 (total number of nodes, except the two biggest domains, for which the size is 5). For all datasets, except creditcard and adult, we have tested \(t\) in the complete range, \(t\in\{0.0,0.2,0.4,0.6,0.8,0.9,1.0,1.1\}\) (the mf only reports results for \(t\geqslant 0.6\)), and in all cases, models both clipped and not clipped. For each dataset, we have set a 10-fold stratified cross-validation experiment, and report the averages for readability (Table 2 in mf gives the results of a Student paired \(t\)-test on error averages for comparison, limit \(p\)-val = 0.1). We also provide two examples of training error averages for domains hillnoise and hillnonoise (Tables A10 and A12).

Implementation details of \(t\)-AdaBoostFirst, regarding file format, we only input a.csv file to \(t\)-AdaBoost. We do not specify a file with feature types as in ARFF files. \(t\)-AdaBoost recognizes the type of each feature from its column content and distinguishes two main types of features: numerical and categorical. The distinction is important to design the splits during decision tree induction: for numerical values, splits are midpoints between two successive observed values. For categorical, splits are partitions of the feature values in two non-empty subsets. Our implementation of \(t\)-AdaBoost (programmed in Java) makes it possible to choose \(t\) not just in the range of values for which we have shown that boosting-compliant convergence is possible (\(t\in[0,1]\)), but also \(t>1\). Because we thus implement AdaBoost (\(t=1\)) but also for \(t>1\), weights can fairly easily become infinite, we have implemented a safe-check during training, counting the number of times the weights become infinite or zero (note that in this latter case, this really is a problem just for AdaBoost because in theory this should never happen unless the weak classifiers achieve perfect (or perfectly wrong) classification), but also making sure leveraging coefficients for classifiers do not become infinite for AdaBoost, a situation that can happen because of numerical approximations in encoding. In our experiments, we have observed that none of these problematic cases did occur (notice that this could not be the case if we were to boost for a large number of iterations). We have implemented algorithm \(t\)-AdaBoost exactly as specified in mf. The weak learner is implemented to train a decision tree in which the stopping criterion is the size of the tree reaching a user-fixed number of nodes. There is thus no pruning. Also, the top-down induction algorithm proceeds by iteratively picking the heaviest leaf in the tree and then choosing the split that minimizes the expected Bayes risk of the tempered loss, computing using the same \(t\) values as for \(t\)-AdaBoost, and with the constraint to not get pure leaves (otherwise, the real prediction at the leaves, which relies on thelink of the loss, would be infinite for AdaBoost). In our implementation of decision-tree induction, when the number of possible splits exceeds a fixed number \(S\) (currently, 2 000), we pick the best split in a subset of \(S\) splits picked at random.

ResultsFirst, one may notice in several plots that the average test error increases with the number of trees. This turns out to be a sign of overfitting, as exemplified for domains hillnonoise and hillnoise, for which we provide the training curves. If we align the training curves at \(T=1\) (the value is different because the splitting criterion for training the tree is different), we notice that the experimental convergence on training is similar for all values of \(t\) (Tables A10 and A12). The other key experimental result, already visible from Table 2 (mf), is that pretty much all tested values of \(t\) are necessary to get the best results. One could be tempted to conclude that \(t\) slightly smaller than \(1.0\) seems to be a good fit from Table 2 (mf), but the curves show that this is more a consequence of the Table being computed for \(J=20\) trees. The case of eeg illustrates this phenomenon best: while small \(t\)-values are clearly the best when there is no noise, the picture is completely reversed when there is training noise. Notice that this ordering is almost reversed on creditcard and adult: when there is noise, small values of \(t\) tend to give better results. Hence, in addition to getting (i) a pruning mechanism that works for all instances of the tempered loss and (ii) a way to guess the right number of models in the ensemble, a good problem to investigate is in fact appropriately tuning \(t\) in a domain-dependent way. Looking at all plots reveals that substantial gains could be obtained with an accurate procedure (over the strategy that would be to always pick a fixed \(t\), _e.g._\(t=1\)).

* clamped models can be very useful to handle overfitting (sonar for \(\eta=0.4\), qsar for \(\eta\geq 0.2\)); this provides another justification to learn clamped models;
* the overall diversity of curves as a function of \(t\) supports the idea that good strategies could in fact tune \(t\) at training time and change its value with iterations.

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table A7: Experiments on \(t\)-AdaBoost comparing with AdaBoost (\(t=1\), bullets) on domain qsar. Conventions follow Table A4.

\begin{table}
\begin{tabular}{l l} \hline \hline training err (not clipped) & training err (clipped) \\ \hline \hline training err (clipped) & training err (clipped) \\ \hline \hline \end{tabular}
\end{table}
Table A10: Experiments on \(t\)-AdaBoost comparing with AdaBoost (\(t=1\), bullets) on domain hillnonoise: training errors displayed for all algorithms using conventions from Table A4. See text for details.

\begin{table}
\begin{tabular}{l l l} \hline \hline  & training err (not clipped) & training err (clipped) \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline  & & \\ \hline \hline  & & \\ \hline \hline  & & \\ \hline \hline  & & \\ \hline \hline  & & \\ \hline \hline  & & \\ \hline \hline  & & \\ \hline \hline \end{tabular}

\end{table}
Table 11: Experiments on \(t\)-AdaBoost comparing with AdaBoost (\(t=1\), bullets) on domain hillnoise. Conventions follow Table 4.

\begin{table}
\begin{tabular}{c c c c}  & \multicolumn{2}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table A13: Experiments on \(t\)-AdaBoost comparing with AdaBoost (\(t=1\), bullets) on domain eeg. Conventions follow Table A4.

\begin{table}
\begin{tabular}{|

Response:
## Review

### Summary

The paper introduces a novel generalization of the AdaBoost algorithm, termed t-AdaBoost, which incorporates a tempering parameter, t, to enhance its performance by utilizing tempered exponential measures. This extension modifies the weight optimization process while retaining the foundational principles of AdaBoost. The authors provide both theoretical insights and experimental validations, indicating that different values of t can yield varying results across datasets. Their findings suggest that tuning the parameter t can lead to improved performance in specific scenarios, thereby highlighting the practical implications of their work in machine learning applications.

### Strengths

- The paper presents a well-structured and original extension of the AdaBoost algorithm.
- Theoretical results demonstrate sound convergence and empirical performance, showing significant improvements over standard AdaBoost in some datasets.
- The integration of tempered exponential measures provides a robust solution to numerical instabilities commonly associated with AdaBoost.
- The paper is well-written and logically organized, making complex concepts accessible.
- The comprehensive analysis includes both theoretical grounding and experimental evidence, enhancing the credibility of the findings.

### Weaknesses

- Certain sections, particularly Algorithms and notations, are confusing and may hinder reader comprehension.
- The choice of notations is inconsistent, leading to ambiguity in understanding key elements of the algorithm.
- There is a lack of clarity regarding the generalization capabilities of the proposed algorithm compared to existing boosting methods.
- Performance sensitivity to the choice of t may pose challenges for practical implementation without established tuning mechanisms.
- Some experimental results lack adequate evaluation of the impact of new loss functions on performance.

### Questions

- Could the authors explore the extension of t-AdaBoost to multiclass settings, similar to original AdaBoost?
- What insights can be drawn regarding the impact of different t values on exponential decay and overfitting, particularly when t < 1?
- Could the authors clarify the significance of certain notations and equations, especially regarding their definitions and implications?
- What methods could be implemented to effectively tune the parameter t in practice?
- Is there a potential relationship between dataset characteristics (e.g., number of examples and features) and the optimal t value?

### Soundness

**Score:** 3

**Description:** 3 = good. The theoretical and empirical analyses are largely sound, although there are some areas of confusion in notation and clarity.

### Presentation

**Score:** 3

**Description:** 3 = good. The overall structure is coherent, but notational inconsistencies and some unclear explanations detract from readability.

### Contribution

**Score:** 3

**Description:** 3 = good. The paper presents a valuable extension to an established algorithm, though the practical implications and generalization capabilities could be better articulated.

### Rating

**Score:** 6

**Description:** 6 = Weak Accept: The paper is technically solid and has moderate-to-high impact potential, with some areas requiring refinement.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents a novel and relevant contribution to the field of machine learning by extending the AdaBoost algorithm through the introduction of a tempering parameter. While there are notable weaknesses in presentation and clarity, the sound theoretical foundation and empirical evidence support the significance of the findings. The work's potential to influence practical applications in boosting algorithms justifies an acceptance, albeit with recommendations for clearer exposition and further exploration of tuning methodologies.