Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Borda Regret Minimization for Generalized Linear Dueling Bandits

Anonymous Author(s)

Affiliation

Address

email

###### Abstract

Dueling bandits are widely used to model preferential feedback prevalent in many applications such as recommendation systems and ranking. In this paper, we study the Borda regret minimization problem for dueling bandits, which aims to identify the item with the highest Borda score while minimizing the cumulative regret. We propose a rich class of generalized linear dueling bandit models, which cover many existing models. We first prove a regret lower bound of order \(\Omega(d^{2/3}T^{2/3})\) for the Borda regret minimization problem, where \(d\) is the dimension of contextual vectors and \(T\) is the time horizon. To attain this lower bound, we propose an explore-then-commit type algorithm for the stochastic setting, which has a nearly matching regret upper bound \(\widetilde{O}(d^{2/3}T^{2/3})\). We also propose an EXP3-type algorithm for the adversarial setting, where the underlying model parameter can change at each round. Our algorithm achieves an \(\widetilde{O}(d^{2/3}T^{2/3})\) regret, which is also optimal. Empirical evaluations on both synthetic data and a simulated real-world environment are conducted to corroborate our theoretical analysis.

## 1 Introduction

Multi-armed bandits (MAB) (Lattimore and Szepesvari, 2020) is an interactive game where at each round, an agent chooses an arm to pull and receives a noisy reward as feedback. In contrast to numerical feedback considered in classic MAB settings, preferential feedback is more natural in various online learning tasks including information retrieval Yue and Joachims (2009), recommendation systems Sui and Burdick (2014), ranking Minka et al. (2018), crowdsourcing Chen et al. (2013), etc. Moreover, numerical feedback is also more difficult to gauge and prone to errors in many real-world applications. For example, when provided with items to shop or movies to watch, it is more natural for a customer to pick a preferred one than scoring the options. This motivates _Dueling Bandits_(Yue and Joachims, 2009), where the agent repeatedly pulls two arms at a time and is provided with feedback being the binary outcome of "duels" between the two arms.

In dueling bandits problems, the outcome of duels is commonly modeled as Bernoulli random variables due to their binary nature. At each round, suppose the agent chooses to compare arm \(i\) and \(j\), then the binary feedback is assumed to be sampled independently from a Bernoulli distribution. For a dueling bandits instance with \(K\) arms, the probabilistic model of the instance can be fully characterized by a \(K\times K\) preference probability matrix with each entry being: \(p_{i,j}=\mathbb{P}\left(\text{arm }i\text{ is chosen over arm }j\right).\)

In a broader range of applications such as ranking, "arms" are often referred to as "items". We will use these two terms interchangeably in the rest of this paper. One central goal of dueling bandits is to devise a strategy to identify the "optimal" item as quickly as possible, measured by either sample complexity or cumulative regret. However, the notion of optimality for dueling bandits is way harder to define than for multi-armed bandits. The latter can simply define the arm with the highest numerical feedback as the optimal arm, while for dueling bandits there is no obvious definition solely dependent on \(\{p_{i,j}|i,j\in[K]\}\).

The first few works on dueling bandits imposed strong assumptions on \(p_{i,j}\). For example, Yue et al. (2012) assumed that there exists a true ranking that is coherent among all items, and the preference probabilities must satisfy both strong stochastic transitivity (SST) and stochastic triangle inequality (STI). While relaxations like weak stochastic transitivity (Falahatgar et al., 2018) or relaxed stochastic transitivity (Yue and Joachims, 2011) exist, they typically still assume the true ranking exists and the preference probabilities are consistent, i.e., \(p_{i,j}>\frac{1}{2}\) if and only if \(i\) is ranked higher than \(j\). In reality, the existence of such coherent ranking aligned with item preferences is rarely the case. For example, \(p_{i,j}\) may be interpreted as the probability of one basketball team \(i\) beating another team \(j\), and there can be a circle among the match advantage relations.

In this paper, we do not assume such coherent ranking exists and solely rely on the _Borda score_ based on preference probabilities. The Borda score \(B(i)\) of an item \(i\) is the probability that it is preferred when compared with another random item, namely \(B(i):=\frac{1}{K-1}\sum_{j\neq i}p_{i,j}\). The item with the highest Borda score is called the _Borda winner_. The Borda winner is intuitively appealing and always well-defined for any set of preferential probabilities. The Borda score also does not require the problem instance to obey any consistency or transitivity, and it is considered one of the most general criteria.

To identify the Borda winner, estimations of the Borda scores are needed. Since estimating the Borda score for one item requires comparing it with every other items, the sample complexity is prohibitively high when there are numerous items. On the other hand, in many real-world applications, the agent has access to side information that can assist the evaluation of \(p_{i,j}\). For instance, an e-commerce item carries its category as well as many other attributes, and the user might have a preference for a certain category (Wang et al., 2018). For a movie, the genre and the plot as well as the directors and actors can also be taken into consideration when making choices (Liu et al., 2017).

Based on the above motivation, we consider _Generalized Linear Dueling Bandits_. At each round, the agent selects two items from a finite set of items and receives a comparison result of the preferred item. The comparisons depend on known intrinsic contexts/features associated with each pair of items. The contexts can be obtained from upstream tasks, such as topic modeling (Zhu et al., 2012) or embedding (Vasile et al., 2016). Our goal is to adaptively select items and minimize the regret with respect to the optimal item (i.e., Borda winner). Our main contributions are summarized as follows:

* We show a hardness result regarding the Borda regret minimization for the (generalized) linear model. We prove a worst-case regret lower bound \(\Omega(d^{2/3}T^{2/3})\) for our dueling bandit model, showing that even in the stochastic setting, minimizing the Borda regret is difficult. The construction and proof of the lower bound are new and might be of independent interest.
* We propose an explore-then-commit type algorithm under the stochastic setting, which can achieve a nearly matching upper bound \(\widetilde{O}(d^{2/3}T^{2/3})\). When the number of items \(K\) is small, the algorithm can also be configured to achieve a smaller regret \(\widetilde{O}\big{(}(d\log K)^{1/3}T^{2/3}\big{)}\).
* We propose an EXP3 type algorithm for linear dueling bandits under the adversarial setting, which can achieve a nearly matching upper bound \(\widetilde{O}\big{(}(d\log K)^{1/3}T^{2/3}\big{)}\).
* We conduct empirical studies to verify the correctness of our theoretical claims. Under both synthetic and real-world data settings, our algorithms can outperform all the baselines in terms of cumulative regret.

NotationIn this paper, we use normal letters to denote scalars, lowercase bold letters to denote vectors, and uppercase bold letters to denote matrices. For a vector \(\mathbf{x}\), \(\|\mathbf{x}\|\) denotes its \(\ell_{2}\)-norm. The weighted \(\ell_{2}\)-norm associated with a positive-definite matrix \(\mathbf{A}\) is defined as \(\|\mathbf{x}\|_{\mathbf{A}}=\sqrt{\mathbf{x}^{\top}\mathbf{A}\mathbf{x}}\). The minimum eigenvalue of a matrix \(\mathbf{A}\) is written as \(\lambda_{\min}(\mathbf{A})\). We use standard asymptotic notations including \(O(\cdot),\Omega(\cdot),\Theta(\cdot)\), and \(\widetilde{O}(\cdot),\widetilde{\Omega}(\cdot),\widetilde{\Theta}(\cdot)\) will hide logarithmic factors. For a positive integer \(N\), \([N]:=\{1,2,\ldots,N\}\).

## 2 Related Work

Multi-armed and Contextual Bandits Multi-armed bandit is a problem of identifying the best choice in a sequential decision-making system. It has been studied in numerous ways with a wide range of applications (Even-Dar et al., 2002; Lai et al., 1985; Kuleshov and Precup, 2014). Contextual linear bandit is a special type of bandit problem where the agent is provided with side information, i.e., contexts, and rewards are assumed to have a linear structure. Various algorithms (Rusmevichientongand Tsitsiklis, 2010; Filippi et al., 2010; Abbasi-Yadkori et al., 2011; Li et al., 2017; Jun et al., 2017) have been proposed to utilize this contextual information.

**Dueling Bandits and Its Performance Metrics** Dueling bandits is a variant of MAB with preferential feedback (Yue et al., 2012; Zoghi et al., 2014, 2015). A comprehensive survey can be found at Bengs et al. (2021). As discussed previously, the probabilistic structure of a dueling bandits problem is governed by the preference probabilities, over which an optimal item needs to be defined. Optimality under the _Borda score_ criteria has been adopted by several previous works (Jamieson et al., 2015; Falahatgar et al., 2017; Heckel et al., 2018; Saha et al., 2021). The most relevant work to ours is Saha et al. (2021), where they studied the problem of regret minimization for adversarial dueling bandits and proved a \(T\)-round Borda regret upper bound \(\tilde{O}(K^{1/3}T^{2/3})\). They also provide an \(\Omega(K^{1/3}T^{2/3})\) lower bound for stationary dueling bandits using Borda regret.

Apart from the Borda score, _Copeland score_ is also a widely used criteria (Urvoy et al., 2013; Zoghi et al., 2014, 2015; Wu and Liu, 2016; Komiyama et al., 2016). It is defined as \(C(i):=\frac{1}{K^{-1}}\sum_{j\neq i}\mathbbm{1}\{p_{i,j}>1/2\}\). A Copeland winner is the item that beats the most number of other items. It can be viewed as a "thresholded" version of Borda winner. In addition to Borda and Copeland winners, optimality notions such as a von Neumann winner were also studied in Ramamohan et al. (2016); Dudik et al. (2015); Balsubramani et al. (2016).

Another line of work focuses on identifying the optimal item or the total ranking, assuming the preference probabilities are consistent. Common consistency conditions include Strong Stochastic Transitivity (Yue et al., 2012; Falahatgar et al., 2017, 2017, 2018; Weak Stochastic Transitivity (Falahatgar et al., 2018; Ren et al., 2019; Wu et al., 2022; Lou et al., 2022), Relaxed Stochastic Transitivity (Yue and Joachims, 2011) and Stochastic Triangle Inequality. Sometimes the aforementioned transitivity can also be implied by some structured models like the Bradley-Terry model. We emphasize that these consistency conditions are not assumed or implicitly implied in our setting.

**Contextual Dueling Bandits** In Dudik et al. (2015), contextual information is incorporated in the dueling bandits framework. Later, Saha (2021) studied a structured contextual dueling bandits setting where each item \(i\) has its own contextual vector \(\mathbf{x}_{i}\) (sometimes called Linear Stochastic Transitivity). Each item then has an intrinsic score \(v_{i}\) equal to the linear product of an unknown parameter vector \(\mathbf{\theta}^{*}\) and its contextual vector \(\mathbf{x}_{i}\). The preference probability between two items \(i\) and \(j\) is assumed to be \(\mu\left(v_{i}-v_{j}\right)\) where \(\mu(\cdot)\) is the logistic function. These intrinsic scores of items naturally define a ranking over items. The regret is also computed as the gap between the scores of pulled items and the best item. While in this paper, we assume that the contextual vectors are associated with item pairs and define regret on the Borda score. In Section A.1, we provide a more detailed discussion showing that the setting considered in Saha (2021) can be viewed as a special case of our model.

## 3 Backgrounds and Preliminaries

### Problem Setting

We first consider the stochastic preferential feedback model with \(K\) items in the fixed time horizon setting. We denote the item set by \([K]\) and let \(T\) be the total number of rounds. At each round \(t\), the agent can pick any pair of items \((i_{t},j_{t})\) to compare and receive stochastic feedback about whether item \(i_{t}\) is preferred over item \(j_{t}\), (denoted by \(i_{t}\succ j_{t}\)). We denote the probability of seeing the event \(i\succ j\) as \(p_{i,j}\in[0,1]\). Naturally, we assume \(p_{i,j}+p_{j,i}=1\), and \(p_{i,i}=1/2\).

In this paper, we are concerned with the generalized linear model (GLM), where there is assumed to exist an _unknown_ parameter \(\mathbf{\theta}^{*}\in\mathbb{R}^{d}\), and each pair of items \((i,j)\) has its own _known_ contextual/feature vector \(\mathbf{\phi}_{i,j}\in\mathbb{R}^{d}\) with \(\|\mathbf{\phi}_{i,j}\|\leq 1\). There is also a fixed known link function (sometimes called comparison function) \(\mu(\cdot)\) that is monotonically increasing and satisfies \(\mu(x)+\mu(-x)=1\), e.g. a linear function or the logistic function \(\mu(x)=1/(1+e^{-x})\). The preference probability is defined as \(p_{i,j}=\mu(\mathbf{\phi}_{i,j}^{\top}\mathbf{\theta}^{*})\). At each round, denote \(r_{t}=\mathbbm{1}\{i_{t}\succ j_{t}\}\), then we have

\[\mathbb{E}[r_{t}|i_{t},j_{t}]=p_{i_{t},j_{t}}=\mu(\mathbf{\phi}_{i_{t},j_{t}}^{\top }\mathbf{\theta}^{*}).\]

Then our model can also be written as

\[r_{t}=\mu(\mathbf{\phi}_{i_{t},j_{t}}^{\top}\mathbf{\theta}^{*})+\epsilon_{t},\]where the noises \(\{\epsilon_{t}\}_{t\in[T]}\) are zero-mean, \(1\)-sub-Gaussian and assumed independent from each other. Note that, given the constraint \(p_{i,j}+p_{j,i}=1\), it is implied that \(\mathbf{\phi}_{i,j}=-\mathbf{\phi}_{j,i}\) for any \(i\in[K],j\in[K]\).

The agent's goal is to maximize the cumulative Borda score. The (slightly modified 1) Borda score of item \(i\) is defined as \(B(i)=\frac{1}{K}\sum_{j=1}^{K}p_{i,j}\), and the Borda winner is defined as \(i^{*}=\operatorname*{argmax}_{i\in[K]}B(i)\). The problem of merely identifying the Borda winner was deemed trivial (Zoghi et al., 2014; Busa-Fekete et al., 2018) because for a fixed item \(i\), uniformly random sampling \(j\) and receiving feedback \(r_{i,j}=\operatorname*{Bernoulli}(p_{i,j})\) yield a Bernoulli random variable with its expectation being the Borda score \(B(i)\). This so-called _Borda reduction_ trick makes identifying the Borda winner as easy as the best-arm identification for \(K\)-armed bandits. Moreover, if the regret is defined as \(\operatorname{Regret}(T)=\sum_{t=1}^{T}(B(i^{*})-B(i_{t}))\), then any optimal algorithms for multi-arm bandits can achieve \(\widetilde{O}(\sqrt{T})\) regret.

Footnote 1: Previous works define Borda score as \(B^{\prime}_{i}=\frac{1}{K-1}\sum_{j\neq i}p_{i,j}\), excluding the diagonal term \(p_{i,i}=1/2\). Our definition is equivalent since the difference between two items satisfies \(B(i)-B_{j}=\frac{K-1}{K}(B^{\prime}_{i}-B^{\prime}_{j})\). Therefore, the regret will be in the same order for both definitions.

However, the above definition of regret does not respect the fact that a pair of items are selected at each round. When the agent chooses two items to compare, it is natural to define the regret so that both items contribute equally. A commonly used regret, e.g., in Saha et al. (2021), has the following form:

\[\operatorname{Regret}(T)=\sum_{t=1}^{T}\big{(}2B(i^{*})-B(i_{t})-B(j_{t})\big{)}, \tag{1}\]

where the regret is defined as the sum of the sub-optimality of both selected arms. Sub-optimality is measured by the gap between the Borda scores of the compared items and the Borda winner. This form of regret deems any classical multi-arm bandit algorithm with Borda reduction vacuous because taking \(j_{t}\) into consideration will invoke \(\Theta(T)\) regret.

Adversarial SettingSaha et al. (2021) considered an adversarial setting for the multi-armed case, where at each round \(t\), the comparison follows a potentially different probability model, denoted by \(\{p^{t}_{i,j}\}_{i,j\in[K]}\). In this paper, we consider its contextual counterpart. Formally, we assume there is an underlying parameter \(\mathbf{\theta}^{*}_{t}\), and at round \(t\), the preference probability is defined as \(p^{t}_{i,j}=\mu(\mathbf{\phi}^{\top}_{i,j}\mathbf{\theta}^{*}_{t})\).

The Borda score of item \(i\in[K]\) at round \(t\) is defined as \(B_{t}(i)=\frac{1}{K}\sum_{j=1}^{K}p^{t}_{i,j}\), and the Borda winner at round \(T\) is defined as \(i^{*}=\operatorname*{argmax}_{i\in[K]}\sum_{t=1}^{T}B_{t}(i)\). The \(T\)-round regret is thus defined as \(\operatorname{Regret}(T)=\sum_{t=1}^{T}\big{(}2B_{t}(i^{*})-B_{t}(i_{t})-B_{t }(j_{t})\big{)}\).

### Assumptions

In this section, we present the assumptions required for establishing theoretical guarantees. Due to the fact that the analysis technique is largely extracted from Li et al. (2017), we follow them to make assumptions to enable regret minimization for generalized linear dueling bandits.

We make a regularity assumption about the distribution of the contextual vectors:

**Assumption 1**.: There exists a constant \(\lambda_{0}>0\) such that \(\lambda_{\min}\big{(}\frac{1}{K^{2}}\sum_{i=1}^{K}\sum_{j=1}^{K}\mathbf{\phi}_{i, j}\mathbf{\phi}^{\top}_{i,j}\big{)}\geq\lambda_{0}\).

This assumption is only utilized to initialize the design matrix \(\mathbf{V}_{\tau}=\sum_{i=1}^{\tau}\mathbf{\phi}_{i_{t},j_{t}}\mathbf{\phi}^{\top}_{i_{t },j_{t}}\) so that the minimum eigenvalue is large enough. We follow Li et al. (2017) to deem \(\lambda_{0}\) as a constant.

We also need the following assumption regarding the link function \(\mu(\cdot)\):

**Assumption 2**.: Let \(\hat{\mu}\) be the first-order derivative of \(\mu\). We have \(\kappa:=\inf_{\|\mathbf{x}\|\leq 1,\|\mathbf{\theta}-\mathbf{\theta}^{*}\|\leq 1} \hat{\mu}(\mathbf{x}^{\top}\mathbf{\theta})>0\).

Assuming \(\kappa>0\) is necessary to ensure the maximum log-likelihood estimator can converge to the true parameter \(\mathbf{\theta}^{*}\)(Li et al., 2017, Section 3). This type of assumption is commonly made in previous works for generalized linear models (Filippi et al., 2010; Li et al., 2017; Faury et al., 2020).

Another common assumption is regarding the continuity and smoothness of the link function.

**Assumption 3**.: \(\mu\) is twice differentiable. Its first and second-order derivatives are upper-bounded by constants \(L_{\mu}\) and \(M_{\mu}\) respectively.

This is a very mild assumption. For example, it is easy to verify that the logistic link function satisfies Assumption 3 with \(L_{\mu}=M_{\mu}=1/4\).

## 4 The Hardness Result

This section presents Theorem 4, a worst-case regret lower bound for the stochastic linear dueling bandits. The proof of Theorem 4 relies on a class of hard instances, as shown in Figure 1. We show that any algorithm will incur a certain amount of regret when applied to this hard instance class. The constructed hard instances follow a stochastic linear model, which is a sub-class of the generalized linear model. Saha et al. (2021b) first proposed a similar construction for finite many arms with no contexts. Our construction is for a contextual setting and the proof of the lower bound takes a rather different route.

For any \(d>0\), we construct the class of hard instances as follows. An instance is specified by a vector \(\mathbf{\theta}\in\{-\Delta,+\Delta\}^{d}\). The instance contains \(2^{d+1}\) items (indexed from 0 to \(2^{d+1}-1\)). The preference probability for an instance is defined by \(p^{\mathbf{\theta}}_{i,j}\) as:

\[p^{\mathbf{\theta}}_{i,j}=\begin{cases}\frac{1}{2},\text{ if }i<2^{d},j<2^{d} \text{ or if }i\geq 2^{d},j\geq 2^{d}\\ \frac{3}{2},\text{ if }i<2^{d},j\geq 2^{d}\\ \frac{4}{4},\text{ if }i\geq 2^{d},j<2^{d}\end{cases}+\langle\mathbf{\phi}_{i,j}, \mathbf{\theta}\rangle,\]

and the \(d\)-dimensional feature vectors \(\mathbf{\phi}_{i,j}\) are given by

\[\mathbf{\phi}_{i,j}=\begin{cases}\mathbf{0},\text{ if }i<2^{d},j<2^{d}\text{ or if }i\geq 2^{d},j\geq 2^{d}\\ \textbf{bit}(i),\text{ if }i<2^{d},j\geq 2^{d}\\ -\textbf{bit}(j),\text{ if }i\geq 2^{d},j<2^{d},\end{cases}\]

where \(\textbf{bit}(\cdot)\) is the (shifted) bit representation of non-negative integers, i.e., suppose \(x\) has the binary representation \(x=b_{0}\times 2^{0}+b_{1}\times 2^{1}+\cdots+b_{d-1}\times 2^{d-1}\), then

\[\textbf{bit}(x)=(2b_{0}-1,2b_{1}-1,\ldots,2b_{d-1}-1)=2\mathbf{b}-1.\]

Note that \(\textbf{bit}(\cdot)\in\{-1,+1\}^{d}\), and that \(\mathbf{\phi}_{i,j}=-\mathbf{\phi}_{j,i}\) is satisfied. The definition of \(p^{\mathbf{\theta}}_{i,j}\) can be slightly tweaked to fit exactly the model described in Section 3 (see Remark 11 in Appendix).

Some calculation shows that the Borda scores of the \(2^{d+1}\) items are:

\[B^{\mathbf{\theta}}(i)=\begin{cases}\frac{5}{8}+\frac{1}{2}\langle\textbf{bit}(i),\mathbf{\theta}\rangle,\text{ if }i<2^{d},\\ \frac{3}{8},\text{ if }i\geq 2^{d}.\end{cases}\]

Intuitively, the former half of items (those indexed from \(0\) to \(2^{d}-1\)) are "good" items (one among them is optimal, others are nearly optimal), while the latter half of items are "bad" items. Under such hard instances, every time one of the two pulled items is a "bad" item, then a one-step regret

Figure 1: Illustration of the hard-to-learn preference probability matrix \(\{p^{\mathbf{\theta}}_{i,j}\}_{i\in[K],j\in[K]}\). There are \(K=2^{d+1}\) items in total. The first \(2^{d}\) items are “good” items with higher Borda scores, and the last \(2^{d}\) items are “bad” items. The upper right block \(\{p_{i,j}\}_{i\leq 2^{d},j\geq 2^{d}}\) is defined as shown in the blue bubble. The lower left block satisfies \(p_{i,j}=1-p_{j,i}\). For any \(\mathbf{\theta}\), there exist one and only best item \(i\) such that \(\textbf{bit}(i)=\textbf{sign}(\mathbf{\theta})\).

\(B^{\mathbf{\theta}}(i^{*})-B^{\mathbf{\theta}}(i)\geq 1/4\) is incurred. To minimize regret, we should thus try to avoid pulling "bad" items. However, in order to identify the best item among all "good" items, comparisons between "good" and "bad" items are necessary. The reason is simply that comparisons between "good" items give no information about the Borda scores as the comparison probabilities are \(p^{\mathbf{\theta}}_{i,j}=\frac{1}{2}\) for all \(i,j<2^{d}\). Hence, any algorithm that can decently distinguish among the "good" items has to pull "bad" ones for a fair amount of times, and large regret is thus incurred. A similar observation is also made by Saha et al. (2021).

This specific construction emphasizes the intrinsic hardness of Borda regret minimization: to differentiate the best item from its close competitors, the algorithm must query the bad items to gain information.

Formally, this class of hard instances leads to the following regret lower bound for both stochastic and adversarial settings:

**Theorem 4**.: For any algorithm \(\mathcal{A}\), there exists a hard instance \(\{p^{\mathbf{\theta}}_{i,j}\}\) with \(T>4d^{2}\), such that \(\mathcal{A}\) will incur expected regret at least \(\Omega(d^{2/3}T^{2/3})\).

The construction of this hard instance for linear dueling bandits is inspired by the worst-case lower bound for the stochastic linear bandit (Dani et al., 2008), which has the order \(\Omega(d\sqrt{T})\), while ours is \(\Omega(d^{2/3}T^{2/3})\). The difference is that for the linear or multi-armed stochastic bandit, eliminating bad arms can make further exploration less expensive. But in our case, any amount of exploration will not reduce the cost of further exploration. This essentially means that exploration and exploitation must be separate, which is also supported by the fact that a simple explore-then-commit algorithm shown in Section 5 can be nearly optimal.

## 5 Stochastic Contextual Dueling Bandit

### Algorithm Description

```
1:Input: time horizon \(T\), number of items \(K\), feature dimension \(d\), feature vectors \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\), exploration rounds \(\tau\), error tolerance \(\epsilon\), failure probability \(\delta\).
2:for\(t=1,2,\ldots,\tau\)do
3: sample \(i_{t}\sim\mathrm{Uniform}([K])\), \(j_{t}\sim\mathrm{Uniform}([K])\)
4: query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
5:endfor
6: Find the G-optimal design \(\pi(i,j)\) based on \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\)
7: Let \(N(i,j)=\left\lceil\frac{d\pi(i,j)}{\epsilon^{2}}\right\rceil\) for any \((i,j)\in\text{supp}(\pi)\), denote \(N=\sum_{i=1}^{K}\sum_{j=1}^{K}N(i,j)\)
8:for\(i\in[K]\), \(j\in[K]\), \(s\in[N(i,j)]\)do
9: set \(t\gets t+1\), set \((i_{t},j_{t})=(i,j)\)
10: query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
11:endfor
12: Calculate the empirical MLE estimator \(\widehat{\mathbf{\theta}}_{\tau+N}\) based on all \(\tau+N\) samples via (2)
13: Estimate the Borda score for each item: \[\widehat{B}(i)=\frac{1}{K}\sum_{j=1}^{K}\mu(\mathbf{\phi}_{i,j}^{\top}\widehat{ \mathbf{\theta}}_{\tau+N}),\qquad\widehat{i}=\operatorname*{argmax}_{i\in[K]} \widehat{B}(i)\]
14: Keep querying \((\widehat{i},\widehat{i})\) for the rest of the time.
```

**Algorithm 1**BETC-GLM

### Algorithm Description

```
1:Input: time horizon \(T\), number of items \(K\), feature dimension \(d\), feature vectors \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\), exploration rounds \(\tau\), error tolerance \(\epsilon\), failure probability \(\delta\).
2:for\(t=1,2,\ldots,\tau\)do
3: sample \(i_{t}\sim\mathrm{Uniform}([K])\), \(j_{t}\sim\mathrm{Uniform}([K])\)
4: query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
5:endfor
6: Find the G-optimal design \(\pi(i,j)\) based on \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\)
7: Let \(N(i,j)=\left\lceil\frac{d\pi(i,j)}{\epsilon^{2}}\right\rceil\) for any \((i,j)\in\text{supp}(\pi)\), denote \(N=\sum_{i=1}^{K}\sum_{j=1}^{K}N(i,j)\)
8:for\(i\in[K]\), \(j\in[K]\), \(s\in[N(i,j)]\)do
9: set \(t\gets t+1\), set \((i_{t},j_{t})=(i,j)\)
10: query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
11:endfor
12: Calculate the empirical MLE estimator \(\widehat{\mathbf{\theta}}_{\tau+N}\) based on all \(\tau+N\) samples via (2)
13: Estimate the Borda score for each item: \[\widehat{B}(i)=\frac{1}{K}\sum_{j=1}^{K}\mu(\mathbf{\phi}_{i,j}^{\top}\widehat{ \mathbf{\theta}}_{\tau+N}),\qquad\widehat{i}=\operatorname*{argmax}_{i\in[K]} \widehat{B}(i)\]
14: Keep querying \((\widehat{i},\widehat{i})\) for the rest of the time.
```

**Algorithm 2**BETC-GLM

### Algorithm Description

```
1:Input: time horizon \(T\), number of items \(K\), feature dimension \(d\), feature vectors \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\), exploration rounds \(\tau\), error tolerance \(\epsilon\), failure probability \(\delta\).
2:for\(t=1,2,\ldots,\tau\)do
3: sample \(i_{t}\sim\mathrm{Uniform}([K])\), \(j_{t}\sim\mathrm{Uniform}([K])\)
4: query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
5:endfor
6: Find the G-optimal design \(\pi(i,j)\) based on \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\)
7: Let \(N(i,j)=\left\lceil\frac{d\pi(i,j)}{\epsilon^{2}}\right\rceil\) for any \((i,j)\in\text{supp}(\pi)\), denote \(N=\sum_{i=1}^{K}\sum_{j=1}^{K}N(i,j)\)
8:for\(i\in[K]\), \(j\in[K]\), \(s\in[N(i,j)]\)do
9: set \(t\gets t+1\), set \((i_{t},j_{t})=(i,j)\)
10: query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
11:endfor
12: Calculate the empirical MLE estimator \(\widehat{\mathbf{\theta}}_{\tau+N}\) based on all \(\tau+N\) samples via (2)
13: Estimate the Borda score for each item: \[\widehat{B}(i)=\frac{1}{K}\sum_{j=1}^{K}\mu(\mathbf{\phi}_{i,j}^{\top}\widehat{ \mathbf{\theta}}_{\tau+N}),\qquad\widehat{i}=\operatorname*{argmax}_{i\in[K]} \widehat{B}(i)\]
14: Keep querying \((\widehat{i},\widehat{i})\) for the rest of the time.
```

**Algorithm 3**BETC-GLM

```
1:Input: time horizon \(T\), number of items \(K\), feature dimension \(d\), feature vectors \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\), exploration rounds \(\tau\), error tolerance \(\epsilon\), failure probability \(\delta\).
2:for\(t=1,2,\ldots,\tau\)do
3: sample \(i_{t}\sim\mathrm{Uniform}([K])\), \(j_{t}\sim\mathrm{Uniform}([K])\)
4: query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
5:endfor
6: Find the G-optimal design \(\pi(i,j)\) based on \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\)
7: Let \(N(i,j)=\left\lceil\frac{d\pi(i,j)}{\epsilon^{2}}\right\rceil\) for any \((i,j)\in\text{supp}(\pi)\), denote \(N=\sum_{i=1}^{K}\sum_{j=1}^{K}N(i,j)\)
18:for\(i\in[K]\), \(j\in[K]\), \(s\in[N(i,j)]\)do
19: set \(t\gets t+1\), set \((i_{t},j_{t})=(i,j)\)
20: query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
21:endfor
22: Calculate the empirical MLE estimator \(\widehat{\mathbf{\theta}}_{\tau+N}\) based on all \(\tau+N\) samples via (2)
23: Estimate the Borda score for each item: \[\widehat{B}(i)=\frac{1}{K}\sum_{j=1}^{K}\mu(\mathbf{\phi}_{i,j}^{\top}\widehat{\mathbf{ \theta}}_{\tau+N}),\qquad\widehat{i}=\operatorname*{argmax}_{i\in[K]}\widehat{B}(i)\]
24: Keep querying \((\widehat{i},\widehat{i})\) for the rest of the time.
```

**Algorithm 4**BETC-GLM

At the high level, Algorithm 1 can be divided into two phases: the exploration phase (Line 2-11) and the exploitation phase (Line 12-14). The exploration phase ensures that the MLE estimator \(\widehat{\mathbf{\theta}}\) is accurate enough so that the estimated Borda score is within \(\widehat{O}(\epsilon)\)-range of the true Borda score (ignoring other quantities). Then the exploitation phase simply chooses the empirical Borda winner to incur small regret.

During the exploration phase, the algorithm first performs "pure exploration" (Line 2-5), which can be seen as an initialization step for the algorithm. The purpose of this step is to ensure the design matrix \(\mathbf{V}_{\tau+N}=\sum_{t=1}^{\tau+N}\mathbf{\phi}_{i_{t},j_{t}}\mathbf{\phi}_{i_{t},j _{t}}^{\top}\) is positive definite.

After that, the algorithm will perform the "designed exploration". Line 6 will find the G-optimal design, which minimizes the objective function \(g(\pi)=\max_{i,j}\|\mathbf{\phi}_{i,j}\|_{\mathbf{\gamma}(\pi)^{-1}}^{2}\), where \(\mathbf{V}(\pi):=\sum_{i,j}\pi(i,j)\mathbf{\phi}_{i,j}\mathbf{\phi}_{i,j}^{\top}\). The G-optimal design \(\pi^{*}(\cdot)\) satisfies \(\|\mathbf{\phi}_{i,j}\|_{\mathbf{V}(\pi^{*})^{-1}}^{2}\leq d\), and can be efficiently approximated by the Frank-Wolfe algorithm (See Remark 8 for a detailed discussion). Then the algorithm will follow \(\pi(\cdot)\) found at Line 6 to determine how many samples (Line 7) are needed. At Line 8-11, there are in total \(N=\sum_{i=1}^{K}\sum_{j=1}^{K}N(i,j)\) samples queried, and the algorithm shall index them by \(t=\tau+1,\tau+2,\ldots,\tau+N\).

At Line 12, the algorithm collects all the \(\tau+N\) samples and performs the maximum likelihood estimation (MLE). For the generalized linear model, the MLE estimator \(\widehat{\mathbf{\theta}}_{\tau+N}\) satisfies:

\[\sum_{t=1}^{\tau+N}\mu(\mathbf{\phi}_{i_{t},j_{t}}^{\top}\widehat{\mathbf{\theta}}_{ \tau+N})\mathbf{\phi}_{i_{t},j_{t}}=\sum_{t=1}^{\tau+N}r_{t}\mathbf{\phi}_{i_{t},j_{t}}, \tag{2}\]

or equivalently, it can be determined by solving a strongly concave optimization problem:

\[\widehat{\mathbf{\theta}}_{\tau+N}\in\operatorname*{argmax}_{\mathbf{\theta}}\sum_{t=1 }^{\tau+N}\bigg{(}r_{t}\mathbf{\phi}_{i_{t},j_{t}}^{\top}\mathbf{\theta}-m(\mathbf{\phi}_{i _{t},j_{t}}^{\top}\mathbf{\theta})\bigg{)},\]

where \(\dot{m}(\cdot)=\mu(\cdot)\). For the logistic link function, \(m(x)=\log(1+e^{x})\). As a special case of our generalized linear model, the linear model has a closed-form solution for (2). For example, if \(\mu(x)=\frac{1}{2}+x\), i.e. \(p_{i,j}=\frac{1}{2}+\mathbf{\phi}_{i,j}^{\top}\mathbf{\theta}^{*}\), then (2) becomes:

\[\widehat{\mathbf{\theta}}_{\tau+N}=\mathbf{V}_{\tau+N}^{-1}\sum_{t=1}^{\tau+N}(r_ {t}-1/2)\mathbf{\phi}_{i_{t},j_{t}},\]

where \(\mathbf{V}_{\tau+N}=\sum_{t=1}^{\tau+N}\mathbf{\phi}_{i_{t},j_{t}}\mathbf{\phi}_{i_{t},j_{t}}^{\top}\).

After the MLE estimator is obtained, Line 13 will calculate the estimated Borda score \(\widehat{B}(i)\) for each item based on \(\widehat{\mathbf{\theta}}_{\tau+N}\), and pick the empirically best one.

### A Matching Regret Upper Bound

Algorithm 1 can be configured to tightly match the worst-case lower bound. The configuration and performance are described as follows:

**Theorem 5**.: Suppose Assumption 1-3 hold and \(T=\Omega(d^{2})\). For any \(\delta>0\), if we set \(\tau=C_{4}\lambda_{0}^{-2}(d+\log(1/\delta))\) (\(C_{4}\) is a universal constant) and \(\epsilon=d^{1/6}T^{-1/3}\), then with probability at least \(1-2\delta\), Algorithm 1 will incur regret bounded by:

\[O\Big{(}\kappa^{-1}d^{2/3}T^{2/3}\sqrt{\log\big{(}T/d\delta\big{)}}\Big{)}.\]

By setting \(\delta=T^{-1}\), the expected regret is bounded as \(\widetilde{O}(\kappa^{-1}d^{2/3}T^{2/3})\).

For linear bandit models, such as the hard-to-learn instances in Section 4, \(\kappa\) is a universal constant. Therefore, Theorem 5 tightly matches the lower bound in Theorem 4, up to logarithmic factors.

**Remark 6** (Regret for Fewer Arms).: In typical scenarios, the number of items \(K\) is not exponentially large in the dimension \(d\). In this case, we can choose a different parameter set of \(\tau\) and \(\epsilon\) such that Algorithm 1 can achieve a smaller regret bound \(\widetilde{O}\big{(}\kappa^{-1}(d\log K)^{1/3}T^{2/3}\big{)}\) with smaller dependence on the dimension \(d\). See Theorem 10 in Appendix A.2.

**Remark 7** (Regret for Infinitely Many Arms).: In most practical scenarios of dueling bandits, it is adequate to consider a finite number \(K\) of items (e.g., ranking items). Nonetheless, BETC-GLMcan be easily adapted to accommodate infinitely many arms in terms of regret. We can construct a covering over all \(\mathbf{\phi}_{i,j}\) and perform optimal design and exploration on the covering set. The resulting regret will be the same as our upper bound, i.e., \(\widetilde{O}(d^{2/3}T^{2/3})\) up to some error caused by the epsilon net argument.

**Remark 8** (Approximate G-optimal Design).: Algorithm 1 assumes an exact G-optimal design \(\pi\) is obtained. In the experiments, we use the Frank-Wolfe algorithm to solve the constraint optimization problem (See Algorithm 5, Appendix G.3). To find a policy \(\pi\) such that \(g(\pi)\leq(1+\varepsilon)g(\pi^{*})\), roughly \(O(d/\varepsilon)\) optimization steps are needed. Such a near-optimal design will introduce a factor of \((1+\varepsilon)^{1/3}\) into the upper bounds.

## 6 Adversarial Contextual Dueling Bandit

This section addresses Borda regret minimization under the adversarial setting. As we introduced in Section 3.1, the unknown parameter \(\mathbf{\theta}_{t}\) can vary for each round \(t\), while the contextual vectors \(\mathbf{\phi}_{i,j}\) are fixed.

Our proposed algorithm, BEXP3, is designed for the contextual linear model. Formally, at round \(t\) and given pair \((i,j)\), we have \(p^{t}_{i,j}=\frac{1}{2}+\langle\mathbf{\phi}_{i,j},\mathbf{\theta}^{*}_{t}\rangle\).

### Algorithm Description

```
1:Input: time horizon \(T\), number of items \(K\), feature dimension \(d\), feature vectors \(\mathbf{\phi}_{i,j}\) for \(i\in[K]\), \(j\in[K]\), learning rate \(\eta\), exploration parameter \(\gamma\).
2:Initialize:\(q_{1}(i)=\frac{1}{K}\).
3:for\(t=1,\ldots,T\)do
4: Sample items \(i_{t}\sim q_{t}\), \(j_{t}\sim q_{t}\).
5: Query pair \((i_{t},j_{t})\) and receive feedback \(r_{t}\)
6: Calculate \(Q_{t}=\sum_{i\in[K]}\sum_{j\in[K]}q_{t}(i)q_{t}(j)\mathbf{\phi}_{i,j}\mathbf{\phi}_{i,j }^{\top},\widehat{\mathbf{\theta}}_{t}=Q_{t}^{-1}\mathbf{\phi}_{i_{t},j_{t}}r_{t}\).
7: Calculate the (shifted) Borda score estimates \(\widehat{B}_{t}(i)=\langle\frac{1}{K}\sum_{j\in[K]}\mathbf{\phi}_{i,j},\widehat{ \mathbf{\theta}}_{t}\rangle\).
8: Update for all \(i\in[K]\), set \[\widetilde{q}_{t+1}(i)=\frac{\exp(\eta\sum_{l=1}^{t}\widehat{B}_{l}(i))}{\sum _{j\in[K]}\exp(\eta\sum_{l=1}^{t}\widehat{B}_{l}(j))};\hskip 28.452756ptq_{t+1}(i )=(1-\gamma)\widetilde{q}_{t+1}(i)+\frac{\gamma}{K}.\]
9:endfor
```

**Algorithm 2** BEXP3

Algorithm 2 is adapted from the DEXP3 algorithm in Saha et al. (2021), which deals with the adversarial multi-armed dueling bandit. Algorithm 2 maintains a distribution \(q_{t}(\cdot)\) over \([K]\), initialized as uniform distribution (Line 2). At every round \(t\), two items are chosen following \(q_{t}\) independently. Then Line 6 calculates the one-sample unbiased estimate \(\widehat{\mathbf{\theta}}_{t}\) of the true underlying parameter \(\mathbf{\theta}^{*}_{t}\). Line 7 further calculates the unbiased estimate of the (shifted) Borda score. Note that the true Borda score at round \(t\) satisfies \(B_{t}(i)=\frac{1}{2}+\langle\frac{1}{K}\sum_{j\in[K]}\mathbf{\phi}_{i,j},\mathbf{ \theta}^{*}_{t}\rangle\). \(\widehat{B}_{t}\) instead only estimates the second term of the Borda score. This is a choice to simplify the proof. The cumulative estimated score \(\sum_{t=1}^{t}\widehat{B}_{l}(i)\) can be seen as the estimated cumulative reward of item \(i\) at round \(t\). In Line 8, \(q_{t+1}\) is defined by the classic exponential weight update, along with a uniform exploration policy controlled by \(\gamma\).

### Upper Bounds

Algorithm 2 can also be configured to tightly match the worst-case lower bound:

**Theorem 9**.: Suppose Assumption 1 holds. If we set \(\eta=(\log K)^{2/3}d^{-1/3}T^{-2/3}\) and \(\gamma=\sqrt{\eta d/\lambda_{0}}=(\log K)^{1/3}d^{1/3}T^{-1/3}\lambda_{0}^{-1/2}\), then the expected regret is upper-bounded by

\[O\big{(}(d\log K)^{1/3}T^{2/3}\big{)}.\]

Note that the lower bound construction is for the linear model and has \(K=O(2^{d})\), thus exactly matching the upper bound.

Experiments

This section compares the proposed algorithm BETC-GLM with existing ones that are capable of minimizing Borda regret. We use random responses (generated from fixed preferential matrices) to interact with all tested algorithms. Each algorithm is run for 50 times over a time horizon of \(T=10^{6}\). We report both the mean and the standard deviation of the cumulative Borda regret and supply some analysis. The following list summarizes all methods we studies in this section, a more complete description of the methods and parameters are available in Appendix E: BETC-GLM(-Match): Algorithm 1 proposed in this paper with different parameters. UCB-Borda: The UCB algorithm (Auer et al., 2002) using _Borda reduction_. DEXP3: Dueling-Exp3 developed by Saha et al. (2021a). ETC-Borda: A simple explore-then-commit algorithm that does not take any contextual information into account. BEXP3: The proposed method for adversarial Borda bandits displayed in Algorithm 2.

Generated Hard CaseWe first test the algorithms on the hard instances constructed in Section 4. We generate \(\mathbf{\theta}^{*}\) randomly from \(\{-\Delta,+\Delta\}^{d}\) with \(\Delta=\frac{1}{4d}\) so that the comparison probabilities \(p_{i,j}^{\mathbf{\theta}^{*}}\in[0,1]\) for all \(i,j\in[K]\). We pick the dimension \(d=6\) and the number of arms is therefore \(K=2^{d+1}=128\). Note the dual usage of \(d\) in our construction and the model setup in Section 3.1. We refer readers to Remark 11 in Appendix B for more details.

As depicted in Figure 1(a), the proposed algorithms (BETC-GLM, BEXP3) outperform the baseline algorithms in terms of cumulative regret when reaching the end of time horizon \(T\). For UCB-Borda, since it is not tailored for the dueling regret definition, it suffers from a linear regret as its second arm is always sampled uniformly at random, leading to a constant regret per round. DEXP3 and ETC-Borda are two algorithms designed for \(K\)-armed dueling bandits. Both are unable to utilize contextual information and thus demand more exploration. As expected, their regrets are higher than BETC-GLM or BEXP3.

Real-world DatasetTo showcase the performance of the algorithms in a real-world setting, we use EventTime dataset (Zhang et al., 2016). In this dataset, \(K=100\) historical events are compared in a pairwise fashion by crowd-sourced workers. We first calculate the empirical preference probabilities \(\widetilde{p}_{i,j}\) from the collected responses, and construct a generalized linear model based on the empirical preference probabilities. The algorithms are tested under this generalized linear model. Due to space limitations, more details are deferred to Appendix F.

As depicted in Figure 1(b), the proposed algorithm BETC-GLM outperforms the baseline algorithms in terms of cumulative regret when reaching the end of time horizon \(T\). The other proposed algorithm BEXP3 performs equally well even when misspecified (the algorithm is designed for linear setting, while the comparison probability follows a logistic model).

## 8 Conclusion and Future Work

In this paper, we introduced Borda regret into the generalized linear dueling bandits setting, along with an explore-then-commit type algorithm BETC-GLM and an EXP3 type algorithm BEXP3. The algorithms can achieve a nearly optimal regret upper bound, which we corroborate with a matching lower bound. The theoretical performance of the algorithms is verified empirically. It demonstrates superior performance compared to other baseline methods.

For future works, due to the fact that our exploration scheme guarantees an accurate estimate in all directions, our work can be extended to solve the top-k recovery or ranking problem, as long as a proper notion of regret can be identified.

Figure 2: The regret of the proposed algorithms (BETC-GLM, BEXP3) and the baseline algorithms (UCB-Borda, DEXP3, ETC-Borda).

## References

* Abbasi-Yadkori et al. (2011)Abbasi-Yadkori, Y., Pal, D. and Szepesvari, C. (2011). Improved algorithms for linear stochastic bandits. In _NIPS_.
* Auer et al. (2002)Auer, P., Cesa-Bianchi, N. and Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. _Machine Learning_**47** 235-256.
* Balsubramani et al. (2016)Balsubramani, A., Karnin, Z., Schapire, R. E. and Zoghi, M. (2016). Instance-dependent regret bounds for dueling bandits. In _Conference on Learning Theory_. PMLR.
* Bengs et al. (2021)Bengs, V., Busa-Fekete, R., El Mesaoudi-Paul, A. and Hullermeier, E. (2021). Preference-based online learning with dueling bandits: A survey. _Journal of Machine Learning Research_**22** 7-1.
* Busa-Fekete et al. (2018)Busa-Fekete, R., Hullermeier, E. and Mesaoudi-Paul, A. E. (2018). Preference-based online learning with dueling bandits: A survey. _ArXiv_**abs/1807.11398**.
* Chen et al. (2013)Chen, X., Bennett, P. N., Collins-Thompson, K. and Horvitz, E. (2013). Pairwise ranking aggregation in a crowdsourced setting. In _Proceedings of the sixth ACM international conference on Web search and data mining_.
* Dani et al. (2008)Dani, V., Hayes, T. P. and Kakade, S. M. (2008). Stochastic linear optimization under bandit feedback. In _Annual Conference Computational Learning Theory_.
* Dudik et al. (2015)Dudik, M., Hofmann, K., Schapire, R. E., Slivkins, A. and Zoghi, M. (2015). Contextual dueling bandits. _ArXiv_**abs/1502.06362**.
* Even-Dar et al. (2002)Even-Dar, E., Mannor, S. and Mansour, Y. (2002). Pac bounds for multi-armed bandit and markov decision processes. In _Annual Conference Computational Learning Theory_.
* Falahatgar et al. (2017a)Falahatgar, M., Hao, Y., Orlitsky, A., Pichapati, V. and Ravindrakumar, V. (2017a). Maxing and ranking with few assumptions. _Advances in Neural Information Processing Systems_**30**.
* Falahatgar et al. (2018)Falahatgar, M., Jain, A., Orlitsky, A., Pichapati, V. and Ravindrakumar, V. (2018). The limits of maxing, ranking, and preference learning. In _International conference on machine learning_. PMLR.
* Falahatgar et al. (2017b)Falahatgar, M., Orlitsky, A., Pichapati, V. and Suresh, A. T. (2017b). Maximum selection and ranking under noisy comparisons. In _International Conference on Machine Learning_. PMLR.
* Faury et al. (2020)Faury, L., Abeille, M., Calauzenes, C. and Fercoq, O. (2020). Improved optimistic algorithms for logistic bandits. In _International Conference on Machine Learning_. PMLR.
* Filippi et al. (2010)Filippi, S., Cappe, O., Garivier, A. and Szepesvari, C. (2010). Parametric bandits: The generalized linear case. _Advances in Neural Information Processing Systems_**23**.
* Heckel et al. (2018)Heckel, R., Simchowitz, M., Ramchandran, K. and Wainwright, M. (2018). Approximate ranking from pairwise comparisons. In _International Conference on Artificial Intelligence and Statistics_. PMLR.
* Jamieson et al. (2015)Jamieson, K., Katariya, S., Deshpande, A. and Nowak, R. (2015). Sparse dueling bandits. In _Artificial Intelligence and Statistics_. PMLR.
* Jun et al. (2017)Jun, K.-S., Bhargava, A., Nowak, R. and Willett, R. (2017). Scalable generalized linear bandits: Online computation and hashing. _Advances in Neural Information Processing Systems_**30**.
* Komiyama et al. (2016)Komiyama, J., Honda, J. and Nakagawa, H. (2016). Copeland dueling bandit problem: Regret lower bound, optimal algorithm, and computationally efficient algorithm. In _International Conference on Machine Learning_. PMLR.
* Kuleshov and Precup (2014)Kuleshov, V. and Precup, D. (2014). Algorithms for multi-armed bandit problems. _arXiv preprint arXiv:1402.6028_.

* Lai et al. (1985)Lai, T. L., Robbins, H. et al. (1985). Asymptotically efficient adaptive allocation rules. _Advances in applied mathematics_**6** 4-22.
* Lattimore and Szepesvari (2020)Lattimore, T. and Szepesvari, C. (2020). _Bandit Algorithms_. Cambridge University Press.
* Li et al. (2017)Li, L., Lu, Y. and Zhou, D. (2017). Provably optimal algorithms for generalized linear contextual bandits. In _International Conference on Machine Learning_. PMLR.
* Liu et al. (2017)Liu, C., Jin, T., Hoi, S. C. H., Zhao, P. and Sun, J. (2017). Collaborative topic regression for online recommender systems: an online and bayesian approach. _Machine Learning_**106** 651-670.
* Lou et al. (2022)Lou, H., Jin, T., Wu, Y., Xu, P., Gu, Q. and Farnoud, F. (2022). Active ranking without strong stochastic transitivity. _Advances in neural information processing systems_.
* Minka et al. (2018)Minka, T. P., Cleven, R. and Zaykov, Y. (2018). Trueskill 2: An improved bayesian skill rating system.
* Ramamohan et al. (2016)Ramamohan, S., Rajkumar, A. and Agarwal, S. (2016). Dueling bandits: Beyond condorcet winners to general tournament solutions. In _NIPS_.
* Ren et al. (2019)Ren, W., Liu, J. K. and Shroff, N. (2019). On sample complexity upper and lower bounds for exact ranking from noisy comparisons. _Advances in Neural Information Processing Systems_**32**.
* Rusmevichientong and Tsitsiklis (2010)Rusmevichientong, P. and Tsitsiklis, J. N. (2010). Linearly parameterized bandits. _Mathematics of Operations Research_**35** 395-411.
* Saha (2021)Saha, A. (2021). Optimal algorithms for stochastic contextual preference bandits. _Advances in Neural Information Processing Systems_**34** 30050-30062.
* Saha et al. (2021a)Saha, A., Koren, T. and Mansour, Y. (2021a). Adversarial dueling bandits. _ArXiv_**abs/2010.14563**.
* Saha et al. (2021b)Saha, A., Koren, T. and Mansour, Y. (2021b). Adversarial dueling bandits. In _International Conference on Machine Learning_. PMLR.
* Sui and Burdick (2014)Sui, Y. and Burdick, J. (2014). Clinical online recommendation with subgroup rank feedback. In _Proceedings of the 8th ACM conference on recommender systems_.
* Urvoy et al. (2013)Urvoy, T., Clerot, F., Feraud, R. and Naamane, S. (2013). Generic exploration and k-armed voting bandits. In _ICML_.
* Vasile et al. (2016)Vasile, F., Smirnova, E. and Conneau, A. (2016). Meta-prod2vec: Product embeddings using side-information for recommendation. In _Proceedings of the 10th ACM conference on recommender systems_.
* Wang et al. (2018)Wang, J., Huang, P., Zhao, H., Zhang, Z., Zhao, B. and Lee (2018). Billion-scale commodity embedding for e-commerce recommendation in alibaba. _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_.
* Wu and Liu (2016)Wu, H. and Liu, X. (2016). Double thompson sampling for dueling bandits. _ArXiv_**abs/1604.07101**.
* Wu et al. (2022)Wu, Y., Jin, T., Lou, H., Xu, P., Farnoud, F. and Gu, Q. (2022). Adaptive sampling for heterogeneous rank aggregation from noisy pairwise comparisons. In _International Conference on Artificial Intelligence and Statistics_. PMLR.
* Yue et al. (2012)Yue, Y., Broder, J., Kleinberg, R. D. and Joachims, T. (2012). The k-armed dueling bandits problem. _J. Comput. Syst. Sci._**78** 1538-1556.
* Yue and Joachims (2009)Yue, Y. and Joachims, T. (2009). Interactively optimizing information retrieval systems as a dueling bandits problem. In _Proceedings of the 26th Annual International Conference on Machine Learning_.
* Yue and Joachims (2011)Yue, Y. and Joachims, T. (2011). Beat the mean bandit. In _International Conference on Machine Learning_.

* Zhang et al. (2016)Zhang, X., Li, G. and Feng, J. (2016). Crowdsourced top-k algorithms: An experimental evaluation. _Proc. VLDB Endow._**9** 612-623.
* Zhu et al. (2012)Zhu, J., Ahmed, A. and Xing, E. P. (2012). Medlda: maximum margin supervised topic models. _J. Mach. Learn. Res._**13** 2237-2278.
* Zoghi et al. (2015)Zoghi, M., Karnin, Z. S., Whiteson, S. and de Rijke, M. (2015). Copeland dueling bandits. In _NIPS_.
* Zoghi et al. (2014)Zoghi, M., Whiteson, S., Munos, R. and de Rijke, M. (2014a). Relative upper confidence bound for the k-armed dueling bandit problem. _ArXiv_**abs/1312.3393**.
* Zoghi et al. (2014b)Zoghi, M., Whiteson, S., Munos, R. and Rijke, M. (2014b). Relative upper confidence bound for the k-armed dueling bandit problem. In _International conference on machine learning_. PMLR.

Response:
## Review

### Summary

This paper investigates the problem of minimizing Borda regret within the context of dueling bandits, specifically under a generalized linear model where each arm pair is associated with a context vector. The authors develop algorithms for both stochastic and adversarial settings, demonstrating that their solutions achieve asymptotic Borda regret that aligns with established lower bounds. While the paper presents some novel contributions, particularly in the form of new lower bounds and empirical results, concerns have been raised about the originality and novelty of the algorithms proposed. Overall, the paper addresses an interesting and significant problem in the field, although some aspects could benefit from further clarification and innovation.

### Strengths

- The paper is well written and easy to read.
- The authors provide a thorough treatment of previous works, illustrating the novelty of their results.
- The proposed algorithms match upper bounds and present interesting constructions for hard instances.
- Experimental results demonstrate empirical performance that outperforms state-of-the-art methods.
- The presentation of algorithms and proofs is reader-friendly and enhances clarity.

### Weaknesses

- The originality of the algorithms is limited, as they appear to be straightforward extensions of existing approaches.
- Concerns about the experimental setup and the convincing nature of the results.
- Some sections vital for understanding the novelty are missing from the main text.
- There are several minor typos and clarity issues present throughout the paper.

### Questions

- Have the authors considered relaxed assumptions that could provide insights into optimality for explore-then-commit algorithms?
- Does the hard instance in Remark 11 fulfill all assumptions 1-3, and if not, could another instance be provided?
- What makes the link function particularly challenging in the context of the BEXP3 algorithm?

### Soundness

**Score:** 3

**Description:** 3 = good; The theoretical arguments are generally sound, but some aspects of the algorithms’ originality and validation could be better articulated.

### Presentation

**Score:** 4

**Description:** 4 = excellent; The paper is well-structured and mostly clear, yet minor issues and typos slightly detract from overall clarity.

### Contribution

**Score:** 3

**Description:** 3 = good; The paper contributes to an important area with notable results, but the novelty of the algorithms is questionable.

### Rating

**Score:** 6

**Description:** 6 = Weak Accept; The paper is technically solid with moderate-to-high impact, but it requires further refinement and clarification in certain areas.

### Paper Decision

**Decision:** Reject

**Reasons:** The decision to reject is based on the lack of originality in the contributions and the concern that the techniques presented largely aggregate existing literature without significant innovation. The paper does not sufficiently meet the standards for acceptance at NeurIPS. Authors are encouraged to address the reviewers' comments and resubmit to a more suitable venue.