# Unconfounded Propensity Estimation for Unbiased Ranking

Dan Luo<EMAIL_ADDRESS>Lehigh UniversityBethlehemPAUSA ,  Lixin Zou†
<EMAIL_ADDRESS>Wuhan UniversityWuhanChina ,  Qingyao Ai
<EMAIL_ADDRESS>Tsinghua UniversityBeijingChina ,  Zhiyu Chen∗
<EMAIL_ADDRESS>Amazon.com, Inc.SeattleWAUSA ,  Chenliang Li
<EMAIL_ADDRESS>Wuhan UniversityWuhanChina ,  Dawei Yin<EMAIL_ADDRESS>Baidu Inc.BeijingChina and  Brian D. Davison<EMAIL_ADDRESS>Lehigh
UniversityBethlehemPAUSA

(2018)

###### Abstract.

The goal of unbiased learning to rank (ULTR) is to leverage implicit user
feedback for optimizing learning-to-rank systems. Among existing solutions,
automatic ULTR algorithms that jointly learn user bias models (_i.e.,_
propensity models) with unbiased rankers have received a lot of attention due
to their superior performance and low deployment cost in practice. Despite
their theoretical soundness, the effectiveness is usually justified under a
weak logging policy, where the ranking model can barely rank documents
according to their relevance to the query. However, when the logging policy is
strong, e.g., an industry-deployed ranking policy, the reported effectiveness
cannot be reproduced. In this paper, we first investigate ULTR from a causal
perspective and uncover a negative result: existing ULTR algorithms fail to
address the issue of propensity overestimation caused by the query-document
relevance confounder. Then, we propose a new learning objective based on
backdoor adjustment and highlight its differences from conventional propensity
models, which reveal the prevalence of propensity overestimation. On top of
that, we introduce a novel propensity model called Logging-Policy-aware
Propensity (LPP) model and its distinctive two-step optimization strategy,
which allows for the joint learning of LPP and ranking models within the
automatic ULTR framework, and actualize the unconfounded propensity estimation
for ULTR. Extensive experiments on two benchmarks demonstrate the
effectiveness and generalizability of the proposed method.

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Unbiased Learning to Rank (ULTR), leveraging implicit user feedback for
optimizing learning-to-rank systems, has been extensively studied in
information retrieval (Joachims, 2002). Usually, directly optimizing ranking
models with click data will suffer from the intrinsic noise and bias in user
interaction. In particular, the position bias (Craswell et al., 2008) that
occurs because users are more likely to examine documents at higher ranks is
known to distort ranking optimization if not handled properly (Joachims et
al., 2005, 2007). 111Position bias is formally modeled by assuming the
examination only depends on the position. Among existing solutions, ULTR
algorithms that jointly estimate click bias and construct unbiased ranking
models, namely the AutoULTR algorithms, have drawn a lot of attention (Wang et
al., 2018; Ai et al., 2018b; Hu et al., 2019). Because they do not need to
conduct separate user studies or online experiments to estimate user bias
(_i.e.,_ the propensity models), and can be deployed on existing systems
without hurting user experiences.

Despite the theoretical soundness, their effectiveness is usually justified
under a weak logging policy, where the ranking model can barely rank documents
correctly according to their relevance to the query. However, when the logging
policy is strong, especially an industry-deployed ranking policy, the reported
effectiveness cannot be reproduced. Since relevant documents are more likely
to be presented on top positions under a strong logging policy, the observed
click-through rates on top positions would be higher than weak logging policy,
_i.e.,_ users click more on the top positions. Therefore, the estimated
propensity on top positions would be larger than actual values, which we refer
to as propensity overestimation problem (Detailed empirical results and
analysis in Section 6.2). Since industrial LTR systems will be updated
dynamically, implicit feedback will only be valuable when the propensity
overestimation is addressed.

In this paper, we investigate propensity overestimation in ULTR through the
lens of causality. By analyzing the causal relations in ULTR, we identify the
confounder, _i.e.,_ query-document relevance, and derive the propensity’s
compositions in existing ULTR methods: (1) the causal effect between the
position and examination, _i.e.,_ the desired propensity; and (2) the
confounding effect caused by the relevance, _i.e.,_ the overestimated part. To
eliminate the confounding effect, a straightforward solution is to adopt
backdoor adjustment (Pearl et al., 2000), which, in ULTR, means building a
propensity model that takes both the position and relevance into account.
However, optimizing this propensity model is non-trivial because separating
ranking and propensity models in AutoULTR algorithms is infeasible when they
share a common input (_i.e.,_ query-document features) and target (_i.e.,_
user clicks) (Yang et al., 2020).

For unconfounded propensity estimation in ULTR, which we refer to as UPE, we
propose a novel propensity model, namely Logging-Policy-aware Propensity
Model, and its distinct two-step optimization strategy: (1) logging-policy-
aware confounding effect learning learns a mapping from query-document feature
vectors to logging policy scores, which captures the confounding effect caused
by the relevance confounder and thereby separates the effects of ranking and
propensity model. (2) joint propensity learning learns a mapping from the
query-document features and position to the examination by locking the
confounding effect part and solely optimizing the position-related parameters.
Thereafter, we are able to conduct unconfounded inference via backdoor
adjustment and actualize AutoULTR. Extensive experiments on two benchmarks
with synthetic clicks on online and offline simulations demonstrate
superiority of UPE.

The contributions of this work can be summarized as follows:

  * •

We propose the propensity overestimation phenomenon and firstly conduct the
causal analysis in ULTR to identify the confounding effect for the
overestimation problem.

  * •

We propose a novel Logging-Policy-aware Propensity Model and its distinct two-
step optimization strategy: (1) logging-policy-aware confounding effect
learning and (2) joint propensity learning, which solves the difficulty of
backdoor adjustment in ULTR.

  * •

We conduct extensive experiments on two benchmarks with synthetic clicks on
online and offline simulations to demonstrate the superiority of our proposal.

##  2\. Related Work

We summarize the related literature on Unbiased Learning to Rank and
Deconfounding in Information Retrieval as follows.

##### Unbiased Learning to Rank

To leverage implicit user feedback for optimizing LTR systems, there are two
streams for ULTR. One school depends on click modeling, which usually
maximizes the likelihood of the observed data, models the examination
probability and infers accurate relevance feedback from user clicks (Craswell
et al., 2008; Chapelle et al., 2009; Chuklin et al., 2016; Mao et al., 2018)
However, click models usually requires multiple times appearances of the same
query-document pair for reliable inference (Mao et al., 2019); thus they may
fall short for tail queries. The other school derives from counterfactual
learning, which treats bias as a counterfactual factor and debiases user
clicks via inverse propensity weighting (Joachims et al., 2017; Wang et al.,
2016). Among them, automatic unbiased learning to rank methods, which jointly
learn user bias models (_i.e.,_ propensity models) with unbiased rankers, have
received a lot of attention due to their superior performance and low
deployment cost.

Based on automatic ULTR methods, recent work has investigated various biased
within ULTR, including position bias (Ai et al., 2018a; Ren et al., 2022; Hu
et al., 2019; Chen et al., 2022b; Vardasbi et al., 2020), contextual position
bias (Chen et al., 2021, 2022a; Zhuang et al., 2021; Yan et al., 2022), trust
bias (Vardasbi et al., 2020, 2021), exploitation bias (Yang et al., 2022),
click unnecessary bias (Wang et al., 2021b), factorizability of the
examination hypothesis (Chen et al., 2022b). Unfortunately, their
effectiveness has only been justified under weak logging policies, which
cannot be reproduced under a strong policy. Our work offers an answer from a
causal perspective by identifying relevance as a confounder, and demonstrating
a propensity overestimation problem. Our proposal not only improves the
ranking performance of ranking models, but also provides an explanation for
the improvement and strong theoretical guarantees of unbiasedness.

##### Deconfounding in Information Retrieval

Recently, causal-aware methods have thrived in information retrieval. In
particular, some efforts have been made to address confounding problems in
recommendation systems. Those methods adopt causal inference to analyze the
root causes of bias problems (Zhang et al., 2021; Wang et al., 2021a; Gupta et
al., 2021; Christakopoulou et al., 2020; Sato et al., 2020) and apply backdoor
adjustment (Pearl et al., 2000) during the training or inference to address
the bias problems. For example, Wang et al. (2021a) identify the distribution
of historical interactions as a confounder for bias amplification. Zhang et
al. (2021) identify popularity as a confounder that affects both item
exposures and user clicks. However, these methods require the confound
variables to be observable, while in ULTR the confounder – document relevance
– is unobservable.

There are also a few efforts that address confounding effects without the need
to be observable (Wang et al., 2020; Liu et al., 2021). For example, Liu et
al. (2021) learn a biased embedding vector with independent biased and
unbiased components in the training phase. In testing, only the unbiased
component is used to deliver more accurate recommendations. Unlike those
methods, extracting separate ranking and propensity models in unbiased
learning to rank is difficult when they share a common input and target. In
summary, these differences make existing deconfounding methods not applicable
in ULTR.

##  3\. Preliminaries

In this section, we formulate ULTR with implicit feedback, and introduce the
inverse propensity weighting for ULTR.

###  3.1. Problem Formulation

Let $\mathcal{D}$ be the universal set of documents, and $\mathcal{Q}$ be the
universal set of queries. For a user-issued query $q\in\mathcal{Q}$, $\pi_{q}$
is the ranked list retrieved for query $q$, $d_{k}\in\pi_{q}$ is the document
presented at position $k$, $\mathbf{x}_{k}\in\mathcal{X}$ and
$r_{k}\in\\{0,1\\}$ are the feature vector and binary relevance of the query
document pair $(q,d_{k})$, respectively. The goal of learning to rank is to
find a mapping function $f$ from a query document feature vector
$\mathbf{x}_{k}$ to its relevance $r_{k}$. In most cases, we are only
concerned with the position of relevant documents ($r_{d}=1$) in retrieval
evaluations (_e.g.,_ MAP, nDCG (Järvelin and Kekäläinen, 2002), ERR (Chapelle
et al., 2009)), so we can formulate the ideal local ranking loss
$\mathcal{L}_{ideal}$ as:

(1)
$\mathcal{L}_{ideal}(f,q|\pi_{q})=\sum_{d_{k}\in\pi_{q},r_{k}=1}\Delta(f(\mathbf{x}_{k}),r_{k}|\pi_{q}),$

where $\Delta$ is a function that computes the individual loss on each
document. An alternative to relevance $r_{k}$ is implicit feedback from users,
such as clicks. If we conduct learning to rank by replacing the relevance
label $r_{k}$ with click label $c_{k}$ in Eq. 1, then the empirical local
ranking loss is derived as follows,

(2)
$\displaystyle\mathcal{L}_{naive}(f,q|\pi_{q})=\sum_{d_{k}\in\pi_{q},c_{k}=1}\Delta(f(\mathbf{x}_{k}),c_{k}|\pi_{q}),$

where $c_{k}$ is a binary variable indicating whether the document at position
$k$ is cliked in the ranked list $\pi_{q}$. However, this naive loss function
is biased, due to factors such as position bias (Joachims et al., 2017). To
address this issue, unbiased learning-to-rank aims to train a ranking model
$f$ with the biased user clicks collected but immune to the bias in the
implicit feedback.

###  3.2. AutoULTR Algorithms

According to the position bias assumption that the examination only depends on
the position, AutoULTR algorithms usually assume the examination and relevance
are independent under a weak logging policy. Thereafter, according to the
Examination Hypothesis (Richardson et al., 2007):

(3) $c_{k}=1\Longleftrightarrow(e_{k}=1\ \ \text{and}\ \ r_{k}=1),$

the problem of learning a propensity model from click data (_i.e.,_ the
estimation of bias in clicks) can be treated as a dual problem of constructing
an unbiased learning-to-rank model; and its unbiasedness has been justified in
Ai et al. (2018a).

Let $e_{k}$ be the binary variables that represent whether document $d_{k}$ is
examined by a user. In AutoULTR, an unbiased ranking system $f$ and a
propensity model $g$ can be jointly learned by optimizing the local ranking
loss as:

(4a)
$\displaystyle\mathcal{L}_{IPW}(f,q|\pi_{q})=\sum_{d_{k}\in\pi_{q},c_{k}=1}\frac{\Delta(f(\mathbf{x}_{k}),c_{k}|\pi_{q})}{g(k)}$
(4b)
$\displaystyle\mathcal{L}_{IRW}(g,q|\pi_{q})=\sum_{d_{k}\in\pi_{q},c_{k}=1}\frac{\Delta(g(k),c_{k}|\pi_{q})}{f(\mathbf{x}_{k})}$

where $g(k)$ is the instantiation of $P(e_{k}=1)$ estimating the propensity at
position $k$. A nice propensity is that both estimations are only affected by
clicked documents $c_{k}=1$, respectively.

##  4\. Causal Analysis on Propensity Overestimation

###  4.1. Causal View of ULTR

(a)

(b)

Figure 1. (a) The causal graph $G$ for ULTR. (b) The causal graph with
intervention $G^{\prime}$ used in UPE. In particular, we apply backdoor
adjustment to cut off the association $X\rightarrow K$ for propensity model
learning. A gray node indicates that the variable is observable.

To illustrate propensity overestimation in ULTR, we first scrutinize its
causal relations and present a causal graph, as shown in Fig. 1(a), which
consists of six variables $\\{X,R,\hat{R},K,E,C\\}$ as follows:

  * •

$R$ represents the real relevance between the query and document. For
simplicity, we assume it is a binary variable with $r\in\\{0,1\\}$. However,
$R$ is unobservable and it is typically estimated by the ranking model in ULTR
algorithms with raw click data.

  * •

$X$ is the representation for query-document pair $(q,d)$, where a particular
value $\mathbf{x}\in\mathbb{R}^{n}$ is the feature vector for the pair.

  * •

$\hat{R}$ with $\hat{r}\in\mathbb{R}$ is the estimated relevance score
generated by the logging policy.

  * •

$K$ denotes $d$’s rank on the search result page, where
$K\in\\{1..\mathcal{K}\\}$ and $\mathcal{K}$ is the maximum number of
observable documents.

  * •

$E$ is a binary variable $e\in\\{0,1\\}$ to indicate whether document $d$ is
examined, which is unobservable and typically estimated by the propensity
model.

  * •

$C$ is a binary variable $c\in\\{0,1\\}$ to denote whether the user clicked
document $d$.

The graph’s edges describe causal relations between variables:

  * •

$X\rightarrow R$: this edge shows that there exists a mapping from query-
document representations to their relevance , which is the goal of unbiased
learning to rank.

  * •

$X\rightarrow\hat{R}$: feature representation determines the estimated
relevance score, based on the logging policy.

  * •

$\hat{R}\rightarrow K$: the logging policy presents a list of documents in
descending order by the estimated relevance scores. Without losing generality,
we assume the estimated relevance score of the document decides its ranked
position, and ignores the influence of comparing with other documents.

  * •

$K\rightarrow E$: due to position bias, the ranked positions affect the chance
that a document is examined by a user.

  * •

$(R,E)\rightarrow C$: the edge follows the examination hypothesis (Richardson
et al., 2007) that a user would only click a document when it is observed by
the user and considered relevant to the user’s need.

According to causal theory (Pearl et al., 2000), query-document feature
representation $X$ is a confounder as $K\leftarrow X\rightarrow C$, which
leads to a spurious correlation in formulating the propensity model.

###  4.2. Analysis of Propensity Overestimation

Based on the causal graph illustrated, we derive the propensity estimand in
existing ULTR methods. As discussed previously, they mainly formulate the
propensity model conditioned on the clicked items as $P(E|K,C)$. In causality
language, this formulation means: $K$ is the cause, and $E$ is the effect.
Thereafter, we can derive propensity estimand $P(E|K,C)$ as follows:

(5a) $\displaystyle\underbrace{P(E|K,C)}_{\text{propensity estimand}}$
$\displaystyle=\sum_{\mathbf{x}}P(E,\mathbf{x}|K,C)$ (5b)
$\displaystyle=\sum_{\mathbf{x}}P(E|\mathbf{x},K,C)P(\mathbf{x}|K,C),$ (5c)
$\displaystyle\varpropto\sum_{\mathbf{x}}\underbrace{P(E|\mathbf{x},K,C)P(\mathbf{x}|C)}_{\text{causal}}\cdot\underbrace{P(K|\mathbf{x},C)}_{\text{confounding}},$

where Eq. 5a is the definition of the law of total probability; Eq. 5b and Eq.
5c follow the Bayes rules, respectively. In particular, the proportion
operation $\varpropto$ does not affect the effectiveness of the proposal,
please refer to Section 5.2.2 for more details.

We can clearly see the propensity estimand $P(E|K,C)$ in existing ULTR methods
consists of two parts: (1) $P(E|\mathbf{x},K,C)P(\mathbf{x}|C)$ that
contributes to a causal effect between the position and examination, _i.e.,_
the desired propensity, which will be illustrated in Eq. 6 later; and (2)
$P(K|\mathbf{x},C)$ that contributes to a confounding effect. Remarkably,
$P(K|\mathbf{x},C)$ fundamentally changes the propensity estimand $P(E|K,C)$,
especially when users’ clicks are collected based on an industry-deployed
logging policy. Suppose K is a top position, a more relevant document will
have a higher chance to be ranked on position K, resulting in a larger value
of $P(K|\mathbf{x},C)$ for that document. Therefore, propensity estimand
$P(E|K,C)$ on observed data will be larger than the actual value. In short,
the query-document relevance confounder $X$ leads to the propensity
overestimation.

According to the justification in Ai et al. (2018a), the unbiasedness of the
propensity model in Eq. 4b requires the unbiasedness of the ranking model, and
vice versa. Since $P(E|K,C)$ is biased, conventional AutoULTR algorithms would
fail to converge to the unbiased ranking and propensity models.

Note that this confounding effect widely exists across all logging policies,
but its effect is unintentionally concealed in the general ULTR setting,
_i.e.,_ clicks are collected under a weak logging policy, where
$P(K|\mathbf{x},c)$ is almost the same for all documents. A weak logging
policy can barely rank documents correctly according to their relevance to the
query, so the confounding effect is minor but non-negligible. Experimental
results in Section 6.4 verify that our solution could obtain a better-
performance ranking model even when the backdoor path leads to a minor
confounding effect under a weak logging policy.

##  5\. Methodology

Figure 2. The workflow of the proposed LPP framework. Its optimization
strategy consists of two steps: (1) confounding effect learning; (2) joint
propensity learning. The unconfounded propensity is obtained by conducting
backdoor adjustment over the LPP model.

In this section, we detail our solution, which is referred to as Unconfounded
Propensity Estimation (UPE). We first resort to the backdoor adjustment (Pearl
et al., 2000; Wang et al., 2020), which enables the causal effect estimation
without causal intervention. Then we propose Logging-policy-aware Propensity
(LPP) model, and its novel learning strategy, namely logging-policy-aware
confounding effect learning and joint propensity learning. Thereafter, we
conduct backdoor adjustment on LPP model for unconfounded propensity
inference.

###  5.1. Backdoor adjustment

Experimentally, one can resolve the impact of relevance confounder $X$ by
collecting intervened data where the document on specific position is forcibly
adjusted to eliminate the impact of the confounder. However, such an
experiment is too costly, and faces the risk of doing harm to user experience
in practice. Therefore, we resort to the causal technique: backdoor adjustment
(Pearl et al., 2000).

According to the theory of backdoor adjustment (Pearl et al., 2000), our
unconfounded propensity estimand is formulated as $P(E|do(K),C)$, where
$do(K)$ can be intuitively seen as cutting off the edge $\hat{R}\rightarrow K$
in $G$, as illustrated in Fig. 1(b). Remarkably, $P(E|do(K),C)$ estimates the
causal effect between position $K$ and examination $E$, rather than $P(E|K,C)$
estimated by existing methods. Now we are ready to derive the specific
expression of backdoor adjustment as

(6a) $\displaystyle P(E|do(K),C)$ $\displaystyle=P_{G^{\prime}}(E|K,C)$ (6b)
$\displaystyle=\sum_{\mathbf{x}}P_{G^{\prime}}(E|\mathbf{x},K,C)P_{G^{\prime}}(\mathbf{x}|K,C)$
(6c)
$\displaystyle=\sum_{\mathbf{x}}P_{G^{\prime}}(E|\mathbf{x},K,C)P_{G^{\prime}}(\mathbf{x}|C)$
(6d) $\displaystyle=\sum_{\mathbf{x}}P(E|\mathbf{x},K,C)P(\mathbf{x}|C),$

where $P_{G^{\prime}}$ denotes the probability function evaluated on
intervened causal graph $G^{\prime}$ in Fig. 1(b). In particular, Eq. 6a is
because of backdoor criterion (Pearl et al., 2000) as the only backdoor path
$K\leftarrow X\rightarrow C$ has been blocked by $do(K)$; Eq. 6b is obtained
by Bayes rules; since $K$ and $\mathbf{x}$ are independent in $G^{\prime}$,
$P_{G^{\prime}}(\mathbf{x}|C)=P_{G^{\prime}}(\mathbf{x}|K,C)$ in Eq. 6c; in
Eq. 6d, $P(E|\mathbf{x},K,C)=P_{G^{\prime}}(E|\mathbf{x},K,C)$ because the
causal relation/association $K\rightarrow E$ and $X\rightarrow E$ are not
changed when cutting off $\hat{R}\rightarrow K$, and
$P(\mathbf{x}|C)=P_{G^{\prime}}(\mathbf{x}|C)$ has the same prior on the two
graphs. To this end, we have demonstrated that the causal component in Eq. 5c
exactly contributes to the causal effect, which is the need for unconfounded
propensity desired.

Unlike estimand $P(E|K,C)$ in existing methods, our estimand $P(E|do(K),C)$
estimates the examination probability for each position with consideration of
every possible value of document $\mathbf{x}$ subject to the prior
$P(\mathbf{x}|C)$ in Eq. 6d, rather than $P(\mathbf{\mathbf{x}}|K,C)$ in
Eq.5c. Therefore, in our method, the documents with high relevance will not
receive high examination probability purely because of a higher rank position
via $P(\mathbf{x}|K,C)$, which addresses propensity overestimation.
Remarkably, the ranking model is unbiased at the presence of an unconfounded
propensity model and learned by an in-principled unbiased algorithm. On the
basis of causal theory, our unconfounded estimand learns the causal effect
between the position and examination; therefore, its unbiasedness can be
guaranteed when jointly works with in-principled unbiased algorithms.

Theoretically, the sample space of $\mathbf{X}$ is infinite, which makes the
calculation of $P(E|do(K),C)$ in Eq. 6d intractable. Therefore, we further
devise an approximation of backdoor adjustment for unconfounded propensity
inference by empirically averaging over the training samples as:

(7)
$P(E|do(K),C)=\frac{1}{|\mathcal{Q}_{b}|\cdot|\pi_{q}|}\sum_{q\in\mathcal{Q}_{b}}\sum_{d_{i}\in\pi_{q}}P(E|\mathbf{x}_{i},K,C),$

where $\mathcal{Q}_{b}$ is a batch of queries in the raw click data, $\pi_{q}$
is the ranked list for query $q$, $|\mathcal{Q}_{b}|$ and $|\pi_{q}|$ denotes
the number of queries within the batch and number of documents in the ranked
list, respectively.

###  5.2. Logging-Policy-aware Propensity Model

To facilitate the backdoor adjustment in Eq. 7, we need to instantiate a
propensity model as $P(E|X,K,C)$. However, it is difficult to extract separate
ranking models and propensity models when they share the common input (_i.e.,_
document) and target (_i.e.,_ user clicks) (Yang et al., 2020). Inspired by
the derivation in Eq. 5, we propose a novel propensity model that considers
both the relevance and position, named Logging-Policy-aware Propensity Model
(LPP) (as shown in Fig. 2), and its novel two-step optimization strategy:
logging-policy-aware confounding effect learning and joint propensity
learning.

Our model consists of three parts: a confounder encoder, a position encoder
and shared feed-forward network. The encoders learn a dense representations
for each query-document pair and its ranking position, respectively.
Afterward, a vector sum integrates relevance and position representations. The
feed-forward network receives the output from the encoder(s) and generates the
final output for optimization with respect to different targets. In
particular, we discuss the design choice of LPP model as follows.

Observable confounder variable. Existing deconfounding methods (Zhang et al.,
2021; Wang et al., 2021a; Yang et al., 2021; Zhan et al., 2022) usually
represent the joint effect by a simple multiplication among each decoupled
factor. Unfortunately, it is not feasible in ULTR because neither $P(E|K,C)$
nor $P(E|\mathbf{x},C)$ is observable. In LPP, we leverage a shared feed-
forward network to capture the joint effect of relevance features $X$ and
position $K$.

The motivation behind LPP model. At first glance, readers may confuse the LPP
in our work and contextual position bias (Yan et al., 2022; Zhuang et al.,
2021; Chen et al., 2021), since they both model $P(E|X,K,C)$. The key
difference between them is: the propensity model desired in our work is
$P(E|do(K),C)$ according to the position-based bias assumption; whereas it is
$P(E|X,K,C)$ according to the contextual position bias. In this work, we need
to build a propensity model $P(E|X,K,C)$ because we need to address propensity
overestimation caused by the relevance confounder as illustrated in Eq. 6.
More importantly, we need to separate the influence over propensity estimation
by the position and relevance from raw click data, which facilitates backdoor
adjustment and obtain the unconfounded propensity estimation $P(E|do(K),C)$.

####  5.2.1. Logging-policy-aware Confounding Effect Learning

The first step is to estimate the confounding effect caused by query-document
features $X$ under a logging policy, which corresponds to $P(K|X,C)$ in Eq.
5c. To this end, we propose logging-policy-aware confounding effect learning,
which learns a mapping from raw document features to logging policy scores
$\hat{R}$. Careful readers may notice that the target is logging policy scores
$\hat{R}$ instead of rank positions $K$. We will compare the effectiveness of
different fitting targets in Section 6.5.1.

Directly learning the mapping in a point-wise way, however, is problematic.
Since the logging policies, _i.e.,_ ranking models, are usually optimized
pairwise or listwise, the logging policy scores may follow different
distributions under different queries. This issue would restrict the
expressive ability of neural encoders.

Therefore, we propose to learn the mapping in a list-wise way, which is
invariant to different score distributions under different queries. Formally,
given a query $q$ and its associated documents $\pi_{q}=[d_{1},\cdots,d_{N}]$,
each feature vector for query-document pair $\mathbf{x}_{i}$ is transformed
into an $n$-dimensional latent feature representation:

(8) $\mathbf{m}_{d_{i}}=\text{Encoder}_{D}(q,d_{i}),\ \ \text{where
}\mathbf{m}_{d_{i}}\in\mathbb{R}^{n}.$

$\text{Encoder}_{D}$ is a confounder encoder that projects the raw query-
document feature vector $\mathbf{x}_{i}$ to latent representations
$\mathbf{m}_{d_{i}}$.

Afterward, the latent feature vector is passed into a feed-forward network,
and generate the predictive scores as:

(9) $\hat{y}^{r}_{d_{i}}=\text{FFN}(\mathbf{m}_{d_{i}}),\ \ \text{where
}\hat{y}^{r}_{d_{i}}\in\mathbb{R},$

FFN is a point-wise feed-forward network that projects the latent
representation to real-value scalar as prediction scores
$P(K|\mathbf{x}_{i},C)$.

Let $y^{r}_{d_{i}}$ be the logging policy score of document $d_{i}$ in query
$q$ that is observed in the logging policy, we optimize in the list-wise
paradigm through an attention rank loss function (Ai et al., 2018a):

(10)
$\mathcal{L}_{r}(g_{\text{pt}}|\pi_{q})=-\sum_{d_{i}\in\pi_{q}}\frac{\exp(y^{r}_{d_{i}})}{\sum_{z\in\pi_{q}}{\exp(y^{r}_{z})}}\cdot\log\frac{\exp(\hat{y}^{r}_{d_{i}})}{\sum_{z\in\pi_{q}}{\exp(\hat{y}^{r}_{z})}},$

where $g_{\text{pt}}$ denotes the parameters including in the confounder
encoder $\text{Encoder}_{D}$ and feed-forward network FFN. To sum up, the
logging-policy-aware confounding effect learning captures confounding
component $P(K|X,C)$ in Eq. 5c, and enables the separation of the ranking and
propensity model from raw clicks.

A powerful confounder encoder is needed. As shown in Eq. 5c, a more accurate
estimation of the confounding effect naturally leads to a more better
unconfounded propensity models, given a logging policy. Thus, an expressive
confounder encoder is needed for separating the ranking and propensity models
from raw clicks, and consequently, the performance of ranking models can be
enhanced.

####  5.2.2. Joint Propensity Learning

Next, we present how to capture the influence of position $K$. In particular,
we propose to learn the mapping from position and query-document feature to
the examination by fixing the confounding effect model and solely tuning the
position-related parameters, as marked lock in Fig. 2. The rationality is that
fixing the confounder encoder and feed-forward network keeps the relevance
confounding effect unchanged; therefore, the position encoder is able to
correctly capture the influence of position over examination.

Formally, we design a position embedding function $\text{Encoder}_{P}$. It
encodes the position of a document that is ranked on position $k$ to a vector
with the same dimension as $\mathbf{m}_{d_{i}}$:

(11) $\mathbf{p}_{k}=\text{Encoder}_{P}(\text{rank}(d_{k})),\ \ \text{where
}\mathbf{p}_{k}\in\mathbb{R}^{n},$

and $\text{rank}(d_{k})$ is the ranked position for document $d_{k}$ generated
by the logging policy. Afterward, we obtain predictions for joint propensity
scores $P(E|K,C)$ through the frozen FFN:

(12) $\hat{y}^{p}_{k}=\text{FFN}(\mathbf{m}_{d_{k}}+\mathbf{p}_{k}).$

Let $y^{p}_{k}$ be the estimated propensity score of the document ranked at
position $k$ by existing ULTR algorithms, which only depends on position $k$.
We solely update the position embedding function $\text{Encoder}_{P}$ during
the optimization via the attention rank loss function, which is formally
defined as,

(13)
$\mathcal{L}_{p}(g_{\text{pos}}|\pi_{q})=-\sum_{d_{k}\in\pi_{q}}\frac{\exp(y^{p}_{k})}{\sum_{z\in\pi_{q}}{\exp(y^{p}_{z})}}\cdot\log\frac{\exp(\hat{y}^{p}_{k})}{\sum_{z\in\pi_{q}}{\exp(\hat{y}^{p}_{z})}},$

where $g_{\text{pos}}$ denotes the parameters of position embedding
$\text{Encoder}_{P}$. To this end, the joint propensity learning captures the
influence of position $K$ by fixing that of confounder $X$.

As shown in Eq. 10 and Eq. 13, the use of the softmax function assumes that
the relevance probabilities and examination probabilities on different
documents in $\pi_{q}$ will sum up to 1, which is not true in practice. This,
however, does not hurt the effectiveness of models training. In fact, the
predicted values of $\hat{y}^{p}_{k}$ have a minor effect on the unconfounded
propensity learning as long as their relative proportions are correct. Due to
this reason, we show normalized propensity against position 10, which reflects
the relative proportion, throughout this paper. Such technique has been widely
applied in existing work, and its effectiveness has been extensively verified
in prior work (Ai et al., 2018b; Cai et al., 2022; Luo et al., 2022).

1

Input : query set $\mathcal{Q}$, an ULTR algorithm $F$

Output : ranking model $f$ and logging-policy-aware propensity model $g$

2

3for  _number of steps for training the unbiased ranking model_ do

4 Sample a batch of queries $Q$ from $\mathcal{Q}$ with ranked lists $\pi_{q}$
and logging policy scores $y^{r}$ per query $q$ ;

5 Estimate propensity scores $y_{k}^{p}$ based on existing ULTR algorithm $F$;

6 Estimate confounding effect $P(K|X,C)$ by updating the confounder encoder
and feed-forward network in LPP model according to Eq. 10 ;

7 Freeze the parameters in the confounder encoder and feed-forward network in
LPP model;

8 Position locked-tuning by updating the position encoder according to Eq. 13;

9 Infer unconfounded propensity $P(E|do(K),C)$ according to backdoor
adjustment approximation according to Eq. 7;

10 Based on unconfounded propensity $P(E|do(K),C)$, update ranking model $f$,
according to Eq. 4a.

11 end for

Algorithm 1 Unconfounded Propensity Estimation for Unbiased Learning to Rank

####  5.2.3. Unconfounded Propensity Inference with Backdoor Adjustment

Given the LPP model, we first estimate the propensity under query-document
relevance confounder:

(14) $\displaystyle
P(E|\mathbf{x}_{i},K,C):=\text{FFN}\big{(}\text{Encoder}_{D}(\mathbf{x}_{i})+\text{Encoder}_{P}(k)\big{)}.$

Afterward, we leverage backdoor adjustment approximation in Eq. 7 to obtain
unconfounded propensity estimation $P(E|do(K),C)$ from raw user clicks, and
integrate with any IPW-based ULTR algorithm to obtain unbiased ranking models.
Given the uncoufounded propensity score, one can conduct any IPW-based ULTR
algorithm with our proposal, as it is a plug-in model, which can seamlessly
integrated into existing automatic ULTR framework. We summarize UPE for ULTR
in Algorithm 1.

##  6\. Experiments

We conduct extensive experiments to demonstrate the effectiveness of our UPE
by investigating the following research questions:

  * •

RQ1: Can empirical results justify the propensity overestimation problem and
how it affects ranking models?

  * •

RQ2: Can UPE address the propensity overestimation problem and obtain an
unbiased ranking model with dynamic training and serving of LTR systems?

  * •

RQ3: Can UPE still address the propensity overestimation problem and obtain an
unbiased ranking model on classic ULTR settings, _i.e.,_ the offline learning
paradigm?

  * •

RQ4: What influence do variant designs have on optimizing the logging-policy-
aware propensity model, and the ranking model?

In particular, we analyze two learning paradigms that are proposed in Ai et
al. (2021). The first one, referred to as the deterministic online paradigm
($\mathit{OnD}$), is an online setting where the displayed ranked list
$\pi_{q}$ is created by the current logging policy, and the ranking model is
updated based on $c_{\pi_{q}}$ collected online. The second one, which is
referred to as the offline paradigm ($\mathit{Off}$), is a classic setting
where we obtain a logging policy, and then both the displayed ranked list
$\pi_{q}$ and the clicks on it $c_{\pi_{q}}$ are fixed and observed in
advance.

Table 1. Overall performance comparison between UPE and the baselines on
Yahoo! and Istella-S datasets with deterministic online learning
($\mathit{OnD}$). “$\ast$” indicates statistically significant improvement
over the best baseline without result randomization.
---
Methods | Yahoo! LETOR | Istella-S
NDCG@K | ERR@K | NDCG@K | ERR@K
K = 1 | K = 3 | K = 5 | K = 10 | K = 1 | K = 3 | K = 5 | K = 10 | K = 1 | K = 3 | K = 5 | K = 10 | K = 1 | K = 3 | K = 5 | K = 10
IPW-Random | 0.693 | 0.702 | 0.722 | 0.767 | 0.352 | 0.432 | 0.452 | 0.469 | 0.667 | 0.637 | 0.660 | 0.720 | 0.596 | 0.705 | 0.722 | 0.728
UPE | $\textbf{0.695}^{\ast}$ | $\textbf{0.703}^{\ast}$ | $\textbf{0.723}^{\ast}$ | $\textbf{0.768}^{\ast}$ | $\textbf{0.355}^{\ast}$ | $\textbf{0.433}^{\ast}$ | $\textbf{0.454}^{\ast}$ | $\textbf{0.469}^{\ast}$ | $\textbf{0.666}^{\ast}$ | $\textbf{0.633}^{\ast}$ | $\textbf{0.657}^{\ast}$ | $\textbf{0.718}^{\ast}$ | $\textbf{0.595}^{\ast}$ | $\textbf{0.704}^{\ast}$ | $\textbf{0.720}^{\ast}$ | $\textbf{0.727}^{\ast}$
Vectorization | 0.670 | 0.678 | 0.702 | 0.753 | 0.343 | 0.423 | 0.446 | 0.460 | 0.663 | 0.630 | 0.653 | 0.711 | 0.593 | 0.701 | 0.716 | 0.724
DLA | 0.671 | 0.677 | 0.701 | 0.750 | 0.345 | 0.423 | 0.445 | 0.460 | 0.663 | 0.629 | 0.653 | 0.712 | 0.592 | 0.701 | 0.717 | 0.724
REM | 0.674 | 0.678 | 0.699 | 0.747 | 0.349 | 0.425 | 0.446 | 0.462 | 0.642 | 0.611 | 0.631 | 0.690 | 0.574 | 0.684 | 0.702 | 0.709
PairD | 0.602 | 0.614 | 0.642 | 0.700 | 0.319 | 0.394 | 0.416 | 0.433 | 0.609 | 0.569 | 0.593 | 0.653 | 0.545 | 0.656 | 0.675 | 0.684
Naive | 0.634 | 0.644 | 0.670 | 0.723 | 0.334 | 0.409 | 0.431 | 0.447 | 0.639 | 0.601 | 0.624 | 0.683 | 0.571 | 0.681 | 0.699 | 0.706

###  6.1. Experimental Settings

##### Datasets

We conduct empirical studies on two of the largest publicly available LTR
datasets:

  * •

Yahoo! LETOR222https://webscope.sandbox.yahoo.com/ comes from the Learn to
Rank Challenge version 2.0 (Set 1), and is one of the largest benchmarks of
unbiased learning to rank (Ai et al., 2018b; Hu et al., 2019). It consists of
29,921 queries and 710K documents. Each query-document pair is represented by
a 700-D feature vector and annotated with 5-level relevance labels (Chapelle
and Chang, 2011).

  * •

Istella-S333http://quickrank.isti.cnr.it/istella-dataset/ contains 33K queries
and 3,408K documents (roughly 103 documents per query) sampled from a
commercial Italian search engine. Each query-document pair is represented by
220 features and annotated with 5-level relevance judgments (Lucchese et al.,
2016).

We follow the predefined data split of training, validation, and testing of
all datasets. The Yahoo! set splits the queries arbitrarily and uses 19,944
for training, 2,994 for validation, and 6,983 for testing. The Istella-S
dataset has been divided into train, validation, and test sets according to a
$60\%-20\%-20\%$ scheme.

##### Click Simulation

We generate synthesized click with a two-step process as in Joachims et al.
(2017) and Ai et al. (2018b). First, we generate the initial ranked list
$\pi_{q}$ for each query $q$ based on learning paradigms, _i.e.,_
$\mathit{OnD}$ and $\mathit{Off}$. Then, we simulate the user browsing process
based on PBM (Joachims et al., 2017) and sample clicks from the initial ranked
list by utilizing the simulation model. The PBM models user browsing behavior
based on the assumption that the bias of a document only depends on its
position:

(15) $P(e_{d_{i}})=\rho_{i}^{\eta},$

where $\rho_{i}$ represents position bias at position $i$ and
$\eta\in[0,+\infty]$ is a parameter controlling the degree of position bias.
The position bias $\rho_{i}$ is obtained from an eye-tracking experiment in
Joachims et al. (2017) and the parameter $\eta$ is set as 1 by default.
Following the methodology proposed by Chapelle et al. (2009), we sample clicks
with:

(16)
$\Pr(r_{d_{i}}=1|\pi_{q})=\epsilon+(1-\epsilon)\frac{2^{y}-1}{2^{y_{\mathrm{max}}}-1},$

where $y\in[0,y_{\mathrm{max}}]$ is the relevance label of the document
$d_{i}$, and $y_{\mathrm{max}}$ is the maximum value of $y$, which is 4 on
both datasets. $\epsilon$ is the noise level, which models click noise such
that irrelevant documents (_i.e.,_ $y=0$) have non-zero probability to be
perceived as relevant and clicked. We fix $\epsilon=0.1$ as the default
setting.

##### Baselines

To demonstrate the effectiveness of our proposed method, we compare it with
baseline methods which are widely used in ULTR problems.

  * •

Vectorization: The Vectorization (Chen et al., 2022b) expands the examination
hypothesis to a vector-based one, which formulates the click probability as a
dot product of two vector functions instead of two scalar functions.

  * •

DLA: The Dual Learning Algorithm (Ai et al., 2018b) treats the problem of
unbiased learning to rank and unbiased propensity estimation as a dual
problem, such that they can be optimized simultaneously.

  * •

REM: The Regression EM model (Wang et al., 2018) uses an EM framework to
estimate the propensity scores and ranking scores.

  * •

PairD: The Pairwise Debiasing (PairD) Model (Hu et al., 2019) uses inverse
propensity weighting for pairwise learning to rank.

  * •

IPW-Random: The Inverse Propensity Weighting (Joachims et al., 2017; Wang et
al., 2016) uses result randomization to estimate the examination probabilities
against positions and optimizes the ranking models accordingly. Its
performance can be considered as the upper bound for learning-to-rank with
user implicit feedback.

  * •

Naive: This model just uses the raw click data to train the ranking model,
without any correction. Its performance can be considered as a lower bound for
the ranking model.

##### Experimental Protocols

We implement UPE and use the baselines in ULTRA (Tran et al., 2021) to conduct
our experiments. In particular, UPE is integrated with DLA, as it is the
state-of-the-art automatic ULTR algorithm. For each query, only the top $N=10$
documents are assumed to be displayed to the users. For both datasets, all
models are trained with synthetic clicks. Following the setting in (Ai et al.,
2018b), the click sessions for training are generated on the fly. We fix the
batch size to 256 and train each model for 10K steps. We use the AdaGrad
optimizer (Duchi et al., 2011) and tune learning rates from 0.01 to 0.05 for
each unbiased learning-to-rank algorithm on the validation dataset.

In the experiments, we train neural networks for our ranking functions. All
reported results are produced using a model with three hidden layers with size
$[512,256,128]$ respectively, with the ELU (Clevert et al., 2016) activation
function and 0.1 dropout (Srivastava et al., 2014). To construct a
sufficiently expressive confounder encoder in LPP model, inspired by SetRank
(Pang et al., 2020), it is configured to have two stacked transformer blocks,
each with 256 hidden units and 8 heads. The hidden size of the document and
position representation are both set to 64. The feed-forward network is a
neural network with two layers with size $[64,256]$.

To evaluate all methods, we use the normalized Discounted Cumulative Gain
(nDCG) (Järvelin and Kekäläinen, 2002) and the Expected Reciprocal Rank (ERR)
(Chapelle et al., 2009). For both metrics, we report the results at ranks 1,
3, 5, and 10 to show the performance of models on different positions.
Following (Ai et al., 2018b), statistical differences are computed based on
the Fisher randomization test (Smucker et al., 2007) with $p\leq 0.05$.

Figure 3. Ranking performance and normalized estimated propensity
($\frac{\text{propensity@K}}{\text{propensity@10}}$) under different logging
policies on Yahoo! LETOR.

###  6.2. Empirical Results for Propensity Overestimation(RQ1)

We first justify the propensity overestimation problem. For a better
illustration, we demonstrate the normalized estimated propensities against
ranking positions and ranking performance, accordingly. Fig. 3(a) investigates
the ranking performance of DLA (Ai et al., 2018b), a state-of-the-art
automatic ULTR method, under weak and strong logging policies, respectively.
It shows that DLA’s ranking performance suffers from a significant drop under
a strong logging policy. Furthermore, in Fig. 3(b), we show the learned
propensity score and indeed observe a propensity overestimation problem in
which normalized estimated propensities for top positions are much larger than
their actual values. This observation confirms the propensity overestimation
problem, and this problem is detrimental to ranking models’ optimization.
Therefore, propensity overestimation needs to be addressed during the dynamic
learning and serving of LTR systems, where the logging policies are not longer
weak.

###  6.3. Dynamic LTR Simulation (RQ2)

In practice, the logging policies are updated periodically: we can collect
sufficient user clicks in a few months and update the logging policy with the
latest ranking policy. To simulate this process, we perform deterministic
online simulation ($OnD$) and update the logging policy after every 2.5K steps
with the most recent ranking model. From Table 1, we can see that UPE
significantly outperforms all the ULTR baseline methods without result
randomization on both datasets. This demonstrates the effectiveness of UPE
with dynamic training and serving of LTR systems. Remarkably, UPE achieves
similar performance to IPW-Random, this indicates that UPE can accurately
estimate the examination probability for each position, yet without the need
for result randomization.

Table 2. Overall performance comparison between UPE and the baselines on
Yahoo! and Istella-S datasets with offline learning ($\mathit{Off}$). “$\ast$”
indicates statistically significant improvement over the best baseline without
result randomization.
---
Methods | Yahoo! LETOR | Istella-S
NDCG@K | ERR@K | NDCG@K | ERR@K
K = 1 | K = 3 | K = 5 | K = 10 | K = 1 | K = 3 | K = 5 | K = 10 | K = 1 | K = 3 | K = 5 | K = 10 | K = 1 | K = 3 | K = 5 | K = 10
IPW-Random | 0.684 | 0.693 | 0.715 | 0.763 | 0.352 | 0.430 | 0.451 | 0.467 | 0.676 | 0.640 | 0.664 | 0.723 | 0.604 | 0.710 | 0.727 | 0.733
UPE | $\textbf{0.687}^{\ast}$ | $\textbf{0.692}^{\ast}$ | $\textbf{0.715}^{\ast}$ | $\textbf{0.762}^{\ast}$ | 0.351 | 0.429 | 0.451 | 0.466 | $\textbf{0.669}^{\ast}$ | $\textbf{0.637}^{\ast}$ | $\textbf{0.659}^{\ast}$ | $\textbf{0.720}^{\ast}$ | $\textbf{0.598}^{\ast}$ | $\textbf{0.706}^{\ast}$ | $\textbf{0.722}^{\ast}$ | $\textbf{0.729}^{\ast}$
Vectorization | 0.681 | 0.689 | 0.711 | 0.759 | 0.350 | 0.428 | 0.449 | 0.465 | 0.663 | 0.630 | 0.652 | 0.714 | 0.593 | 0.699 | 0.718 | 0.722
DLA | 0.682 | 0.688 | 0.710 | 0.758 | 0.350 | 0.427 | 0.449 | 0.464 | 0.662 | 0.629 | 0.653 | 0.713 | 0.592 | 0.699 | 0.717 | 0.723
REM | 0.655 | 0.662 | 0.685 | 0.735 | 0.342 | 0.418 | 0.440 | 0.456 | 0.609 | 0.570 | 0.589 | 0.637 | 0.545 | 0.653 | 0.672 | 0.681
PairD | 0.659 | 0.664 | 0.688 | 0.739 | 0.337 | 0.415 | 0.438 | 0.453 | 0.609 | 0.590 | 0.621 | 0.687 | 0.543 | 0.664 | 0.684 | 0.692
Naive | 0.662 | 0.666 | 0.688 | 0.739 | 0.338 | 0.416 | 0.438 | 0.454 | 0.630 | 0.607 | 0.634 | 0.699 | 0.562 | 0.679 | 0.697 | 0.705

To provide a more insightful understanding for the benefit of UPE, we also
illustrate the learning curve of estimated propensity in Fig. 4. In
particular, we investigate the normalized estimated propensity against
position 1 on UPE, compared with DLA, the state-of-the-art automatic ULTR
algorithm. We select position 1 for illustration because it suffers from
propensity overestimation most (will see Fig. 3). The “ground truth” is
computed by $\frac{\rho_{1}}{\rho_{10}}$, where $\rho_{i}$ is the position
bias defined in Eq.15.

We can see that DLA suffers from propensity overestimation, as evidenced by
its learning curves deviating from the ground truth during dynamic training.
This demonstrates that existing ULTR methods are unable to leverage implicit
feedback to enhance the ranking model during dynamic training and serving of
LTR systems, due to the issue of propensity overestimation. In contrast, UPE’s
propensity estimation is robust to changes in the logging policy, and its
curves generally align with the ground truth. As more user clicks are
collected, the ranking performance improves, highlighting the effectiveness of
UPE in enhancing the ranking model during training. This nice property
actualizes the value of implicit feedback in the practical ULTR scenario,
_i.e.,_ the LTR systems are updated dynamically. Remarkably, though the
performance of logging policy varies from scratch to being in high accuracy,
UPE can accurately estimate propensity in a consistent manner. This suggests
that the superiority of UPE is not related to the performance of logging
policies, but the accurate estimation for confounding effect under a logging
policy, which justifies the necessity of an expressive confounder encoder in
LPP.

It is worth explaining a counter-intuitive observation: the proposed UPE
outperforms the ideal IPW-Random (theoretical upper bound), which uses result
randomization to estimate the examination probability against positions. While
result randomization would produce the accurate estimation of the true
position bias and leads to the optimality of IPW-Random in theory, it also
introduces large variance in practice. The “unexpected” results observed in
Table 1 and Table 2 are mostly due to such variances.

Figure 4. The learning curve of normalized estimated propensity for position 1
with deterministic online learning ($\mathit{OnD}$).

###  6.4. Offline Simulation (RQ3)

To investigate the generalizability of UPE, we conduct experiments on the
commonly used ULTR setting, _i.e.,_ offline learning paradigm ($\mathit{Off}$)
as in Joachims et al. (2017) and Ai et al. (2018b). Unlike the online
paradigm, the ranked lists are generated by a Rank SVM model (Joachims, 2006)
that trained with $1\%$ of the training data with real relvance judgements,
_i.e.,_ weak logging policy.

Table 2 shows the experimental results. We can see that UPE also achieves the
best performance among all baseline methods without result randomization, and
UPE achieves similar performance as IPW-Random on the offline paradigm
$\mathit{Off}$. Besides, UPE outperforms the best baseline methods without
result randomization in most of the metrics. This observation indicates that
UPE can effectively address propensity overestimation even when there exists a
minor confounding effect by query-document relevance confounder $X$ under a
weak logging policy, and more importantly such confounding effect is indeed
non-negligible.

To validate the effectiveness of UPE in propensity estimation under different
logging policies, we have also demonstrated the overall distribution of
normalized estimated propensity against position in Fig. 3. The results
demonstrate the high generalizability of UPE, which can consistently obtain
unconfounded propensity estimation under both strong and weak policies. Its
ranking performance is also shown in Fig. 3, where UPE can consitenly achieve
the SOTA ranking performance under both weak and strong policies.

###  6.5. Ablation Study (RQ4)

Our ablation studies investigate how variant designs of LPP affect its
optimization, and consequentially affect the ranking performance.

####  6.5.1. Is it more beneficial to model $P(\hat{R}|\mathbf{x},C)$ than
$P(K|\mathbf{x},C)$ in propensity model training?

In this section, we compare different fitting targets for LPP optimization. To
instantiate $P(K|\mathbf{x},C)$, we transform the ranked positions as
relevance labels for ranking optimization as in Zhang et al. (2019):

(17a) MRR-UPE $\displaystyle:\ \ \text{MRR-LPP}@K=\frac{1}{K}$ (17b) DCG-UPE
$\displaystyle:\ \ \text{DCG-LPP}@K=\frac{1}{\log_{2}{(K+1)}}.$

Table 3. The performance of different fitting targets in LLP optimization on
Yahoo! LETOR with deterministic online learning ($\mathit{OnD}$). $``\ast"$
indicates statistically significant improvement over the best baseline.
---
Methods | NDCG@K | ERR@K
K = 1 | K = 3 | K = 5 | K = 10 | K = 1 | K = 3 | K = 5 | K = 10
UPE | $\textbf{0.695}^{\ast}$ | $\textbf{0.703}^{\ast}$ | $\textbf{0.723}^{\ast}$ | $\textbf{0.768}^{\ast}$ | $\textbf{0.355}^{\ast}$ | $\textbf{0.433}^{\ast}$ | $\textbf{0.454}^{\ast}$ | $\textbf{0.469}^{\ast}$
MRR-UPE | 0.688 | 0.694 | 0.715 | 0.762 | 0.352 | 0.429 | 0.451 | 0.466
DCG-UPE | 0.686 | 0.691 | 0.714 | 0.760 | 0.350 | 0.428 | 0.451 | 0.465

The experimental results are summarized in Table 3. We can see that UPE
significantly outperforms the two variants, _i.e.,_ MRR-UPE and DCG-UPE. This
observation confirms that the logging policy scores are more informative than
the ranked positions, because logging policy scores not only provide the order
of the documents, _i.e.,_ ranked positions, but also their relevance
strengths.

####  6.5.2. Is the two-step optimization strategy, _i.e.,_ logging-policy-
aware confounding effect learning and joint propensity learning, indispensable
for LPP optimization?

We argue that naively applying existing automatic ULTR methods cannot extract
separated ranking model $P(R|X,C)$ and propensity model $P(E|K,X,C)$ from raw
click data in UPE. In this section, we justify this argument, and demonstrate
the indispensability of our novel two-step optimization.

We refer to the naive propensity learning framework as UPEN, which optimizes
$P(R|X,C)$ and $P(E|K,X,C)$ from raw click data alternatively. For better
illustration, we show the logarithm of the normalized estimated propensity at
positions 1, 2, 3 and 9 in Fig. 5(a). We can observed that from step $3000$,
the estimated propensity by UPEN against position 1 has been much smaller than
that against position 9, and those at other positions are almost identical to
that at position 9. It indicates that UPEN fails to learn the correct
examination probability for each position, which should have a larger value
for a higher-ranked position. We side-by-side present the ranking performance
curve of UPEN in Fig. 5(b). Starting from step $3000$, nDCG@10 of UPEN does
not increase; this means that collecting more click data does not improve the
ranking model because the propensity model has failed in separating the impact
of relevance and position on propensity. This phenomenon also verifies that an
accurate propensity model is indeed necessary for optimizing ranking models.

Figure 5. The learning curve of normalized estimated propensity and ranking
performance of UPE and UPED on Yahoo! LETOR with deterministic online learning
($\mathit{OnD}$).

Unlike UPED, the proposed logging-policy-aware confounding effect learning and
joint propensity learning enable us to obtain an unconfounded propensity
through backdoor adjustment, which correctly estimates the causal effect
between the propensity and position, as has been shown in Fig. 4. Moreover,
the learning curve of UPE in Fig. 5(b) shows that the ranking performance of
our proposal consistently improves when more click data are collected.
Therefore, the two-step optimization strategy is necessary for LPP
optimization, it enables the separation of ranking and propensity models from
raw clicks.

##  7\. Conclusion

In this work, we investigate unbiased learning to rank through the lens of
causality and identify query-document representation as a confounder, which
leads to propensity overestimation. For unconfounded propensity
overestimation, we propose a novel propensity model, _i.e.,_ Logging-Policy-
aware Propensity Model, and its distinct two-step optimization strategy: (1)
logging-policy-aware confounding effect learning, which captures the
confounding effect caused by the query-document feature confounder and thereby
separates ranking and propensity models from raw clicks; and (2) joint
propensity learning, which learns the mapping from position and query-document
feature to the examination by fixing the confounding effect model and solely
tuning the position-related parameters. Given the fine-tuned LPP model, we
conduct backdoor adjustment for unconfounded propensity estimation, which
serves for ULTR. Extensive experiments on two benchmarks with synthetic clicks
with online and offline simulations validate the effectiveness of our proposal
in addressing propensity overestimation and improving ranking performance. A
natural future direction would be to extend current work to pairwise learning
to explore the feasibility of UPE across more ULTR frameworks.

## References

  * (1)
  * Ai et al. (2018a) Qingyao Ai, Keping Bi, Jiafeng Guo, and W. Bruce Croft. 2018a.  Learning a Deep Listwise Context Model for Ranking Refinement. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_.
  * Ai et al. (2018b) Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W. Bruce Croft. 2018b.  Unbiased Learning to Rank with Unbiased Propensity Estimation. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_.
  * Ai et al. (2021) Qingyao Ai, Tao Yang, Huazheng Wang, and Jiaxin Mao. 2021\.  Unbiased Learning to Rank: Online or Offline?  _ACM Trans. Inf. Syst._ 39, 2 (2021), 21:1–21:29.
  * Cai et al. (2022) Yinqiong Cai, Jiafeng Guo, Yixing Fan, Qingyao Ai, Ruqing Zhang, and Xueqi Cheng. 2022\.  Hard Negatives or False Negatives: Correcting Pooling Bias in Training Neural Ranking Models. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022_.
  * Chapelle and Chang (2011) Olivier Chapelle and Yi Chang. 2011.  Yahoo! Learning to Rank Challenge Overview. In _Proceedings of the Yahoo! Learning to Rank Challenge, held at ICML 2010, Haifa, Israel, June 25, 2010_.
  * Chapelle et al. (2009) Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009.  Expected reciprocal rank for graded relevance. In _Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, China, November 2-6, 2009_.
  * Chen et al. (2022a) Mouxiang Chen, Chenghao Liu, Zemin Liu, and Jianling Sun. 2022a.  LBD: Decouple Relevance and Observation for Individual-Level Unbiased Learning to Rank. In _Advances in Neural Information Processing Systems_ , Vol. 35.
  * Chen et al. (2022b) Mouxiang Chen, Chenghao Liu, Zemin Liu, and Jianling Sun. 2022b.  Scalar is Not Enough: Vectorization-based Unbiased Learning to Rank. In _KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022_ , Aidong Zhang and Huzefa Rangwala (Eds.). ACM, 136–145.
  * Chen et al. (2021) Mouxiang Chen, Chenghao Liu, Jianling Sun, and Steven C. H. Hoi. 2021.  Adapting Interactional Observation Embedding for Counterfactual Learning to Rank. In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_.
  * Christakopoulou et al. (2020) Konstantina Christakopoulou, Madeleine Traverse, Trevor Potter, Emma Marriott, Daniel Li, Chris Haulk, Ed H. Chi, and Minmin Chen. 2020.  Deconfounding User Satisfaction Estimation from Response Rate Bias. In _RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22-26, 2020_.
  * Chuklin et al. (2016) Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2016\.  Click Models for Web Search and their Applications to IR: WSDM 2016 Tutorial. In _Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, San Francisco, CA, USA, February 22-25, 2016_.
  * Clevert et al. (2016) Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Craswell et al. (2008) Nick Craswell, Onno Zoeter, Michael J. Taylor, and Bill Ramsey. 2008.  An experimental comparison of click position-bias models. In _Proceedings of the International Conference on Web Search and Web Data Mining, WSDM 2008, Palo Alto, California, USA, February 11-12, 2008_.
  * Duchi et al. (2011) John C. Duchi, Elad Hazan, and Yoram Singer. 2011\.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.  _J. Mach. Learn. Res._ (2011).
  * Gupta et al. (2021) Priyanka Gupta, Ankit Sharma, Pankaj Malhotra, Lovekesh Vig, and Gautam Shroff. 2021.  CauSeR: Causal Session-based Recommendations for Handling Popularity Bias. In _CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021_.
  * Hu et al. (2019) Ziniu Hu, Yang Wang, Qu Peng, and Hang Li. 2019\.  Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm. In _The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019_.
  * Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002\.  Cumulated gain-based evaluation of IR techniques.  _ACM Trans. Inf. Syst._ (2002).
  * Joachims (2002) Thorsten Joachims. 2002\.  Optimizing search engines using clickthrough data. In _Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23-26, 2002, Edmonton, Alberta, Canada_. ACM, 133–142.
  * Joachims (2006) Thorsten Joachims. 2006\.  Training linear SVMs in linear time. In _Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006_.
  * Joachims et al. (2005) Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005\.  Accurately interpreting clickthrough data as implicit feedback. In _SIGIR 2005: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Salvador, Brazil, August 15-19, 2005_.
  * Joachims et al. (2007) Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007.  Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search.  _ACM Trans. Inf. Syst._ 25, 2 (2007), 7\.
  * Joachims et al. (2017) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017\.  Unbiased Learning-to-Rank with Biased Feedback. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM 2017, Cambridge, United Kingdom, February 6-10, 2017_.
  * Liu et al. (2021) Dugang Liu, Pengxiang Cheng, Hong Zhu, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. 2021.  Mitigating Confounding Bias in Recommendation via Information Bottleneck. In _RecSys ’21: Fifteenth ACM Conference on Recommender Systems, Amsterdam, The Netherlands, 27 September 2021 - 1 October 2021_.
  * Lucchese et al. (2016) Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Salvatore Trani. 2016.  Post-Learning Optimization of Tree Ensembles for Efficient Ranking. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, July 17-21, 2016_.
  * Luo et al. (2022) Dan Luo, Lixin Zou, Qingyao Ai, Zhiyu Chen, Dawei Yin, and Brian D. Davison. 2022\.  Model-based Unbiased Learning to Rank.  _CoRR_ abs/2207.11785 (2022).
  * Mao et al. (2019) Jiaxin Mao, Zhumin Chu, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019.  Investigating the Reliability of Click Models. In _Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2019, Santa Clara, CA, USA, October 2-5, 2019_.
  * Mao et al. (2018) Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018\.  Constructing Click Models for Mobile Search. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_.
  * Pang et al. (2020) Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. 2020\.  SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_.
  * Pearl et al. (2000) Judea Pearl et al. 2000\.  Models, reasoning and inference.  _Cambridge, UK: CambridgeUniversityPress_ 19 (2000), 2.
  * Ren et al. (2022) Yi Ren, Hongyan Tang, and Siwen Zhu. 2022.  Unbiased Learning to Rank with Biased Continuous Feedback. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022_. ACM, 1716–1725.
  * Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007\.  Predicting clicks: estimating the click-through rate for new ads. In _Proceedings of the 16th International Conference on World Wide Web, WWW 2007, Banff, Alberta, Canada, May 8-12, 2007_.
  * Sato et al. (2020) Masahiro Sato, Sho Takemori, Janmajay Singh, and Tomoko Ohkuma. 2020.  Unbiased Learning for the Causal Effect of Recommendation. In _RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22-26, 2020_.
  * Smucker et al. (2007) Mark D. Smucker, James Allan, and Ben Carterette. 2007\.  A comparison of statistical significance tests for information retrieval evaluation. In _Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management, CIKM 2007, Lisbon, Portugal, November 6-10, 2007_.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _J. Mach. Learn. Res._ 15, 1 (2014), 1929–1958.
  * Tran et al. (2021) Anh Tran, Tao Yang, and Qingyao Ai. 2021.  ULTRA: An Unbiased Learning To Rank Algorithm Toolbox. In _CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021_.
  * Vardasbi et al. (2021) Ali Vardasbi, Maarten de Rijke, and Ilya Markov. 2021\.  Mixture-Based Correction for Position and Trust Bias in Counterfactual Learning to Rank. In _CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021_ , Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (Eds.). ACM, 1869–1878.
  * Vardasbi et al. (2020) Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. 2020\.  When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank. In _CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020_.
  * Wang et al. (2021b) Nan Wang, Zhen Qin, Xuanhui Wang, and Hongning Wang. 2021b.  Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction. In _WSDM ’21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12, 2021_. ACM, 481–489.
  * Wang et al. (2021a) Wenjie Wang, Fuli Feng, Xiangnan He, Xiang Wang, and Tat-Seng Chua. 2021a.  Deconfounded Recommendation for Alleviating Bias Amplification. In _KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021_.
  * Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016.  Learning to Rank with Selection Bias in Personal Search. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, July 17-21, 2016_.
  * Wang et al. (2018) Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018\.  Position Bias Estimation for Unbiased Learning to Rank in Personal Search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018_.
  * Wang et al. (2020) Yixin Wang, Dawen Liang, Laurent Charlin, and David M. Blei. 2020\.  Causal Inference for Recommender Systems. In _RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22-26, 2020_.
  * Yan et al. (2022) Le Yan, Zhen Qin, Honglei Zhuang, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2022\.  Revisiting Two-tower Models for Unbiased Learning to Rank. In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022_ , Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (Eds.).
  * Yang et al. (2020) Tao Yang, Shikai Fang, Shibo Li, Yulan Wang, and Qingyao Ai. 2020.  Analysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank. In _CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020_. 2277–2280.
  * Yang et al. (2022) Tao Yang, Chen Luo, Hanqing Lu, Parth Gupta, Bing Yin, and Qingyao Ai. 2022\.  Can Clicks Be Both Labels and Features?: Unbiased Behavior Feature Collection and Uncertainty-aware Learning to Rank. In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022_. ACM, 6–17.
  * Yang et al. (2021) Xun Yang, Fuli Feng, Wei Ji, Meng Wang, and Tat-Seng Chua. 2021.  Deconfounded Video Moment Retrieval with Causal Intervention. In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_.
  * Zhan et al. (2022) Ruohan Zhan, Changhua Pei, Qiang Su, Jianfeng Wen, Xueliang Wang, Guanyu Mu, Dong Zheng, Peng Jiang, and Kun Gai. 2022.  Deconfounding Duration Bias in Watch-time Prediction for Video Recommendation. In _KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022_. 4472–4481.
  * Zhang et al. (2019) Junqi Zhang, Jiaxin Mao, Yiqun Liu, Ruizhe Zhang, Min Zhang, Shaoping Ma, Jun Xu, and Qi Tian. 2019\.  Context-Aware Ranking by Constructing a Virtual Environment for Reinforcement Learning. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019_.
  * Zhang et al. (2021) Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021.  Causal Intervention for Leveraging Popularity Bias in Recommendation. In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_.
  * Zhuang et al. (2021) Honglei Zhuang, Zhen Qin, Xuanhui Wang, Michael Bendersky, Xinyu Qian, Po Hu, and Dan Chary Chen. 2021.  Cross-Positional Attention for Debiasing Clicks. In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_.