The Simpson's Paradox in the Offline Evaluation of Recommendation Systems

Recommendation systems are often evaluated based on user's interactions that were collected from an existing, already deployed recommendation system.
In this situation, users only provide feedback on the exposed items and they may not leave feedback on other items since they have not been exposed to them by the deployed system.
As a result, the collected feedback dataset that is used to evaluate a new model is influenced by the deployed system, as a form of closed loop feedback.
In this paper, we show that the typical offline evaluation of recommender systems suffers from the so-called Simpson's paradox. Simpson's paradox is the name given to a phenomenon observed when a significant
trend appears in several different sub-populations of observational data but disappears or is even reversed when these sub-populations are combined together.
Our in-depth experiments based on stratified sampling reveal that a very small minority of items that are frequently exposed by the deployed system plays a confounding factor in the offline evaluation of recommendation systems.
In addition, we propose a novel evaluation methodology that takes into account the confounder, i.e the deployed system's characteristics.
Using the relative comparison of many recommendation models as in the typical offline evaluation of recommender systems, and based on the Kendall rank correlation coefficient, we show that our proposed evaluation methodology exhibits statistically significant improvements of 14% and 40% on the examined open loop datasets (Yahoo! and Coat), respectively,  in reflecting the true ranking of systems with an open loop (randomised) evaluation in comparison to the standard evaluation.

<concept_desc>Information systems Recommender systems</concept_desc>

[500]Information systems Recommender systems

§ INTRODUCTION

Recommendation systems are typically measured in either an online (A/B testing or interleaving) or an offline manner. However, the offline evaluation of recommender systems remains the most widely reported evaluation in the literature due to the difficulties in deploying research systems for online evaluation <cit.>. In reality, it is hard to reliably evaluate the effectiveness of a recommendation model in an offline fashion using logs of historical user interactions <cit.>. The issue is that there may be confounders, i.e. variables that affect both the exposure of items and the outcomes (e.g. ratings) <cit.>. Confounding problems occur when a platform attempts to model user behaviour without taking into account the selection bias in the exposed recommended items <cit.>, leading to unintended consequences. In this situation, it is difficult to distinguish which user interactions stem from the users' true preferences and which are influenced by the deployed recommender system.

Typically, the offline evaluation of recommendation systems has two stages: (1) gathering a collection of user's feedback from a deployed system; and (2) the use of such feedback to empirically evaluate and compare different recommendation models.
The first stage can be subject to different types of confounders, either initiated by the item selection behaviour of users or through the actions of the deployed recommendation system <cit.>.
For instance, aside from other interaction mechanisms such as search, users are more likely to interact with those items that they have been exposed to.
In this way, using historical interactions in the offline evaluation of a new recommender model, where those interactions have been obtained from the deployed recommender system, forms a closed loop feedback <cit.>, i.e. the deployed recommender system[Naturally, a recommender system relies on a recommendation model or algorithm. Hence, in this paper, we interchangeably use system and model.] has a direct effect on the collected feedback, which is in turn used for the offline evaluation of other recommendation models.
Hence, the new recommendation model is evaluated in terms of how much it tends to mimic the interactions collected by the deployed model, rather than how well it meets the true preferences of users. On the other hand, in an open loop (randomised) scenario, the deployed model is a random recommendation model, i.e. random items are exposed to the users. Therefore, the feedback loop between the deployed model and the new recommendation model is broken and the deployed model has no effect on the collected feedback dataset and, consequently, it has also no effect on the offline evaluation of any new model based on the collected feedback dataset.
However, it is naturally not practical to expose users to random items due to the potential for degraded user experience.
Therefore, the training and evaluation of recommendation systems based on closed loop feedback is a critical problem.

Simpson's paradox is a phenomenon in statistics, in which a significant trend appears in several different groups of observational data but disappears or even reverses when these groups are combined together <cit.>. In recommendation scenarios,
the Simpson's paradox occurs when the association between the exposure (i.e. the recommended items) and the outcome (i.e. the user's explicit or implicit feedback) is investigated but the exposure and outcome are strongly influenced by a third variable. In statistics, if the observed feedback is a product of Simpson's paradox, the stratification of the feedback according to the confounding variable causes the disappearance of the paradox.
We argue that in the case of recommendation systems, this confounding variable is the deployed model (or system) from which the interaction data were collected, a.k.a. closed loop feedback <cit.>.
-1 Our key aims in this paper are to provide an in-depth investigation of the impact of closed loop feedback on the offline evaluation of recommendation systems and to provide a robust solution to the problem. In particular, we argue that the feedback datasets collected from a deployed model are biased towards the deployed model's characteristics, leading to the conclusions being prone to the Simpson's paradox. We thoroughly investigate the effect of the confounding variable (i.e. the deployed model's characteristics) on the offline evaluation of recommendation systems where significant trends are observed, which then disappear or even reverse when reporting observations from the classical offline setting. In addition, we propose a new evaluation methodology that addresses the Simpson's paradox in order to conduct a sounder offline evaluation of recommendation systems.

To better understand the subtleties of the tackled problem, consider a deployed recommendation model that promotes a specific group of items (e.g. popular items[It is well-known in the literature that recommendation algorithms and public datasets favor popular items <cit.>.]) – there are a few head items that are widely exposed to the users that obtain a large number of interactions, in contrast to a long tail of items with only a few interactions.
When we evaluate a new recommendation model based on feedback that was collected from the aforementioned deployed model without taking into account (or adjusting) how frequently different items were exposed, the evaluation process is confounded by the deployed model's characteristics, i.e. the performance of any model that has a tendency to exhibit similar properties to the already deployed model will likely be over-estimated.
In this situation, we might choose to deploy a new model that matches the properties of the already deployed model, while it might be less effective than another model from the actual users' perspective.
In this paper, we thoroughly investigate the consequences of this problem on the conclusions drawn from the standard offline evaluation of recommender systems and propose a new methodology to address the issue.  In particular, the contributions of this paper are two-fold:

* We present an in-depth analysis of the standard offline evaluation of recommender systems, showing that it indeed suffers from the Simpson's paradox.
Our in-depth experiments on four well-known closed loop and open loop (randomised) datasets reveal that a very small minority of items that are over-exposed by the deployed model play a confounding factor in a standard offline evaluation, i.e. significant trends in a given recommender system's performance that are observed in the majority of user-item interactions actually disappear or even reverse in the standard offline evaluation when we merge this majority feedback with a minority confounder.

* -1 To address this problem, we propose a new propensity-based stratified evaluation method in comparison to both the standard offline holdout evaluation <cit.> and the recently introduced counterfactual evaluation <cit.> methods. Our experiments on a broad range of recommendation models and evaluation metrics reveal that the proposed propensity-based stratified evaluation method better represents the actual utility of the examined recommendation models.

The remainder of the paper is organised as follows: In Section <ref>, we position our contributions in comparison to the existing literature while Section <ref> describes the current offline evaluation methods.
Section <ref> presents the Simpson's paradox and how it affects the offline evaluation of recommender systems; Section <ref> describes the new proposed evaluation method; Section <ref> and Section <ref> present the experimental setup as well as the conducted experiments and their analysis, respectively. Finally, Section <ref> summarises our findings.

§ RELATED WORK

Our work is inspired by three lines of related research, ranging from tackling bias in learning to rank methods (Section <ref>), through algorithmic confounding or closed loop feedback (Section <ref>), to recent work on counterfactual learning and evaluation (Section <ref>).

§.§ Unbiased Learning to Rank

Users' feedback in recommendation systems is reminiscent of clickthrough data in information retrieval (IR). Previous research <cit.> has shown that there is a strong dependency between the documents[These correspond to items in recommendation systems.] presented to the user, and those for which the system receives feedback, i.e. higher-ranked documents obtain more clicks (a phenomenon denoted in the literature as position/presentation bias).
For example, consider a movie recommendation system that aims to learn the users' preferences by observing their watching history. The system can infer which movie(s) they prefer over other programs at a particular time, but these are relative preferences in relation to the exposed movies, but do not constitute preferences on an absolute scale.
Previous research in information retrieval (IR) <cit.> has revealed that such feedback does not reliably reflect retrieval quality while nonetheless still providing relative user preferences in paired experiments.
Classical offline evaluation setups in IR use pooling to gather a comprehensive set of possibly relevant documents for assessment, thereby preventing bias in the relevance assessments towards a single system (known as incompleteness).
However, pooling is not a suitable approach for addressing incompleteness in the evaluation of recommendation systems, due to the contextual and personalised nature of users' recommendation needs.

-1 Recent, unbiased learning to rank approaches aim to learn reliable relevance signals from noisy and biased clickthrough data either with the help of click models or through counterfactual methods <cit.>.
These models typically require a large quantity of clicks, which makes them difficult to apply in systems where click data is highly sparse due to the personalised nature of recommendation systems <cit.>.
On the other hand, these models are designed to tackle position/presentation bias in IR systems. However,
although both position and presentation bias can influence the collected feedback from a deployed recommendation system, recommendation systems mostly suffer from selection bias <cit.>.
In this paper, we investigate the origin of selection bias in recommendation systems and provide a novel evaluation method that takes into account the role of the deployed recommendation system.

§.§ Algorithmic Confounding

Recommendation systems are often evaluated with interaction data obtained from users that have already been exposed to a deployed algorithm. This creates a feedback loop <cit.> (a.k.a. algorithmic confounding <cit.> or closed loop feedback <cit.>).
Using simulations on synthetic data, Chaney et al. <cit.> revealed that this feedback loop in recommendation models encourages similar users to interact with the same set of items, thereby homogenising their behavior, relative to an open loop (randomised) platform where random items are exposed to the user.
In contrast to the simulation experiments, Jadidinejad et al. <cit.> investigated the effect of closed loop feedback in both the training and offline evaluation of recommendation models based on an open loop (randomised) dataset. They found that there is a strong correlation between the deployed model from which the users' feedback were collected and a new model that is trained or evaluated based on the collected feedback dataset. For example, when popular items are favoured by the deployed algorithm, the observed interactions are biased towards popular items. In this situation, the performance of any algorithm that has a tendency to recommend popular items might be over-estimated <cit.>.
In this paper, we propose a theoretical framing of feedback loops in recommendation systems based on the Simpson's paradox, as well as formulate and validate hypotheses about the implications of closed loop feedback in the offline evaluation of recommendation systems. In addition, we propose a novel evaluation methodology, which takes into account the confounding role of the deployed model and show its benefits in comparison to both the standard offline holdout evaluation <cit.> and the recently introduced counterfactual evaluation <cit.> methods.

§.§ Counterfactual Learning and Evaluation

Although researchers in IR have proposed novel approaches for both learning <cit.> and evaluating <cit.> from user's clickthrough data, many researchers and practitioners still focus on evaluating recommendation systems in terms of accuracy metrics on datasets that were collected from an already deployed system <cit.>.
On the other hand, several recent studies <cit.> have attempted to estimate the actual utility of a recommendation model with propensity-weighting techniques commonly used in the off-policy evaluation approaches in Reinforcement Learning with the aim of achieving an unbiased performance estimator of biased observations.
Propensity in this context is the probability that the deployed model has exposed item $i$ to user $u$ by the time of feedback collection.
The intuition in counterfactual evaluation is that each feedback is individually weighted based on its propensity.
The key to a fair evaluation in the counterfactual approaches lies in accurately logging propensities in an open loop (randomised) dataset where random items are exposed to the user. However, most current public recommendation datasets do not introduce a randomisation element <cit.>, making it practically difficult to effectively deploy the counterfactual approaches for evaluating recommender systems.
In the absence of open loop (randomised) feedback, the estimated propensities are noisy and skewed. As a result, weighting each feedback individually based on its propensity score can lead to a high variance in the estimation of the model's effectiveness, as reported in several previous studies <cit.>.
Hence, compared to previous works on propensity weighting <cit.>, our model does not weight each feedback individually based on its propensity. We instead propose a novel evaluation method based on stratification, which considers each feedback in the corresponding stratum based on the estimated propensities.
As a result, as our experiments show, our proposed evaluation method is more robust in coping with these noisy estimated propensities.

§ OFFLINE EVALUATION METHODS

In this section, we summarise the current offline evaluation methods commonly used in the literature, namely the standard holdout evaluation (Section <ref>) and the counterfactual evaluation (Section <ref>).

§.§ Holdout Evaluation

The purpose of the holdout evaluation <cit.> is to test a model on a different data from what it was trained on. In this method, the dataset is randomly divided into a training set and a test set, the latter being the holdout data. The effectiveness of a model ($\hat{R}$) for the predicted item ranking $\hat{Z}$ can be calculated as follows on the holdout data <cit.>:

\begin{equation}
\hat{R}_{\mathrm{holdout}}(\hat{Z})=\frac{1}{|\mathcal{U}|} \sum_{u \in \mathcal{U}} \frac{1}{\left|\mathcal{S}_{u}^{*}\right|} \sum_{i \in \mathcal{S}_{u}^{*}} c(\hat{Z}_{u, i})
\end{equation}

where $\mathcal{S}_{u}^{*}$ is the set of items (among all the items in $\mathcal{I}$) that are exposed to the user $u \in \mathcal{U}$, $\hat{Z}_{u, i}$ is the predicted ranking of the exposed item $i$ for user $u$, and the function $c$ denotes any top-k scoring metric, such as normalised Discounted Cumulative Gain at rank cutoff k (nDCG@k).

The holdout evaluation method is the standard and most common evaluation method in the field of recommendation systems because of its simplicity and flexibility <cit.>. However, if the dataset is biased towards a group of items, the holdout test set will be a biased sample of the user-item interactions <cit.>. Therefore, the holdout evaluation does not necessarily represent the actual effectiveness of the examined models, i.e. the expected outcome of the holdout evaluation does not necessarily conform to the open loop evaluation <cit.>:

\begin{equation}
\mathbb{E}_{O}\left[\hat{R}_{\mathrm{holdout}}(\hat{Z})\right] \neq R(\hat{Z})
\end{equation}

-1 where $R$ is the actual performance of the predicted ranking of items $\hat{Z}$. In this situation, the effectiveness of any model that captures the bias in the original dataset might be over-estimated <cit.>. For example, when popular items are favoured by the deployed algorithm, the observed interactions are biased towards popular items. As a result, the performance of any algorithm that has a tendency to recommend popular items might be over-estimated <cit.>.
An expensive solution is to create an open loop (randomised) test set from the experiments, i.e. to ask the users to provide feedback for randomly selected items. Another alternative and more practical solution consists in using the propensity-based methods <cit.> (e.g. counterfactual evaluation), which we describe in the following section.

§.§ Counterfactual Evaluation

The problem of evaluating recommendation models using historical feedback is similar to counterfactual evaluation in reinforcement learning. In this setting, the goal is to evaluate a new recommendation model based on a feedback dataset that is collected from a deployed recommendation model.
Inverse Propensity Scoring (IPS) leverages importance sampling to account for the fact that the feedback, which was collected from the deployed recommendation model is not uniformly random. Theoretically, the IPS evaluation method is able to evaluate a new model independently from the deployed model (the confounder). In the context of recommendation systems, the IPS evaluation method was proposed for both explicit <cit.> and implicit <cit.> feedback. The key idea of IPS is to reweight feedback based on the propensities of the logged items using importance sampling. The propensity $p_{u,i}$ in this context is the probability that item $i$ would have been shown to the user $u$ by the time the feedback data was collected.
The effectiveness of a model ($\hat{R}$) for the predicted item ranking $\hat{Z}$ and the provided propensity scores $P$ can be calculated as follows <cit.>:

\begin{equation}
\hat{R}_{\mathrm{IPS}}(\hat{Z} | P)=\frac{1}{|\mathcal{U}|} \sum_{u \in \mathcal{U}} \frac{1}{\left|\mathcal{S}_{u}^{\final{*}}\right|} \sum_{i \in \mathcal{S}_{u}^{*}} \frac{c(\hat{Z}_{u, i})}{p_{u, i}}
\end{equation}

The intuition behind the IPS method is that the propensity score $p_{u,i}$ is a balancing factor: conditioned on the propensity score, the observed feedback distribution will be adjusted to the examined model.
Unlike the holdout evaluation $\hat{R}_{\mathrm{holdout}}$, the IPS evaluation method is theoretically unbiased ($\mathbb{E}_{O}\left[\hat{R}_{\mathrm{IPS}}(\hat{Z})\right] = R(\hat{Z})$) if and only if we can sample open loop feedback from all possible items $\mathcal{I}$ uniformly at random <cit.>. However, in various situations, it may be difficult or even impossible to expose users to random items due to obvious business constraints (users are unlikely to tolerate that the system keeps showing them random non-relevant items). Moreover, the imbalance of the assigned propensities will allow for some feedback to be weighted disproportionately. As a result, a major difficulty in applying the IPS evaluation method in practice is its high variance in the estimation of a given model's effectiveness <cit.>.
In the following, we investigate the impact of our proposed propensity-based stratified evaluation in comparison to both the holdout and counterfactual evaluation.

§ SIMPSON'S PARADOX

Simpson's paradox is the name given to an observed phenomenon in statistics, in which a significant trend appears in several different groups of observational data but disappears or even reverses when these groups are combined together.
This topic has been widely studied in the causal inference literature <cit.>.
In this phenomenon, an apparent paradox arises because aggregated data can support a conclusion that is opposite to the conclusions from the same stratified data before aggregation.
The Simpson's paradox occurs when the association between two variables is investigated but these variables are strongly influenced by a confounding variable. When the data is stratified according to the confounding variable, the paradox is revealed showing paradoxical conclusions.
In this situation, the  use of a significance test might identify unsound conclusions made across a specific stratum; however, as we show later in Section <ref>, significance tests may not possibly identify such trends. Indeed, in a recommender system evaluation scenario, testing occurs over users, while the paradox we discuss here usually involves the user-item feedback generation process. On the other hand, the Simpson's paradox can be resolved when causal relations are appropriately addressed in statistical modeling <cit.>.

Simpson's paradox examples in medical studies (a) and recommendation systems (b).
Kidney stone example <cit.>. Each entry represents the number of responses out of the total number of patients while the success rates are shown in brackets.

Treatment A     Treatment B
2*Size $<$ 2cm     Group 1     Group 2

081/870 (93%)      234/270 (87%)
2*Size $\geq$ 2cm     Group 3     Group 4

192/263 (73%)      055/800 (69%)
Both     273/350 (78%)     289/350 (83%)

Simpson's paradox in the offline evaluation of recommender systems. Each entry represents the effectiveness of the examined model on the corresponding stratum. * denotes a significant difference compared to the other model (paired t-test, p <0.05).

***Model A***     ***Model B***
2*Q1 (99%)     Group 1     Group 2

0.339     0.350*
2*Q2 (1%)     Group 3     Group 4

0.695*     0.418
Both     0.373*     0.369

In this section, to illustrate the Simpson's paradox, we present a real-life example from a seminal paper <cit.> comparing the success rates of two treatments for the kidney stone disease.[Treatments A and B correspond to `All open procedures' and
`Percutaneous nephrolithotomy' in the original paper <cit.>.]
The objective is to find out which treatment is more effective based on observational studies.
Charig et al. <cit.> randomly sampled 350 patients who were exposed to each treatment and reported the success rates in Table <ref>.
A plausible conclusion is that treatment B is more effective than treatment A (83% vs. 78% recovery rate). On the other hand, the paradoxical conclusion is that the success rates reverse when the stone size is taken into account, i.e. treatment A is more effective for both small (93% vs. 87%) and large stone sizes (73% vs. 69%).
Charig et al. <cit.> argued that the treatment (A vs. B) and the outcome (success vs. failure) are strongly associated with a third confounding variable (namely, the stone size):

Doctors tend to opt for favoring treatment B for less severe cases (i.e. small stones) while favouring treatment A for more severe cases (i.e. large stones).

Table <ref> (data from <cit.>) validates the above hypothesis, i.e. the majority of patients who were exposed to treatment A had large stones (263 out of 350 random patients in Group 3) while the majority of patients who were exposed to treatment B had small stones (270 out of 350 random patients in Group 2). As a result, when the samples are randomly selected from the patients who were exposed to treatments A or B, the sampling process is not purely random, i.e. the samples are systematically skewed towards the patients who have severe cases for measuring treatment A, while sampling mild cases for measuring treatment B. This is a well-known phenomena in causal analysis, which is also known as the Simpson's paradox <cit.>.

Table <ref> shows an illustrative example of the Simpson's paradox in the offline evaluation of recommendation systems.[This example is actually a highlight of our later experiments, detailed further in Section <ref>.]
We are interested to evaluate the effectiveness of two recommendation models, depicted as A and B in Table <ref>. Both models are evaluated on the same dataset and using the same evaluation metric.
The standard offline evaluation on the examined dataset shows that model A significantly outperforms model B, according to a paired t-test. However, stratifying the examined dataset into two strata (Q1 and Q2) reveals that indeed model B is the significantly better model for 99% of the examined dataset while model A is statistically superior only for a small minority (1%) of the examined dataset. The statistical superiority of model B in 99% of the examined dataset disappears when aggregating the Q1 and Q2 strata together.
In the following section, we present how the Simpson's paradox affects the offline evaluation of recommendation systems, discuss the causes of such a paradox and propose a new evaluation methodology to address it.

§ PROPENSITY-BASED STRATIFIED EVALUATION

block = [draw, fill=blue!20, rectangle,
minimum height=2.5em, minimum width=6em]
output = [coordinate]
pinstyle = [pin edge=to-,thin,black]

[auto, node distance=2cm]
[name=input] (input) ;
[block, right of=input] (recsys) RecSys;
[block, right of=recsys, pin=[pinstyle]above:Other Confounding Variables,
node distance=3cm] (user) User;

[->] (recsys) – node[name=u] $e$ (user);
[output, right of=user] (output) ;

[->] (input) – node $r$ (recsys);
[->] (user) – node [name=r] $r$(output);
[-] (r) – ++ (0,-2.1) node [xshift=-85,yshift=10] Closed Loop Feedback -|  ++ (-6.40,1.89);

Data flow in a recommender system <cit.>.

(1) at (0,0) [align=center] RecSys' Characteristics
(2) [below of =1, left of =1, align=center, xshift=-3em, yshift=-2em] Closed Loop Feedback
(3) [below of =1, right of =1, align=center, xshift=+3em, yshift=-2em] Offline Evaluation

[->] (2) edge  (3);
[->] (1) edge  (2);
[->,dashed] (1) edge  (3);

Assumed coarse-grained causal diagram.

Closed loop feedback in recommendation systems.

When creating a dataset for the offline evaluation of a recommender system, the users' feedback can be collected not only from interactions with items suggested by the deployed recommendation system but also through other modalities such as interactions occurring when browsing the items' catalogue or when following links to sponsored items. It is not straightforward to distinguish the source of the users' feedback since no public datasets provide sufficient data to establish the sources of the users' feedback. Hence, in this paper we focus our investigation on the main source of user's feedback, namely the deployed system <cit.>.  In order to demonstrate the Simpson's paradox in recommendation systems, we need a causal hypothesis similar to Hypothesis <ref> in Section <ref>.

Figure <ref> shows the information flow of a typical recommendation system where the users' feedback is collected through the deployed system.
The deployed recommender system component (denoted by RecSys) filters items for the target user (e.g. by suggesting a ranked list of items) depicted as the exposure ($e$).
On the other hand, the user's recorded preferences (e.g. ratings or clicks) towards items (depicted as $r$) are leveraged as interaction data to train and/or evaluate the next-generation of the recommendation model. Hence, since the users' clicks are obtained on items exposed by the RecSys[Their clicks may also be affected by other confounding variables, such as temporal factors and other browsing interactions with the item catalogue, as illustrated in Figure <ref>.], the model itself influences the generation of data that is used to either train or evaluate it <cit.>.
The system presented in Figure <ref> is a dynamic system where a simple associative reasoning about the system is difficult because each component influences the other. Figure <ref> shows the corresponding causal diagram in such a closed loop feedback scenario. The solid lines represent an explicit/observed relationship between the causes and effects while the dashed line represents an implicit/unobserved relationship.
As mentioned above, in the case of recommendation systems, the main confounding variable is the deployed model from which the interaction data were collected <cit.>.
Our aim is to evaluate the effectiveness of a recommendation model ($Y$) based on closed loop feedback ($r$) collected from a deployed model while the analysis is influenced by the main confounder, i.e. the deployed model's characteristics. In this situation, it is difficult to distinguish those users' interactions that stem from the users' true preferences from those that are influenced by the deployed recommendation model. As a result, in the scenario where the users' feedback is typically collected through the deployed system we postulate that the offline evaluation of recommendation models based on closed loop feedback datasets is strongly influenced by the underlying deployed recommendation model:

Closed loop feedback collected from a deployed recommendation model is skewed towards the deployed model's characteristics (i.e. the exposed items) and the deployed model's characteristics play a confounding factor in the offline evaluation of recommendation models.

The core problem is that the deployed recommendation model is attempting to model the underlying user preferences, making it difficult to make claims about user behaviour (or use the behaviour data) without accounting for algorithmic confounding <cit.>.
On the other hand, if the RecSys component in Figure <ref> is a random model, then the interconnection between the User and the RecSys components in Figure <ref> will be removed and the RecSys component has no effect on the collected feedback dataset, a situation known as open loop feedback, i.e. no item receives more or less expected exposure compared to any other item in the collection <cit.>.

As shown in Figure <ref>, if the deployed model's characteristics ($e$) could be identified and measured, the offline evaluation ($Y$)
would be independent of the confounder.
Therefore, in order to validate the effect of closed loop feedback in a recommendation system (Hypothesis <ref>), we have to quantify the deployed model's characteristics. To this end, following <cit.>, we define the propensity score as follows:

The propensity score $p_{u,i}$ is the tendency of the deployed model (depicted as RecSys in Figure <ref>) to expose item $i\in\mathcal{I}$ to user $u\in\mathcal{U}$.

The propensity $p_{u,i}$ is the probability that item $i$ is exposed to user $u$ by the deployed model under a closed loop feedback scenario[In Section <ref> we propose a method to estimate the propensity scores in our experiments.].
This value quantifies a system's deviation from an unbiased open loop exposure  scenario, where random items are exposed to the user and the deployed model has no effect on the collected feedback.
The propensity score $p_{u,i}$ allows us to design and analyse the offline evaluation of recommendation models  based on the observed closed loop feedback so that it mimics some of the particular characteristics of the open loop scenario.

Stratification is a well-known approach to identify and estimate causal effects by first identifying the underlying strata before  investigating causal effects in each stratum <cit.>.
The general intuition is to stratify on the confounding variable and to investigate the potential outcome in each stratum.
As a result, it becomes possible to measure the potential outcome irrespective of the confounding variable.
In this situation, a marginalisation over the confounding variable
can be leveraged as a combined estimate.
As mentioned before, the hypothetical confounding variable in recommender systems is the deployed model's characteristics (Hypothesis <ref>).  Definition <ref> quantifies this variable as propensities. In this situation, stratification on the propensity scores allows us to analyse the effect of the deployed model's characteristics on the offline evaluation of recommendation models.

For the sake of simplicity, suppose that we have a single categorical confounding variable $\mathrm{X}$.
If we stratify the observed outcome based on the possible values of $\mathrm{X}$, then the expected value of the potential outcome ($\mathrm{Y}$) is defined as follows:

\begin{equation}
\mathrm{E}(\mathrm{Y})=\sum_{x} \mathrm{E}\left(\mathrm{Y} | \mathrm{X}=x\right) \mathrm{P}(\mathrm{X}=x)
\label{eq:marginalisation}
\end{equation}

where $\mathrm{E}\left(\mathrm{Y} | \mathrm{X}=x\right)$ is the conditional expectation of the observed outcome given  stratum $x$, and $\mathrm{P}(\mathrm{X}=x)$ is the marginal distribution of $x$.
For example, in the kidney stone example presented in Table <ref>, the confounding variable is the kidney stone size based on which our observations were stratified ($X=\{small, large\}$)[Stone size is a continuous variable. In this example, we cast it into a categorical variable for a more clear illustration.] and the potential outcome is the treatment effect. We can calculate the expected value of each treatment based on Equation (<ref>). For example, the expected value of treatment A can be calculated as follows:

\begin{equation}
\mathrm{E}(\mathrm{A})= \mathrm{E}\left(\mathrm{A} | \mathrm{X}=small\right) \mathrm{P}(\mathrm{X}=small)+\mathrm{E}\left(\mathrm{A} | \mathrm{X}=large\right) \mathrm{P}(\mathrm{X}=large)
\label{eq:marginalisation_kidney_example}
\end{equation}

Based on the numbers in Table <ref> and Equation (<ref>), the expected value of treatment A (resp. B) is calculated to be 0.832 and 0.782, respectively, i.e. $\mathrm{E}(\mathrm{A}) > \mathrm{E}(\mathrm{B})$, which better estimates the actual performance of the treatments.
In a similar manner, for the recommendation example in Table <ref>, the expected values of model A and model B are calculated to be 0.343 and 0.351, respectively, i.e. $\mathrm{E}(\mathrm{B}) > \mathrm{E}(\mathrm{A})$.

As mentioned above, in recommender systems, the main hypothetical confounding variable is the deployed model (Hypothesis <ref>). This variable is quantified as propensity scores (Definition <ref>). Propensity is a continuous variable. In this paper, we cast it into a categorical variable by sorting and segmenting the propensity scores into a predefined number of strata, i.e. Q1 and Q2 strata in Table <ref>.
In both cases of Tables <ref> and  <ref>, stratification based on the hypothetical confounding variable and marginalisation on the examined strata using Equation (<ref>), does resolve the Simpson's paradox. For example, in Table <ref>, model B should be recognised as the superior model by any reasonable evaluation outcome since it performs better than model A for 99% of the user-item feedback on the examined dataset. This important trend is captured by the proposed stratified evaluation while it is completely reversed in the standard offline evaluation when combining the Q1 and Q2 strata together.
In the following section, we investigate the effect of the Simpson's paradox and the usefulness of our proposed propensity-based stratified evaluation in the offline evaluation of recommendation systems. Specifically, we investigate the following research questions:

To what extent is the offline evaluation of recommendation systems affected by the deployed model's characteristics in a closed loop feedback scenario?

Here, our objective is to assess the confounding effect of the deployed model's characteristics on the offline evaluation of recommendation models, as depicted in Figure <ref>.
In this regard, similar to the schematic examples presented in Section <ref>, we are interested to stratify the observed closed loop feedback based on the deployed model's characteristics. Such a stratified analysis enables us to assess the presence of the Simpson’s paradox in the standard offline evaluation of recommendation systems.
As such, we investigate significant trends in the relative performances of many different recommendation models where a trend is observed in the majority of strata, but the trend disappears or even reverses in the standard offline evaluation.

Can the propensity-based stratified evaluation help in better estimating the actual model’s performance when conducting a comparative offline evaluation?

-1 Our objective in the above research question is to assess the effectiveness of the proposed propensity-based stratified evaluation in the offline evaluation of recommendation models. As shown in Equation (<ref>), we can leverage marginalisation over the distribution of the confounding variable as a combined estimation of the potential outcome. In this research question, we are interested to assess how well this estimation is correlated with the open loop (randomised) evaluation where the deployed model is a random recommendation model, i.e. random items are exposed to the user.
Specifically, given a particular evaluation method (open loop, closed loop, etc.) we measure the effectiveness of many recommendation models based on standard rank-based evaluation metrics (e.g. nDCG) and record the relative ranking of those models in terms of the used metric. We then investigate the correlation between the relative rankings of the examined models, comparing the rankings obtained from an open loop (randomised) setup to those obtained using our proposed propensity-based evaluation method.
We leverage suitable significance tests, as described in Section <ref>, to determine if the comparative performances of the examined models predicted by the proposed propensity-based stratified evaluation method better correlates with the open loop (randomised) evaluation in comparison to the standard offline holdout evaluation.
Answering this research question provides a better understanding about the actual effectiveness of a given recommendation model in a closed loop scenario.

§ EXPERIMENTAL SETUP

(x) at (0,0) [draw, align=center] Open Loop (Random) Evaluation
(y) [draw, below of =1, left of =1, align=center, xshift=-5em, yshift=-4em] The Baseline Evaluation
(z) [draw, below of =1, right of =1, align=center, xshift=+5em, yshift=-4em] The Proposed Evaluation

(y) edge node $\tau_{XY}$ (x);
(z) edge node[above right] $\tau_{XZ}$ (x);
(y) edge node $\tau_{YZ}$  (z);

The dependent correlations significance test based on Steiger's method <cit.>. The effectiveness of each evaluation method ($Y$ and $Z$) is measured based on Kendall's $\tau$ rank correlation with the open loop (randomised) evaluation ($X$). The significance difference between the baseline evaluation method ($Y$) and the proposed evaluation method ($Z$) is measured based on Steiger's method <cit.>.

-1 In the following, we conduct experiments to address the two aforementioned research questions.
Figure <ref> presents the general structure of our experiments.
Each evaluation method ($X$, $Y$ and $Z$) ranks various examined recommendation models based on their relative  performances.
We use Kendall's $\tau$ rank correlation coefficient to measure the correlation between the relative order of the examined models in each evaluation method ($Y$ or $Z$) in comparison to the ground truth, i.e. open loop (randomised) evaluation ($X$). These correlation values are depicted as $\tau_{XY}$ and $\tau_{XZ}$ in Figure <ref>.
In addition, we use Steiger's method <cit.> to test the difference between two evaluation methods for significance, e.g. the baseline evaluation method ($Y$) and the proposed evaluation method ($Z$). We compare the effectiveness of our proposed propensity-based stratified evaluation with both the standard offline holdout and the counterfactual evaluation methods as baselines.
In the following, we describe the details of our experimental setup including the used datasets and evaluation metrics (Section <ref>), the examined recommendation models (Section <ref>) and how we estimate the propensities (Section <ref>) in our experiments.

§.§ Datasets and Evaluation Metrics

We use four datasets, which are widely used in the offline evaluation of recommendation system, namely the MovieLens <cit.>, Netflix <cit.>, Yahoo! music rating <cit.> and Coat shopping <cit.> datasets.
MovieLens contains 1m ratings from 6k users and 4k items, Netflix[<https://www.kaggle.com/netflix-inc/netflix-prize-data/>] contains 607k ratings from 10k users and 5k items, while the Yahoo! dataset contains 300K ratings given by 15.4k users for 1k items.
Furthermore, the Coat shopping dataset simulates the shopping for a coat in an online store. It contains about 7k ratings given by 290 users for 300 items.
All datasets were collected from an unknown deployed recommendation system in a closed loop scenario.
In addition, the Yahoo! dataset has a unique feature: a subset of users (5.4k) were asked to rate 10 randomly selected items (open loop scenario). Therefore, for a subset of users (5.4k), we have both the closed loop and open loop (random) feedback.
Similarly, for the Coat dataset, each user was asked to provide ratings for 24 self-selected (closed loop) items and 16 randomly picked (open loop) items.
However, in a real-world practical evaluation setup, the training of the models should be based on the collected closed loop feedback, while the actual evaluation could be based on the open loop (random) feedback because of the difficulties of collecting open loop feedback in the real-world, as discussed in Section 1.
We use the normalised Discounted Cumulative Gain (nDCG@k) metric for different cut-offs ($k=\{5, 10, 20, 30, 100\}$), while we use nDCG, without a rank cutoff, to denote the case where the metric is computed on the ranking of all items for each user.
The user's rating value for all datasets is an integer $r\in[0,5]$. Our objective is to rank the most relevant items for each user. Hence, we binarise all explicit rating values by keeping the highly recommended items ($r\geq4$).
In the closed loop evaluation of the examined models, we use 80% of the closed loop dataset (MovieLens, Yahoo! and Coat) for training and a 20% random split for testing, as is typical in RecSys evaluation <cit.>. On the other hand, for the open loop evaluation (Yahoo! and Coat), we evaluate the trained models on all the provided open loop (randomised) feedback.

§.§ Recommendation Models

In our experiments, we use the implementations provided by the Cornac framework[<https://github.com/PreferredAI/cornac>] to evaluate the following models[For ease of reference and compatibility, we use the same acronyms as used by the Cornac framework to denote the baselines.]:

* BO: a simple baseline model that recommends a random permutation of all available items to each user, regardless of their preferences.

* GA: a baseline model that recommends the global average rating to each and every user, regardless of their preferences.

* POP: a baseline model that ranks items based on their popularity, i.e. the number of times a specific item is rated, and suggests these items to every user regardless of their preferences.

* MF: Matrix Factorisation <cit.> is a rating prediction model that represents both users and items as latent vectors. This model is optimised to predict the observed rating for each user-item pair.

* PMF: Probabilistic Matrix Factorisation <cit.> is an extension of Matrix Factorisation <cit.> for large, sparse and imbalanced datasets. There are two variants of this algorithm depending on how the users' preferences are modelled, i.e. linear and non-linear.
We use both linear and non-linear variants in our experiments.

* SVD++: Singular Value Decomposition <cit.> is another rating prediction model that factorises the user-item rating matrix to a product of two lower rank matrices, i.e. user factors and item factors.

* WMF: Weighted Matrix Factorisation <cit.> is another extension of Matrix Factorisation <cit.> that models confidence and uncertainty in addition to preferences.

* NMF: Non-negative Matrix Factorisation <cit.> factorises the user-item rating matrix into user and item latent matrices, with the property that all matrices have no negative elements.

* BPR: Bayesian Personalised Ranking <cit.> is a pair-wise ranking model. The BPR model is trained based on uniform negative sampling, i.e. we randomly sample items not interacted with as negative instances for each user. We use both normal and weighted variants in our experiments.

* MMMF: Maximum Margin Matrix Factorisation <cit.> is a type of matrix factorisation model that is optimised for a soft margin (Hinge) ranking loss.

* NCF: Neural Collaborative Filtering <cit.> is a general neural network framework for collaborative filtering. We use Multi-Layer Perceptron (MLP), Generalised Matrix Factorisation (GMF) and Neural Matrix Factorisation (NeuMF) in our experiments.

We are interested in the relative order of the models' performances within closed loop and open loop scenarios.
Therefore, following previous research <cit.>, we vary the hyperparameter controlling the size of the latent factors in {10, 20, …, 100}, for each model that supports this hyperparameter[All the examined models support a latent variable hyperparameter except BO, GA, POP and MLP.]. This leads to a total of 104 models that are evaluated in our experiments. Each examined model differs on the algorithm or the latent variable size. When discussing a particular model, we denote the hyperparameter value in superscript - e.g. MF40 represents the MF model using latent factor size $m=40$.
For reproducibility, our code and the used datasets are available from: <https://github.com/terrierteam/stratified_recsys_eval>.

§.§ Estimating Propensity Scores

The propensity score $p_{u,i}$ is defined as the tendency of the deployed model to expose item $i$ to user $u$ (Definition <ref>).
Since it is not realistic for the deployed model to expose each and every item to the user, we have to estimate the propensity scores $p_{u,i}$ for various user and item pairs.
Yang et al. <cit.> proposed a simple method to estimate the propensity scores based on the following simplified assumption:

The propensity score is user independent, i.e. $p_{u,i}=p_{*,i}$. This assumption was made to address the lack of auxiliary user information in public datasets.

The user independent propensity score $p_{*,i}$ can be estimated using a two-step generative process <cit.>:

\begin{equation}
p_{*,i}=p_{*,i}^{select} * p_{*,i}^{interact|select}
\end{equation}

where $p_{*,i}^{select}$ is the prior probability that item $i$ is recommended by the deployed model and $p_{*,i}^{interact|select}$ is the conditional probability that the user interacts with item $i$ given that it is recommended.
Based on a set of mild assumptions, we can estimate the user independent propensity score $p_{*,i}$ as follows <cit.>:

\begin{equation}
\hat{p}_{*, i} \propto\left(n_{i}^{*}\right)^{\left(\frac{\gamma+1}{2}\right)}
\label{eq:estimated_props}
\end{equation}

-1 where $n_{i}^{*}$ is the total number of times item $i$ is interacted with and $\gamma$ is a parameter that affects the propensity distributions over items with different observed popularity.
The power-law parameter $\gamma$ affects the propensity distributions over items and depends on the examined dataset.
Following previous research <cit.>,
we estimate the $\gamma$ parameter using maximum likelihood for each dataset.

§ EXPERIMENTAL RESULTS AND ANALYSIS

We proposed a propensity-based stratified evaluation method in Section <ref> that takes into account the confounding role of the deployed model in the offline evaluation of recommendation systems.
In Section <ref>, we proposed to use Kendall's $\tau$ correlation coefficient to quantify the similarity between the relative order of many examined recommendation models in the proposed propensity-based stratified evaluation method as well as in the open loop (randomised) evaluation.
In the following, we conduct experiments with respect to the two research questions stated in Section <ref>, concerning the existence of the Simpson's paradox in the standard offline evaluation (Section <ref>) and the effectiveness of the proposed propensity-based stratified evaluation method (Section <ref>).

§.§ RQ <ref>: Investigating Simpson's Paradox

An instance of Simpson's paradox in the offline evaluation of recommender systems using nDCG on the MovieLens dataset <cit.>. * denotes a significant difference compared to the other model (paired t-test, p <0.05).

Evaluation Method     BPR10     WMF10     Number of Ratings     Number of Items
Holdout     0.373*     0.369*     200,018     3,449
IPS     0.207*     0.204*     200,018     3,449
Stratified     0.344*     0.351*     200,018     3,449

1r|Q1     0.339*     0.350*     197,600 (99%)     3,445

1r|Q2     0.695*     0.418*     2,418 (1%)     4

An instance of Simpson's paradox in the offline evaluation of recommender systems using nDCG on the Netflix dataset <cit.>. * denotes a significant difference compared to the other model (paired t-test, p <0.05).

Evaluation Method     MMMF100     MF50     Number of Ratings     Number of Items
Holdout     0.269*     0.263*     121,460     3,962
IPS     0.246*     0.245*     121,460     3,962
Stratified     0.239*     0.241*     121,460     3,962

1r|Q1     0.236*     0.242*     109,452 (90%)     3,944

1r|Q2     0.271*     0.229*     12,008 (10%)     18

Research Question <ref> investigates the confounding effect of the deployed model on the offline evaluation of recommendation systems based on closed loop feedback datasets as mentioned in Hypothesis <ref>. In Section <ref>, we presented a simple statistical method to represent the deployed model's characteristics as propensity scores. In the following, we use the estimated propensity scores $\hat{p}_{*, i}$ to partition the examined dataset into two equal size strata, namely the Q1 and Q2 strata. Since we estimate the propensity scores based on the total number of times each item is interacted with according to Equation (<ref>), the Q1 and Q2 strata represent the users' interactions with the long-tail and head items, respectively.

First, we highlight some instances of the Simpson's paradox in both the examined closed loop (MovieLens and Netflix) and open loop (Yahoo! and Coat) datasets.
Tables <ref> & <ref> compare the effectiveness of the examined evaluation methods (namely the holdout, IPS and our proposed stratified evaluation methods) on the MovieLens and Netflix datasets.
For the sake of simplicity, in the following, we focus on the Movielens dataset, analysing two models (BPR10, WMF10) and one metric (nDCG)[In Section <ref>, we compare the effectiveness of all the examined models for different nDCG cut-offs based on the Kendall's $\tau$ rank correlation coefficient.].
We observe that BPR10 is significantly better than WMF10 using both the holdout and counterfactual (IPS) evaluation methods.
However, the stratified analysis on the same test dataset (Q1 and Q2 strata) reveals that WMF10 is significantly better than BPR10 for the Q1 stratum while BPR10 is the dominant model for the Q2 stratum.
In addition, we observe that the Q1 and Q2 strata cover 99% and 1% of the user-item interactions, respectively. In fact, WMF10 is significantly better than BPR10 for 99% of the examined test dataset (i.e. the Q1 stratum) but this trend reverses when we combine it with the 1% feedback in the Q2 stratum, as represented in the holdout evaluation. Therefore, the real question is whether BPR10 does perform better than WMF10 as recognised by the holdout evaluation method?
The stratified analysis reveals that WMF10 should be recognised as the superior model by any reasonable evaluation method as recognised by our proposed stratified evaluation. We note that BPR10 is the superior model only for 1% of the feedback dataset and both the holdout and IPS evaluation methods are influenced by a very small minority of user-item interactions in the test dataset, i.e. 1% of user-item interactions in the Q2 stratum.
This stratum corresponds  to only 4 items among 3,499 items in the MovieLens dataset. The same pattern is also observed for the Netflix dataset between the MMMF100 and MF50 models presented in Table <ref>.

An instance of Simpson's paradox in the offline evaluation of recommender systems using nDCG on the Yahoo! dataset <cit.>. * denotes a significant difference compared to the other model (paired t-test, p <0.05).

Evaluation Method     BPR40     MF40     Number of Ratings     Number of Items
Holdout      0.270*     0.266*     62,341     1,000
IPS      0.238*     0.239*     62,341     1,000
Stratified      0.249*     0.256*     62,341     1,000

1r|Q1     0.218*     0.250*     57,467 (92%)     995

1r|Q2     0.622*     0.330*     4,874  (8%)*     5
Open Loop     0.169*     0.172*     54,000     1,000

An instance of Simpson's paradox in the offline evaluation of recommender systems using nDCG on the Coat dataset <cit.>. * denotes a significant difference compared to the other model (paired t-test, p <0.05).

Evaluation Method     GMF20     SVD20     Number of Ratings     Number of Items
Holdout      0.245*     0.244*     1,392     286
IPS      0.235*     0.237*     1,392     286
Stratified      0.227*     0.230*     1,392     286

1r|Q1     0.199*     0.215*     1,301 (93%)     281

1r|Q2     0.612*     0.434*     **91 (7%)*     5
Open Loop     0.248*     0.264*     1,392     286

Yahoo! Dataset

Coat Dataset

The correlation of the examined models between the open loop evaluation (x axis) and the specified closed loop evaluation (y axis) in the Yahoo! dataset (a) and the Coat dataset (b). All Kendall's $\tau$ correlations are statistically significant according to Steiger's method <cit.> (p <0.05). The orange line represents the best fit for the open/closed loop data based on a linear regression model.

-1 In the MovieLens and Netflix datasets, we do not have access to open loop feedback, i.e. feedback that was collected from a randomised exposure to items. As a result, we cannot measure the actual performances of the models in an open loop scenario in comparison to the corresponding closed loop scenario. As mentioned in Section <ref>, we have some open loop feedback for all the users in the Coat dataset and a fraction of users in the Yahoo! dataset. Tables <ref> and <ref> present performance comparisons of two pairs of models on the Yahoo! and Coat datasets. In particular, in Table <ref>, we compare BPR40 and MF40 using the nDCG evaluation metric, and observe that BPR40 is significantly better than MF40 using the classical holdout evaluation method. On the other hand, the IPS evaluation method favours MF40 but the difference is not significant based on the paired t-test ($p <0.05$).
However, the stratified analysis based on the estimated propensities (the Q1 and Q2 strata) reveals that MF40 is significantly better than BPR40 for the Q1 stratum, which covers 92% of feedback in the examined test dataset while BPR40 is the dominant model for the Q2 stratum, which covers only 8% of feedback in the closed loop test dataset.
Indeed, MF40 significantly
outperforms BPR40 for the majority of feedback and items (92% and 99.5%, respectively) in the examined test dataset.
Therefore, we argue that MF40 should be recognised as a better model by any reasonable evaluation method as recognised by the proposed stratified evaluation method and the open loop evaluation where random items were exposed to the users.
When we evaluate the models based on the aggregation of the Q1 and Q2 strata (i.e. holdout evaluation), a small minority of 5 items out of 1,000 total items corresponding to only 8% of the total user-item interactions in the Yahoo! dataset, plays a confounding factor in the offline evaluation of the examined models.
When considering the Coat dataset (Table <ref>), we also observe the same phenomenon when assessing the effectiveness of the GMF20 and SVD20 recommendation models. The observed phenomenon in Table <ref>, Table <ref> and Table <ref> can be explained by the fact that all the examined datasets (MovieLens, Netflix, Yahoo! and Coat) were collected from a closed loop scenario. Therefore, the collected closed loop dataset is skewed towards the deployed model's characteristics (as captured in the Q2 stratum in Table <ref>, Table <ref> and Table <ref>).

Next, to fully answer Research Question 1, we assess the generalisation of the above observations, by verifying the prevalence of the Simpson's paradox in the evaluation of all our 104 models. Figure <ref> shows the correlation of all the examined models (104 models as described in Section <ref>) between the open loop evaluation and the specified closed loop evaluation for both the Yahoo! and Coat datasets.
We observe that the models exhibit an imbalanced performance on the Q1 and Q2 strata, i.e. the nDCG scores for the Q1 stratum are not in proportion to the Q2 stratum ($nDCG_{Q2} \gg nDCG_{Q1}$).
On the other hand, the Kendall's $\tau$ correlation between the Q1 stratum and the open loop evaluation is much higher compared to the corresponding Q2 stratum ($\tau_{Q1} \gg \tau_{Q2}$).
Specifically, for the Coat dataset (Figure <ref>), the closed loop evaluation based on the Q2 stratum has a negative correlation with the open loop evaluation.
As a result, combining these two heterogeneous strata in the holdout evaluation – as shown in Figure <ref> and Figure <ref> – without taking into account that the Q1 stratum covers the majority of feedback (92% and 93% of the total feedback in the Yahoo! and Coat datasets, respectively), leads to unexpected consequences[This is in contrast to the Q2 stratum that covers only the remaining feedback, which is a small fraction of the dataset.]. In particular, the Kendall's $\tau$ correlation between the open loop evaluation and the holdout evaluation is significantly lower than the corresponding correlation between the Q1 stratum and the open loop evaluation, i.e. $0.62 < 0.72$ and $0.20 < 0.51$ for the Yahoo! and the Coat datasets, respectively.
This answers Research Question <ref>: the offline holdout evaluation of recommendation systems is influenced by the deployed model's characteristics (as captured in the Q2 stratum), leading to a conclusion prone to the Simpson's paradox.

To summarise, in this section, we highlighted the existence of the Simpson's paradox in the standard offline evaluation of recommendation systems based on both closed loop (MovieLens and Netflix) and open loop (Yahoo! and Coat) datasets.
In the following, we investigate the effectiveness of our proposed propensity-based stratified evaluation in addressing the issue.

§.§ RQ <ref>: Assessing the Proposed Propensity-based Stratified Evaluation Method

-1 In this section, we investigate the extent to which the proposed propensity-based stratified evaluation method (Section <ref>) leads to conclusions that are aligned with the results obtained from the open loop (randomised) evaluation, compared with both the counterfactual (IPS) and the holdout evaluation methods (Section <ref>). Indeed, our objective is to investigate to what extent each evaluation method correlates with the open loop evaluation. In contrast, we consider  open loop evaluation to be more akin to online evaluation (A/B testing or interleaving) where the evaluation process is not influenced by any confounders.
As mentioned in Section <ref>, we make use of the Kendall's $\tau$ rank correlation coefficient between the examined evaluation method and the open loop evaluation for all the examined recommendation models.

Kendall's $\tau$ rank correlation coefficient between the examined evaluation methods and the open loop (randomised) evaluation on the Yahoo! <cit.> and Coat <cit.> datasets. 1/2/3/4/5 denote a significant difference compared to the indicated evaluation method (Steiger's method <cit.>, p <0.05).

Dataset     Metric     Holdout1     IPS2     Stratified3     Q14     Q25
6*[origin=c]90Yahoo!     nDCG@5       *0.5993/4/5/0     0.6414/5/0/0     0.7001/4/5/0     0.4071/2/3/0     *0.4141/2/3/0

nDCG@10      *0.6934/5/0/0     0.7044/5/0/0     0.6974/5/0/0     0.5591/2/3/0     *0.5011/2/3/0

nDCG@20      *0.7294/5/0/0     0.7214/5/0/0     0.7164/5/0/0     0.6361/2/3/0     *0.5231/2/3/0

nDCG@30      *0.6523/5/0/0     0.6533/5/0/0     0.7131/2/5/0     0.6735/0/0/0     *0.4501/2/3/4

nDCG@100     *0.7013/4/5/0     0.7243/5/0/0     0.7741/2/5/0     0.7731/5/0/0     *0.3481/2/3/4

nDCG         *0.6223/4/5/0     0.6443/4/5/0     0.7101/2/5/0     0.7221/2/5/0     *0.3361/2/3/4
6*[origin=c]90Coat     nDCG@5     -0.0403/4/5/0     0.0513/4/5/0     0.1161/2/4/5     0.3621/2/3/5     -0.2881/2/3/4

nDCG@10     *0.1213/4/5/0     0.1653/4/5/0     0.2191/2/4/5     0.4181/2/3/5     -0.2401/2/3/4

nDCG@20     *0.1264/5/0/0     0.1184/5/0/0     0.1724/5/0/0     0.4501/2/3/5     -0.1951/2/3/4

nDCG@30     *0.1654/5/0/0     0.1594/5/0/0     0.2134/5/0/0     0.4671/2/3/5     -0.2071/2/3/4

nDCG@100     *0.3273/4/5/0     0.3603/4/5/0     0.3931/2/4/5     0.5471/2/3/5     -0.1861/2/3/4

nDCG     *0.2023/4/5/0     0.2253/4/5/0     0.2831/2/4/5     0.5091/2/3/5     -0.2081/2/3/4

Table <ref> shows the Kendall's $\tau$ rank correlation coefficient between the examined evaluation methods and the ground truth, i.e. open loop (randomised) evaluation.
On analysing the table, we observe that the
$\tau$ correlation between the holdout evaluation and the open loop evaluation is medium-to-strong ($0.599\leq \tau \leq 0.729$) for the Yahoo! dataset, and weaker (and $-0.40 \leq \tau \leq 0.327$) for the Coat dataset, with correlations on both datasets rising slightly for larger cut-offs.
Indeed, we note that the Coat dataset used a different data generation process compared to the MovieLens, Netflix and Yahoo! datasets <cit.> (i.e. making use of crowdsourcing users rather than real users). In addition, the number of examined users and items are lower than those in the Yahoo! dataset.
This finding agrees with previous research <cit.> who found that the offline evaluation based on standard rank-based metrics
does not necessarily correlate with the online evaluation (A/B testing or interleaving) results and researchers have questioned the validity of offline evaluation <cit.>.

Next, we consider the Kendall's $\tau$ correlation between the counterfactual evaluation (IPS) and the open loop evaluation methods. Table <ref> reveals that the IPS evaluation method performs slightly better than the holdout evaluation, i.e. the correlation values for IPS are greater than the holdout evaluation method for the majority of nDCG cut-offs in both datasets. However, none of these correlation differences are statistically significant according to the Steiger's test.
Therefore, we conclude that the counterfactual (IPS) evaluation method does not better represent unbiased user preferences during the evaluation than the holdout evaluation for the examined datasets. This is because in the IPS evaluation method, each feedback is weighted individually based on its propensity. The feedback sampling process, i.e. the process from which the closed loop feedback was collected, is not a random process. As a result, the sampled feedback is skewed and the corresponding estimated propensity scores are imbalanced.
In this situation, reweighting the feedback based on its disproportionate propensities leads to high variance  in the models' performance estimation using the IPS evaluation method <cit.>.

We now consider the correlations exhibited by the proposed stratified evaluation in Table <ref>. This reveals that the stratified evaluation performs significantly better than both the holdout and IPS evaluation methods[The holdout and IPS significance tests are represented as 1/2, respectively in the Stratified column of Table <ref>.] for the majority of nDCG cut-offs in both datasets, specifically at deeper cut-offs (Steiger's significance test <cit.>, $p<0.05$).
For example, considering nDCG as the evaluation metric, we observe that the proposed stratified evaluation method performs significantly better than both the holdout evaluation and the IPS evaluation, i.e. $0.710 > 0.622$ and $0.710 > 0.644$ for the Yahoo! dataset, while $0.283 > 0.202$ and $0.283 > 0.225$ for the Coat dataset.
Overall, we find that our propensity-based stratified evaluation method performs significantly better than both the holdout and counterfactual evaluation methods for deeper nDCG cut-offs in both datasets, which answers Research Question <ref>.

However, the performances of the holdout evaluation, the IPS evaluation and the proposed propensity-based stratified evaluation method are not significantly different for shallow nDCG cut-offs ($10\leq k \leq20$ and $20\leq k \leq30$ for the Yahoo! dataset and the Coat dataset, respectively) while
the observed correlations generally increase for deeper nDCG cut-offs.
The increased sensitivity for deeper rank cut-offs supports previous research <cit.> on the greater robustness to sparsity and the discriminative power of deeper cut-offs on the evaluation of recommendation systems. Valcarce et al. <cit.> found that since only a few items are exposed to the user during the feedback sampling process, the use of deeper cut-offs allow researchers to perform more robust and discriminative evaluations of recommender systems. Similar observations were made by <cit.> concerning the use of deeper rank cut-offs for training listwise learning to rank techniques for web search.

legend style=at=(0.5,-0.15),
anchor=north,legend columns=-1,
nodes near coords,
nodes near coords align=vertical,
xlabel=Number of strata,
ylabel=Kendall's $\tau$,
(1, 0.202)
(2, 0.282)
(3, 0.235)
(4, 0.244)
(5, 0.285)
(6, 0.276)
(7, 0.268)
(8, 0.249)
(9, 0.300)
(10, 0.251)
[red,sharp plot,dashed,nodes near coords=] coordinates
[above, red] at (axis cs:10,0.266) $0.266 \pm 0.021$;
[above, red, font=] at (axis cs: 2,0.007+0.282) $\ast$;
[above, red, font=] at (axis cs: 5,0.007+0.285) $\ast$;
[above, red, font=] at (axis cs: 6,0.007+0.276) $\ast$;
[above, red, font=] at (axis cs: 7,0.007+0.268) $\ast$;
[above, red, font=] at (axis cs: 9,0.007+0.300) $\ast$;

Coat Dataset

legend style=at=(0.5,-0.15),
anchor=north,legend columns=-1,
nodes near coords,
nodes near coords align=vertical,
xlabel=Number of strata,
ylabel=Kendall's $\tau$,
(1, 0.622)
(2, 0.710)
(3, 0.722)
(4, 0.713)
(5, 0.718)
(6, 0.728)
(7, 0.720)
(8, 0.718)
(9, 0.683)
(10, 0.639)
[red,sharp plot,dashed,nodes near coords=] coordinates
[above, red] at (axis cs:10,0.703) $0.706 \pm 0.026$;
[above, red, font=] at (axis cs: 2,0.007+0.710) $\ast$;
[above, red, font=] at (axis cs: 3,0.007+0.722) $\ast$;
[above, red, font=] at (axis cs: 4,0.007+0.713) $\ast$;
[above, red, font=] at (axis cs: 5,0.007+0.718) $\ast$;
[above, red, font=] at (axis cs: 6,0.007+0.728) $\ast$;
[above, red, font=] at (axis cs: 7,0.007+0.720) $\ast$;
[above, red, font=] at (axis cs: 8,0.007+0.718) $\ast$;

Yahoo! Dataset

The effect of the number of strata ($X$) on the performance of the proposed stratified evaluation.
The horizontal and vertical axes correspond to the number of strata and the correlation to the open loop (randomised) evaluation, respectively. $\ast$ denotes a significant difference compared to the standard holdout evaluation method ($X=1$) based on Steiger's method <cit.>, $p <0.05$. The horizontal dashed line represents the mean correlations for $2 \leq X \leq 10$.

Further experiments on the feedback sub-populations (the Q1 and Q2 strata in Table <ref>) reveal that the offline evaluation of the examined models, which is only based on the long-tail feedback (Q1 stratum) better correlates with the open loop evaluation.
Specifically, for the Coat dataset, the offline evaluation of the examined models based on the Q2 stratum has a negative correlation with the open loop evaluation.
This supports previous research such as Cremonesi et al. <cit.>, who found that the very few top popular items skew the rank-based evaluation of the recommendation systems. They suggested to exclude the extremely popular items while evaluating the effectiveness of recommendation models.

While Table <ref> demonstrates the effectiveness of the proposed stratified evaluation method for two sub-populations (Q1 and Q2 stratum), we next examine the effect of the number of strata used. In particular,
Figure <ref> demonstrates the effect of the number of strata in the proposed stratified evaluation for both the Coat and Yahoo! datasets. The horizontal axes ($X=\{1,2,...,10\}$) represent the number of strata while the vertical axes represent the correlation between the proposed stratified evaluation method and the open loop (randomised) evaluation for all the 104 examined models. When we have only one stratum (i.e. $X=1$ in Figure <ref> and Figure <ref>), the proposed stratified evaluation method corresponds to the holdout evaluation presented in Table <ref>.
We observe that for all the examined number of strata ($2 \leq X \leq 10$) in both datasets, the proposed stratified evaluation method better correlates with the open loop (randomised) evaluation compared with the holdout evaluation (i.e. $X=1$). However, the number of strata has a marginal effect on the correlation between the proposed stratified evaluation and the open loop (randomised) evaluation, i.e. the mean correlations between the proposed stratified evaluation ($ 2 \leq X \leq 10$) for the Coats and Yahoo! datasets are $0.266 \pm 0.021$ and $0.706 \pm 0.026$, respectively, compared to $0.202$ and $0.622$ correlations in the holdout evaluation ($X=1$).
In addition, for $2 \leq X \leq 10$, the majority of cases (5 and 7 out of 9 for the Coats and Yahoo! datasets, respectively) demonstrate significantly higher correlations than the holdout evaluation.
Note that the number of strata to use for each dataset can be determined using the stratification principle <cit.>, which favours the use of strata such that within each stratum, the users' feedback is as similar as possible.
Although different datasets have different levels of closed loop effect depending on the deployed model's characteristics,
our experiments reveal that without further information about the deployed model, stratifying closed loop feedback into sub-populations based on the estimated propensities
allow researchers to account for the closed loop effect from the collected feedback dataset.

§ CONCLUSIONS

The first key step in evaluating recommendation models based on feedback datasets is to understand how the feedback data was generated.
As for any causal analysis, we started from a hypothesis about the data generating process (Hypothesis <ref>).
We investigated the influence of the deployed recommendation model on the closed loop feedback data generation process. Next, we studied the impact of this collected closed loop feedback data on the offline evaluation of a new recommendation model.
Our in-depth experiments based on both the closed loop and open loop (randomised) feedback datasets revealed that the standard holdout evaluation method suffers from the Simpson's paradox, i.e. a very small minority of items, which are repeatedly exposed by the deployed model plays a confounding factor in the holdout evaluation where all feedback is aggregated together without accounting for the deployed model's confounding effect.
Consequently, our experiments highlighted that when evaluating a recommendation model in an offline fashion, the test set should be chosen carefully in order to free it from any confounders, and specifically from the closed loop algorithmic effect. Our findings on four well-known closed loop and open loop (randomised) feedback datasets provided a theoretical and experimental explanation of the  concerns regarding the offline evaluation of recommendation systems observed in the recent literature <cit.>.
In addition, we proposed a novel propensity-based stratified evaluation method, which takes into account the role of the deployed model in the offline evaluation to more accurately estimate the performance of a given new recommendation model. The proposed model significantly outperforms the classical holdout evaluation and the counterfactual evaluation methods for deeper nDCG cut-offs, i.e. the proposed model better represents the actual relative order of the examined models using an open loop setup.
Our work aims to bring attention to the challenges of the offline evaluation based on closed loop feedback datasets and to show how the application of causal reasoning techniques, through our proposed propensity-based stratified evaluation method, allows to counter the confounding effect of the deployed model so as to obtain a more accurate evaluation of a recommender system.

-1 This work has been supported by EPSRC grant EP/R018634/1: Closed-Loop Data Science for Complex, Computationally- & Data-Intensive Analytics.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Ai, Bi, Luo, Guo, and CroftAi
et al2018a]
authorpersonQingyao Ai, personKeping Bi,
personCheng Luo, personJiafeng Guo, and
personW. Bruce Croft. year2018a.
Unbiased Learning to Rank with Unbiased Propensity
Estimation. In booktitleProceedings of the 41st
International ACM SIGIR Conference on Research and Development in Information
Retrieval (seriesSIGIR '18).

[Ai, Mao, Liu, and CroftAi
et al2018b]
authorpersonQingyao Ai, personJiaxin Mao,
personYiqun Liu, and personW. Bruce Croft.
Unbiased Learning to Rank: Theory and Practice. In
booktitleProceedings of the 27th ACM International
Conference on Information and Knowledge Management
(seriesCIKM '18).

[Alstott, Bullmore, and PlenzAlstott
et al2014]
authorpersonJeff Alstott, personEd
Bullmore, and personDietmar Plenz.
powerlaw: A Python Package for Analysis of
Heavy-Tailed Distributions.
journalPLOS ONE volume9,
number1 (year2014).

[Anderson, Maystre, Anderson, Mehrotra, and
LalmasAnderson et al2020]
authorpersonAshton Anderson, personLucas
Maystre, personIan Anderson, personRishabh Mehrotra,
and personMounia Lalmas. year2020.
Algorithmic Effects on the Diversity of Consumption
on Spotify. In booktitleProceedings of The Web Conference
2020 (seriesWWW '20).

authorpersonPeter C. Austin.
An Introduction to Propensity Score Methods for
Reducing the Effects of Confounding in Observational Studies.
journalMultivariate Behavioral Research
volume46, number3 (year2011).

[Cañamares, Castells, and
MoffatCañamares et al2020]
authorpersonRocío Cañamares,
personPablo Castells, and personAlistair Moffat.
Offline evaluation options for recommender
journalInformation Retrieval Journal
volume23, number4 (year2020).

[Chaney, Stewart, and EngelhardtChaney
et al2018]
authorpersonAllison J. B. Chaney,
personBrandon M. Stewart, and personBarbara E.
Engelhardt. year2018.
How Algorithmic Confounding in Recommendation
Systems Increases Homogeneity and Decreases Utility. In
booktitleProceedings of the 12th ACM Conference on
Recommender Systems (seriesRECSYS '18).

[Charig, Webb, Payne, and WickhamCharig
et al1986]
authorpersonC R Charig, personD R Webb,
personS R Payne, and personJ E Wickham.
Comparison of treatment of renal calculi by open
surgery, percutaneous nephrolithotomy, and extracorporeal shockwave
journalBMJ volume292,
number6524 (year1986).

[Cremonesi, Koren, and TurrinCremonesi
et al2010]
authorpersonPaolo Cremonesi, personYehuda
Koren, and personRoberto Turrin.
Performance of Recommender Algorithms on Top-n
Recommendation Tasks. In booktitleProceedings of the Fourth
ACM Conference on Recommender Systems (seriesRECSYS

[Diaz, Mitra, Ekstrand, Biega, and
CarteretteDiaz et al2020]
authorpersonFernando Diaz, personBhaskar
Mitra, personMichael D. Ekstrand, personAsia J.
Biega, and personBen Carterette.
Evaluating Stochastic Rankings with Expected
Exposure. In booktitleProceedings of the 29th ACM
International Conference on Information & Knowledge Management
(seriesCIKM '20).

[Gruson, Chandar, Charbuillet, McInerney,
Hansen, Tardieu, and CarteretteGruson et al2019]
authorpersonAlois Gruson, personPraveen
Chandar, personChristophe Charbuillet, personJames
McInerney, personSamantha Hansen, personDamien
Tardieu, and personBen Carterette.
Offline Evaluation to Make Decisions About
PlaylistRecommendation Algorithms. In booktitleProceedings
of the Twelfth ACM International Conference on Web Search and Data Mining
(seriesWSDM '19).

[Harper and KonstanHarper and
authorpersonF. Maxwell Harper and
personJoseph A. Konstan. year2015.
The MovieLens Datasets: History and Context.
journalACM Trans. Interact. Intell. Syst.
volume5, number4 (year2015).

[He, Liao, Zhang, Nie, Hu, and ChuaHe
et al2017]
authorpersonXiangnan He, personLizi Liao,
personHanwang Zhang, personLiqiang Nie,
personXia Hu, and personTat-Seng Chua.
Neural Collaborative Filtering. In
booktitleProceedings of The Web Conference
(seriesWWW '17).

[Herlocker, Konstan, Terveen, and
RiedlHerlocker et al2004]
authorpersonJonathan L. Herlocker,
personJoseph A. Konstan, personLoren G. Terveen,
and personJohn T. Riedl. year2004.
Evaluating Collaborative Filtering Recommender
journalACM Trans. Inf. Syst. volume22,
number1 (year2004).

[Hu, Koren, and VolinskyHu
et al2008]
authorpersonY. Hu, personY. Koren, and
personC. Volinsky. year2008.
Collaborative Filtering for Implicit Feedback
Datasets. In booktitleProceedings of the 8th IEEE
International Conference on Data Mining (seriesICDM

[Jadidinejad, Macdonald, and OunisJadidinejad
et al2019]
authorpersonAmir H. Jadidinejad, personCraig
Macdonald, and personIadh Ounis.
How Sensitive is Recommendation Systems' Offline
Evaluation to Popularity?. In booktitleProceedings of the
Workshop on Offline Evaluation for Recommender Systems (REVEAL '19).

[Jadidinejad, Macdonald, and OunisJadidinejad
et al2020]
authorpersonAmir H. Jadidinejad, personCraig
Macdonald, and personIadh Ounis.
Using Exploration to Alleviate Closed-Loop Effects
in Recommender Systems. In booktitleProceedings of the 43nd
International ACM SIGIR Conference on Research and Development in Information
Retrieval (seriesSIGIR '20).

[Jiang, Chiappa, Lattimore, György, and
KohliJiang et al2019]
authorpersonRay Jiang, personSilvia
Chiappa, personTor Lattimore, personAndrás
György, and personPushmeet Kohli.
Degenerate Feedback Loops in Recommender Systems.
In booktitleProceedings of the 2019 AAAI/ACM Conference on
AI, Ethics, and Society (seriesAIES ’19).

[Joachims and SwaminathanJoachims and
authorpersonThorsten Joachims and
personAdith Swaminathan. year2016.
Counterfactual Evaluation and Learning for Search,
Recommendation and Ad Placement. In booktitleProceedings of
the 39th International ACM SIGIR Conference on Research and Development in
Information Retrieval (seriesSIGIR '16).

[Joachims, Swaminathan, and SchnabelJoachims
et al2017]
authorpersonThorsten Joachims, personAdith
Swaminathan, and personTobias Schnabel.
Unbiased Learning-to-Rank with Biased Feedback. In
booktitleProceedings of the Tenth ACM International
Conference on Web Search and Data Mining (seriesWSDM

[Kievit, Frankenhuis, Waldorp, and
BorsboomKievit et al2013]
authorpersonRogier A. Kievit,
personWillem E. Frankenhuis, personLourens J.
Waldorp, and personDenny Borsboom.
Simpson's paradox in psychological science: a
practical guide.
journalFrontiers in psychology
volume4 (year2013).

authorpersonYehuda Koren.
Factorization Meets the Neighborhood: A
Multifaceted Collaborative Filtering Model. In
booktitleProceedings of the 14th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining
(seriesKDD '08).

[Koren, Bell, and VolinskyKoren
et al2009]
authorpersonYehuda Koren, personRobert
Bell, and personChris Volinsky.
Matrix Factorization Techniques for Recommender
journalComputer volume42,
number8 (year2009).

[Lee and SeungLee and Seung2000]
authorpersonDaniel D. Lee and
personH. Sebastian Seung. year2000.
Algorithms for Non-Negative Matrix Factorization.
In booktitleProceedings of the 13th International Conference
on Neural Information Processing Systems (seriesNIPS

[Macdonald, Santos, and OunisMacdonald
et al2013]
authorpersonCraig Macdonald,
personRodrygo L. Santos, and personIadh Ounis.
The Whens and Hows of Learning to Rank for Web
journalInf. Retr. volume16,
number5 (year2013).

[Marlin and ZemelMarlin and Zemel2009]
authorpersonBenjamin M. Marlin and
personRichard S. Zemel. year2009.
Collaborative Prediction and Ranking with
Non-random Missing Data. In booktitleProceedings of the
Third ACM Conference on Recommender Systems (seriesRECSYS

[Oosterhuis and de RijkeOosterhuis and
de Rijke2020]
authorpersonHarrie Oosterhuis and
personMaarten de Rijke. year2020.
Policy-Aware Unbiased Learning to Rank for Top-k
Rankings. In booktitleProceedings of the 43nd International
ACM SIGIR Conference on Research and Development in Information Retrieval
(seriesSIGIR '20).

[Ovaisi, Ahsan, Zhang, Vasilaky, and
ZhelevaOvaisi et al2020]
authorpersonZohreh Ovaisi, personRagib
Ahsan, personYifan Zhang, personKathryn Vasilaky,
and personElena Zheleva. year2020.
Correcting for Selection Bias in Learning-to-Rank
Systems. In booktitleProceedings of The Web Conference
(seriesWWW '20).

authorpersonJudea Pearl.
Comment: Understanding Simpson’s Paradox.
journalThe American Statistician
volume68, number1 (year2014).

[Radlinski, Kurup, and JoachimsRadlinski
et al2008]
authorpersonFilip Radlinski, personMadhu
Kurup, and personThorsten Joachims.
How Does Clickthrough Data Reflect Retrieval
Quality?. In booktitleProceedings of the 17th ACM
Conference on Information and Knowledge Management
(seriesCIKM '08).

[Radlinski, Kurup, and JoachimsRadlinski
et al2011]
authorpersonFilip Radlinski, personMadhu
Kurup, and personThorsten Joachims.
booktitleEvaluating Search Engine Relevance with
Click-Based Metrics.

[Rendle, Freudenthaler, Gantner, and
Schmidt-ThiemeRendle et al2009]
authorpersonSteffen Rendle, personChristoph
Freudenthaler, personZeno Gantner, and personLars
Schmidt-Thieme. year2009.
BPR: Bayesian Personalized Ranking from Implicit
Feedback. In booktitleProceedings of the Twenty-Fifth
Conference on Uncertainty in Artificial Intelligence
(seriesUAI '09).

[Rossetti, Stella, and ZankerRossetti
et al2016]
authorpersonMarco Rossetti, personFabio
Stella, and personMarkus Zanker.
Contrasting Offline and Online Results when
Evaluating Recommendation Algorithms. In
booktitleProceedings of the 10th ACM Conference on
Recommender Systems (seriesRECSYS '16).

[Salakhutdinov and MnihSalakhutdinov and
authorpersonRuslan Salakhutdinov and
personAndriy Mnih. year2007.
Probabilistic Matrix Factorization. In
booktitleProceedings of the 20th International Conference on
Neural Information Processing Systems (seriesNIPS’07).

[Sammut and WebbSammut and Webb2010]
editorpersonClaude Sammut and
personGeoffrey I. Webb (Eds.). year2010.
booktitleHoldout Evaluation, Encyclopedia of Machine

[Schnabel, Swaminathan, Singh, Chandak, and
JoachimsSchnabel et al2016]
authorpersonTobias Schnabel, personAdith
Swaminathan, personAshudeep Singh, personNavin
Chandak, and personThorsten Joachims.
Recommendations As Treatments: Debiasing Learning
and Evaluation. In booktitleProceedings of the 33rd
International Conference on International Conference on Machine Learning -
Volume 48 (seriesICML'16).

authorpersonHarald Steck.
Training and Testing of Recommender Systems on Data
Missing Not at Random. In booktitleProceedings of the 16th
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
(seriesKDD '10).

authorpersonHarald Steck.
Item Popularity and Recommendation Accuracy. In
booktitleProceedings of the Fifth ACM Conference on
Recommender Systems (seriesRECSYS '11).

authorpersonJames H Steiger.
Tests for comparing elements of a correlation
journalPsychological Bulletin
volume87, number2 (year1980).

[Sun, Khenissi, Nasraoui, and ShaftoSun
et al2019]
authorpersonWenlong Sun, personSami
Khenissi, personOlfa Nasraoui, and personPatrick
Shafto. year2019.
Debiasing the Human-Recommender System Feedback
Loop in Collaborative Filtering. In booktitleCompanion
Proceedings of The 2019 World Wide Web Conference
(seriesWWW '19).

[Swaminathan and JoachimsSwaminathan and
authorpersonAdith Swaminathan and
personThorsten Joachims. year2015.
Batch Learning from Logged Bandit Feedback through
Counterfactual Risk Minimization.
journalJournal of Machine Learning Research
volume16 (year2015).

authorpersonSteven K. Thompson.
booktitleStratified Sampling.
publisherJohn Wiley & Sons, Ltd, Chapter 11,

[Valcarce, Bellogín, Parapar, and
CastellsValcarce et al2018]
authorpersonDaniel Valcarce, personAlejandro
Bellogín, personJavier Parapar, and
personPablo Castells. year2018.
On the Robustness and Discriminative Power of
Information Retrieval Metrics for top-N Recommendation. In
booktitleProceedings of the 12th ACM Conference on
Recommender Systems (seriesRECSYS '18).

[Wang, Bendersky, Metzler, and NajorkWang
et al2016]
authorpersonXuanhui Wang, personMichael
Bendersky, personDonald Metzler, and personMarc
Najork. year2016.
Learning to Rank with Selection Bias in Personal
Search. In booktitleProceedings of the 39th International
ACM SIGIR Conference on Research and Development in Information Retrieval
(seriesSIGIR '16).

[Wang, Liang, Charlin, and BleiWang
et al2018]
authorpersonYixin Wang, personDawen
Liang, personLaurent Charlin, and
personDavid M. Blei. year2018.
The Deconfounded Recommender: A Causal Inference
Approach to Recommendation.
journalArXiv e-prints, Article
articlenoarXiv:1808.06581 (year2018).

[Weimer, Karatzoglou, and SmolaWeimer
et al2008]
authorpersonMarkus Weimer, personAlexandros
Karatzoglou, and personAlex Smola.
Improving Maximum Margin Matrix Factorization.
journalMach. Learn. volume72,
number3 (year2008).

[Yang, Cui, Xuan, Wang, Belongie, and
EstrinYang et al2018]
authorpersonLongqi Yang, personYin Cui,
personYuan Xuan, personChenyang Wang,
personSerge Belongie, and personDeborah Estrin.
Unbiased Offline Recommender Evaluation for
Missing-not-at-random Implicit Feedback. In
booktitleProceedings of the 12th ACM Conference on
Recommender Systems (seriesRECSYS '18).