# Estimating and Penalizing Induced Preference Shifts in Recommender Systems

Micah Carroll  Dylan Hadfield-Menell  Stuart Russell  Anca Dragan

###### Abstract

The content that a recommender system (RS) shows to users influences them.
Therefore, when choosing which recommender to deploy, one is implicitly also
choosing to induce specific internal states in users. Even more, systems
trained via long-horizon optimization will have direct incentives to
manipulate users, e.g. shift their preferences so they are easier to satisfy.
In this work we focus on induced preference shifts in users. We argue that –
before deployment – system designers should: estimate the shifts a recommender
would induce; evaluate whether such shifts would be undesirable; and even
actively optimize to avoid problematic shifts. These steps involve two
challenging ingredients: _estimation_ requires anticipating how hypothetical
policies would influence user preferences if deployed – we do this by using
historical user interaction data to train predictive user model which
implicitly contains their preference dynamics; _evaluation_ and _optimization_
additionally require metrics to assess whether such influences are
manipulative or otherwise unwanted – we use the notion of “safe shifts”, that
define a trust region within which behavior is safe. In simulated experiments,
we show that our learned preference dynamics model is effective in estimating
user preferences and how they would respond to new recommenders. Additionally,
we show that recommenders that optimize for staying in the trust region can
avoid manipulative behaviors while still generating engagement.

Machine Learning, ICML

##  1 Introduction

Recommender systems (RSs) generally show users feeds of items with the goal of
maximizing their engagement, and users choose what to click on based on their
preferences. Importantly, the recommender’s actions are not independent of
changes in users’ internal states: simple changes in the content displayed to
users have been shown to affect user’s behavior (Wilhelm et al., 2018;
Hohnhold et al., 2015), mood (Kramer et al., 2014), beliefs (Allcott et al.,
2020), and preferences (Epstein & Robertson, 2015).

Given the dependence between users’ internal state and the recommender system,
when a system designer chooses a specific recommender algorithm (policy), they
are implicitly also choosing how to influence user’s behaviors, mood,
preferences, etc. While traditionally RS policies have been myopic (tended at
satisfying users’ current desires), optimizing long-term user engagement has
been a growing trend – typically via reinforcement learning (Afsar et al.,
2021); these non-myopic policies are commonly referred to as long-term value,
or LTV, systems. However, these policies will have incentives to manipulate
users as a side-effect (Albanie et al., 2017; Krueger et al., 2020; Evans &
Kasirzadeh, 2021): for example, certain preferences are easier to satisfy than
others, leading to more potential for engagement – this could be because of
availability of more content for some preferences compared to others, or
because strong preferences for a particular type of content lead to higher
engagement than more neutral ones. This can make LTV systems a particularly
poor choice for avoiding undesired shifts in user’s behaviors, moods,
preferences, etc. While it has been proposed to prevent the RS from reasoning
about manipulation pathways (e.g., by keeping it myopic) (Krueger et al.,
2020; Farquhar et al., 2022), even such systems can influence users in
systematically undesirable ways (Jiang et al., 2019; Mansoury et al., 2020;
Chaney et al., 2018).

In this work, we focus on user’s preference shifts, which is a particularly
challenging problem that has been gaining more attention lately (Franklin et
al., 2022). Any recommender policy will have some influence on user
preferences. While this might seem cause for concern, the degree to which
undesirable and manipulative preference influence occurs in practice has yet
to be measured. Moreover, no framework has yet been proposed to _explicitly
account for a policy’s influence on users when choosing which policy to
deploy_. We attempt to propose such a framework here – requiring us to tackle
two critical problems.

Figure 1: Preferences induced by different RS policies across the same cohort
of users. In our simulated environment, preferences (y axis) are in 1D, and
change over time (x axis) as users interact with the policy. On the right, the
RL system drives preferences to one spot. A myopic policy (center) has a
similar effect but less concentrated. These shifts are different from what we
introduce as the “natural” evolution of user preferences.

First, we need to anticipate what preference shifts a RS policy will cause
_before it is deployed_. This alone would enable system designers to generate
visualizations such as those in Fig. 1 (explained in more depth in our
experiments): these could surface whether certain RS policies tend to induce
shifts that are significantly different from others, and allow to assess
whether such shifts seem manipulative or otherwise unwanted. We study how to
do this using historical user interaction data, and show how to train a
predictive model that implicitly captures preference dynamics. Once trained,
our model estimates a user’s (latent) preferences induced by their
interactions with a previous RS policy, and then uses that to estimate how
that user’s preferences would evolve under a new policy.

The second critical problem is having quantitative metrics for evaluating
whether an evolution of preferences is actually unwanted: a computable metric
not only simplifies evaluation, but also enables recommenders to actively
optimize against unwanted shifts. The issue arises from the fact that standard
engagement metrics are preference-change-agnostic: they do not directly assign
value to preference shifts (or shifts in any other aspect of the user’s state,
for that matter). Even if a system were to completely overwrite a user’s
preferences, as long as the user is engaged, the system would be “performing
well”. We thus introduce some preliminary metrics which assign value to
preference changes – based on how the users’ preferences would have changed in
the absence of the recommender – in an attempt to isolate the recommender’s
influence over the user’s shifts.

For such metrics, instead of defining desirable or undesirable shifts
directly, we provide a framework for conservatively defining “safe shifts”: we
non-exhaustively list certain shifts that we trust not to be particularly
problematic, and measure the extent to which a policy deviates from them. If
the shifts induced by the policy at hand differ significantly from the safe
ones, that should warrant more investigation. A candidate for safe shifts that
we introduce is how user preferences would shift if they were interacting with
a random recommender – which we call “natural preference shifts”. One can
think of this as an approximation to not having a recommender system at all.
Fig. 1 (left) shows this natural preference evolution for our running example,
and how user preferences stay somewhat diffuse but drift towards the opposite
mode that the RL and myopic policies push them to. Note that while our metrics
can effectively penalize undersired shifts, it comes at a cost: natural
shifts, and in fact any lists of safe shifts that we define, are unlikely to
be exhaustive, which means the approach will conservatively penalize policies
that might in reality be safe.

To demonstrate both the estimation of preference shifts and their evaluation,
we set up a testing environment in which we emulate ground truth user behavior
by drawing from a model of preference shift from prior work (Bernheim et al.,
2021). We first show that our learned preference shift estimation model –
trained using historical user interaction data – can correctly anticipate user
preference shifts almost as well as knowing the true preference dynamics.
Additionally, we show qualitatively that in this environment, RL and even
myopic recommenders lead to potentially undesired shifts. Further, we find
that our evaluation metric can correctly flag which policies will produce
undesired shifts, and evaluates the RL policy from Fig. 1 as 35% worse than
the myopic one, which is in turn 40% worse than our policy which is penalized
for manipulating user’s preferences. Our results also suggest that evaluating
our metric using the trained estimation model correlates to using ground truth
preference dynamics, and that optimizing for safe shifts does lead to higher
scoring (more safe) policies.

Although this work just scratches the surface of finding the right metrics for
unwanted preference shifts and evaluating them in real systems, our results
already have implications for the development of recommender systems: in order
to ethically use recommenders at scale, we must take active steps to measure
and penalize how such systems shift users’ internal states. In turn, we offer
a roadmap for how one might be able to do so by learning from user interaction
data, and put forward a framework for specifying “safe shifts” for detecting
and controlling such incentives.

##  2 Related Work

RS effects on users’ internal states. A variety of previous work considers how
RSs algorithms might affect users: influencing user’s preferences for
e-commerce purposes (Häubl & Murray, 2003; Cosley et al., 2003; Gretzel &
Fesenmaier, 2006), altering people’s moods for psychology research (Kramer et
al., 2014), changing populations’ opinions or behaviors (Matz et al., 2017),
exacerbate (Hasan et al., 2018) cases of addiction to social media
(Andreassen, 2015), or increase polarization (Stray, 2021). There have been
three main types of approaches to quantitatively estimating effects of RSs’
policies on users: 1) analyzing static datasets of interactions directly
(Nguyen et al., 2014; Ribeiro et al., 2019; Juneja & Mitra, 2021; Li et al.,
2014), 2) simulating interactions between users and RSs based on hand-crafted
models of user dynamics (Chaney et al., 2018; Bountouridis et al., 2019; Jiang
et al., 2019; Mansoury et al., 2020; Yao et al., 2021; Ie et al., 2019a), or
3) using access to real users and estimating effects through direct
interventions (Holtz et al., 2020; Matz et al., 2017). We see our approach as
an improvement on 2), in that we propose to implicitly learn user dynamics
instead of hand-specifying them. While we still assume a known choice model
(how users choose content based on their preferences), such assumption is much
less restrictive than assuming fully known dynamics.

Neural networks for recommendation and human modeling. While data-driven
models of human behavior have been used widely in real-world RS as click
predictors (Zhang et al., 2019; Covington et al., 2016; Cheng et al., 2016;
Okura et al., 2017; Mudigere et al., 2021; Zhang et al., 2014; Wu et al.,
2017) and for simulating human behavior in the context RL RSs’ training (Chen
et al., 2019; Zhao et al., 2019; Bai et al., 2020; Shi et al., 2018), to our
knowledge they have not been previously used for explicitly simulating and
quantifying the effect on users of hypothetical recommenders. We emphasize how
human models can also be used as simulation mechanisms by anticipating RSs’
policies impact on users’ behaviors and, as enabled by our method, even
preferences.

RL for RS. Using RL to train LTV RSs has recently seen a dramatic increase of
interest (Afsar et al., 2021), with some notable work led by YouTube and
Facebook (Ie et al., 2019b; Chen et al., 2020; Gauci et al., 2019) – which
have led to significant real-world engagement increases.

Side effects and safe shifts. Our work starts from a similar question to that
of the side effects literature (Krakovna et al., 2019; Kumar et al., 2020),
applied to preference change: given that the reward function will not fully
account for the cost (or value) of preference shifts, how do we prevent
undesired preference-shift side effects? Our notion of safe shifts corresponds
to the choice of baseline in this literature.

##  3 Preliminaries

Setup. We model users as having time-indexed preferences
$u_{t}\in\mathbb{R}^{d}$, which assign a scalar value to every possible item
of content $x\in\mathbb{R}^{d}$: the value to the user derived from the
engagement with item $x_{t}$ under $u_{t}$ is modeled as being given by
$\hat{r}_{t}(u_{t})=u_{t}^{T}x_{t}$. The user’s preferences $u_{t}$ together
with other variables constitute the user’s internal state $z_{t}$, which
comprises a sufficient statistic for their long-term behavior, but which our
method will not explicitly model. At every time step the user sees a slate
$s_{t}$ (a list of items) produced by a recommender policy $\pi$, and chooses
an item $x_{t}$. The policy $\pi$ maps history of slates and choices so far,
$s_{0:t},x_{0:t}$, to each new slate $s_{t+1}$: that is
$s_{t+1}\sim\pi(s_{0:t},x_{0:t})$. Upon making a choice, the user’s internal
state updates to $z_{t+1}$.

User choice model. We are interested in inferring and predicting preferences,
which we do not observe directly. As such, we make an assumption – justified
in Appendix B – about how user behavior (the user’s choice $x_{t}$) relates to
their current preferences $u_{t}$: that is, we assume the form of
$\mathbb{P}(x_{t}=x|s_{t},u_{t})$ is known.

Preference evolution as an NHMM. The dynamics of the user’s time-indexed
internal state $z_{t}$ – which we don’t assume to be known – depend on the
previous state $z_{t-1}$ and on the last choice of slate by the recommender
$s_{t}$ (which in turn depends on history because the policy $\pi$ uses
history). This makes our setup a Non-homogeneous HMM (NHMM) (Hughes et al.,
1999; Rabiner, 1986), with hidden state $z_{t}$ and time-dependent dynamics
$\mathbb{P}(z_{t+1}|z_{t},s_{t})$ (the choice of slate $s_{t}$ by $\pi$ will
affect the future internal state). See Appendix D for more information. We
will be using the NHMM inferences as an oracle benchmark for our methods’
performance.

##  4 Estimating Users’ Preferences

Our proposal boils down to the following: before deploying a new recommender
policy $\pi^{\prime}$, we first need to be able to understand how that policy
would change preferences and behavior of users. More formally, for a set of
$N$ users, we assume access to historical data of their interaction with a RS
policy $\pi$: $\mathcal{D}_{j}=\\{s^{\pi}_{0:T},x^{\pi}_{0:T}\\}$ for every
user $j$. For any user, we are interested in estimating how their preferences
would evolve if further interactions occurred with a new policy
$\pi^{\prime}$, which we denote $u^{\pi^{\prime}}_{H}$ (where $H>T$).

###  4.1 Estimation under known user dynamics

We first describe here how one could solve this problem optimally if one had
oracle access to the true user internal state dynamics, and later relax this
assumption.

Assuming the user internal state dynamics $\mathbb{P}(z_{t+1}|z_{t},s_{t})$ to
be known, our goal is – as mentioned above – to estimate what their
preferences would be at a future timestep $H$ if policy $\pi^{\prime}$ were to
be deployed going forward, that is, $u_{H}^{\pi^{\prime}}$. Given the
assumption of known internal state dynamics, every component of the NHMM is
known, meaning that we can perform exact inference over
$u_{H}^{\pi^{\prime}}$. We can do so via a simple extension of HMM prediction:
using the history of interactions $(s^{\pi}_{0:T},x^{\pi}_{0:T})$ to estimate
$\mathbb{P}(z^{\pi^{\prime}}_{H}|s^{\pi}_{0:T},x^{\pi}_{0:T})$ as shown in
Appendix E.1. As the preferences $u^{\pi^{\prime}}_{H}$ are just one component
of the internal state $z_{H}^{\pi^{\prime}}$, one can trivially recover a
posterior over the preferences specifically.

Figure 2: Future preference estimation model. Given slates and choices up
until the current timestep $s_{0:t},x_{0:t}$, we train a network to predict
beliefs over the next preferences $u_{t+1}$ which – together with the next
slate $s_{t+1}$ – induce a distribution over content choices through the
choice model. We can supervise the training with the actual choices users made
for slates – the network will learn to output preference beliefs which induce
similar choices to those in the data.

###  4.2 Estimation under unknown user dynamics

In practice, one will not have access to an explicit representation of the
user’s internal state $z_{t}$, let alone its dynamics. We thus attempt to
approximate the NHMM estimation task we are interested in by implicitly
learning user preference dynamics from their past interaction data. As seen in
Fig. 2, we can train a neural network on $\\{\mathcal{D}_{j}\\}_{1:N}$ such
that, given a user’s past interaction data ($s^{\pi}_{0:t},x^{\pi}_{0:t}$,
where $t\leq T$), the model outputs a belief over the next-timestep
preferences $\mathbb{P}(u_{t+1}|s^{\pi}_{0:t},x^{\pi}_{0:t})$. Using the known
choice model assumption from Sec. 3, we map the belief over preferences,
together with the new slate $s_{t+1}$, to a distribution over content items:
$\mathbb{P}(x_{t+1}|s_{0:t+1},x_{0:t})=\sum_{u_{t+1}}\mathbb{P}(x_{t+1}|u_{t+1},s_{t+1})\mathbb{P}(u_{t+1}|s_{0:t},x_{0:t})$.

We now have a model which approximates the inference
$\mathbb{P}(u_{t+1}|s^{\pi}_{0:t},x^{\pi}_{0:t})$ for any $t\leq T$ – that is,
it provides us beliefs over the preferences $u_{t+1}$ resulting from the
user’s interactions so far with $\pi$. However, for the problem we stated at
the beginning of Sec. 4, we are interested in estimating the preferences that
would be induced if a different policy $\pi^{\prime}$ were to be deployed
going forward after $\pi$: formally, we are interested in
$\mathbb{P}(u^{\pi^{\prime}}_{H}|s^{\pi}_{0:T},x^{\pi}_{0:T})$ where $H>T$.

To obtain a belief over $u^{\pi^{\prime}}_{H}$ for a user $j$, we can sample
simulated user preference trajectories from the model to obtain a Monte Carlo
estimate of the desired distribution – as shown in Fig. 3. We use the data
$\mathcal{D}_{j}=\\{s^{\pi}_{0:T},x^{\pi}_{0:T}\\}$ as input to the model to
obtain a belief over the next-timestep preferences $u_{T+1}$. We then use such
belief and a slate $s^{\pi^{\prime}}_{T+1}$ sampled from policy $\pi^{\prime}$
to simulate the user’s choice $x^{\pi^{\prime}}_{T+1}$. Treating this extra
(simulated) step of interaction history for the user (the slate
$s^{\pi^{\prime}}_{T+1}$ and choice $x^{\pi^{\prime}}_{T+1}$) as part of the
observed history so far, we repeat this process to obtain preference estimates
under $\pi^{\prime}$ for future timesteps. By simulating multiple future
trajectories, we can obtain beliefs over the expected preferences a user would
have at any future timestep. See Algorithm 1 for the exact procedure. In
Appendix E.1 we show that this procedure does in fact approximate the desired
estimate.

One challenge with this method is that it requires the learned preference
predictor model to generalize to histories collected under $\pi^{\prime}$,
even though no training data was collected with $\pi^{\prime}$. We believe
that there is still reason to be hopeful given that real-world datasets of
user interaction are usually collected under many different deployed policies.
This suggests that as long as the histories induced by $\pi^{\prime}$ are not
too dissimilar from those in the historical data used for training, the
network should be able to generalize to predict preference evolutions induced
by $\pi^{\prime}$ too.

Figure 3: Simulating future user preferences and choices. By iteratively using
the future preferences estimation model (Fig. 2) by inputting the observables
(shaded, e.g. $x^{\pi}_{0:6},s^{\pi}_{0:6}$), one can simulate how an existing
user’s preferences would evolve in the future if they interacted with same
policy $\pi$ or a different policy $\pi^{\prime}$ (by selecting future slates
with $\pi^{\prime}$).

##  5 Quantifying unwanted shifts and optimizing to avoid them

Given the estimates of preference shifts under different policies that Sec. 4
enables us to compute, it would be useful to have quantitative metrics for
whether they are undesirable: this would enable to automatically evaluate
policies and even actively optimize to avoid such shifts. To obviate the
difficulty of explicitly defining unwanted shifts, we limit ourselves to
defining some shifts that we somewhat trust – “safe shifts” – and flag
preference shifts that largely differ from these safe shifts as potentially
unwanted. This is similar to defining a region of the space which we trust in
a risk-averse manner. The underlying philosophy is that “we don’t know what
good shifts are, but as long as you stay close to [family of safe shifts],
things shouldn’t go too bad”. To do this, we need both a notion of shifts we
trust (“safe shifts”), and a notion of distance between different evolutions
of preferences.

Notation. We denote the $\pi$-induced engagement under “safe-shift
preferences” $u_{t}^{\text{safe}}$ as
$\hat{r}_{t}(u_{t}^{\text{safe}},\pi)=(x_{t}^{\pi})^{T}u^{\text{safe}}_{t}$.
This means that $\pi$ was used to select the slate from which the user picked
the item $x_{t}^{\pi}$, but $x_{t}^{\pi}$’s engagement is considered relative
to different preferences $u_{t}^{\text{safe}}$ than those the user would have
developed under $\pi$.

Distance between shifts. Consider a known preference-trajectory
$u_{0:T}^{\pi}$ induced by $\pi$. We choose a metric between shifts such that
engagement for the items $x_{0:T}^{\pi}$ chosen under policy $\pi$ is also
high under the preferences one would have had under safe shifts
$u_{0:T}^{\text{safe}}$. We operationalize this notion of distance between
$\pi$-induced shifts and safe shifts $u_{0:T}^{\text{safe}}$ as
$D(u_{0:T}^{\pi},u_{0:T}^{\text{safe}})=\sum_{t}\mathbb{E}\big{[}\hat{r}_{t}(u_{t}^{\pi},\pi)-\hat{r}_{t}(u_{t}^{\text{safe}},\pi)\big{]}=\sum_{t}\mathbb{E}_{x_{t}^{\pi}}\big{[}(x_{t}^{\pi})^{T}u^{\pi}_{t}-(x_{t}^{\pi})^{T}u^{\text{safe}}_{t}\big{]}$.
However, this operationalization could easily be substituted with others. In
the case that safe shifts are random variables (a family of safe shifts), we
consider the expected distance
$\mathbb{E}_{U_{0:T}^{\text{safe}}}\big{[}D(u_{0:T}^{\pi},U_{0:T}^{\text{safe}})\big{]}$.

Safe shifts: initial preferences. As a first (very crude) proposal for safe
shifts, we can consider using the initial preferences $u_{0}$ as our safe
shifts $u_{0:T}^{\text{safe}}$: any deviation from the initial preferences
$u_{0}$ will be flagged as potentially problematic with
$D(u_{0:T}^{\pi},u_{0})$.

Safe shifts: natural preference shifts (NPS). One limitation of the above
metric is that not all preference shifts are unwanted: people routinely change
their preferences and behavior “naturally”. But what does “naturally” even
mean? We propose an idealized notion of natural shifts, by asking how
preferences would evolve if the user were “omniscient” and have full agency
over their preference evolution process, i.e. if would have access to all
content directly and had the ability to process it, unhindered by a small and
biased slate offered by the RS. Unfortunately this is impractical, as we can
never get data from such hypothetical users that can attend to all content
when choosing what to consume. As an approximation, we consider _random_
slates, which at least eliminates the agency of the RS policy (which in turn
can change the user’s belief about the distribution of available content). We
therefore operationalize “natural preferences” $u^{\pi_{\text{rnd}}}$ as the
preferences which the user would have interacting with a random recommender
$\pi^{\text{rnd}}$, and we use
$\mathbb{E}_{U_{0:T}^{\text{rnd}}}\big{[}D(u_{0:T}^{\pi},U_{0:T}^{\pi_{\text{rnd}}})\big{]}$
as the metric.

Using safe shift metrics. For how one can estimate such metrics, see Appendix
F. Once computed, they can be used both for evaluating preference shifts or
for recommender optimization (as alternate proxies for “value” relative to
simple engagement). However, clearly such metrics also fall short of fully
capturing “value”: we don’t want a system that actively tries to keep the user
preferences static (as would result from blindly optimizing
$D(u_{0:T}^{\pi},u_{0})$). Additionally, as we can’t perfectly operationalize
what Natural Preference Shifts would look like, it would also be incorrect to
simply optimize to induce shifts that a random recommender system would.

Penalized objective. By considering a weighted sum of these metrics (as we
will do in the penalized RL objective below), we try to lead the system to
perform well under a variety of relatively-reasonable definitions of value,
some of which are not preference-shift-agnostic. One can think of this as
hedging our bets as to what the value (or cost) of induced preference shifts
should be, and making explicit that it should not be zero, as current systems
assume.

Penalized RS training. By using the method from Sec. 4, one obtains a human
model which can be used to simulate human choices (in addition to
preferences). Similarly to previous work, we can use this human model as a
simulator for RL training of recommender systems (Chen et al., 2019; Zhao et
al., 2019; Bai et al., 2020). During training, we can compute the metrics
defined above penalize the current policy for causing any shifts that we have
not explicitly identified as “safe” – in what can be considered a “risk-
averse” design choice. We incorporate these metrics in the training of a new
policy $\pi$ by adding the two distance metrics from above to the basic
objective $\mathbb{E}\big{[}\hat{r}_{t}(u_{t}^{\pi})\big{]}$ of maximizing
long-term engagement, leading to the updated objective
$\mathbb{E}\big{[}\hat{r}_{t}(u_{t}^{\pi})+\nu^{\prime}_{1}\
\hat{r}_{t}(u_{0},\pi)+\nu^{\prime}_{2}\
\hat{r}_{t}(u_{t}^{\pi_{\text{rnd}}},\pi)\big{]}$ where
$\nu^{\prime}_{1},\nu^{\prime}_{2}$ are hyperparameters. This objective can be
optimized either myopically or via long-horizon (RL) optimization. See
Appendix G for more details.

##  6 Experimental Setup

Figure 4: Ground truth human dynamics. At each timestep, the user will receive
a slate $s_{t}$. Given the user’s preferences $u_{t}$, the slate $s_{t}$
induces a distribution over item choices $\mathbb{P}(x_{t}|s_{t},u_{t})$ from
which the user samples an item $x_{t}$ and receives an engagement value
$\hat{r}_{t}$ (unobserved by the RS). Additionally, $s_{t}$ induces a belief
over the future slates in the user ${b^{H}_{t}(s)}$. In turn ${b^{H}_{t}(s)}$
– together with $u_{t}$ – induce a distribution over next timestep user
preferences, from which $u_{t+1}$ is sampled.

Why simulation? To test our method, we need to evaluate both recommenders
which interact with users (rendering static datasets of user interaction
unsuitable), as well as test our evaluation metrics themselves – which are
defined based on internal preferences (for which we never get ground truth in
real interactions). We thus create a testing environment in which we can
emulate user behavior and have access to their ground truth preferences. Like
previous work (Chaney et al., 2018; Bountouridis et al., 2019; Jiang et al.,
2019; Mansoury et al., 2020; Yao et al., 2021), we simulate both a
recommendation environment and human behavior. However, unlike such
approaches, we use simulated human behavior for evaluation purposes only: our
human model is learned exclusively from data that would be observed in a real-
world RS (slates and choices), i.e. data of users (in our case the simulated
users) interacting with previous RS policies – meaning our approach could be
applied to real user data of this form. A fundamental advantage of testing our
method in a simulated environment is that we can actually evaluate how well
our model is able to recover the preferences of our “ground truth” users,
giving us insights about how our methods could perform with real users.

Ground truth human dynamics. See Fig. 4 for a summary of the ground truth
human dynamics we use for testing our method (and Fig. 12 for more info on our
environment setup). Following prior work (Ie et al., 2019b; Chen et al., 2019)
we assume that users choose items in proportion to (the exponentiated)
engagement under their current preferences, i.e. that
$\mathbb{P}(x_{t}=x|s_{t},u_{t})$ is given by the conditional logit model. In
our setup, this reduces to
$\mathbb{P}(x_{t}=x|s_{t},u_{t})\propto\mathbb{P}(s_{t}=x)e^{\beta_{c}x^{T}u_{t}}$,
with an additional term $\mathbb{P}(s_{t}=x)$ which takes into account how
prevalent each item is in the slate – and we assume such choice model to be
also known by our method. We adapt (Bernheim et al., 2021) to be our ground
truth human preference dynamics. On a high-level, at each timestep users
choose their next-timestep preferences to be more “convenient” ones – trading
off between choosing preferences that they expect will lead them to higher
engagement and maintaining engagement under current preferences. See Appendix
H.1 for a proof of the logit model reduction, and more information about the
ground truth human.

Simulated environment setup. For ease of interpretation of the results, in our
experiments we only consider a cohort of users whose initial preferences are
concentrated around preference $u=130\degree$. To showcase preference-
manipulation incentives to make users more predictable, we make the choice-
stochasticity temperature $\beta_{c}$ a function of part of preference space
one is in, with local optima $\beta_{c}(80\degree)=1$ and
$\beta_{c}(270\degree)=4$. This causes these areas in preference space to be
attractor points, as RSs are able to lead to higher engagement when users act
less stochastically (see Appendix H.1 for more details).

Dataset. For all our experiments, we use a dataset of 10k trajectories (each
of length 10), collected under a mixture of policies described in Appendix
H.3. 7.5k trajectories are used for training our models and 2.5k for computing
the validation losses and accuracies reported in Sec. 7.1.

Training human models and penalized RS policies. For training our human
models, we use a bidirectional transformer model similar to that of Sun et al.
(2019), and only assume access to a dataset of historical interactions
$s_{0:T},x_{0:T}$. We train myopic and RL RS policies $\pi^{\prime}$ using PPO
(Schulman et al., 2017; Liang et al., 2018) and restrict the action space to 6
possible slate distributions for ease of training. For penalized training we
give the three metrics $\hat{r}_{t}(u_{t}^{\pi})$,
$\hat{r}_{t}(u_{0},\pi^{\prime})$,
$\hat{r}_{t}(u_{t}^{\pi_{\text{rnd}}},\pi^{\prime})$ equal weight. For more
details on human model and RL training, see respectively Appendix H.2 and
Appendix G.

Figure 5: Validation losses and accuracies on held-out trajectories for the
preference prediction task, averaged across timesteps. Both under the correct
choice model and with some mis-specification, preference prediction performs
similarly to oracle NHMM estimation (which additionally has access to the
preference dynamics). Figure 6: Natural Preference Shifts (induced by
$\pi^{\text{rnd}}$) among a cohort of 1000 users (left) vs. a Monte Carlo
estimate using 1000 simulated user interaction trajectories obtained with our
model and Algorithm 1 (right).

##  7 Results

In Sec. 7.1 we validate that our method from Sec. 4 can estimate user’s
preferences and predict their evolution under alternate policies; then, in
Sec. 7.2, we turn to validating the metrics and penalized RL training approach
from Sec. 5.

###  7.1 Estimating user preferences

Oracle baseline. We use the NHMM estimates computed with full access to the
human preference dynamics as a baseline for our preference and behavior
estimates. Given that we can compute them through exact inference, such
estimates are the best one could possibly hope for.

Estimating preferences. We first verify that, given snippets of past
interactions from the validation trajectories, the preference prediction model
(Sec. 4) is able to predict the next timestep preferences (and observations,
as induced by the predicted preferences). For each interaction sequence in the
validation set we estimate the $t$th timestep preferences and observations
based on the previous interactions (for all $1\leq t\leq 10$). We also perform
this same inference on the same data with a random predictor, and with our
oracle baseline: exact NHMM inference. We report the average values for the
prediction losses and the accuracies in Fig. 5.

Robustness to mis-specified choice models. Our method requires the
specification of a user choice model, which will be challenging in practice.
In Fig. 5 we also show how the quality of our estimates withstands a mis-
specified choice model (described in Appendix H.2). We found that validation
loss for choice predictions (which would be observed in real-world
experiments, unlike the preference prediction loss and accuracy) was a good
indicator of the amount of model mis-specification: reducing choice model mis-
specification reduces choice prediction validation loss. This could be used to
guide the design of better choice models.

Figure 7: With penalized RL training, the preferences induced by the learned
policies are closer to the NPS and initial preferences, as desired. Myopic
systems will only greedily be pursuing the penalized objective, leading to
somewhat arbitrary behavior.

Imagining Natural Preference Shifts (NPS). In the experiment above, we were
validating our model on data from the same distribution (and RS policy) as
during training. To test the performance further, we try to now see whether
our model is able to estimate how alternate policies would affect the
evolution of user preferences on a population level, if they were to be
deployed (for which we had no data at training time). We use Algorithm 1 to
simulate preference trajectories of users interacting with $\pi^{\text{rnd}}$
(setting $T=0$, i.e. without considering any past interaction context). We
then average the predicted preference distributions across the simulated
users, giving us estimated preference evolutions similar to the actual ones
induced by $\pi^{\text{rnd}}$ (i.e. the Natural Preference Shifts) – shown in
Fig. 6. This shows that in our experimental setup, our method is able to
capture the preference dynamics of a user in ways that generalize to alternate
policies; that is, the model is able to estimate preference evolutions under
different RS policies from the ones from which the training interaction data
was obtained.

###  7.2 Evaluating undesirable shifts and penalizing them

Qualitative effects of different policies. Firstly, we show the qualitative
effect of using different classes of RS policies in our simulated setup in
Fig. 1. We see how interacting with the unpenalized RL policy drives users
strongly away from their initial preferences and concentrates them in a
specific region of preference space (in bright yellow) – in a behavior that
seems potentially concerning (what if the preference type were a political
axis?). The myopic policy (center), shows the same type of effect but much
less pronounced.

Safe shift metrics and sum of rewards. As delineated in Sec. 5, we use the
initial preferences $u_{0}$ and Natural Preference Shifts
$u_{0:T}^{\pi^{\text{rnd}}}$ of a user as our “safe shift” proxies, which give
us alternate evaluations of trajectories $\hat{r}_{t}(u_{0},\pi^{\prime})$ and
$\hat{r}_{t}(u_{t}^{\pi_{\text{rnd}}},\pi^{\prime})$ relative to simple
current-preference engagement $\hat{r}_{t}(u_{t}^{\pi^{\prime}})$. We consider
the sum of these metrics as a better proxy for “true value” than any one of
these metrics alone. $\pi^{\prime}$ here denotes the trained policy.

Hypotheses about metrics. We now turn to our hypotheses regarding metrics. We
hypothesize that our metrics are able to: (H1) identify whether policies will
induce potentially unwanted preference shifts, and (H2) incentivize better
behavior when added as a penalty during training.

Learned human model confound. To validate H1 with real users, one would have
to rely on an approximate computation of the metrics (estimated evaluation)
with a learned user models of preference dynamics (as computing the metrics
requires estimating preferences). Additionally, to validate H2, one would
likely train with simulated interactions from the learned user model (training
in simulation) – as explained in Sec. 5. The quality of the learned user model
would thus affect the evaluation of the quality of the choice of metrics
themselves – which would constitute a confound.

H1 under oracle dynamics access. In order to deconfound our experiments from
the errors in our estimated human dynamics, we first test our hypotheses
assuming oracle access to users and their dynamics for the purposes of
training and evaluation. We first hypothesize that (H1.1) by computing the
metrics exactly using the preference estimates obtained through NHMM inference
(oracle evaluation), our metrics are able to flag potentially unwanted
preference shifts. We find this to be the case by comparing the oracle
evaluation metric values (left of Table 1, “unpenalized”) and the actual
preference shifts induced by the various RSs we consider (Fig. 1): while
unpenalized RL performs better ($7.49$ vs $5.71$) than myopic for engagement
$\hat{r}_{t}(u_{t}^{\pi^{\prime}})$, it performs worse with respect to our
safe shift metrics. This matches Fig. 1, where RL has more undesired effects.

H2 under oracle dynamics access. We additionally hypothesize that (H2.1) such
metrics (still computed exactly, in oracle evaluation) can be used for
training penalized LTV systems which avoid the most extreme unwanted
preference shifts. For this training, allow ourselves on-policy human
interaction data with the ground truth users, as if the RL happened directly
in the real world – we call this oracle training. For the penalized RL RS, the
cumulative metric value (“Sum” in Table 1) increases substantially, although
it is at the slight expense of instantaneous engagement (Table 1).
Qualitatively, we see that the induced preference shifts caused by the RL
system seem closer to “safe shifts” (Fig. 7), supporting H1.1 in that high
metric values qualitatively match shifts that seem more desirable.

Overall, we see that with oracle access, the metrics capture what we see as
qualitatively undesired shifts, and that using them for penalized RL produces
policies with slightly lower engagement, but drastically better at avoiding
such shifts.

H1 and H2 with learned user dynamics. In practice we would not have access to
the underlying user dynamics, or potentially even the ability to interact on-
policy with users as we train RL policies, due to the high cost and risk of
negative side effects for collecting data with unsafe policies. Therefore, we
wish to show that even with estimated metrics and simulated interactions
(based on the learned models), (H1.2) estimated evaluation and (H2.2) training
in simulation (described above) are still able to respectively flag unwanted
shifts and penalize manipulative RL behaviors. Table 2 shows that – although
the estimated metrics can differ from the ground truth ones somewhat
substantially (see Oracle Eval. at the top vs Estimated Eval. at the bottom) –
importantly the relative ordering of the policies ranked by our estimated
values stay the same: the penalized RL policy trained in simulation actually
has (under oracle evaluation) higher cumulative reward than the unpendalized
one, and the estimated evaluation keeps the same ranking (even though is more
optimistic about the unpenalized reward).This provides some initial evidence
that, even without observing preferences in practice, one could successfully
optimize – and assess – recommenders that penalize preference-shifts.

Table 1: Results under oracle training and evaluation. Both here an in Table 2 we report the cumulative reward under various metrics, averaged across 1000 trajectories (standard errors are all $<0.1$). By looking at the safe shift metrics $\hat{r}(u_{0})$ and $\hat{r}(u_{t}^{\text{rnd}})$, we see that penalized systems stay significantly closer to safe shifts than unpenalized ones. |  | Oracle Training
---|---|---
|  | Unpenalized | Penalized
|  | Myopic | RL | Myopic | RL
Oracle Eval | $\hat{r}_{t}(u_{t}^{\pi^{\prime}})$ | 5.71 | 7.49 | 6.20 | 5.28
$\hat{r}_{t}(u_{0},\pi^{\prime})$ | 1.99 | -0.08 | 3.61 | 6.21
$\hat{r}_{t}(u_{t}^{\pi_{\text{rnd}}},\pi^{\prime})$ | 2.01 | -1.09 | 3.10 | 4.57
Sum | 9.69 | 6.33 | 12.90 | 16.05

##  8 Discussion

Summary. In conclusion, our contributions are: 1) proposing a method to
estimate the preference shifts which would be induced by recommender system
policies before deployment; 2) a framework for defining safe shifts, which can
be used to evaluate whether preference shifts might be problematic; 3) showing
how one can use such metrics to optimize recommenders which penalize unwanted
shifts. As dynamics of preference are learned (rather than handcrafted), our
method has hope to be applied to real user data. While there is no ground
truth for human preferences, verifying the model’s ability to anticipate
behavior can give confidence in using it to evaluate and penalize undesired
preference shifts. We acknowledge that this is only a first step, tested in an
idealized setting with relatively strong assumptions. However, we hope this
can be a starting point for further research which focuses on relaxing such
assumptions and making these ideas applicable to the complexity of real RSs.

Limitations and future work. To validate our estimation method and metrics, we
required ground truth access to user preferences – leading us to conduct our
experiments in simulation. While the ground truth model of dynamics we use is
inspired by the econometrics literature, we cannot guarantee that our results
would translate when the method is applied to real users with real preference
dynamics. We expect real users to have more complex dynamics, with preference
changes mediated by beliefs other than just about the distribution of content,
such as beliefs about the world; while our method makes no assumptions about
the structure of this internal space, it might require a lot more (and more
diverse) data to capture these effects. Further, we also chose the
experimental setup such that an RL system would act on its incentives to
manipulate preferences – not all real settings will necessarily have that
property. Moreover, crucially our method requires designing a user choice
model: one could obviate this by predicting behavior and defining metrics on
behavior, but this would mean losing the latent preference structure.
Additionally, even what we call “preferences” – a latent for instantaneous
behavior – is limiting as it doesn’t lend itself to capturing long-term
preferences.

Table 2: Effect of estimating evaluations and simulating training for LTV. We see that the estimated evaluations of trained systems strongly correlate with the oracle evaluations (and importantly, maintain their relative orderings). A similar effect occurs when training in simulation rather than by training with on-policy data collected from real users. |  | Oracle Training | Training in Sim.
---|---|---|---
|  | Unpen. | Penal. | Unpen. | Penal.
Orac. Eval | $\hat{r}_{t}(u_{t}^{\pi^{\prime}})$ | 7.49 | 5.28 | 6.40 | 5.48
$\hat{r}_{t}(u_{0},\pi^{\prime})$ | -0.08 | 6.21 | -1.24 | 5.61
$\hat{r}_{t}(u_{t}^{\pi_{\text{rnd}}},\pi^{\prime})$ | -1.09 | 4.57 | -1.83 | 4.43
Sum | 6.33 | 16.05 | 3.36 | 15.52
Est. Eval | $\hat{r}_{t}(u_{t}^{\pi^{\prime}})$ | 5.58 | 5.42 | 6.49 | 5.78
$\hat{r}_{t}(u_{0},\pi^{\prime})$ | 1.28 | 5.57 | -0.80 | 4.94
$\hat{r}_{t}(u_{t}^{\pi_{\text{rnd}}},\pi^{\prime})$ | 2.05 | 3.88 | 1.48 | 4.41
Sum | 8.91 | 14.87 | 7.17 | 15.15

#### Acknowledgments

This work was funded by ONR YIP. We thank David Krueger, Jonathan Stray,
Stephanie Milani, Lawrence Chan, Scott Emmons, Smitha Milli, Aditi
Raghunathan, Orr Paradise, Kenneth Lai, Gloria Liu, Emily Chao, and Mihaela
Curmei for helpful discussions at various stages of the project.

## References

  * Afsar et al. (2021) Afsar, M. M., Crump, T., and Far, B.  Reinforcement learning based recommender systems: A survey.  _arXiv:2101.06286 [cs]_ , January 2021.  URL http://arxiv.org/abs/2101.06286.  arXiv: 2101.06286.
  * Albanie et al. (2017) Albanie, S., Shakespeare, H., and Gunter, T.  Unknowable Manipulators: Social Network Curator Algorithms.  _arXiv:1701.04895 [cs, stat]_ , January 2017.  URL http://arxiv.org/abs/1701.04895.  arXiv: 1701.04895.
  * Allcott et al. (2020) Allcott, H., Braghieri, L., Eichmeyer, S., and Gentzkow, M.  The Welfare Eﬀects of Social Media.  pp. 121, 2020.
  * Andreassen (2015) Andreassen, C. S.  Online Social Network Site Addiction: A Comprehensive Review.  _Current Addiction Reports_ , 2(2):175–184, June 2015.  ISSN 2196-2952.  doi: 10.1007/s40429-015-0056-9.  URL https://doi.org/10.1007/s40429-015-0056-9.
  * Bai et al. (2020) Bai, X., Guan, J., and Wang, H.  Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation.  _arXiv:1911.03845 [cs, stat]_ , January 2020.  URL http://arxiv.org/abs/1911.03845.  arXiv: 1911.03845.
  * Bernheim et al. (2021) Bernheim, B. D., Braghieri, L., Martínez-Marquina, A., and Zuckerman, D.  A theory of chosen preferences.  _American Economic Review_ , 111(2):720–54, 2021\.
  * Bilmes (1998) Bilmes, J. A.  A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models.  pp. 15, 1998.
  * Bountouridis et al. (2019) Bountouridis, D., Harambam, J., Makhortykh, M., Marrero, M., Tintarev, N., and Hauff, C.  SIREN: A Simulation Framework for Understanding the Effects of Recommender Systems in Online News Environments.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , pp. 150–159, Atlanta GA USA, January 2019. ACM.  ISBN 978-1-4503-6125-5.  doi: 10.1145/3287560.3287583.  URL https://dl.acm.org/doi/10.1145/3287560.3287583.
  * Chaney et al. (2018) Chaney, A. J. B., Stewart, B. M., and Engelhardt, B. E.  How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility.  _Proceedings of the 12th ACM Conference on Recommender Systems_ , pp. 224–232, September 2018.  doi: 10.1145/3240323.3240370.  URL http://arxiv.org/abs/1710.11214.  arXiv: 1710.11214.
  * Chen et al. (2020) Chen, M., Beutel, A., Covington, P., Jain, S., Belletti, F., and Chi, E.  Top-K Off-Policy Correction for a REINFORCE Recommender System.  _arXiv:1812.02353 [cs, stat]_ , November 2020.  URL http://arxiv.org/abs/1812.02353.  arXiv: 1812.02353.
  * Chen et al. (2019) Chen, X., Li, S., Li, H., Jiang, S., Qi, Y., and Song, L.  Generative Adversarial User Model for Reinforcement Learning Based Recommendation System.  _arXiv:1812.10613 [cs, stat]_ , December 2019.  URL http://arxiv.org/abs/1812.10613.  arXiv: 1812.10613.
  * Cheng et al. (2016) Cheng, H.-T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G., Chai, W., Ispir, M., Anil, R., Haque, Z., Hong, L., Jain, V., Liu, X., and Shah, H.  Wide & Deep Learning for Recommender Systems.  _arXiv:1606.07792 [cs, stat]_ , June 2016.  URL http://arxiv.org/abs/1606.07792.  arXiv: 1606.07792.
  * Cosley et al. (2003) Cosley, D., Lam, S. K., Albert, I., Konstan, J. A., and Riedl, J.  Is seeing believing? how recommender system interfaces affect users’ opinions.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , CHI ’03, pp. 585–592, New York, NY, USA, April 2003. Association for Computing Machinery.  ISBN 978-1-58113-630-2.  doi: 10.1145/642611.642713.  URL https://doi.org/10.1145/642611.642713.
  * Covington et al. (2016) Covington, P., Adams, J., and Sargin, E.  Deep Neural Networks for YouTube Recommendations.  In _Proceedings of the 10th ACM Conference on Recommender Systems - RecSys ’16_ , pp. 191–198, Boston, Massachusetts, USA, 2016. ACM Press.  ISBN 978-1-4503-4035-9.  doi: 10.1145/2959100.2959190.  URL http://dl.acm.org/citation.cfm?doid=2959100.2959190.
  * Epstein & Robertson (2015) Epstein, R. and Robertson, R. E.  The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections.  _Proceedings of the National Academy of Sciences_ , 112(33):E4512–E4521, August 2015.  doi: 10.1073/pnas.1419828112.  URL https://www.pnas.org/doi/10.1073/pnas.1419828112.  Publisher: Proceedings of the National Academy of Sciences.
  * Evans & Kasirzadeh (2021) Evans, C. and Kasirzadeh, A.  User Tampering in Reinforcement Learning Recommender Systems.  _arXiv:2109.04083 [cs]_ , September 2021.  URL http://arxiv.org/abs/2109.04083.  arXiv: 2109.04083.
  * Farquhar et al. (2022) Farquhar, S., Carey, R., and Everitt, T.  Path-Specific Objectives for Safer Agent Incentives.  _arXiv:2204.10018 [cs, stat]_ , April 2022.  URL http://arxiv.org/abs/2204.10018.  arXiv: 2204.10018.
  * Franklin et al. (2022) Franklin, M., Ashton, H., Gorman, R., and Armstrong, S.  Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AI.  _arXiv:2203.10525 [cs]_ , March 2022.  URL http://arxiv.org/abs/2203.10525.  arXiv: 2203.10525.
  * Gauci et al. (2019) Gauci, J., Conti, E., Liang, Y., Virochsiri, K., He, Y., Kaden, Z., Narayanan, V., Ye, X., Chen, Z., and Fujimoto, S.  Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform.  _arXiv:1811.00260 [cs, stat]_ , September 2019.  URL http://arxiv.org/abs/1811.00260.  arXiv: 1811.00260.
  * Gretzel & Fesenmaier (2006) Gretzel, U. and Fesenmaier, D.  Persuasion in Recommender Systems.  _International Journal of Electronic Commerce_ , 11(2):81–100, December 2006.  ISSN 1086-4415.  doi: 10.2753/JEC1086-4415110204.  URL https://doi.org/10.2753/JEC1086-4415110204.
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F., Wiebe, M., Peterson, P., Gérard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E.  Array programming with NumPy.  _Nature_ , 585(7825):357–362, September 2020\.  doi: 10.1038/s41586-020-2649-2.  URL https://doi.org/10.1038/s41586-020-2649-2.
  * Hasan et al. (2018) Hasan, M. R., Jha, A. K., and Liu, Y.  Excessive use of online video streaming services: Impact of recommender system use, psychological factors, and motives.  _Computers in Human Behavior_ , 80:220–228, March 2018\.  ISSN 07475632.  doi: 10.1016/j.chb.2017.11.020.  URL https://linkinghub.elsevier.com/retrieve/pii/S0747563217306581.
  * Hohnhold et al. (2015) Hohnhold, H., O’Brien, D., and Tang, D.  Focusing on the Long-term: It’s Good for Users and Business.  In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pp. 1849–1858, Sydney NSW Australia, August 2015. ACM.  ISBN 978-1-4503-3664-2.  doi: 10.1145/2783258.2788583.  URL https://dl.acm.org/doi/10.1145/2783258.2788583.
  * Holtz et al. (2020) Holtz, D., Carterette, B., Chandar, P., Nazari, Z., Cramer, H., and Aral, S.  The Engagement-Diversity Connection: Evidence from a Field Experiment on Spotify.  _arXiv:2003.08203 [cs]_ , March 2020.  URL http://arxiv.org/abs/2003.08203.  arXiv: 2003.08203.
  * Hughes et al. (1999) Hughes, J. P., Guttorp, P., and Charles, S. P.  A Non-Homogeneous Hidden Markov Model for Precipitation Occurrence.  _Journal of the Royal Statistical Society. Series C (Applied Statistics)_ , 48(1):15–30, 1999.  URL http://www.jstor.org/stable/2680815.
  * Häubl & Murray (2003) Häubl, G. and Murray, K. B.  Preference Construction and Persistence in Digital Marketplaces: The Role of Electronic Recommendation Agents.  _Journal of Consumer Psychology_ , 13(1):75–91, January 2003.  ISSN 1057-7408.  doi: 10.1207/S15327663JCP13-1&2˙07.  URL http://www.sciencedirect.com/science/article/pii/S1057740803701788.
  * Ie et al. (2019a) Ie, E., Hsu, C.-w., Mladenov, M., Jain, V., Narvekar, S., Wang, J., Wu, R., and Boutilier, C.  RecSim: A Configurable Simulation Platform for Recommender Systems.  _arXiv:1909.04847 [cs, stat]_ , September 2019a.  URL http://arxiv.org/abs/1909.04847.  arXiv: 1909.04847.
  * Ie et al. (2019b) Ie, E., Jain, V., Wang, J., Narvekar, S., Agarwal, R., Wu, R., Cheng, H.-T., Lustman, M., Gatto, V., Covington, P., McFadden, J., Chandra, T., and Boutilier, C.  Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology.  _arXiv:1905.12767 [cs, stat]_ , May 2019b.  URL http://arxiv.org/abs/1905.12767.  arXiv: 1905.12767.
  * Jiang et al. (2019) Jiang, R., Chiappa, S., Lattimore, T., György, A., and Kohli, P.  Degenerate Feedback Loops in Recommender Systems.  _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pp. 383–390, January 2019.  doi: 10.1145/3306618.3314288.  URL http://arxiv.org/abs/1902.10730.  arXiv: 1902.10730.
  * Juneja & Mitra (2021) Juneja, P. and Mitra, T.  Auditing E-Commerce Platforms for Algorithmically Curated Vaccine Misinformation.  pp. 27, 2021.
  * Kaelbling et al. (1998) Kaelbling, L. P., Littman, M. L., and Cassandra, A. R.  Planning and acting in partially observable stochastic domains.  _Artificial Intelligence_ , 101(1-2):99–134, May 1998.  ISSN 00043702.  doi: 10.1016/S0004-3702(98)00023-X.  URL https://linkinghub.elsevier.com/retrieve/pii/S000437029800023X.
  * Krakovna et al. (2019) Krakovna, V., Orseau, L., Kumar, R., Martic, M., and Legg, S.  Penalizing side effects using stepwise relative reachability.  _arXiv:1806.01186 [cs, stat]_ , March 2019.  URL http://arxiv.org/abs/1806.01186.  arXiv: 1806.01186.
  * Kramer et al. (2014) Kramer, A. D. I., Guillory, J. E., and Hancock, J. T.  Experimental evidence of massive-scale emotional contagion through social networks.  _Proceedings of the National Academy of Sciences_ , 111(24):8788–8790, June 2014.  ISSN 0027-8424, 1091-6490.  doi: 10.1073/pnas.1320040111.  URL https://www.pnas.org/content/111/24/8788.  Publisher: National Academy of Sciences Section: Social Sciences.
  * Krueger et al. (2020) Krueger, D., Maharaj, T., and Leike, J.  Hidden Incentives for Auto-Induced Distributional Shift.  _arXiv:2009.09153 [cs, stat]_ , September 2020.  URL http://arxiv.org/abs/2009.09153.  arXiv: 2009.09153.
  * Kumar et al. (2020) Kumar, R., Uesato, J., Ngo, R., Everitt, T., Krakovna, V., and Legg, S.  REALab: An Embedded Perspective on Tampering.  _arXiv:2011.08820 [cs]_ , November 2020.  URL http://arxiv.org/abs/2011.08820.  arXiv: 2011.08820.
  * Li et al. (2014) Li, L., Zheng, L., Yang, F., and Li, T.  Modeling and broadening temporal user interest in personalized news recommendation.  _Expert Systems with Applications_ , 41(7):3168–3177, June 2014.  ISSN 09574174.  doi: 10.1016/j.eswa.2013.11.020.  URL https://linkinghub.elsevier.com/retrieve/pii/S0957417413009329.
  * Liang et al. (2018) Liang, E., Liaw, R., Moritz, P., Nishihara, R., Fox, R., Goldberg, K., Gonzalez, J. E., Jordan, M. I., and Stoica, I.  RLlib: Abstractions for Distributed Reinforcement Learning.  _arXiv:1712.09381 [cs]_ , June 2018.  URL http://arxiv.org/abs/1712.09381.  arXiv: 1712.09381.
  * Lu & Yang (2016) Lu, Z. and Yang, Q.  Partially observable markov decision process for recommender systems.  _arXiv preprint arXiv:1608.07793_ , 2016.
  * MacLeod & Campbell (1992) MacLeod, C. and Campbell, L.  Memory accessibility and probability judgments: An experimental evaluation of the availability heuristic.  _Journal of Personality and Social Psychology_ , 63(6):890–902, 1992.  ISSN 1939-1315(Electronic),0022-3514(Print).  doi: 10.1037/0022-3514.63.6.890.  Place: US Publisher: American Psychological Association.
  * Mansoury et al. (2020) Mansoury, M., Abdollahpouri, H., Pechenizkiy, M., Mobasher, B., and Burke, R.  Feedback Loop and Bias Amplification in Recommender Systems.  _arXiv:2007.13019 [cs]_ , July 2020.  URL http://arxiv.org/abs/2007.13019.  arXiv: 2007.13019.
  * Matz et al. (2017) Matz, S. C., Kosinski, M., Nave, G., and Stillwell, D. J.  Psychological targeting as an effective approach to digital mass persuasion.  _Proceedings of the National Academy of Sciences_ , 114(48):12714–12719, November 2017.  ISSN 0027-8424, 1091-6490.  doi: 10.1073/pnas.1710966114.  URL http://www.pnas.org/lookup/doi/10.1073/pnas.1710966114.
  * Mladenov et al. (2019) Mladenov, M., Meshi, O., Ooi, J., Schuurmans, D., and Boutilier, C.  Advantage Amplification in Slowly Evolving Latent-State Environments.  _arXiv:1905.13559 [cs, stat]_ , May 2019.  URL http://arxiv.org/abs/1905.13559.  arXiv: 1905.13559.
  * Mudigere et al. (2021) Mudigere, D., Hao, Y., Huang, J., Tulloch, A., Sridharan, S., Liu, X., Ozdal, M., Nie, J., Park, J., Luo, L., Yang, J. A., Gao, L., Ivchenko, D., Basant, A., Hu, Y., Yang, J., Ardestani, E. K., Wang, X., Komuravelli, R., Chu, C.-H., Yilmaz, S., Li, H., Qian, J., Feng, Z., Ma, Y., Yang, J., Wen, E., Li, H., Yang, L., Sun, C., Zhao, W., Melts, D., Dhulipala, K., Kishore, K. R., Graf, T., Eisenman, A., Matam, K. K., Gangidi, A., Chen, G. J., Krishnan, M., Nayak, A., Nair, K., Muthiah, B., khorashadi, M., Bhattacharya, P., Lapukhov, P., Naumov, M., Qiao, L., Smelyanskiy, M., Jia, B., and Rao, V.  High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models.  _arXiv:2104.05158 [cs]_ , April 2021.  URL http://arxiv.org/abs/2104.05158.  arXiv: 2104.05158.
  * Nguyen et al. (2014) Nguyen, T. T., Hui, P.-M., Harper, F. M., Terveen, L., and Konstan, J. A.  Exploring the filter bubble: the effect of using recommender systems on content diversity.  In _Proceedings of the 23rd international conference on World wide web - WWW ’14_ , pp. 677–686, Seoul, Korea, 2014. ACM Press.  ISBN 978-1-4503-2744-2.  doi: 10.1145/2566486.2568012.  URL http://dl.acm.org/citation.cfm?doid=2566486.2568012.
  * Okura et al. (2017) Okura, S., Tagami, Y., Ono, S., and Tajima, A.  Embedding-based News Recommendation for Millions of Users.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pp. 1933–1942, Halifax NS Canada, August 2017. ACM.  ISBN 978-1-4503-4887-4.  doi: 10.1145/3097983.3098108.  URL https://dl.acm.org/doi/10.1145/3097983.3098108.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  Pytorch: An imperative style, high-performance deep learning library.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 32_ , pp. 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Rabiner (1986) Rabiner, L. R.  An Introduction to Hidden Markov Models.  pp. 13, 1986.
  * Ribeiro et al. (2019) Ribeiro, M. H., Ottoni, R., West, R., Almeida, V. A. F., and Meira, W.  Auditing Radicalization Pathways on YouTube.  _arXiv:1908.08313 [cs]_ , December 2019.  URL http://arxiv.org/abs/1908.08313.  arXiv: 1908.08313.
  * Russell & Norvig (2002) Russell, S. and Norvig, P.  Artificial intelligence: a modern approach.  2002\.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shi et al. (2018) Shi, J.-C., Yu, Y., Da, Q., Chen, S.-Y., and Zeng, A.-X.  Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning.  _arXiv:1805.10000 [cs]_ , May 2018.  URL http://arxiv.org/abs/1805.10000.  arXiv: 1805.10000.
  * Stray (2021) Stray, J.  Designing Recommender Systems to Depolarize.  _arXiv:2107.04953 [cs]_ , July 2021.  URL http://arxiv.org/abs/2107.04953.  arXiv: 2107.04953.
  * Sun et al. (2019) Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., and Jiang, P.  BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer.  _arXiv:1904.06690 [cs]_ , August 2019.  URL http://arxiv.org/abs/1904.06690.  arXiv: 1904.06690.
  * Sunehag et al. (2015) Sunehag, P., Evans, R., Dulac-Arnold, G., Zwols, Y., Visentin, D., and Coppin, B.  Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions.  _arXiv:1512.01124 [cs]_ , December 2015.  URL http://arxiv.org/abs/1512.01124.  arXiv: 1512.01124.
  * Sutton & Barto (1998) Sutton, R. S. and Barto, A. G.  _Reinforcement learning: an introduction_.  Adaptive computation and machine learning. MIT Press, Cambridge, Mass, 1998.  ISBN 978-0-262-19398-6.
  * Wilhelm et al. (2018) Wilhelm, M., Ramanathan, A., Bonomo, A., Jain, S., Chi, E. H., and Gillenwater, J.  Practical Diversified Recommendations on YouTube with Determinantal Point Processes.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , pp. 2165–2173, Torino Italy, October 2018. ACM.  ISBN 978-1-4503-6014-2.  doi: 10.1145/3269206.3272018.  URL https://dl.acm.org/doi/10.1145/3269206.3272018.
  * Wu et al. (2017) Wu, C.-Y., Ahmed, A., Beutel, A., Smola, A. J., and Jing, H.  Recurrent recommender networks.  In _Proceedings of the tenth ACM international conference on web search and data mining_ , pp. 495–503, 2017.
  * Yao et al. (2021) Yao, S., Halpern, Y., Thain, N., Wang, X., Lee, K., Prost, F., Chi, E. H., Chen, J., and Beutel, A.  Measuring Recommender System Effects with Simulated Users.  _arXiv:2101.04526 [cs]_ , January 2021.  URL http://arxiv.org/abs/2101.04526.  arXiv: 2101.04526.
  * Zhang et al. (2019) Zhang, S., Yao, L., Sun, A., and Tay, Y.  Deep Learning based Recommender System: A Survey and New Perspectives.  _ACM Computing Surveys_ , 52(1):1–38, February 2019.  ISSN 0360-0300, 1557-7341.  doi: 10.1145/3285029.  URL http://arxiv.org/abs/1707.07435.  arXiv: 1707.07435.
  * Zhang et al. (2014) Zhang, Y., Dai, H., Xu, C., Feng, J., Wang, T., Bian, J., Wang, B., and Liu, T.-Y.  Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks.  _arXiv:1404.5772 [cs]_ , July 2014.  URL http://arxiv.org/abs/1404.5772.  arXiv: 1404.5772.
  * Zhao et al. (2019) Zhao, X., Xia, L., Zou, L., Yin, D., and Tang, J.  Toward Simulating Environments in Reinforcement Learning Based Recommendations.  _arXiv:1906.11462 [cs]_ , September 2019.  URL http://arxiv.org/abs/1906.11462.  arXiv: 1906.11462.

##  Appendix A Notation

  * •

$x_{t}\in\mathbb{R}^{d}$: time-indexed user choice of content from a slate.
$x_{t}^{\pi}$ indicates a choice that was made from a slate that was produced
by policy $\pi$.

  * •

$s_{t}\in\mathbb{R}^{d}$: time-indexed slate chosen by the recommender system.
See Fig. 12 for how slates are represented. $s_{t}^{\pi}$ indicates a slate
that was sampled from policy $\pi$.

  * •

$\pi:(s_{0:k},x_{0:k})\rightarrow s_{k+1}$: a recommender system policy.
$\pi^{\text{rnd}}$ denotes a policy which selects slates randomly – so that
the distribution of content matches the distribution of the slate.

  * •

$u_{t}\in\mathbb{R}^{d}$: time-indexed user preferences.
$u_{t}^{\pi^{\prime}}$ is the preferences a user has at timestep $t$ after
interacting with policy $\pi^{\prime}$. From the context, it should be clear
whether such preferences are estimated future or counterfactual preferences,
described in Appendix C.

  * •

$z_{t}$: time-indexed user’s internal state. Policy superscripts are used in
the same way as for preferences.

  * •

$\hat{r}_{t}(u_{t})=u_{t}^{T}x_{t}$ is the reward function which captures user
engagement. The expression
$\hat{r}_{t}(u_{t}^{\text{safe}},\pi)=(x_{t}^{\pi})^{T}u^{\text{safe}}_{t}$ is
used when the preferences used to evaluate the content are different from the
policy: This means that $\pi$ was used to select the slate from which the user
picked the item $x_{t}^{\pi}$, but $x_{t}^{\pi}$’s engagement is considered
relative to potentially different preferences than those the user would have
developed under $\pi$.

  * •

$b_{0:t}^{\pi}(u_{t+1})$: the belief over a user’s preferences at timestep
$t+1$, as induced by slates sampled under $\pi$ $s^{\pi}_{0:t},x^{\pi}_{0:t}$.
Formally, this is equivalent to
$\mathbb{P}(u_{t+1}|s^{\pi}_{0:t},x^{\pi}_{0:t})$.

  * •

$b_{0:t}^{\pi}(x_{t+1})$: the belief over a user’s choice at timestep $t+1$,
as induced by slates sampled under $\pi$. Formally equivalent to:
$\mathbb{P}(x_{t+1}|s_{0:t+1},x_{0:t})$.

  * •

$\hat{P}_{f}:(s_{0:k},x_{0:k})\rightarrow(b_{0:k}(u_{k+1}),b_{0:k}(x_{k+1}))$:
the future preference estimator model described in Sec. 4.

  * •

$\hat{P}_{i}:(s_{0:k-1},x_{0:k-1})\rightarrow b_{0:k}(u_{0})$ initial
preference estimator described in Appendix C.

  * •

$\hat{P}_{c}:(s_{0:k-1},x_{0:k-1},b_{0:k}(u_{0}))\rightarrow(b_{0:k}(u_{k+1}),b_{0:k}(x_{k+1}))$
counterfactual preference estimator described in Appendix C.

##  Appendix B Known choice model assumption

Here we try to justify why assuming a known choice model is almost a necessity
given the problem that we set out to solve: without this (or some other
assumption), estimating user preferences and learning their dynamics seems too
difficult. Looking at our problem as a NHMM gives us insight as to such
difficulty. See Appendix D for the casting of our problem to the NHMM
formalism.

Our goal is to infer preferences (part of the hidden state), given only
observations, but no user choice model (which is part of the observation model
of the underlying NHMM). Algorithms such as Baum-Welch (Bilmes, 1998) have
been proposed for jointly learning both the transition model of the hidden
state, and the observation model of a HMM. This type of algorithm could be
extended to the NHMM setting, providing a promising direction. However, the
Baum-Welch algorithm assumes the transition dynamics of the hidden state to be
linear, and the dimension of the hidden state to be known. We expect real
user’s dynamics to be non-linear (and so are the ones that we consider in our
experiments). Moreover, we don’t want to constrain our method to require
knowledge of the dimension of the hidden state (the user’s internal state) –
which will be unknown in practice.

In light of this, assumption the nature of the choice model to be known seems
instrumental to render preference inference possible in our setting. As shown
in Sec. 7.1, it might sometimes be possible to detect whether the choice model
is mis-specified if it leads to higher validation loss on choice prediction
(which is observed) than alternate choice model hypotheses. This could guide a
process of iteratively refining one’s choice model. An important thing to note
is that – although limiting – such assumption is already an improvement
relative to all previous work related to preference shifts that we are aware
of (Krueger et al., 2020; Jiang et al., 2019; Mansoury et al., 2020; Chaney et
al., 2018; Evans & Kasirzadeh, 2021), which assumes the whole user preference
dynamics to be known.

##  Appendix C Counterfactual Preferences Estimation

In Sec. 4 we develop a methodology to train a human model that can be used for
both: 1) predicting how existing users’ preferences would evolve if new
policies were to be deployed to them; 2) simulating entire user preference
trajectories under desired RS policies; this is useful, but suffers from a
problem – users have already been biased by the previous policies, and their
preferences might have already e.g. shifted to extremes; forward prediction
will thus miss certain undesired effects. Another useful task would be to
“turn back time” – and ask for a specific user, “how would have their
preferences shifted if we exposed them to $\pi^{\prime}$ _from the very
beginning_ of their interactions with the platform?”. To differentiate these
problems in this section, we call the methodology developed in Sec. 4 future
preference estimation; instead we call the problem described just above the
counterfactual preference estimation problems.

While the methodology described in Sec. 4 is sufficient to estimate the
expected preferences and behaviors a user will have in the future, it cannot
be readily used to estimate counterfactual preferences and behaviors: given a
set of past interactions $\mathcal{D}_{j}$ collected under policy $\pi$, we
cannot predict what the preferences would have been for this user if an
alternate policy $\pi^{\prime}$ had been used from the first timestep of
interaction instead of $\pi$ – we denote these counterfactual preferences as
$u_{0:T}^{\pi^{\prime}}$ (note that in this section, we are overburdening the
notation for the superscript to mean counterfactual preferences under a policy
– this can be distiguished from the usual notation in Appendix A of future
preferences under a policy by whether the time subscripts are larger or
smaller than $T$). On a high-level, to perform the counterfactual estimation
task we want to extract as much information as possible about the user’s
initial internal state from the historical interactions $\mathcal{D}_{j}$ we
have available for them: even though such interactions were collected with a
different policy $\pi$, they will still contain information about the user’s
initial state (including their initial preferences before interacting with the
RS). Then, based on this belief about the user’s initial internal state, we
want to obtain a user-specific estimate of the effect that another policy
$\pi^{\prime}$ would have had on their preferences.

As done in Sec. 4, we first describe how one could solve this problem if one
had oracle access to the true user internal state dynamics, and then relax
this assumption showing how one can learn to perform the inference
approximately given only observable interaction data.

Estimation under known internal state dynamics.

Under oracle access to the internal state dynamics, we can first obtain the
belief over the initial state of a given user
$\mathbb{P}(z_{0}|s_{0:T}^{\pi},x^{\pi}_{0:T})$ via NHMM smoothing. This is a
simple extension of HMM smoothing (Russell & Norvig, 2002) which can be
derived using the same steps used in Appendix E.1. We can then roll out the
human model forward dynamics with the fixed policy $\pi^{\prime}$ instead of
$\pi$:
$\mathbb{P}(z^{\pi^{\prime}}_{t}|s_{0:T}^{\pi},x^{\pi}_{0:T})=\sum_{z_{0}}\mathbb{P}(z^{\pi^{\prime}}_{t}|z_{0})\mathbb{P}(z_{0}|s_{0:T}^{\pi},x^{\pi}_{0:T})$.

Estimation under unknown internal state dynamics.

Figure 8: Simulating counterfactuals. To simulate what the user’s preferences
would have been under a different policy $\pi^{\prime}$, we first can use the
initial preference model to estimate initial preferences based on later
observables. We can then use the estimated initial preferences to simulate
counterfactual preference trajectories for the user under $\pi^{\prime}$ using
a conditioned future preference estimation model – where estimates are
conditioned on the recovered initial preference belief.

Without oracle access to the internal state dynamics, again we try to learn to
perform this NHMM counterfactual task approximately. One challenge in
obtaining supervision for the task is that in our dataset of interactions we
never get to see true counterfactuals. We get around this by decomposing the
task into two parts, for which we use two separate models: (1) an initial
preference estimation network, which is used to estimate the initial internal
state for the user (based on the interaction data
$x_{0:T}^{\pi},s_{0:T}^{\pi}$ under $\pi$ which we have available), i.e. train
a predictor which approximates the NHMM smoothing distribution
$\mathbb{P}(u_{0}|x_{0:T}^{\pi},s_{0:T}^{\pi})$ which we will denote here as
$b^{\pi}_{0:T}(u_{0})$ (the belief over initial preferences); and then (2) an
counterfactual preference estimation network which is used to estimate
$u_{0:T}^{\pi^{\prime}}$ conditional on the belief of the initial preferences.

The models for the two tasks are trained with the same method used for
predicting future preference estimation (Fig. 2) but with different inputs and
supervision signal (Fig. 8). For task (1), the network is trained to predict
the _initial_ (instead of future) preferences of the user based on later
context $x_{1:T}^{\pi},s_{1:T}^{\pi}$. While the network predicts
$\mathbb{P}(u_{0}|x_{1:T}^{\pi},s_{1:T}^{\pi})$, we can recover the correct
smoothing estimate as
$\mathbb{P}(u_{0}|x_{0:T}^{\pi},s_{0:T}^{\pi})\propto\mathbb{P}(x_{0}^{\pi}|s_{0}^{\pi},u_{0})\mathbb{P}(u_{0}|x_{1:T}^{\pi},s_{1:T}^{\pi})$
(see Appendix E.2).

The network for task (2) is obtained by changing the prediction network to
also condition on the recovered initial preferences as shown in 8, enabling us
to make predictions of the form
$\mathbb{P}\big{(}u_{k+1}^{\pi^{\prime}}|b^{\pi}_{0:T}(u_{0}),x_{0:k}^{\pi^{\prime}},s_{0:k}^{\pi^{\prime}}\big{)}$.
For training, we first recover initial preference beliefs for each user $j$ in
the data $\mathcal{D}_{j}$ with the initial preference estimation model (which
we assume has already been trained). We then train this counterfactual
estimation network to reconstruct the user $j$’s actual interactions (under
$\pi$) simply based on this initial preference estimate (similarly to the case
in Sec. 4.2 but by additionally conditioning on the initial preference
belief). This teaches the network to leverage the information contained user’s
initial preferences estimate to better estimate their preferences and
behaviors based on what their interactions have been so far. In practice, we
train these models using a transformer architecture similar to (Sun et al.,
2019), and we detail in Appendix E.2 why this is a good fit for our tasks.

At inference time, we recover an estimate of initial preferences based on
interaction data ($s_{0:T}^{\pi^{\prime}},x_{0:T}^{\pi^{\prime}}$) of a new
user with a new recommender policy $\pi^{\prime}$ with model (1), and then can
estimate the preferences such user would have had under a new policy
$\pi^{\prime\prime}$ with model (2). Such counterfactual preference estimate
is obtained with Monte Carlo simulations similarly to Sec. 4.2 with the
difference that the network is also conditioned on the initial preferences
estimate. See Algorithm 2 for the full algorithm and Appendix E.2 for why this
approximates the NHMM task.

We now have a way to estimate – for a new policy $\pi^{\prime}$ – what we
would expect its impact would have been on a user relative to having deployed
$\pi$ (from which we have observational data; note that similarly to future
preferences estimation, this procedure too can suffer if $\pi^{\prime}$
induces a strong distribution shift relative to the training data).

##  Appendix D Casting our problem to a Non-Homogeneous Markov Model

In our NHMM instance, the observation for each timestep is the corresponding
($s_{t}$, $x_{t}$) pair, the hidden state is $z_{t}$, and the probability of
an observation is given by the joint probability of the policy choosing a
specific slate given the history so far
$\mathbb{P}(s_{t}|s_{0:t-1},x_{0:t-1})=\mathbb{P}\big{(}\pi(s_{0:t-1},x_{0:t-1})=s_{t}\big{)}$
and the user choosing a specific item from that slate
$\mathbb{P}(x_{t}|s_{t})$. The dynamics model is instead given by
$\mathbb{P}(z_{t+1}|z_{t},s_{0:t-1},x_{0:t-1})=\sum_{s_{t+1}}\mathbb{P}(z_{t+1}|z_{t},s_{t})\mathbb{P}(s_{t}|s_{0:t-1},x_{0:t-1})$
(the output of $\pi$ depends on the history of slates and choices so far, but
we omit them from the notation for simplicity). Note that if the policy $\pi$
were deterministic, the dynamics model of the NHMM reduces to
$\mathbb{P}(z_{t+1}|z_{t},s_{t})$, as conditioning on $s_{t}$ or on the full
history is equivalent. We use this notation throughout the main text of the
paper for simplicity.

##  Appendix E Preference estimation: additional information

###  E.1 Future preference estimation

Future preference estimation with known dynamics

The Hidden Markov Model (HMM) “prediction task” corresponds to the inference
of the hidden state of the system at the next timestep given the history so
far: $\mathbb{P}(h_{T}|o_{0:k})$ where $h$ and $o$ are respectively the hidden
state and the observations of the HMM, and $T>k$. We will provide here a proof
sketch of how the HMM prediction task can be easily extended to NHMM
prediction. On a high level, by taking into account the time-dependent
dynamics (computing forward inferences differently for every timestep),
inference tasks in a NHMM should be no different than those in an HMM. We
illustrate this more formally below for the prediction task.

The prediction inference in HMMs is performed by recursively repeating two
steps: first, obtaining the filtering estimate which incorporates the latest
observation as
$\mathbb{P}(h_{t}|o_{0:t})=\alpha\mathbb{P}(o_{t}|h_{t})\mathbb{P}(h_{t}|o_{0:t-1})$,
where $\alpha$ is a constant; then using the filtering estimate for prediction
without the addition of new evidence,
$\mathbb{P}(h_{t+1}|o_{0:t})=\sum_{t}\mathbb{P}(h_{t+1}|h_{t})\mathbb{P}(h_{t}|o_{0:t})$.
Once all evidence is incorporated and we have $\mathbb{P}(h_{k}|o_{0:k})$, one
can repeatedly apply the second step to obtain $\mathbb{P}(h_{T}|o_{0:k})$ –
the quantity we set out to obtain. This can be thought of as simply
propagating the belief over the hidden state one timestep at a time in the
future using the dynamics model $\mathbb{P}(h_{t+1}|h_{t})$. See (Russell &
Norvig, 2002) for more information.

This is where the difference with the HMM comes in: the forward model will be
time dependent, as the output of the policy $\pi$ depends on the history so
far. By using the same exact approach as in the paragraph above (with this
modification in the prediction step), one can perform the same inference also
in a NHMM. In the HMM notation, we would have
$\mathbb{P}(h_{t+1}|o_{0:t})=\sum_{t}\mathbb{P}(h_{t+1}|h_{t},o_{0:t})\mathbb{P}(h_{t}|o_{0:t})$,
where we cannot drop the evidence term $o_{0:t}$ in the forward model.

One peculiarity of our specific task is that we want to infer the preferences
in the future under a different policy $\pi^{\prime}$, even though our
evidence is under a different policy $\pi$. Note that this is not an issue as
long as $\pi$ and $\pi^{\prime}$ are known, as it just means that the time-
dependent dynamics will be different in the timesteps in which the policy
$\pi^{\prime}$ was used.

Future preference estimation with Algorithm 1 and why it approximates NHMM
prediction.

Algorithm 1 is used to obtain the belief over the preferences a user would
have at timestep $H$ – assuming that the user interacted for $T$ timesteps
with a policy $\pi$ (for which we have actual interaction data), and then
interacted from timestep $T$ to $H$ with a policy $\pi^{\prime}$. We refer to
this belief as
$b_{0:T}^{\pi}(u_{H})=\mathbb{P}(u_{H}^{\pi^{\prime}}|s_{0:T}^{\pi},x^{\pi}_{0:T})$.

The first model inference pass in Algorithm 1 resulting in $b_{0:T}(u_{T+1})$
will be a distribution over $u_{T+1}$ conditioned on
$s_{0:T}^{\pi},x^{\pi}_{0:T}$. A minimizer for this prediction problem would
be for the network to represent the actual belief that would be obtained by
performing the prediction step in the NHMM (described in Sec. 4.1). This is
because the expected cross-entropy between user choices and the induced choice
distribution (induced by the preference belief prediction) would be minimized.

One limitation is that there are possibly multiple beliefs over preferences
that induce the same exact distribution over choices: in that sense, the
correct preference belief might be unidentifiable. Experimentally, we don’t
find this to be a problem: the choice of having the representation of the
preference belief be a mixture of Von Mises distributions (see the “Preference
estimation model form” heading later in this section) – which can be thought
of as Normals over a circle – is already a good enough inductive bias to be
able to recover good preference beliefs.

In the NHMM, note that the unrolling the forward model (i.e. computing
$\mathbb{P}(u^{\pi^{\prime}}_{H}|z_{T+1}^{\pi})$) could be computed exactly,
or be performed with Monte Carlo estimation, by sampling many internal states
$z_{T+1}^{\pi}$ from the forward prediction distribution, and then sampling
internal state evolutions according to the dynamics. In our algorithm, by
sampling choices and slates then conditioning on them, we are approximating
the Monte Carlo estimation approach. When simulating each user choice, the
model can be thought of as implicitly sampling a preference for this simulated
user (from the preference belief) and then sampling a choice from the
corresponding choice distribution. This means that each simulation rollout
should be equivalent to “sampling a user” (according to the distribution of
users in the data) and then sampling their choices.

Algorithm 1 Predicting future user preferences at timestep $H$ under RS policy
$\pi^{\prime}$

Input: past interactions $x_{0:T}^{\pi}$, $s_{0:T}^{\pi}$, policy
$\pi^{\prime}$, future preference estimator $\hat{P}_{f}$, horizon $H$, number
of Monte Carlo simulations $N$

{if no past interaction data is given}

if $x_{0:T}=s_{0:T}=\emptyset$ then

Sample slate $s_{0}\sim\pi^{\prime}(\emptyset,\emptyset)$

Obtain belief over initial preferences and choice
$b_{\emptyset}(u_{0}),b_{\emptyset}(x_{0})=\hat{P}_{f}(\emptyset,\emptyset)$

Simulate a user choice $x_{0}\sim b_{\emptyset}(x_{0})$ {we now have past
interactions with $T=0$}

end if

for $i=0$; $i<N$; $i++$ do

while $k=T$; $k<H$; $k++$ do

$b_{0:k}(u_{k+1}),b_{0:k}(x_{k+1})=\hat{P}_{f}(s_{0:k},x_{0:k})$ {estimate
pref. and choices}

$s_{k+1}\sim\pi^{\prime}(x_{0:k},s_{0:k})$ {sample next timestep slate}

$x_{k+1}\sim b_{0:k}(x_{k+1})$ {simulate a user choice}

Add $x_{k+1}$ and $s_{k+1}$ to the current simulated trajectory’s history

end while

end for

Average $b_{0:k-1}(u_{k})$ and $b_{0:k-1}(x_{k})$ across the $N$ futures (for
each $k$, with $T<k\leq H$)

Return: Belief over future pref. $b^{\pi}_{0:T}(u_{k})$ and behaviors
$b^{\pi}_{0:T}(x_{k})$ for each $k$ s.t. $T<k\leq H$.

###  E.2 Counterfactual preference estimation

Initial preference prediction model output correction

Below, we show that one can recover the smoothing estimate
$\mathbb{P}(u_{0}|x_{0:t},s_{0:t})$ from the predicted preferences
$\mathbb{P}(u_{0}|x_{1:t},s_{1:t})$ which will be a biased estimate of the
initial preferences (as it does not incorporate the information from timestep
$t=0$) . Note that:

$\displaystyle\mathbb{P}(u_{0}|x_{0:t},s_{0:t})$
$\displaystyle=\frac{\mathbb{P}(u_{0}|x_{0},s_{0})\mathbb{P}(x_{1:t},s_{1:t}|u_{0})}{\mathbb{P}(x_{0:t},s_{0:t})}=\frac{\mathbb{P}(u_{0}|x_{0},s_{0})}{\mathbb{P}(x_{0:t},s_{0:t})}\frac{\mathbb{P}(u_{0}|x_{1:t},s_{1:t})\mathbb{P}(x_{1:t},s_{1:t})}{\mathbb{P}(u_{0})}$
(1)
$\displaystyle=\frac{\mathbb{P}(x_{0},s_{0}|u_{0})\mathbb{P}(u_{0}|x_{1:t},s_{1:t})\mathbb{P}(x_{1:t},s_{1:t})}{\mathbb{P}(x_{0:t},s_{0:t})\mathbb{P}(x_{0},s_{0})}\propto\mathbb{P}(x_{0},s_{0}|u_{0})\mathbb{P}(u_{0}|x_{1:t},s_{1:t})$
(2)

The first equality is given by the definition of smoothing applied to $t=0$
(Russell & Norvig, 2002). The second equality is obtained by using Bayes Rule
on the backwards message $\mathbb{P}(x_{1:t},s_{1:t}|u_{0})$, and the third is
obtained by using Bayes Rule on $\mathbb{P}(u_{0}|x_{0},s_{0})$. Finally, we
can ignore $\mathbb{P}(x_{1:t},s_{1:t})$, $\mathbb{P}(x_{0:t},s_{0:t})$, and
$\mathbb{P}(x_{0},s_{0})$ as they are constants.

Counterfactual preference estimation with Algorithm 2 and why it approximates
NHMM prediction.

A similar argument to that in Appendix E.1 can be made as to why the initial
preference estimation model would approximate the corresponding NHMM smoothing
task.

However, for the second step of counterfactual preference estimation, one
issue arises. Relative to having access to the full dynamics of the internal
state, when performing approximate inference with our model we lose some
information: we are only able to recover a belief over the initial
preferences, whereas the NHMM with full dynamics access would be able to
recover a belief over the full internal state of the user. This will reduce
the accuracy of our counterfactual estimation, but is the best we can do
without further assumptions.

Mathematically, we approximate the NHMM target distribution
$\mathbb{P}(u_{T}^{\pi^{\prime}}|x_{0:t}^{\pi},s_{0:t}^{\pi})$ as:

$\displaystyle\mathbb{P}(u_{T}^{\pi^{\prime}}|x_{0:t}^{\pi},s_{0:t}^{\pi})$
$\displaystyle\approx\sum_{u_{0}}\mathbb{P}(u_{T}^{\pi^{\prime}}|u_{0})\mathbb{P}(u_{0}|x_{0:t}^{\pi},s_{0:t}^{\pi})=\mathbb{P}\big{(}u_{T}^{\pi^{\prime}}|b^{\pi}_{0:t}(u_{0})\big{)}$
(3)
$\displaystyle=\sum_{x_{0:T-1}^{\pi^{\prime}},s_{0:T-1}^{\pi^{\prime}}}\mathbb{P}(x_{0:T}^{\pi^{\prime}},s_{0:T}^{\pi^{\prime}})\mathbb{P}\big{(}u_{T}^{\pi^{\prime}}|b^{\pi}_{0:t}(u_{0}),x_{0:T}^{\pi^{\prime}},s_{0:T}^{\pi^{\prime}}\big{)}$
(4)

where we the last expression is approximated with a Monte Carlo estimate
detailed in Algorithm 2111This algorithm is not actually used in it’s pure
form in the experiments. Our choice of “safe policy” $\pi_{\text{rnd}}$
happens to choose constant slates (i.e. a uniform distribution no matter the
history), so there is a shortcut to the procedure: by simply setting the
inputted slated for counterfactual prediction to be uniforms, and predicting
preferences without any user choices, one can train the model to directly
output the belief over counterfactual preferences for any timestep, across the
whole userbase. (similarly to what was done in Algorithm 1). To do well at
this second trajectory reconstruction task, the network necessarily needs to
learn how to make best use of the belief over initial preferences, and
implicitly learn their dynamics, as for the future preference estimation task.

Algorithm 2 Predicting counterfactual user preferences at timestep $T$ under
policy $\pi^{\prime}$, given $t$ timesteps of interaction data with $\pi$.

Inputs: past interactions $x_{0:t}^{\pi}$, $s_{0:t}^{\pi}$, policy
$\pi^{\prime}$, future, initial, and counterfactual preference estimators
$\hat{P}_{f},\hat{P}_{i},\hat{P}_{c}$, horizon $T$, constant $N$

$b^{\pi}_{0:t}(u_{0})=\hat{P}_{i}(s_{0:t}^{\pi},x_{0:t}^{\pi})${initial pref.
belief given interactions with $\pi$}

$\hat{P}_{f}=\hat{P}_{c}(b=b^{\pi}_{0:t}(u_{0})\big{)}$ {future pref predictor
conditioned on init belief}

$\big{(}b^{\pi}_{0:k-1}(u^{\pi^{\prime}}_{k}),b^{\pi}_{0:k-1}(x^{\pi^{\prime}}_{k})\big{)}_{0<k\leq
T}=$Algorithm
1$\Big{(}\emptyset,\emptyset,\pi^{\prime},\hat{P}_{f},T,N\Big{)}$

Return: Distributions of counterfactual preferences and behaviors under policy
$\pi^{\prime}$

Preference estimation model form

When considering all models described in Sec. 4 and in Appendix 2, we have
three models for preference estimation: respectively initial, counterfactual,
and future preference estimators $\hat{P}_{i},\hat{P}_{c},\hat{P}_{f}$. Any
sequence model, such as RNNs or transformers, would be appropriate for these
tasks that have variable number of inputs. We choose to use transformers, as
described in Appendix H.2.

One detail of note that was omitted from Fig. 2 is that – to enable to
represent multi-modal beliefs over preference space – we let the models’
output be parameters of multiple Von Mises distributions and additionally some
weights $w$. The $w$ weighted average of these distributions will form the
predicted belief over $u_{t}$.

##  Appendix F Computing metrics

For computing the metrics defined in Sec. 5 for estimated future preferences
of a user, one can use the approach delineated in Sec. 4. If one wants instead
to compute the metrics for counterfactual preferences of a user, one should
use the methodology from Appendix C. Computing metrics for counterfactual
metrics is necessary in the case of penalized RL training: we want to know,
for the user at hand and the current simulated trajectory under the policy
$\pi^{\prime}$ we are training, what the preferences of such a user would have
been under an alternate safe policy $\pi^{\text{safe}}$. See Algorithm 3 for
more details.

##  Appendix G Penalized RL

The underlying MDP

One could cast the recommendation problem as a POMDP (Lu & Yang, 2016;
Mladenov et al., 2019) in which the state of the environment is hidden and
contains the user’s internal state, which evolves over time. Equivalently, one
can consider the belief-MDP induced by the recommender POMDP (Kaelbling et
al., 1998), and approximate a solution to such belief-MDP via Deep-RL with a
policy trained with observation histories as input (this is theoretically
sufficient for the policy to recover a belief over the current hidden state
and take the optimal action). The action space will be given by the space of
possible slates that the RS can choose. The reward signal will be the expected
reward for the current timestep
$\mathbb{E}\big{[}\hat{r}_{t}(u_{t}^{\pi})\big{]}$ (or with the extra terms
for the proxies in the case of penalized training). The introduction of
expectation can be thought of as expected SARSA (Sutton & Barto, 1998), as
argued in (Ie et al., 2019b).

Penalized RL training

The full set of steps to run RL training are as follows: once the human models
described in Sec. 4 are trained, one can use them to simulate user
trajectories and compute penalty metrics for such trajectories (see Algorithm
3). One can then optimize the RL policy based on the on-policy simulated
trajectory rollouts.

Algorithm 3 Generating a trajectory for RL training and computing metrics

Input: Initial, counterfactual, and future preference estimators
$\hat{P}_{i},\hat{P}_{c},\hat{P}_{f}$; a policy $\pi$, a safe policy
$\pi_{\text{safe}},$ a horizon $H$, a constant $N$.

Sample slate $s^{\pi}_{0}\sim\pi(\emptyset,\emptyset)$ and imagine user choice
$x^{\pi}_{0}\sim\hat{P}_{i}(x_{0}|\emptyset,\emptyset)$

for $t=1$; $t\leq H$; $t++$ do

$b^{\pi}_{0:t-1}(u_{0})=\hat{P}_{i}(s^{\pi}_{0:t-1},x^{\pi}_{0:t-1})$ {current
belief over initial preferences}

$b^{\pi}_{0:t-1}(u_{t}),b^{\pi}_{0:t-1}(x_{t})=\hat{P}_{f}(s^{\pi}_{0:t-1},x^{\pi}_{0:t-1})$
{belief over pref and choices}

$\mathbb{E}[b^{\pi}_{0:t-1}(u^{\pi_{\text{safe}}}_{t})]\in$ Algorithm
2$\big{(}t,\pi_{\text{safe}},\hat{P}_{c},\hat{P}_{i},x^{\pi}_{0:t-1},s^{\pi}_{0:t-1}\big{)}$

{belief over counterfactual preferences under safe policy for this user}

$s^{\pi}_{t}\sim\pi(x^{\pi}_{0:t-1},s^{\pi}_{0:t-1})$ {sample slate}

$x^{\pi}_{t}\sim b^{\pi}_{0:t-1}(x_{t})${imagine a user choice}

$D_{t}(u_{t}^{\pi},u_{t}^{\text{safe}})=\mathbb{E}_{u^{\pi}_{t},u^{\pi_{\text{safe}}}_{t},x^{\pi}_{t}}\big{[}(x_{t}^{\pi})^{T}u^{\pi}_{t}-(x_{t}^{\pi})^{T}u^{\pi_{\text{safe}}}_{t}\big{]}$
{compute penalty metric(s) for timestep}

$r^{RL}_{t}=\mathbb{E}[r_{t}]+D_{t}(u_{t}^{\pi},u_{t}^{\text{safe}})$

end forReturn: User-RS interactions and training rewards for the simulated
trajectory

Reduction of distances to final penalized objective

Note that the full penalized RL objective
$\bigg{(}\sum_{t}^{T}\mathbb{E}\big{[}\hat{r}_{t}(u_{t}^{\pi})\big{]}\bigg{)}-\nu_{1}D(u_{0:T}^{\pi},u_{0})-\nu_{2}D(u_{0:T}^{\pi},u_{0:T}^{\pi_{\text{rnd}}})$
for our choice of distance function $D$ reduces to
$\sum_{t}^{T}\mathbb{E}\big{[}\hat{r}_{t}(u_{t}^{\pi})+\nu^{\prime}_{1}\
\hat{r}_{t}(u_{0},\pi)+\nu^{\prime}_{2}\
\hat{r}_{t}(u_{t}^{\pi_{\text{rnd}}},\pi)\big{]}$ for some choice of
$\nu^{\prime}_{1},\nu^{\prime}_{2}$ which can be treated as hyperparameters of
how much we want to value the engagement under each safe policy preferences
relative to the engagement under the main policy.

##  Appendix H Experiment details

###  H.1 Ground truth users

Reduction of logit model to our case.

In our experimental setup, the traditional conditional logit model
$\mathbb{P}(x_{t}=x|s_{t},u_{t})=\frac{e^{\beta_{c}x^{T}u_{t}}}{\sum_{x\in
s}e^{\beta_{c}x^{T}u_{t}}}$ doesn’t apply directly in this form, as we
consider slates to be distributions rather than sets of discrete items.
Intuitively, user’s choices should still depend on the slate: the proportion
of a certain item in the current slate (one can think of this as the
proportion of a certain item type), should influence the probability of the
user of selecting that item (type). We operationalize this as
$\mathbb{P}(x_{t}=x|s_{t},u_{t})\propto\mathbb{P}(s_{t}=x)e^{\beta_{c}x^{T}u_{t}}$,
with an additional term $\mathbb{P}(s_{t}=x)$ which takes into account the
proportion of each item (type). Note that this also mathematically corresponds
to the generalization of the traditional logit model: when the slate is
discrete, $\mathbb{P}(s_{t}=x)$ will simply be an indicator for whether the
item is in the slate, leading to the traditional logit model form.

Feed belief update

Our ground truth user has a belief over future slates ${b^{H}_{t}(s)}$. Users’
initial belief matches the content feature distribution itself
${b^{H}_{0}(s)}=\mathcal{D}$. After receiving a slate $s_{t}$, the user’s
induces a belief ${b^{H}_{t}(s)}\propto s_{t}^{3}$ over the future slates in
the user, i.e. the user will expect the next feeds to look like the most
common items in the current feed, as a result of availability bias (MacLeod &
Campbell, 1992).

Lack of no-op choice.

While the assumption that user must pick item from the slate is unrealistic,
this could be resolved by adding an extra no-op choice to every slate (Sunehag
et al., 2015).

Preference shift model.

Our preference shift model is inspired by (Bernheim et al., 2021), but adapted
to our experimental setup as described below. The choice of preferences is
modulated by a “mindset flexibility” parameter $\lambda$ which captures how
open they are to modifying their current preferences. Users assign value to
the choice of next-timestep preferences $u_{t+1}$ as:
$V\big{(}u_{t},{b^{H}_{t}(s)},u_{t+1},\lambda\big{)}=\mathbb{E}_{{x_{t+1}\sim{b^{H}_{t}(s)}}}[\lambda\hat{r}_{t+1}(u_{t})+(1-\lambda)\hat{r}_{t+1}(u_{t+1})]$,
where with $\hat{r}_{t+1}(u_{t})$ indicates the engagement value obtained by
the user under the choice $x_{t+1}$ and preference $u_{t}$. Users pick their
next timestep preferences also according to the conditional logit model, but
over their expected value of such preferences choices:
$\mathbb{P}(u_{t+1}|{b^{H}_{t}(s)},u_{t},\lambda)\propto
e^{\beta_{d}V(u_{t},{b^{H}_{t}(s)},u_{t+1},\lambda)}$. Intuitively, users
update their preference to more “convenient” ones – ones that they expect will
lead them to higher engagement value. The main change we introduce from the
original model is incorporating the belief over future feeds.

Ground truth human parameters for experiments ($\beta$, $\lambda$, $u_{0}$,
etc.).

The users’ preference-flexibility parameter is given by $\lambda=0.9$, and
their initial preferences are drawn from a normal distribution222Technically
one should use Von Mises distributions – a distribution similar to normals,
but for which the domain is a circle. As Von Mises distributions are not
implemented in numpy (Harris et al., 2020), for simplicity we use clipped
normal distributions (disregarding probability mass beyond $180\degree$ in
either direction) in all places except for the the transformer output (which
uses PyTorch’s (Paszke et al., 2019) Von Mises implementation)  with mean
$u=130\degree$ and standard deviation $20\degree$. The temperature parameter
$\beta$ is set as mentioned in the main text and shown in Fig. 9.

As users obtain higher engagement value when they act less stochastically,
these portions of preference space form attractor points as can be seen in all
policies in Fig. 1. Also in Fig. 1 we see that while naturally preferences
shift to mostly focus on one of these modes, some RS policies drive
preferences to the other mode. While preferences naturally tend to shift
towards one of these modes, some RS policies drive preferences to the other
mode. As the engagement does not correspond to actual value, converging to the
higher local optimum of engagement ($u=270\degree$ instead of $u=80\degree$)
is not necessarily desirable.

Figure 9: How the temperature parameter $\beta$ varies across the preference
space for our ground truth humans, which defines the true choice model. Figure
10: How the temperature parameter $\beta$ varies across the preference space
for our mis-specified choice model used for the mis-specification robustness
experiments in Sec. 7.1 and Appendix I.

###  H.2 Learned human models

Figure 11: BERT representation of the inference tasks. While our method is
compatible with any sequence model, we choose to use a BERT transformer
models. Left: estimation of the user’s future preferences and choice at t=2,
given the interaction history history so far (this modality of prediction is
closest in setup to Sun et al. (2019)). Middle: recovering a belief over the
initial preferences and choice of the user based on later interactions. Right:
conditioning on the estimate of initial preferences ($\dagger$) recovered from
the smoothing network one can estimate counterfactual preferences and choices
under slates (*) (chosen from a policy $\pi^{\prime}$ we are interested in)
and imagined choices (not shown due to space).

For our learned human models, we use the BERT transformer architecture
(similarly to (Sun et al., 2019)) with 2 layers, 2 attention heads, 4 sets of
Von Mises distribution parameters, a learning rate of $0.00003$, batch size of
$500$, and $100$ epochs. We train on the data described in Sec. H.3.

In architecture, we use a similar form to that of BERT4Rec (Sun et al., 2019)
for ease of performing inference on future or past preferences given contexts
of interaction, as described in Fig. 11. We mask all inputs that should not be
used for prediction.

Mis-specification model. For the mis-specified human choice model experiments,
we set the beta parameters of the choice model across the preference space as
shown in Fig. 10 (in contrast to the true values, shown in Fig. 9). We found
that increasing the mis-specification further led to worse results as
expected.

###  H.3 Simulated dataset

Figure 12: a) $\rightarrow$ b) The content can be mapped to an empirical
distribution over feature space $\mathcal{D}$ in $\mathbb{R}^{d}$. We consider
dimension $d=2$ for ease of visualization. Restricting preferences and choices
to be unit vectors, one can think of them as a points on a circle: the
engagement value $\hat{r}_{t}$ will thus be related to the angle $\theta$
between $u_{t}$ and $x_{t}$. c) We discretize this circular preference and
feature space into $n=36$ bins (i.e. binning the angles) which enables to
visualize distributions over preferences and over content features as
histograms over angles. d) We model slates $s_{t}$ as categorical
distributions over the discretized $n$-bin feature space.

See Fig. 12 for a summary of how content is generally instantiated in our
setup.

We set the distribution of content in such a way that it forms a uniform
distribution across features, that is
$\mathcal{D}=\text{Uniform}(0\degree,360\degree)$. We simulate historical user
interaction data with a mixed policy $\pi$ which is similar (but not equal to)
a random policy in half the rollouts and for the other half is goal-directed:

  * •

Half of the data is created with a RS policy which chooses an action uniformly
among a set of possible slates (distributions over the feature space with
means $0\degree,10\degree,...,340\degree,350\degree$, and standard deviations
equal to $30\degree$ or $60\degree$). Each of these slate types can be thought
of as a slate which contain mostly one specific type of content.

  * •

The other half of the data is created with a RS which chooses
$s_{t}=\mathcal{D}=\text{Uniform}(0\degree,360\degree)$ 80% of the time (i.e.
the slate that would be chosen by the random RS $\pi_{\text{rnd}}$), and
chooses a random action from the same set of possible slates as above the
remaining 20% of the time.

This is to simulate the setting in which the safe policy we are interested in
(the random NPS policy) is similar to previously deployed ones that are
represented in the data (although not the same, so the network still has to
generalize across RS policies at test time) – but also not quite the same,
requiring some amount of generalization.

###  H.4 RS training

For RL optimization, we use PPO (Schulman et al., 2017) trained with Rllib
(Liang et al., 2018). The action space of the recommender system is given by
distributions over feature space with means
$0\degree,60\degree,...,260\degree,320\degree$, and standard deviations equal
to $60\degree$. As observations to the system, we provide the current slate,
the previous user choice, and the current estimates from the HMM for
smoothing, filtering, and natural preference shift counterfactual
distributions, in order to increase training speed (note that these will not
change the optimal policy). All policies are recurrent so they are able to
reason about the history of interactions so far.

For training our myopic policies, we use the same exact infrastructure as
above, but set $\gamma=0$, similarly to previous work (Krueger et al., 2020).

We use batch size $1200$, minibatch size $600$, $4$ parallel workers, $0.005$
learning rate, $50$ gradient updates per minibatch per iteration, policy
function clipping parameter of $0.5$, value function clipping parameter of
$50$ and loss coefficient of $8$, with an LSTM network with $64$ cell size.
$\gamma=0$ for the myopic training and $\gamma=0.99$ for long-horizon RL
training. Training runs in less than $30$ minutes for each condition on a
MacBook Pro 16” (2020).

##  Appendix I Results

We report here additional experimental results on for the preference
estimation task, this time using the other models described in Fig. 11: the
initial preference estimation network and the counterfactual preference
estimation network.

Quality of initial preference estimates. The setup for this experiment is
identical to that of Fig. 5, except that now we are evaluating the initial
preference predictor which approximates the inference
$\mathbb{P}(u_{0}|x_{0:T}^{\pi},s_{0:T}^{\pi})$. We show the results in Fig.
13. Both under the correct choice model and with some mis-specification,
preference prediction performs similarly to oracle estimation. For the initial
preference prediction, mis-specification seems to have a less detrimental
effect relative to Fig. 3-Fig. 14. The preference losses for our model are
(slightly) higher than oracle. Preference prediction accuracies are very
slightly higher than those of the oracle by random variability.

Quality of counterfactual preference estimates. We now turn to evaluating the
quality of our counterfactual preference estimation model. As mentioned in
Appendix C, such a model is trained to predict the preferences and choices of
users for which we have seen interactions for, based on the approximate
smoothing estimate obtained by the initial preference estimation model. To
test this model in a harder setting we consider oracle access to
counterfactual trajectories of the users in the normal dataset, while
interacting with a random recommender policy $\pi^{\text{rnd}}$. We obtain the
smoothing estimates from the usual trajectories present in the validation data
(the ones from the section above), but query the counterfactual network to
predict preferences for the counterfactual NPS trajectories for each user. We
then can compute validation losses and accuracies based on this counterfactual
dataset that we pre-computed. The results for this experiment are in Fig. 14.
Interestingly, we see that generally our network does somewhat worse than in
the other settings considered. This is likely due to 1) the approximate nature
of the smoothing estimate which is used to perform the counterfactual task,
and 2) the task that requires estimation for preference-evolutions induced by
a policy that is different from the one seen at training time.

Figure 13: Equivalent setup to Fig. 3, except computed using the initial
preference estimation network from Fig. 11. Error bars are standard deviations
over 3 seeds.
Figure 14: Similar setup to Fig. 3-Fig. 13, except computed using the
counterfactual preference estimation network from Fig. 11. More details in
Appendix I. Error bars are standard deviations over 3 seeds.