# Personalized Reward Learning with
Interaction-Grounded Learning (IGL)

Jessica Maghakian
Stony Brook University
&Paul Mineiro
Microsoft Research NYC
&Kishan Panaganti
Texas A&M University
&Mark Rucker
University of Virginia &Akanksha Saran
Microsoft Research NYC
&Cheng Tan
Microsoft Research NYC

###### Abstract

In an era of countless content offerings, recommender systems alleviate
information overload by providing users with personalized content suggestions.
Due to the scarcity of explicit user feedback, modern recommender systems
typically optimize for the same fixed combination of implicit feedback signals
across all users. However, this approach disregards a growing body of work
highlighting that (i) implicit signals can be used by users in diverse ways,
signaling anything from satisfaction to active dislike, and (ii) different
users communicate preferences in different ways. We propose applying the
recent Interaction Grounded Learning (IGL) paradigm to address the challenge
of learning representations of diverse user communication modalities. Rather
than taking a fixed, human-designed reward function, IGL is able to learn
personalized reward functions for different users and then optimize directly
for the latent user satisfaction. We demonstrate the success of IGL with
experiments using simulations as well as with real-world production traces.

##  1 Introduction

From shopping to reading the news, modern Internet users have access to an
overwhelming amount of content and choices from online services. Recommender
systems offer a way to improve user experience and decrease information
overload by providing a customized selection of content. A key challenge for
recommender systems is the rarity of explicit user feedback, such as ratings
or likes/dislikes (Grčar et al., 2005). Rather than explicit feedback,
practitioners typically use more readily available implicit signals, such as
clicks (Hu et al., 2008), webpage dwell time (Yi et al., 2014), or inter-
arrival times (Wu et al., 2017) as a proxy signal for user satisfaction. These
implicit signals are used as the reward objective in recommender systems, with
the popular Click-Through Rate (CTR) metric as the gold standard for the field
(Silveira et al., 2019). However, directly using implicit signals as the
reward function presents several issues.

_Implicit signals do not directly map to user satisfaction._ Although clicks
are routinely equated with user satisfaction, there are examples of
unsatisfied users interacting with content via clicks. Clickbait exploits
cognitive biases such as caption bias (Hofmann et al., 2012) or the curiosity
gap (Scott, 2021) so that low quality content attracts more clicks. Direct
optimization of the CTR degrades user experience by promoting clickbait items
(Wang et al., 2021). Recent work shows that users will even click on content
that they know a priori they will dislike. In a study of online news reading,
Lu et al. (2018a) discovered that 15% of the time, users would click on
articles that they strongly disliked. Similarly, although longer webpage dwell
times are associated with satisfied users, a study by Kim et al. (2014) found
that dwell time is also significantly impacted by page topic, readability and
content length.

_Different users communicate in different ways._ Demographic background is
known to have an impact on the ways in which users engage with recommender
systems. A study by Beel et al. (2013) shows that older users have CTR more
than 3x higher than their younger counterparts. Gender also has an impact on
interactions, e.g. men are more likely to leave dislikes on YouTube videos
than women (Khan, 2017). At the same time, a growing body of work shows that
recommender systems do not provide consistent performance across demographic
subgroups. For example, multiple studies on ML fairness in recommender systems
show that women on average receive less accurate recommendations compared to
men (Ekstrand et al., 2018; Mansoury et al., 2020). Current systems are also
unfair across different age brackets, with statistically significant
recommendation utility degradation as the age of the user increases (Neophytou
et al., 2022). The work of Neophytou et al. identifies usage features as the
most predictive of mean recommender utility, hinting that the inconsistent
performance in recommendation algorithms across subgroups arises from the
differences in how users interact with the recommender system.

These challenges motivate the need for personalized reward functions. However,
extensively modeling the ways in which implicit signals are used or how
demographics impact interaction style is costly and inefficient. Current
state-of-the-art systems utilize reward functions that are manually engineered
combinations of implicit signals, typically refined through laborious trial
and error methods. Yet as recommender systems and their users evolve, so do
the ways in which users implicitly communicate preferences. Any extensive
models or hand tuned reward functions developed now could easily become
obsolete within a few years time.

To this end, we propose Interaction Grounded Learning (IGL) Xie et al. (2021)
for personalized reward learning (IGL-P). IGL is a learning paradigm where a
learner optimizes for unobservable rewards by interacting with the environment
and associating observable feedback with the true latent reward. Prior IGL
approaches assume the feedback either depends on the reward alone Xie et al.
(2021), or on the reward and action Xie et al. (2022). These methods are
unable to disambiguate personalized feedback which depends on the context.
Other approaches such as reinforcement learning and traditional contextual
bandits suffer from the choice of reward function. However our proposed
personalized IGL, IGL-P, resolves the 2 above challenges while making minimal
assumptions about the value of observed user feedback. Our new approach is
able to incorporate both explicit and implicit signals, leverage ambiguous
user feedback and adapt to the different ways in which users interact with the
system.

Our Contributions: We present IGL-P, the first IGL strategy for context-
dependent feedback, the first use of inverse kinematics as an IGL objective,
and the first IGL strategy for more than two latent states.Our proposed
approach provides an alternative to agent learning methods which require hand-
crafted reward functions. Using simulations and real production data, we
demonstrate that IGL-P is able to learn personalized rewards when applied to
the domain of online recommender systems, which require at least 3 reward
states.

##  2 Problem Setting

###  2.1 Contextual Bandits

The contextual bandit (Auer et al., 2002; Langford & Zhang, 2007) is a
statistical model of myopic decision making which is pervasively applied in
recommendation systems (Bouneffouf et al., 2020). IGL operates via reduction
to contextual bandits, hence, we briefly review contextual bandits here.

The contextual bandit problem proceeds over $T$ rounds. At each round
$t\in[T]$, the learner receives a context $x_{t}\in\mathcal{X}$ (the context
space), selects an action $a_{t}\in\mathcal{A}$ (the action space), and then
observes a reward $r_{t}(a_{t})$, where $r_{t}:\mathcal{A}\to[0,1]$ is the
underlying reward function. We assume that for each round $t$, conditioned on
$x_{t}$, $r_{t}$ is sampled from a distribution
$\mathbb{P}_{r_{t}}(\cdot\mid{}x_{t})$. A contextual bandit algorithm attempts
to minimize the cumulative regret

$\displaystyle\mathrm{\mathbf{Reg}}_{\mathsf{CB}}(T)\vcentcolon=\sum_{t=1}^{T}r_{t}(\pi^{\star}(x_{t}))-r_{t}(a_{t})$
(1)

relative to an optimal policy $\pi^{\star}$ over a policy class $\Pi$.

In general, both the contexts $x_{1},\ldots,x_{T}$ and the distributions
$\mathbb{P}_{r_{1}},\ldots,\mathbb{P}_{r_{T}}$ can be selected in an
arbitrary, potentially adaptive fashion based on the history. In the sequel we
will describe IGL in a stochastic environment, but the reduction induces a
nonstationary contextual bandit problem, and therefore the existence of
adversarial contextual bandit algorithms is relevant.

###  2.2 Interaction Grounded Learning

IGL is a problem setting in which the learner’s goal is to optimally interact
with the environment with no explicit reward to ground its policies. IGL
extends the contextual bandit framework by eliding the reward from the
learning algorithm and providing feedback instead (Xie et al., 2021). We
describe the stochastic setting where $(x_{t},r_{t},y_{t})\sim D$ triples are
sampled iid from an unknown distribution; the learner receives the context
$x_{t}\in\mathcal{X}$, selects an action $a_{t}\in\mathcal{A}$, and then
observes the feedback $y_{t}(a_{t})$, where $y_{t}:\mathcal{A}\to[0,1]$ is the
underlying feedback function. Note $r_{t}(a_{t})$ is never revealed to the
algorithm: nonetheless, the regret notion remains the same as Eq. 1. An
information-theoretic argument proves assumptions relating the feedback to the
underlying reward are necessary to succeed (Xie et al., 2022).

####  2.2.1 Specialization to Recommendation

For specific application in the recommendation domain, we depart from prior
art in IGL (Xie et al., 2021; 2022) in two ways: first, in the assumed
relationship between feedback and underlying reward; and second, in the number
of latent reward states. For the remainder of the paper, we represent the
users as $x$ (i.e., as context), content items as action $a$, the user
satisfaction with recommended content as $r$, and the feedback signals in
response to the recommended content (user interactions with the system
interface) as $y$.

recommender system$x$$a$I like this!$r$$y$ Figure 1: IGL in the recommender
system setting. The learner observes the context $x$, plays an action $a$, and
then observes a feedback $y$ (that is dependent on the latent reward $r$), but
not $r$ itself.

##### Feedback Dependence Assumption

Xie et al. (2021) assumed full contextual independence of the feedback on the
context and chosen action, i.e. $y\perp x,a|r$. For recommender systems, this
implies that all users communicate preferences identically for all content. In
a subsequent paper, Xie et al. (2022) loosen the full conditional independence
by considering context conditional independence, i.e. $y\perp x|a,r$. For our
setting, this corresponds to the user feedback varying for combinations of
preference and content, but remaining consistent across all users. Neither of
these two assumptions are natural in the recommendation setting because
different users interact with recommender systems in different ways.(Beel et
al., 2013; Shin, 2020). In this work, we assume $y\perp a|x,r$, i.e., the
feedback $y$ is independent of the displayed content $a$ given the user $x$
and their disposition toward the displayed content $r$. Thus, we assume that
users may communicate in different ways, but a given user expresses
satisfaction, dissatisfaction and indifference to all content in the same way.

##### Number of Latent Reward States

Prior work demonstrates that a binary latent reward assumption, along with an
assumption that rewards are rare under a known reference policy, are
sufficient for IGL to succeed. Specifically, optimizing the contrast between a
learned policy and the oblivious uniform policy is able to succeed when
feedback is both context and action independent Xie et al. (2021); and
optimizing the contrast between the learned policy and all constant-action
policies succeeds when the feedback is context independent Xie et al. (2022).

Although the binary latent reward assumption (e.g., satisfied or dissatisfied)
appears reasonable for recommendation scenarios, it fails to account for user
indifference versus user dissatisfaction. This observation was first motivated
by our production data, where a 2 state IGL policy would sometimes maximize
feedback signals with obviously negative semantics. Assuming users ignore most
content most of the time (Nguyen et al., 2014), negative feedback can be as
difficult to elicit as positive feedback, and a 2 state IGL model is unable to
distinguish between these extremes. Hence, we posit a minimal latent state
model for recommender systems involves 3 states: (i) $r=1$, when users are
satisfied with the recommended content, (ii) $r=0$, when users are indifferent
or inattentive, and (iii) $r=-1$, when users are dissatisfied. We motivate the
3 state model with a simulated experiment in Section A.1.

##  3 Derivations

Prior approaches to IGL use contrastive learning objectives (Xie et al., 2021;
2022), but the novel feedback dependence assumption in the prior section
impedes this line of attack. Essentially, given arbitrary dependence upon $x$,
learning must operate on each example in isolation without requiring
comparison across examples. This motivates attempting to predict the current
action from the current context and the currently observed feedback, i.e.,
inverse kinematics.

##### Inverse Kinematics

Traditionally in robotics and computer animation, inverse kinematics is the
mathematical process used to recover the movement (action) of a robot/object
in the world from some other data such as the position and orientation of a
robot manipulator/video of a moving object (in our case, the other data is
context and feedback). We motivate our inverse kinematics strategy using exact
expectations. When acting according to any policy $P(a|x)$, we can imagine
trying to predict the action taken given the context and feedback; the
posterior distribution is

$\displaystyle P(a|y,x)$ $\displaystyle=\frac{P(a|x)P(y|a,x)}{P(y|x)}$
$\displaystyle\left(\text{Bayes rule}\right)$ (2)
$\displaystyle=P(a|x)\sum_{r}\frac{P(y|r,a,x)}{P(y|x)}P(r|a,x)$
$\displaystyle\left(\text{Total Probability}\right)$
$\displaystyle=P(a|x)\sum_{r}\frac{P(y|r,x)}{P(y|x)}P(r|a,x)$
$\displaystyle\left(y\perp a|x,r\right)$
$\displaystyle=P(a|x)\sum_{r}\frac{P(r|y,x)}{P(r|x)}P(r|a,x)$
$\displaystyle\left(\text{Bayes rule}\right)$
$\displaystyle=\sum_{r}P(r|y,x)\left(\frac{P(r|a,x)P(a|x)}{\sum_{a}P(r|a,x)P(a|x)}\right).$
$\displaystyle\left(\text{Total Probability}\right)$

We arrive at the inner product between a reward decoder term ($P(r|y,x$)) and
a reward predictor term ($P(r|a,x)$).

##### Extreme Event Detection

Direct extraction of a reward predictor using maximum likelihood on the action
prediction problem with Eq. 2 is frustrated by two identifiability issues:
first, this expression is invariant to a permutation of the rewards on a
context dependent basis; and second, the relative scale of two terms being
multiplied is not uniquely determined by their product. To mitigate the first
issue, we assume $\sum_{a}P(r=0|a,x)P(a|x)>\frac{1}{2}$, i.e., nonzero rewards
are rare under $P(a|x)$; and to mitigate the second issue, we assume the
feedback can be perfectly decoded, i.e., $P(r|y,x)\in\\{0,1\\}$. Under these
assumptions, we have

$\displaystyle r=0\implies P(a|y,x)$
$\displaystyle=\frac{P(r=0|a,x)P(a|x)}{\sum_{a}P(r=0|a,x)P(a|x)}$
$\displaystyle\leq 2P(r=0|a,x)P(a|x)\leq 2P(a|x).$ (3)

Eq. 3 forms the basis for our extreme event detector: anytime the posterior
probability of an action is predicted to be more than twice the prior
probability, we deduce $r\neq 0$.

Note a feedback merely being apriori rare or frequent (i.e., the magnitude of
$P(y|x)$ under the policy $P(a|x)$) does not imply that observing such
feedback will induce an extreme event detection; rather the feedback must have
a probability that strongly depends upon which action is taken. Because
feedback is assumed conditionally independent of action given the reward, the
only way for feedback to help predict which action is played is via the
(action dependence of the) latent reward.

##### Extreme Event Disambiguation

With 2 latent states, $r\neq 0\implies r=1$, and we can reduce to a standard
contextual bandit with inferred rewards
$\mathop{\mathbbm{1}}(P(a|y,x)>2P(a|x))$. With 3 latent states, $r\neq
0\implies r=\pm 1$, and additional information is necessary to disambiguate
the extreme events. We assume partial reward information is available via a
“definitely negative” function111“Definitely positive” information can be
incorporated analogously.
$\texttt{DN}:\mathcal{X}\times\mathcal{Y}\to\\{-1,0\\}$ where
$P(\texttt{DN}(x,y)=0|r=1)=1$ and $P(\texttt{DN}(x,y)=-1|r=-1)>0$. This
reduces extreme event disambiguation to one-sided learning (Bekker & Davis,
2020) applied only to extreme events, where we try to predict the underlying
latent state given $(x,a)$. We assume partial labelling is selected completely
at random Elkan & Noto (2008) and treat the (constant) negative labelling
propensity $\alpha$ as a hyperparameter. We arrive at our 3-state reward
extractor

$\displaystyle\rho(x,a,y)$ $\displaystyle=\begin{cases}0&P(a|y,x)\leq
2P(a|x)\\\ -\alpha^{-1}&P(a|y,x)>2P(a|x)\text{ and }\texttt{DN}(x,y)=-1\\\
1&\text{otherwise}\end{cases},$ (4)

equivalent to Bekker & Davis (2020, Equation 11). Setting $\alpha=1$ embeds
2-state IGL.

Algorithm 1 IGL; Inverse Kinematics; 2 or 3 Latent States; On or Off-Policy.

0: Contextual bandit algorithm CB-Alg.

0: Calibrated weighted multiclass classification algorithm MC-Alg.

0: Definitely negative oracle DN. # 3SCOMMENT$\texttt{DN}(\ldots)=0$ for 2
state IGL

0: Negative labelling propensity $\alpha$. # 3SCOMMENT$\alpha=1$ for 2 state
IGL

0: Action set size $K$.

1: $\pi\leftarrow\textrm{new }\texttt{CB-Alg}$.

2: $\texttt{IK}\leftarrow\textrm{new }\texttt{MC-Alg}$.

3: for $t=1,2,\dots;$ do

4: Observe context $x_{t}$ and action set $A_{t}$ with $|A_{t}|=K$.

5: if On-Policy IGL then

6: $P(\cdot|x_{t})\leftarrow\pi.\textrm{predict}(x_{t},A_{t})$.

7: Play $a_{t}\sim P(\cdot|x_{t})$ and observe feedback $y_{t}$.

8: else

9: Observe $\left(x_{t},a_{t},y_{t},P(\cdot|x_{t})\right)$.

10: $w_{t}\leftarrow 1/(KP(a_{t}|x_{t}))$. # 3SCOMMENTSynthetic uniform
distribution

11:
$\hat{P}(a_{t}|y_{t},x_{t})\leftarrow\texttt{IK}.\textrm{predict}((x_{t},y_{t}),A_{t},a_{t})$.
# 3SCOMMENTPredict action probability

12: if $K\hat{P}(a_{t}|y_{t},x_{t})\leq 2$ then # 3SCOMMENT$\hat{r}_{t}=0$

13: $\pi.\textrm{learn}(x_{t},a_{t},A_{t},r_{t}=0)$

14: else # 3SCOMMENT$\hat{r}_{t}\neq 0$

15: if $\texttt{DN}(\ldots)=0$ then

16: $\pi.\textrm{learn}(x_{t},a_{t},A_{t},r_{t}=1,P(\cdot|x_{t}))$

17: else # 3SCOMMENTDefinitely negative

18: $\pi.\textrm{learn}(x_{t},a_{t},A_{t},r_{t}=-\alpha^{-1},P(\cdot|x_{t}))$

19: $\texttt{IK}.\textrm{learn}((x_{t},y_{t}),A_{t},a_{t},w_{t})$.

##### Implementation Notes

In practice, $P(a|x)$ is known but the other probabilities are estimated.
$\hat{P}(a|y,x)$ is estimated online using maximum likelihood on the problem
predicting $a$ from $(x,y)$, i.e., on a data stream of tuples $((x,y),a)$. The
current estimates induce $\hat{\rho}(x,a,y)$ based upon the plug-in version of
Eq. 4. In this manner, the original data stream of $(x,a,y)$ tuples is
transformed into stream of $\left(x,a,\hat{r}=\hat{\rho}(x,a,y)\right)$ tuples
and reduced to a standard online contextual bandit problem.

As an additional complication, although $P(a|x)$ is known, it is typically a
good policy under which rewards are not rare (e.g., offline learning with a
good historical policy; or acting online according to the policy being learned
by the IGL procedure). Therefore, we use importance weighting to synthesize a
uniform action distribution $P(a|x)$ from the true action distribution.222When
the number of actions is changing from round to round, we use importance
weighting to synthesize a non-uniform action distribution with low rewards,
but we elide this detail for ease of exposition. Ultimately we arrive at the
procedure of Algorithm 1.

##  4 Empirical Evaluations

Evaluation Settings: Evaluation settings include simulation using a supervised
classification dataset, online news recommendation on Facebook, and a
production image recommendation scenario.

Abbreviations: Algorithms are denoted by the following abbreviations:
Personalized IGL for 2 latent states (IGL-P(2)); Personalized IGL for 3 latent
states (IGL-P(3)); Contextual Bandits for the Facebook news setting that
maximizes for emoji, non-like click-based reactions (CB-emoji); Contextual
Bandits for the Facebook news setting that maximizes for comment interactions
(CB-comment); Contextual Bandits for the production recommendation scenario
that maximizes for CTR (CB-Click).

General Evaluation Setup:  At each time step $t$, the context $x_{t}$ is
provided from either the simulator (Section 4.1, Section 4.2) or the logged
production data (Section 4.3). The learner then selects an action $a_{t}$ and
receives feedback $y_{t}$. In these evaluations, each user provides feedback
in exactly one interaction and different user feedback signals are mutually
exclusive, so that $y_{t}$ is a one-hot vector. In simulated environments, the
ground truth reward is sometimes used for evaluation but never revealed to the
algorithm.

###  4.1 Covertype IGL Simulation

To highlight that personalized IGL can distinguish between different user
communication styles, we create a simulated 2-state IGL scenario from a
supervised classification dataset. First, we apply a supervised-to-bandit
transform to convert the dataset into a contextual bandit simulation (Bietti
et al., 2021), i.e., the algorithm is presented the example features as
context, chooses one of the classes as an action, and experiences a binary
reward which indicates whether or not it matches the example label. In the IGL
simulation, this reward is experienced but not revealed to the algorithm.
Instead, the latent reward is converted into a feedback signal as follows:
each example is assigned one of $N$ different user ids and the user id is
revealed to the algorithm as part of the example features. The simulated user
will generate feedback in the form of one of $M$ different word ids. Unknown
to the algorithm, the words are divided equally into “good” and “bad” words,
and the users are divided equally into “normal” and “bizarro” users. “Normal”
users indicate positive and zero reward via “good” and “bad” words
respectively, while “bizarro” users employ the exact opposite communication
convention.

Figure 2: The proposed personalized IGL algorithm successfully disambiguates
both different user communication styles and different event semantics. See
Section 4.1 for details.

We simulated using the Covertype (Blackard & Dean, 1999) dataset with
$M=N=100$, and an (inverse kinematics) model class which embedded both user
and word ids into a 2 dimensional space. Fig. 2 demonstrates that both the
user population and the words are cleanly separated into two latent groups.
Additional results showcasing the learning curves for inverse kinematics,
reward and policy learning are shown in Section A.2 along with more detailed
description of the model parameters.

###  4.2 Fairness in Facebook News Recommendation

Personalized reward learning is the key to more fair recommender systems.
Previous work (Neophytou et al., 2022) suggests that inconsistent performance
in recommender systems across user subgroups arises due to differences in user
communication modalities. We now test this hypothesis in the setting of
Facebook news recommendation. Our simulations are built on a dataset
(Martinchek, 2016) of all posts by the official Facebook pages of 3 popular
news outlets (Fox News, The Huffington Post and TIME Magazine) that span the
political spectrum. Posts range from May to November 2016 and contain text
content information, as well as logged interaction counts, which include
comments and shares, as well as diverse click-based reactions (see Fig. 3).

Figure 3: Facebook click-based reactions: like, love, haha, wow, sad and angry
(image source: Meta ). The reactions allow users to engage with content using
diverse communication signals.

Constructing a hand-engineered reward signal using these feedbacks is
difficult, and Facebook itself came under fire for utilizing reward weights
that disproportionately promote toxic, low quality news. One highly criticized
iteration of the reward ranking algorithm treated emoji reactions as five
times more valuable than likes (Merrill & Oremus, 2021). Future iterations of
the ranking algorithm promoted comments, in an attempt to bolster “meaningful
social interactions” (Hagey & Jeff Horwitz, 2021). Our experiments evaluate
the performance of CB algorithms using these two reward functions, referring
to them as CB-emoji and CB-comment.

We model the news recommendation problem as a 3 latent reward state problem,
with readers of different news outlets as different contexts. Given a curated
selection of posts, the goal of the learner is to select the best article to
show to the reader. The learner can leverage action features including the
post type (link, video, photo, status or event) as well as embeddings of the
text content that were generated using pre-trained transformers (Reimers &
Gurevych, 2019). User feedback is drawn from a fixed probability distribution
(unknown to the algorithms) that depends on the user type and latent reward of
the chosen action. As an approximation of the true latent reward signal, we
use low dimensional embeddings of the different news outlets combined with
aggregate statistics from the post feedback to categorize whether the users
had a positive ($r=1$), neutral ($r=0$) or negative experience ($r=-1$) with
the post. This categorization is not available to the evaluated algorithms.
Finally, we implement IGL-P(3) with the angry reaction as a negative oracle to
disambiguate the positive and negative reward states.

(a) Average fraction of rewards that are positive

(b) Average fraction of rewards that are negative

Figure 4: IGL-P(3) uses personalized reward learning to achieve fair news
recommendations across diverse reader bases, while CB policies based on
rewards used in practice by Facebook perform inconsistently, with subsets of
users receiving both fewer high quality recommendations and more low quality
recommendations. Standard error on all averages shown is $<0.005$.

Fig. 4 shows the results of our online news recommendation experiments. While
the performance of both CB-emoji and CB-comment varies significantly across
the different reader groups, IGL-P(3) maintains relatively stable performance
for both positive and negative rewards. The CB algorithm that maximizes emoji
reactions achieves the best performance for TIME readers at the cost of very
bad performance for the Fox News and Huffington Post Readers. On the other
hand, the CB algorithm that maximizes comment engagement achieves best
performance with Fox News readers, however it still performs worse than
IGL-P(3). Finally, our simulations show that the CB-comment objective that was
introduced to decrease low quality news actually significantly _increased_ it
for the Fox News reader population. Additional details of this experiment,
including model parameters and learning curves are available in Section A.3.

###  4.3 Production Results

Algorithm | Clicks | Likes | Dislikes
---|---|---|---
IGL-P(3) | $[0.999,1.067,1.152]$ | $[0.985,1.029,1.054]$ | $[0.751,1.072,1.274]$
IGL-P(2) | $[0.926,1.005,1.091]$ | $[0.914,0.949,0.988]$ | $[1.141,1.337,1.557]$
Table 1: Relative metrics lift over a production baseline. The production baseline uses a hand-engineered reward function which is not available to IGL algorithms. Shown are point estimates and associated bootstrap 95% confidence regions. IGL-P(2) erroneously increases dislikes to the detriment of other metrics. IGL-P(3) is equivalent to the hand-engineered baseline. Algorithm | Clicks | Likes | Dislikes
---|---|---|---
IGL-P(3) | $[1.000,1.010,1.020]$ | $[1.006,1.026,1.049]$ | $[0.890,0.918,0.955]$
CB-Click | $[0.968,0.979,0.990]$ | $[0.935,0.959,0.977]$ | $[1.234,1.311,1.348]$
Table 2: Relative metrics lift over a CB policy that uses the click feedback
as the reward. Point estimates and associated bootstrap 95% confidence regions
are reported. IGL-P(3) significantly outperforms CB-Click at minimizing user
dissatisfaction. Surprisingly, IGL-P(3) even outperforms CB-Click with respect
to click feedback, providing evidence for the proposed latent state
model.444Data for Table 1 and Table 2 are drawn from different date ranges due
to compliance limitations, so the IGL-P(3) performance need not match.

Our production setting is a real world image recommendation system that serves
hundreds of millions of users. In our recommendation system interface, users
provide feedback in the form of clicks, likes, dislikes or no feedback. All
four signals are mutually exclusive and the user only provides one feedback
after each interaction. For these experiments, we use data that spans millions
of interactions. The production baseline is a contextual bandit algorithm with
a hand-engineered multi-task reward function which dominates approaches that
only use click feedback555The utility of multi-task learning for
recommendation systems is well-established, e.g., Chen et al. (2021); Lu et
al. (2018b); Chen et al. (2019).. Consequently, any improvements over the
production policy imply improvement over any bandit algorithm optimizing for
click feedback.

We implement IGL-P(2) and IGL-P(3) and report the performance as relative lift
metrics over the production baseline. Unlike the simulation setting, we no
longer have access to the user’s latent reward after each interaction. As a
result, we evaluate IGL by comparing all feedback signals. An increase in both
clicks and likes, and a decrease in dislikes, are considered desirable
outcomes. Table 1 shows the results of our empirical study. IGL-P(2) exhibits
an inability to avoid extreme _negative_ events. Although the true latent
state is unknown, IGL-P(2) is Pareto-dominated due to an increase in dislikes.
IGL-P(3) does not exhibit this pathology. These results indicate the utility
of a $3$ latent state model in real world recommendation systems. To clearly
illustrate the improvement of IGL-P(3) and provide an additional benchmark, we
also report that IGL-P(3) significantly outperforms a contextual bandit policy
CB-Click using the industry standard CTR reward in Table 2.

##  5 Related Work

###  5.1 Eliminating reward engineering

Online learning algorithms in supervised learning, contextual bandits,
reinforcement learning (Bishop & Nasrabadi, 2006; Kober et al., 2013; Zhou,
2015; Bzdok et al., 2018; Hoi et al., 2021; Brunton & Kutz, 2022; Alsubari et
al., 2022) all optimize over hand-crafted scalar rewards. Partial label
learning (PLL) approaches (Feng & An, 2019; Gong et al., 2022; Wang et al.,
2022) in supervised learning learn strategies when the underlying ground-truth
label are noisy. Branches of reinforcement learning like inverse reinforcement
learning (Fu et al., 2017; Arora & Doshi, 2021), imitation learning (Hussein
et al., 2017; Oh et al., 2018), robust reinforcement learning (Zhou et al.,
2021; Panaganti et al., 2022) exists that learn from historical data without
access to the underlying true reward models. However these methods require
expert demonstration data, are incapable of generalizing Chen et al. (2022) to
different feedback signals, and generally need expensive compute and enormous
sampling capabilities for learning. In this context, IGL (Xie et al., 2021;
2022) is a novel paradigm, where a learner’s goal is to optimally interact
with the environment with no explicit reward or demonstrations to ground its
policies. Our proposed approach to IGL, IGL-P(3) further enables learning of
personalized reward functions for different feedback signals.

###  5.2 Recommender Systems

Recommender systems are a well-studied field due to their direct link to
product revenues (Naumov et al., 2019; Steck et al., 2021). The rapid growth
of online content has generated interest in ML-based solutions (Li et al.,
2011) that are able to offer more diverse personalized recommendations to
internet users. Traditional vanilla recommendation approaches can be divided
into three types. Content-based approaches maintain representations for users
based on their content and recommend new content with good similarity metrics
for particular users (Balabanović & Shoham, 1997; IJntema et al., 2010; Kompan
& Bieliková, 2010; Lops et al., 2019; Argyriou et al., 2020; Javed et al.,
2021). In contrast, collaborative filtering approaches employ user rating
predictions based on historical consumed content and underlying user
similarities (Balabanović & Shoham, 1997; Schafer et al., 2007; Hu et al.,
2008; Argyriou et al., 2020; Steck & Liang, 2021). Finally, there are hybrid
approaches that combine the previous two contrasting approaches to better
represent user profiles for improved recommendations (Balabanović & Shoham,
1997; Funakoshi & Ohguro, 2000; Burke, 2007; Argyriou et al., 2020; Javed et
al., 2021). Our work is a significant departure from these approaches, in that
_we learn representations of user’s communication style_ via their content
interaction history for improved diverse personalized recommendations.

Our work is significantly different from these in the following ways: (i) we
propose a novel personalized IGL algorithm based on the inverse kinematics
strategy as described in Section 3, (ii) we leverage the capability of IGL
learning rewards based off of implicit and explicit feedback signals and avoid
the costly, inefficient, status quo process of reward engineering, and (iii)
we formulate and propose a new recommendation system based on the IGL
paradigm. The works that are closest to ours are Xie et al. (2021; 2022) which
introduce and solve for the IGL paradigm under different assumptions. However,
we propose a personalized IGL algorithm for recommendation systems with
improved reward predictor models, more practical assumptions on feedback
signals, and more intricacies described in Section 2.2.1.

##  6 Discussion

We evaluated the proposed personalized IGL approach (IGL-P) in three different
settings: (1) A simulation using a supervised classification dataset shows
that IGL-P can learn to successfully distinguish between different
communication modalities; (2) A simulation for online news recommendation
based on real data from Facebook users shows that IGL-P leverages insights
about different communication modalities to learn better policies and achieve
fairness with consistent performance among diverse user groups; (3) A real-
world experiment deployed in an image recommendation product showcases that
the proposed method outperforms the hand-engineered reward baseline, and
succeeds in a practical application.

This work assumes that users may communicate in different ways, but a given
user expresses (dis)satisfaction or indifference to all content in the same
way. This assumption was critical to deriving the inverse kinematics approach,
but in practice user feedback can also depend upon content (Freeman et al.,
2020). IGL with arbitrary joint content-action dependence of feedback is
intractable, but plausibly there exists a tractable IGL setting with a
constrained joint content-action dependence, we are very much interested in
exploring as part of future work. Furthermore, although we established the
utility of a three state model over a two state model in our experiments,
perhaps more than three states is necessary for more complex recommendation
scenarios.

Although we consider the application of recommender systems, personalized
reward learning can benefit any application suffering from a one-size-fits-all
approach. One candidate domain is the detection of underlying medical
conditions and development of new treatments and interventions. Systematic
misdiagnoses in subgroups with symptoms different than the “norm” are
tragically standard in medicine. IGL-P can empower medical practitioners to
address systematic inequality by tuning diagnostic tools via personalization
and moving past an oblivious diagnostic approach. Other potential applications
are learned brain-computer and human-computer interfaces (Xie et al., 2022).
One example is a self-calibrating eye tracker where people’s response to
erroneous agent control is idiosyncratic (Gao et al., 2021).

## References

  * Alsubari et al. (2022) S Nagi Alsubari, Sachin N Deshmukh, A Abdullah Alqarni, Nizar Alsharif, TH Aldhyani, Fawaz Waselallah Alsaade, and Osamah I Khalaf.  Data analytics for the identification of fake reviews using supervised learning.  _CMC-Computers, Materials & Continua_, 70(2):3189–3204, 2022.
  * Argyriou et al. (2020) Andreas Argyriou, Miguel González-Fierro, and Le Zhang.  Microsoft recommenders: Best practices for production-ready recommendation systems.  In _Companion Proceedings of the Web Conference 2020_ , pp. 50–51. Association for Computing Machinery, 2020.  ISBN 9781450370240.
  * Arora & Doshi (2021) Saurabh Arora and Prashant Doshi.  A survey of inverse reinforcement learning: Challenges, methods and progress.  _Artificial Intelligence_ , 297:103500, 2021.
  * Auer et al. (2002) Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire.  The nonstochastic multiarmed bandit problem.  _SIAM journal on computing_ , 32(1):48–77, 2002\.
  * Balabanović & Shoham (1997) Marko Balabanović and Yoav Shoham.  Fab: content-based, collaborative recommendation.  _Communications of the ACM_ , 40(3):66–72, 1997\.
  * Beel et al. (2013) Joeran Beel, Stefan Langer, Andreas Nürnberger, and Marcel Genzmehr.  The impact of demographics (age and gender) and other user-characteristics on evaluating recommender systems.  In _International Conference on Theory and Practice of Digital Libraries_ , pp. 396–400. Springer, 2013.
  * Bekker & Davis (2020) Jessa Bekker and Jesse Davis.  Learning from positive and unlabeled data: A survey.  _Machine Learning_ , 109(4):719–760, 2020.
  * Bietti et al. (2021) Alberto Bietti, Alekh Agarwal, and John Langford.  A contextual bandit bake-off.  _J. Mach. Learn. Res._ , 22:133–1, 2021.
  * Bishop & Nasrabadi (2006) Christopher M Bishop and Nasser M Nasrabadi.  _Pattern recognition and machine learning_ , volume 4.  Springer, 2006.
  * Blackard & Dean (1999) Jock A Blackard and Denis J Dean.  Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables.  _Computers and electronics in agriculture_ , 24(3):131–151, 1999.
  * Bouneffouf et al. (2020) Djallel Bouneffouf, Irina Rish, and Charu Aggarwal.  Survey on applications of multi-armed and contextual bandits.  In _2020 IEEE Congress on Evolutionary Computation (CEC)_ , pp. 1–8. IEEE, 2020.
  * Brunton & Kutz (2022) Steven L Brunton and J Nathan Kutz.  _Data-driven science and engineering: Machine learning, dynamical systems, and control_.  Cambridge University Press, 2022.
  * Burke (2007) Robin Burke.  Hybrid web recommender systems.  _The adaptive web_ , pp. 377–408, 2007.
  * Bzdok et al. (2018) Danilo Bzdok, Martin Krzywinski, and Naomi Altman.  Machine learning: supervised methods.  _Nature methods_ , 15(1):5, 2018.
  * Chen et al. (2021) Lei Chen, Jie Cao, Guixiang Zhu, Youquan Wang, and Weichao Liang.  A multi-task learning approach for improving travel recommendation with keywords generation.  _Knowledge-Based Systems_ , 233:107521, 2021.
  * Chen et al. (2022) Xiaocong Chen, Lina Yao, Xianzhi Wang, Aixin Sun, and Quan Z Sheng.  Generative adversarial reward learning for generalized behavior tendency inference.  _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * Chen et al. (2019) Zhongxia Chen, Xiting Wang, Xing Xie, Tong Wu, Guoqing Bu, Yining Wang, and Enhong Chen.  Co-attentive multi-task learning for explainable recommendation.  In _IJCAI_ , pp. 2137–2143, 2019.
  * Ekstrand et al. (2018) Michael D Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera.  All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness.  In _Conference on fairness, accountability and transparency_ , pp. 172–186. PMLR, 2018.
  * Elkan & Noto (2008) Charles Elkan and Keith Noto.  Learning classifiers from only positive and unlabeled data.  In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 213–220, 2008.
  * Feng & An (2019) Lei Feng and Bo An.  Partial label learning with self-guided retraining.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pp. 3542–3549, 2019.
  * Freeman et al. (2020) Cole Freeman, Hamed Alhoori, and Murtuza Shahzad.  Measuring the diversity of facebook reactions to research.  _Proceedings of the ACM on Human-Computer Interaction_ , 4(GROUP):1–17, 2020.
  * Fu et al. (2017) Justin Fu, Katie Luo, and Sergey Levine.  Learning robust rewards with adversarial inverse reinforcement learning.  _arXiv preprint arXiv:1710.11248_ , 2017.
  * Funakoshi & Ohguro (2000) Kaname Funakoshi and Takeshi Ohguro.  A content-based collaborative recommender system with detailed use of evaluations.  In _KES’2000. Fourth International Conference on Knowledge-Based Intelligent Engineering Systems and Allied Technologies. Proceedings (Cat. No. 00TH8516)_ , volume 1, pp. 253–256. IEEE, 2000.
  * Gao et al. (2021) Jensen Gao, Siddharth Reddy, Glen Berseth, Nicholas Hardy, Nikhilesh Natraj, Karunesh Ganguly, Anca D. Dragan, and Sergey Levine.  X2T: training an x-to-text typing interface with online learning from user feedback.  In _9th International Conference on Learning Representations, ICLR_ , 2021.
  * Gong et al. (2022) Xiuwen Gong, Dong Yuan, and Wei Bao.  Partial label learning via label influence function.  In _International Conference on Machine Learning_ , pp. 7665–7678. PMLR, 2022.
  * Grčar et al. (2005) Miha Grčar, Dunja Mladenič, Blaž Fortuna, and Marko Grobelnik.  Data sparsity issues in the collaborative filtering framework.  In _International workshop on knowledge discovery on the web_ , pp. 58–76. Springer, 2005.
  * Hagey & Jeff Horwitz (2021) Keach Hagey and Jeff Jeff Horwitz.  Facebook tried to make its platform a healthier place. it got angrier instead.  _The Wall Street Journal_ , 2021.  URL https://www.wsj.com/articles/facebook-algorithm-change-zuckerberg-11631654215.
  * Hofmann et al. (2012) Katja Hofmann, Fritz Behr, and Filip Radlinski.  On caption bias in interleaving experiments.  In _Proceedings of the 21st ACM international conference on Information and knowledge management_ , pp. 115–124, 2012.
  * Hoi et al. (2021) Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao.  Online learning: A comprehensive survey.  _Neurocomputing_ , 459:249–289, 2021.
  * Hu et al. (2008) Yifan Hu, Yehuda Koren, and Chris Volinsky.  Collaborative filtering for implicit feedback datasets.  In _2008 Eighth IEEE international conference on data mining_ , pp. 263–272. Ieee, 2008.
  * Hussein et al. (2017) Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne.  Imitation learning: A survey of learning methods.  _ACM Computing Surveys (CSUR)_ , 50(2):1–35, 2017\.
  * IJntema et al. (2010) Wouter IJntema, Frank Goossen, Flavius Frasincar, and Frederik Hogenboom.  Ontology-based news recommendation.  In _Proceedings of the 2010 EDBT/ICDT Workshops_ , pp. 1–6, 2010\.
  * Javed et al. (2021) Umair Javed, Kamran Shaukat, Ibrahim A Hameed, Farhat Iqbal, Talha Mahboob Alam, and Suhuai Luo.  A review of content-based and context-based recommendation systems.  _International Journal of Emerging Technologies in Learning (iJET)_ , 16(3):274–306, 2021.
  * Khan (2017) M Laeeq Khan.  Social media engagement: What motivates user participation and consumption on YouTube?  _Computers in human behavior_ , 66:236–247, 2017.
  * Kim et al. (2014) Youngho Kim, Ahmed Hassan, Ryen W White, and Imed Zitouni.  Modeling dwell time to predict click-level satisfaction.  In _Proceedings of the 7th ACM international conference on Web search and data mining_ , pp. 193–202, 2014.
  * Kober et al. (2013) Jens Kober, J Andrew Bagnell, and Jan Peters.  Reinforcement learning in robotics: A survey.  _The International Journal of Robotics Research_ , 32(11):1238–1274, 2013.
  * Kompan & Bieliková (2010) Michal Kompan and Mária Bieliková.  Content-based news recommendation.  In _International conference on electronic commerce and web technologies_ , pp. 61–72. Springer, 2010.
  * Langford & Zhang (2007) John Langford and Tong Zhang.  The epoch-greedy algorithm for contextual multi-armed bandits.  _Advances in neural information processing systems_ , 20(1):96–1, 2007.
  * Li et al. (2011) Lihong Li, Wei Chu, John Langford, and Xuanhui Wang.  Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms.  In _Proceedings of the Forth International Conference on Web Search and Web Data Mining, WSDM 2011, Hong Kong, China_ , pp. 297–306. ACM, 2011.
  * Lops et al. (2019) Pasquale Lops, Dietmar Jannach, Cataldo Musto, Toine Bogers, and Marijn Koolen.  Trends in content-based recommendation.  _User Modeling and User-Adapted Interaction_ , 29(2):239–249, 2019.
  * Lu et al. (2018a) Hongyu Lu, Min Zhang, and Shaoping Ma.  Between clicks and satisfaction: Study on multi-phase user preferences and satisfaction for online news reading.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pp. 435–444, 2018a.
  * Lu et al. (2018b) Yichao Lu, Ruihai Dong, and Barry Smyth.  Why i like it: multi-task learning for recommendation and explanation.  In _Proceedings of the 12th ACM Conference on Recommender Systems_ , pp. 4–12, 2018b.
  * Mansoury et al. (2020) Masoud Mansoury, Himan Abdollahpouri, Jessie Smith, Arman Dehpanah, Mykola Pechenizkiy, and Bamshad Mobasher.  Investigating potential factors associated with gender discrimination in collaborative recommender systems.  In _The Thirty-Third International Flairs Conference_ , 2020.
  * Martinchek (2016) Patrick Martinchek.  2012-2016 Facebook Posts.  https://data.world/martinchek/2012-2016-facebook-posts, 2016.
  * Merrill & Oremus (2021) Jeremy Merrill and Will Oremus.  Five points for anger, one for a ‘like’: How facebook’s formula fostered rage and misinformation.  _The Washington Post_ , 2021.  URL https://www.washingtonpost.com/technology/2021/10/26/facebook-angry-emoji-algorithm/.
  * (46) Meta.  Reactions now available globally.  _Facebook News_.  URL https://about.fb.com/news/2016/02/reactions-now-available-globally/.
  * Naumov et al. (2019) Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al.  Deep learning recommendation model for personalization and recommendation systems.  _arXiv preprint arXiv:1906.00091_ , 2019.
  * Neophytou et al. (2022) Nicola Neophytou, Bhaskar Mitra, and Catherine Stinson.  Revisiting popularity and demographic biases in recommender evaluation and effectiveness.  In _European Conference on Information Retrieval_ , pp. 641–654. Springer, 2022.
  * Nguyen et al. (2014) Tien T Nguyen, Pik-Mai Hui, F Maxwell Harper, Loren Terveen, and Joseph A Konstan.  Exploring the filter bubble: the effect of using recommender systems on content diversity.  In _Proceedings of the 23rd international conference on World wide web_ , pp. 677–686, 2014.
  * Oh et al. (2018) Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee.  Self-imitation learning.  In _International Conference on Machine Learning_ , pp. 3878–3887. PMLR, 2018.
  * Panaganti et al. (2022) Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh.  Robust reinforcement learning using offline data.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=AK6S9MZwM0.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019\.
  * Schafer et al. (2007) J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen.  Collaborative filtering recommender systems.  In _The adaptive web_ , pp. 291–324. Springer, 2007.
  * Scott (2021) Kate Scott.  You won’t believe what’s in this paper! Clickbait, relevance and the curiosity gap.  _Journal of pragmatics_ , 175:53–66, 2021.
  * Shin (2020) Donghee Shin.  How do users interact with algorithm recommender systems? The interaction of users, algorithms, and performance.  _Computers in Human Behavior_ , 109:106344, 2020.
  * Silveira et al. (2019) Thiago Silveira, Min Zhang, Xiao Lin, Yiqun Liu, and Shaoping Ma.  How good your recommender system is? A survey on evaluations in recommendation.  _International Journal of Machine Learning and Cybernetics_ , 10(5):813–831, 2019.
  * Steck & Liang (2021) Harald Steck and Dawen Liang.  Negative interactions for improved collaborative filtering: Don’t go deeper, go higher.  In _Fifteenth ACM Conference on Recommender Systems_ , pp. 34–43, 2021.
  * Steck et al. (2021) Harald Steck, Linas Baltrunas, Ehtsham Elahi, Dawen Liang, Yves Raimond, and Justin Basilico.  Deep learning for recommender systems: A netflix case study.  _AI Magazine_ , 42:7–18, Nov. 2021.  doi: 10.1609/aimag.v42i3.18140.  URL https://ojs.aaai.org/index.php/aimagazine/article/view/18140.
  * Wang et al. (2022) Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao.  Pico: Contrastive label disambiguation for partial label learning.  _arXiv preprint arXiv:2201.08984_ , 2022.
  * Wang et al. (2021) Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua.  Clicks can be cheating: Counterfactual recommendation for mitigating clickbait issue.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pp. 1288–1297, 2021\.
  * Wu et al. (2017) Qingyun Wu, Hongning Wang, Liangjie Hong, and Yue Shi.  Returning is believing: Optimizing long-term user engagement in recommender systems.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , pp. 1927–1936, 2017.
  * Xie et al. (2021) Tengyang Xie, John Langford, Paul Mineiro, and Ida Momennejad.  Interaction-Grounded Learning.  In _International Conference on Machine Learning_ , pp. 11414–11423. PMLR, 2021.
  * Xie et al. (2022) Tengyang Xie, Akanksha Saran, Dylan J Foster, Lekan Molu, Ida Momennejad, Nan Jiang, Paul Mineiro, and John Langford.  Interaction-Grounded Learning with Action-inclusive Feedback.  _arXiv preprint arXiv:2206.08364_ , 2022.
  * Yi et al. (2014) Xing Yi, Liangjie Hong, Erheng Zhong, Nanthan Nan Liu, and Suju Rajan.  Beyond clicks: dwell time for personalization.  In _Proceedings of the 8th ACM Conference on Recommender systems_ , pp. 113–120, 2014.
  * Zhou (2015) Li Zhou.  A survey on contextual multi-armed bandits.  _arXiv preprint arXiv:1508.03326_ , 2015.
  * Zhou et al. (2021) Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn.  Finite-sample regret bound for distributionally robust offline tabular reinforcement learning.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 3331–3339. PMLR, 2021.

##  Appendix A Appendix

###  A.1 Motivating the 3 State Model for Recommender Systems

Through a simulated experiment, we demonstrate that the traditional 2 latent
state reward model used by prior IGL methods (Xie et al., 2021; 2022) is not
sufficient for the recommender system scenario. We further illustrate that a 3
latent reward state model can overcome the limitations of the two latent state
model, and achieve personalized reward learning for recommendations.

Simulator Design. Before the start of each experiment, user profiles with
fixed latent rewards for each action are generated. The users are also
assigned predetermined communication styles, so the probability of emitting a
given signal conditioned on the latent reward remains static throughout the
duration of the experiment. For the available feedback, users can provide
feedback using five signals: (1) like, (2) dislike, (3) click, (4) skip and
(5) none. The feedback includes a mix of explicit (likes, dislikes) and
implicit (clicks, skips, none) signals. Despite receiving no human input on
the assumed meaning of the implicit signals, we will demonstrate that IGL can
determine which feedback are associated with which latent state. In addition
to policy optimization, IGL can also be a tool for automated feature
discovery. To reveal the qualitative properties of the approach, the simulated
probabilities for observing a particular feedback given the reward are chosen
so that they can be perfectly decoded, i.e., each feedback has a nonzero
emission probability in exactly one latent reward state. Production data does
not obey this constraint (e.g., accidental emissions of all feedback occur at
some rate): theoretical analysis of our approach without perfectly decodable
rewards is a topic for future work.

IGL-P(2). We implement Algorithm 1 for 2 latent states as IGL-P(2). The
experiment here shows the following two results about IGL-P(2): (i) it is able
to succeed in the scenario when there are 2 underlying latent rewards and (ii)
it can no longer do so when there are 3 latent states. Fig. 5 shows the
simulator setup used, where clicks and likes are used to communicate
satisfaction, and dislikes, skips and no feedback (none) convey (active or
passive) dissatisfaction.

(a) 2 latent state model

(b) 3 latent state model

Figure 5: Simulator settings for 2 state and 3 state latent model. In Fig. 5a,
$\mathbf{r=0}$ corresponds to anything other than the user actively enjoying
the content, whereas in Fig. 5b, lack of user enjoyment is split into
indifference and active dissatisfaction.

Fig. 6 shows the distribution of rewards for IGL-P(2) as a function of the
number of iterations, for both the 2 and 3 latent state model. When there are
only 2 latent rewards, IGL-P(2) consistently improves; however for 3 latent
states, IGL-P(3) oscillates between $r=1$ and $r=-1$, resulting in much lower
average user satisfaction. The empirical results demonstrate that although
IGL-P(2) can successfully identify and maximize the rare feedbacks it
encounters, it is unable to distinguish between satisfied and dissatisfied
users.

(a) Two latent states

(b) Three latent states

Figure 6: Performance of IGL-P(2) in simulated environment. Although IGL-P(2)
is successful with the 2 state simulator, it fails on the 3 state simulator
and oscillates between attempting to maximize $\mathbf{r=1}$ and
$\mathbf{r=-1}$.

IGL-P(3): Personalized Reward Learning for Recommendations. Since IGL-P(2) is
not sufficient for the recommendation system setting, we now explore the
performance of IGL-P(3). Using the same simulator as Fig. 5b, we evaluated
IGL-P(3). Fig. 7a demonstrates the distribution of the rewards over the course
of the experiment. IGL-P(3) quickly converged, and because of the partial
negative feedback for dislikes, never attempted to maximize the $r=-1$ state.
Even though users used the ambiguous skip signal to express dissatisfaction
80% of the time, IGL-P(3) was still able to learn user preferences.

In order for IGL-P(3) to succeed, the algorithm requires direct grounding from
the dislike signal. We next examined how IGL-P(3) is impacted by increased or
decreased presence of user dislikes. Fig. 7b was generated by varying the
probability $p$ of users emitting dislikes given $r=-1$, and then averaging
over 10 experiments for each choice of $p$. While lower dislike emission
probabilities are associated with slower convergence, IGL-P(3) is able to
overcome the increase in unlabeled feedback and learn to associate the skip
signal with user dissatifaction. Once the feedback decoding stabilizes,
regardless of the dislike emission probability, IGL-P(3) enjoys strong
performance for the remainder of the experiment.

(a) Ground truth learning curves, $\mathbf{P(\text{dislike}|r=-1)=0.2}$.

(b) Effect of varying $\mathbf{P(\text{dislike}|r=-1)}$.

Figure 7: Performance of IGL-P(3) in simulated environment. In Fig. 7a,
IGL-P(3) successfully maximizes user satisfaction while minimizing
dissatisfaction. Fig. 7b demonstrates how IGL-P(3) is robust to varying the
frequency of partial information received, although more data is needed for
convergence when “definitely bad” events are less frequent.

###  A.2 Additional Results for the Covertype IGL Simulation

Model Parameters For The Covertype IGL Simulation. For the Covertype IGL
simulation we trained both a CB and IK model (as shown in Algorithm 1). We use
online multiclass learning to implement the IK model, where the classes are
all the possible actions and the label is the action played. Both CB and IK
are linear logistic regression models implemented in PyTorch, trained using
the cross-entropy loss. Both models used Adam to update their weights with a
learning rate of $2.5e^{-3}$. The first layer of weights for both models had
$512$ weights that were initialized using a Cauchy distribution with
$\sigma=0.2$. The learning curves are shown in Figure 8b

(a) Reward learning.

(b) Accuracy of the inverse kinematics model used for the intermediate task of
action prediction.

Figure 8: Learning curves for the simulated experiment in Sec. 4.1.

###  A.3 Additional Results for Facebook News Simulation

Model Parameters For The Facebook News Simulation. For the Facebook news
simulation there were 2 CB models and an IK model. All models used Adam to
update their weights with a learning rate of $10^{-3}$, batch size 100, and
cross-entropy loss function. The reward learning curves are shown in Figures
9,10 and 11.

Figure 9: Reward learning curves for IGL-P(3) in Facebook news recommendation
experiment. Over time, IGL-P(3) learns the difference between the three states
and tries to minimize $r=0$ and $r=-1$ while maximizing $r=1$ across all
contexts.

Figure 10: Reward learning curves for CB-emoji in Facebook news recommendation
experiment. In two of the three different contexts (The Huffington Post
readers and TIME readers), CB-emoji tries to and succeeds in maximizes $r=-1$.
Figure 11: Reward learning curves for CB-comment in Facebook news
recommendation experiment. Although CB-comment performs successfully with TIME
readers, it maximizes the unhappiness of Fox News readers and promotes
indifference in The Huffington Post readers.