# Integrating Offline Reinforcement Learning with Transformers for Sequential
Recommendation

Xumei Xi  School of Operations Research and Information Engineering, Cornell
University  Work done while at Bloomberg.  Yuke Zhao  Bloomberg  Quan Liu
Bloomberg  Liwen Ouyang  Bloomberg  Yang Wu

<EMAIL_ADDRESS>{yzhao826, qliu268<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We consider the problem of sequential recommendation, where the current
recommendation is made based on past interactions. This recommendation task
requires efficient processing of the sequential data and aims to provide
recommendations that maximize the long-term reward. To this end, we train a
farsighted recommender by using an offline RL algorithm with the policy
network in our model architecture that has been initialized from a pre-trained
transformer model. The pre-trained model leverages the superb ability of the
transformer to process sequential information. Compared to prior works that
rely on online interaction via simulation, we focus on implementing a fully
offline RL framework that is able to converge in a fast and stable way.
Through extensive experiments on public datasets, we show that our method is
robust across various recommendation regimes, including e-commerce and movie
suggestions. Compared to state-of-the-art supervised learning algorithms, our
algorithm yields recommendations of higher quality, demonstrating the clear
advantage of combining RL and transformers.

##  1 Introduction

Recommender systems encompass almost all facets of everyday life, from
applications in the entertainment industry like movie recommendations [10] to
merchandise suggestions [35]. Especially with the emergence of big data,
navigating the boundless sea of information without the guidance of a
recommender can be challenging for customers. Therefore, designing a
recommender that produces relevant and helpful suggestions tailored for a
variety of user profiles is gaining more and more attention in recent years.
From a business perspective, a good recommender should also adapt to a user’s
evolving tastes over time, in order to maintain customer engagement.

Traditional latent factor based methods including matrix factorization [22,
16] aim to decompose the sparse user-item interaction matrix as a set of
feature vectors representing users and items. The more overlap one particular
user has with an item in the feature space, the more likely it is that we
recommend this item to the user. However, the static nature of matrix
factorization makes it unsuitable for predicting recommendations in a
sequential manner, since the method is oblivious to the chronological order of
the interaction history. More recently, viewing the recommendation process
from a sequential perspective has been adopted by several works [41, 21],
since it takes into account session-based user profiles in a more systematic
way compared to previous methods like matrix factorization. Whether the model
architecture is convolutional neural network (CNN) [41] or recurrent neural
network (RNN) [21], the methodologies are all considered supervised learning.
One key element is still missing here: modeling the dynamics of user-item
interaction. More specifically, sequential supervised learning methods
overlook the interactive and evolving nature of recommender systems. As a
result, they make decisions that may be optimal in the short term but
suboptimal in the long run.

To tackle the aforementioned issue, we utilize the framework of reinforcement
learning (RL), which has undergone remarkable development in the field of
interactive games [36] and robotic control [39]. The two main parts of RL are
the agent and the environment. The agent is the decision maker, while the
environment represents a potentially unknown set of rules applied to the
agent. The goal of the agent is to learn a decision rule, referred to as a
policy, that maximizes the reward accumulated over time, by repeatedly
interacting with the environment to collect data. Unlike supervised learning,
the RL agent learns by exploring the environment through trial and error.
Another important distinction is that the current action of the RL agent
influences not only the immediate results but also the long-term ones, as
pointed out in [40]. Consequently, RL is perhaps a more appropriate framework
than supervised learning for recommender systems, as it captures the
interactive and evolving aspects. There have been a growing number of works in
the recommender community that deploy RL to provide sequential recommendations
in different settings [27, 24, 47]. The focus of our paper is to integrate the
algorithmic framework of offline RL with the transformer architecture.

In this paper, we model the recommendation process as a sequential decision
making problem and utilize offline RL combined with transformers to obtain a
recommendation strategy that achieves good long-term performance. The offline
RL framework makes sure that we only infer from previously collected data
without the need to perform further online interaction. For interactive games,
it is natural to learn online by trial and error since all interactions can be
made in a controlled environment. For recommender systems, however, online
interactions can be risky and time-consuming. Revenue loss is also possible if
the deployed policy is not properly trained. On the other hand, building a
realistic and reliable simulation environment from the offline data to mimic
the real environment is highly non-trivial. As a result, an offline approach
is more direct and trustworthy compared to online and simulation-based
methods. We list our main contributions as follows.

  * •

We propose a fully offline RL-based recommendation framework, with an
efficient actor-critic algorithm under the hood. The specific algorithm we use
is Critic Regularized Regression (CRR) [46] that incorporates value
regularization in a computationally efficient way, to avoid bootstrapping
error commonly encountered in offline RL.

  * •

We harness the capacity of transformers to provide our RL algorithm with high-
quality, sequentially encoded item embeddings. The transformer model we use is
the distilled GPT-2 (DistilGPT2) [34], the smaller version of Generative Pre-
trained Transformer 2 (GPT-2) [33]. To the best of our knowledge, we are the
first to integrate offline RL with transformers in recommender systems.

  * •

To better integrate RL and transformers to facilitate learning, we propose a
two-stage training strategy. In the first stage, we train the policy via
supervised learning on a simple next-item prediction task. In the second
stage, we continue to train the policy (actor) alongside the critic that
estimates the Q function. The rationale of our two-stage strategy is two-fold.
The first stage (supervised learning) helps stabilize the training of RL to
converge faster111Our experiment shows that RL training without the first
stage will suffer from slow convergence or instability early on. The next-item
prediction task is both simple and effective to boost convergence. . The
second stage (RL) improves the policy obtained from the first stage and
alleviates overfitting. Through extensive experiments, we show that our method
significantly outperforms supervised learning methods that use sequential
modeling.

##  2 Related Work

In this section, we give an overview of various approaches proposed for
recommender systems. For the purpose of this paper, we categorize the methods
into two groups based on whether the method uses RL or not. Most non-RL
approaches are supervised learning methods, where the task is to estimate
unobserved interactions based on a sparse subset of observed interactions. On
the other hand, RL-based methods model the recommendation process as a Markov
decision process (MDP) and use various algorithms for inference.

###  2.1 Non-RL Methods

We first discuss supervised learning methods. Content-based filtering [28]
matches the characteristics of the items with the user profile, which
represents the user’s interests. Collaborative filtering methods recommend
items based on the preferences of users who share similar tastes. Compared to
content-based filtering, collaborative filtering relies on information about
similar users as guidance, making the users implicitly collaborate with one
another, whereas content-based filtering does not. A popular and powerful
example in collaborative filtering is matrix factorization [22, 16, 44]. The
sparsely observed user-item interaction matrix is decomposed as a set of
feature vectors for users and items. Based on this decomposition, we estimate
each unobserved interaction via the inner product of the corresponding user
feature vector and the item feature vector. The implicit assumption of matrix
factorization is that the user-item interaction satisfies a linear
relationship. Beyond linear interaction, we can use deep neural networks to
allow for more complex relations, as demonstrated in [15, 17].

In addition to supervised learning, the application of contextual multi-armed
bandits to recommender systems [26, 5, 50] provides a new direction to
explore, by considering the interactive nature of the system. To balance the
trade-off between exploration and exploitation, the method of Thompson
sampling is used in [5] and upper confidence bound is employed in [26]. The
work [50] proposes a dynamical context drift model on the reward function to
tackle the problem of time-varying preference.

###  2.2 RL-based Methods

Compared to the aforementioned methods, the benefit of adopting the RL
perspective is two-fold. Firstly, it incorporates an important dynamic that
the current recommendation has an impact on future ones. For example, if a
user likes one movie from a trilogy, the probability of them favoring the
other movies in the trilogy increases. Secondly, the RL agent is oriented
toward maximizing the long-term reward, rather than the immediate one. This
property makes RL more effective at keeping users engaged for a longer period.
Because of the dynamic nature of RL, we need to be aware of potential
distribution shift during training, which is central to many ideas in
algorithmic design. We will discuss more on this in Subsection 3.2.

In terms of specific RL algorithms, they can be classified into two groups:
model-based and model-free methods. Model-based methods first predict the
environment dynamics and then find the optimal policy, while model-free
methods directly learn the optimal policy without modeling the environment.
One example of model-based methods applied to recommender systems is [37]. As
the reconstruction of the dynamics could be costly, researchers also adopt
model-free methods. We can further divide the model-free methods into three
main categories: value-based, policy-based, and actor-critic methods. Value-
based methods find the optimal value function and calculate the optimal policy
based on the value function. Policy-based methods directly learn the optimal
policy, bypassing value function estimation. Actor-critic methods optimize
both the value function and the policy simultaneously. For recommender
systems, several papers [51, 24, 11, 12] use Deep Q-Network (DQN), a popular
value-based method. Deep Deterministic Policy Gradient (DDPG) is a powerful
policy-based method that has been adopted by [45, 27]. Actor-critic methods
are present in [8, 13, 49]. We comment that over the past few years, there
have been a growing number of works applying RL to recommender systems. For a
more comprehensive review, we refer the reader to a recent survey [1].

##  3 Preliminaries

In this section, we provide some preliminaries which set the stage for later
sections. We first describe the decision process formulation for recommender
systems in Subsection 3.1. Then, we introduce the notion of offline RL and the
associated challenges it presents in Subsection 3.2. Specifically, we discuss
the general framework of actor-critic methods for offline RL and the method we
choose. Lastly, we discuss the transformer architecture in Subsection 3.3.

###  3.1 Markov Decision Process (MDP) for Recommender Systems

We formulate the MDP as a tuple
$(\mathcal{S},\mathcal{A},r,P,\mu_{0},\gamma)$, where $\mathcal{S}$ is the
state space, $\mathcal{A}$ is the action space, $r$ is the immediate reward,
$P$ represents the state transition, $\mu_{0}\in\Delta(\mathcal{S})$ is the
initial state distribution and $\gamma$ is the discount factor. In the context
of recommender systems, we view the users as the environment providing
feedback and the recommender as the agent that aims to find a good
recommendation policy, by interacting with the users. In the sequel, we
explain what each element of $(\mathcal{S},\mathcal{A},r,P,\mu_{0},\gamma)$
represents in our setting.

The action space $\mathcal{A}$ for the agent corresponds to an item candidate
pool of appropriate size and each action $a$ is an item recommended to the
user. Since several RL algorithms struggle to scale to large and discrete item
spaces, item embeddings [27] are often adopted to convert the discrete item
space to a continuous one. Another way to handle large item space is to use
hierarchical structures [6, 11], which requires prior categorization of items.
The reward function $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ represents
the feedback provided by the user after receiving the recommendation. In
practice, the reward can be a numerical rating or a binary variable
representing click or purchase. For different datasets, the construction of
rewards might differ. We defer the specifics of the reward function to
Subsection 5.4.

It is less clear what the state should be. Ideally, the state should contain
information such as user preference and interaction history, based on which
the recommendation is made. There are many ways to design state
representation. The algorithm from [24] models the state as the user feature
vector obtained from matrix factorization. In the same spirit, the deep
reinforcement learning based recommendation framework (DRR) in [27] proposed
three different structures, based on concatenating feature interactions
between items, or between users and items. Other works that take on a more
sequential perspective directly model the state as the most recent items a
certain user has interacted with [47]. To further differentiate between the
items with positive feedback (high rating, purchase) and negative feedback
(low rating, skipped), one can simply include the complete feedback history
[11], categorize the items into two subgroups [51], or only include positive
items [13]. The aforementioned works all use a fixed sequence length for user
history, with padding if needed. A recent paper [2] proposes a novel sequence
length adaptation. In this paper, we model the state as a fixed length of the
most recent positive items from the user’s interaction history, due to its
simplicity and effectiveness.

Because our state only includes the positive items, the state transition
happens if and only if a positive rating is received. When the rating is
negative, the state remains unchanged. Suppose the current state is a vector
$\boldsymbol{s}_{t}=(s^{1}_{t},s^{2}_{t},\dots,s^{\ell}_{t})$ with $\ell\geq
1$ being the sequence length and each $s^{i}_{t}$ for all $i\in[\ell]$
corresponds to an item ranking from old to new. We have the following state
transition rule conditioned on the feedback of the current action $a_{t}$:

$\boldsymbol{s}_{t+1}=\begin{cases}(s^{2}_{t},\dots,s^{\ell}_{t},a_{t}),&\text{if
}r(\boldsymbol{s}_{t},a_{t})>0\\\
\boldsymbol{s}_{t},&\text{otherwise.}\end{cases}$ (1)

We note that the transition rule is deterministic conditioned on the immediate
reward. Namely, $P(\boldsymbol{s}_{t+1}|\boldsymbol{s}_{t})=1$, following the
formula (1).

Lastly, the discount factor $\gamma\in(0,1)$ governs how much we care about
future rewards. To see that, we define some functions related to the notion of
value. For a policy $\pi:\mathcal{S}\to\Delta(\mathcal{A})$, we define the
value function $V^{\pi}:\mathcal{S}\to\mathbb{R}$ as

$V^{\pi}(\boldsymbol{s})=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(\boldsymbol{s}_{t},a_{t})\;\Big{|}\;\boldsymbol{s}_{0}=\boldsymbol{s}\right],$
(2)

which represents the expected cumulative reward the agent collects if the
initial state is $\boldsymbol{s}_{0}=\boldsymbol{s}$ and the agent follows
policy $\pi$. The state-action value function (Q function) is similarly
defined as

$Q^{\pi}(\boldsymbol{s},a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(\boldsymbol{s}_{t},a_{t})\;\Big{|}\;(\boldsymbol{s}_{0},a_{0})=(\boldsymbol{s},a)\right],$
(3)

which is the expected cumulative reward when starting from the designated
state-action pair $(\boldsymbol{s},a)$. The goal of RL is to find the optimal
policy such that the value averaged over the initial state distribution
$\mu_{0}$ is maximized:

$\max_{\pi}\;J^{\pi}=\mathbb{E}_{\boldsymbol{s}\sim\mu_{0}}\left[V^{\pi}(\boldsymbol{s})\right].$

Directly from the definitions (2) and (3), we can see that the agent is more
myopic when $\gamma$ is small. When $\gamma$ is close to one, the agent takes
the future reward into account and the resulting optimal policy is more
farsighted. For future reference, we define the advantage function as

$A^{\pi}(\boldsymbol{s},a)=Q^{\pi}(\boldsymbol{s},a)-V^{\pi}(\boldsymbol{s}),$
(4)

which measures the potential benefit we might get if we update the current
policy $\pi$. When $A^{\pi}(\boldsymbol{s},a)>0$, it means the current policy
is not optimal and the action $a$ should be preferred at state
$\boldsymbol{s}$. When $A^{\pi}(\boldsymbol{s},a)<0$, it implies action $a$ is
less preferable at state $\boldsymbol{s}$.

###  3.2 Offline RL

In the setting of offline RL, we get a previously collected dataset
$\mathcal{D}=\\{(s,a,s^{\prime},r)\\}$, and further interaction with the
environment is prohibited. When generating the dataset $\mathcal{D}$, we can
assume the agent follows a behavior policy $\pi^{\beta}$. Since further
exploration is not allowed, there is a limit to which the agent can improve
upon the behavior policy. Intuitively, if there exists a region in the
environment that is not well explored during data collection, the agent
trained on offline data is not expected to perform well inside that particular
region. Therefore, the learning goal for offline RL is to output a policy that
is _optimal given the data_. We also point out that the presence of
distribution shift in offline RL needs careful treatment to avoid
overconfidence in less explored regions. In the sequel, we first give a high-
level review of actor-critic methods in offline RL. Afterward, we introduce
the specific algorithm we adopt for the recommender system problem.

####  3.2.1 Actor-Critic Methods for Offline RL

The actor-critic framework has two main components: the actor updates $\pi$ to
improve the policy and the critic approximates $Q^{\pi}$ to evaluate the
policy. We parameterize the Q function and the policy as
$Q_{\theta}(\cdot,\cdot)$ and $\pi_{\phi}(\cdot|\cdot)$. Based on the offline
dataset $\mathcal{D}$, we perform the Q update on $\theta$ and policy update
on $\phi$:

$\displaystyle\theta^{k+1}$
$\displaystyle\leftarrow\underset{\theta}{\operatorname{argmin}\;}\textstyle\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{D}}\Big{[}\big{(}r(s,a)+\gamma\mathbb{E}_{a^{\prime}\sim\pi_{\phi_{k}}(\cdot|s^{\prime})}[Q_{\theta^{k}}(s^{\prime},a^{\prime})]-Q_{\theta}(s,a)\big{)}^{2}\Big{]}$
(5) $\displaystyle\phi^{k+1}$
$\displaystyle\leftarrow\underset{\phi}{\operatorname{argmax}\;}\mathbb{E}_{s\sim\mathcal{D},a\sim\pi_{\phi}(\cdot|s)}\left[Q_{\theta^{k+1}}(s,a)\right].$
(6)

The Q update (5) performs temporal difference (TD) learning and the policy
update (6) sets the policy greedily according to the estimated Q function.

One of the fundamental difficulties of offline RL is the distribution shift.
Namely, the state-action visitation measure induced by the behavior policy can
be very different from that induced by the learned policy. As a result, the
policy evaluation can be highly off-policy, which in turn affects policy
improvement. If implemented naively, actor-critic methods often suffer from
extrapolation error when evaluating out-of-distribution state-action pairs.
Researchers have proposed different ideas to address the distribution shift.
We will review some of the ideas here. Interested readers may refer to a
survey [25]. Importance sampling [32] corrects the bias in the estimation
caused by distribution shift but requires good coverage of the behavior policy
and suffers from high variance. It has been applied to recommender systems by
[8]. The distribution shift can also be handled by adding constraints to the
policy update. For example, Bootstrapping Error Accumulation Reduction (BEAR)
proposed in [23] constrains the current policy to be close to the behavior
policy in terms of maximum mean discrepancy. Other works [38, 29] consider
regularizing the learned policy through KL divergence. For application in
recommender systems, the paper [13] adopts a trust region method using KL
divergence. The idea of using uncertainty estimation of the Q function can
regularize the target values and produce conservative estimators. Uncertainty
can be estimated by taking the variance across value estimations of an
ensemble of Q functions [48]. An example of using value penalized Q-learning
in recommender systems can be found in [12]. In this paper, we adopt the
algorithm called CRR from [46], which is a policy-regularized actor-critic
method. The regularization is achieved by constraining the KL divergence
between the learned policy and the behavior policy. We choose this method
because of its simple yet effective update. As we will see shortly, the KL
divergence constraint provides a computationally efficient modification to the
original actor-critic update, which helps us save memory and time when working
on gigantic datasets. At the same time, CRR pairs up well with transformers
and converges fast, as demonstrated in Section 5.

####  3.2.2 Critic Regularized Regression (CRR)

We introduce a recently proposed offline RL algorithm CRR [46] and explain the
reasons for choosing CRR for our application. CRR modifies the policy update
(6) in the standard actor-critic framework by value-filtered regression.
Specifically, we update the policy parameter according to

$\underset{\phi}{\operatorname{argmax}\;}\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[f(Q_{\theta},\pi_{\phi},s,a)\log\pi_{\phi}(a|s)\right],$
(7)

where the filter $f$ is a non-negative and scalar function that is
monotonically increasing in the first argument $Q_{\theta}$. The filter
decides what state-action pair we encourage the algorithm to update,
preventing $\pi$ to use the actions that are not present in the dataset. In
[46], they consider two different filters: binary filter
$f=\boldsymbol{1}[\hat{A}(s,a)>0]$ and exponential filter
$f=\exp\left(\hat{A}(s,a)/\beta\right)$. Here, $\boldsymbol{1}$ is the
indicator function, $\beta>0$ is a hyperparameter controlling the filter
strength, and $\hat{A}$ denotes the estimation of the advantage function
defined in (4). In our implementation, we estimate the advantage function by
averaging over $m$ realizations of the current policy:

$\hat{A}_{k}(s,a)=Q_{\theta^{k}}(s,a)-\frac{1}{m}\sum_{j=1}^{m}Q_{\theta^{k}}(s,a^{j}),\quad\text{with
}a^{j}\sim\pi_{\phi^{k}}(\cdot|s).$ (8)

From another perspective, the maximization program (7) is solving the maximum
weighted likeliho103od, where the weight is given by the filter function $f$.
By using the filter, we put more weight on state-action pairs that have a
larger advantage. As we discussed before, a large advantage indicates that the
policy needs more emphasis on the particular state-action pair. Compared to
behavior cloning ($f\equiv 1$), using a filter that varies depending on the
advantage estimation is more flexible and less conservative. Rather than
fitting the behavior policy, CRR can better distinguish between the good and
bad actions taken by the behavior policy. Since for the recommender system
problem, the dataset is often collected by carrying out a combination of
different policies, it is imperative for the algorithm to distinguish the good
actions from the bad ones. By using a filter that encourages good actions and
penalizes bad ones, CRR achieves exactly what we need. In our experiment, we
choose the exponential filter since it is softer and smoother compared to the
binary one. As a result, we do not run the risk of filtering out too many
samples. Therefore, the policy update is

$\phi^{k+1}\leftarrow\underset{\phi}{\operatorname{argmax}\;}\mathbb{E}_{\mathcal{D}}\left[\exp\left(\hat{A}_{k}(s,a)/\beta\right)\log\pi_{\phi_{k}}(a|s)\right].$
(9)

The Q update stays unchanged, i.e. we still use formula (5). There is also a
connection to policy regularization in CRR. As pointed out by both [46] and
[30], we comment that the policy update (9) is equivalent to optimizing the
Lagrangian with the form
$\mathcal{L}_{\beta}(\pi_{\phi^{k}})=\mathbb{E}_{\mathcal{D}}\left[\hat{A}_{k}(s,a)\right]+\beta\left(\varepsilon-
D_{KL}(\pi_{\phi^{k}}(\cdot|s)\;||\;\pi_{\beta}(\cdot|s))\right)$. The learned
policy is constricted to be close to the behavior policy in terms of KL
divergence. With this regularization, we avoid the potential over-
extrapolation error often observed in offline RL. From this perspective, the
CRR update (9) is essentially the closed-form solution of the KL-divergence
regularized policy iteration, which explains its efficacy at addressing
distribution shift.

###  3.3 Transformers

Introduced in [42], transformers initiated a major paradigm shift and have
become the go-to model for processing sequential data in natural language
processing (NLP) and computer vision (CV) tasks. Before the emergence of
transformers, recurrent neural networks (RNNs) such as long short-term memory
(LSTM) [19] and gated recurrent units (GRUs) [9] were the common models for
processing sequential data. The limitations of RNNs are two-fold. RNNs process
data in sequential order and the previous information is stored in a state
vector which is updated after reading the current token. As the sequence
progresses, the vanishing gradient problem is exacerbated, rendering the model
unable to scale to longer sequences and synthesize distant knowledge. On the
other hand, RNNs are difficult to parallelize since the data must be processed
one by one. The first limitation can be addressed by adding self-attention
mechanisms, which allow all previous states to be accessed. With a learned
weight that represents the relevance to a previous state, RNNs with attention
can retrieve related but far-off information which previously was
inaccessible. As a result, the attention mechanism improves the performance of
RNNs by integrating information over longer horizons. However, the second
limitation still persists for RNNs with attention: the sequential nature of
RNNs prohibits the utilization of modern GPUs with powerful parallel computing
capabilities. Transformers resolve the inefficiency of RNNs by getting rid of
the recurrent structure altogether. The success of transformers shows that the
self-attention mechanism works well all by itself without the recurrent
architecture. Transformers can be parallelized during training, opening the
door to pre-trained language models such as GPT (Generative Pre-trained
Transformer) [33, 4]. The pre-trained models have substantial generalization
potential in the sense that they possess strong zero-shot and few-shot
learning capacities after being trained on massive and diverse datasets like
the Wikipedia Corpus, WebText, and Common Crawl.

In our setting, we adopt the transformer architecture precisely because of its
ability to synthesize data over long horizons (which is a consequence of self-
attention) and the benefit of parallel computing (which is due to non-
recurrent architecture). Unlike some NLP tasks that require sequence-to-
sequence models, our goal is to predict a single item to recommend based on
previous interactions. Hence, we adopt an encoder-only model which takes the
user-item interaction history as input, embeds the history into a latent
space, position encodes the embeddings and at last, passes them through a
self-attention unit. The particular transformer we use is DistilGPT2 [34],
which is a distilled version of the smallest GPT-2 [33] with around 82 million
trainable parameters. The idea of knowledge distillation [18] is used to
compress a large model, referred to as the teacher, to a more compact one, the
student, that can reproduce the behavior of the teacher. By distilling the
GPT-2, the resulting model has similar performance with less memory cost and
faster computation.

Our proposed method, which combines RL and transformers, is not the first
attempt at this particular integration. The recently introduced decision
transformer [7] views the problem of RL as a conditional sequence model and
thereby harnesses the simplicity and scalability of the transformer
architecture. Another recent work [31] modifies the architecture of
transformers and significantly improves the stability and learning speed in
the regime of RL. Compared to our work, we comment that the aforementioned
papers did not address the recommender systems as we do.

##  4 Proposed Approach

In this section, we formally introduce our method that combines offline RL and
transformers to tackle the problem of sequential recommendation. The overall
framework consists of two stages. In the first stage, we only train the actor
(policy agent) by performing supervised learning on a next-item prediction
task, where the loss is computed via cross-entropy loss. The initial training
of the actor helps stabilize the RL algorithm used in the next stage to
converge faster. In the second stage, we run CRR introduced in Subsection
3.2.2 to train the actor and the critic (value agent) simultaneously until
convergence.

###  4.1 Algorithm

We denote the user-item interaction history as $\mathcal{H}=\\{(u,i,r)\\}$,
where $u$ is the user, $i$ is the item and $r$ is the feedback (e.g. rating,
click, purchase). Let $\mathcal{I}$ denote the full item space and $I$ denote
its cardinality. Based on the history $\mathcal{H}$, we pre-process the data
$\mathcal{H}$ to match the MDP formulation introduced in Subsection 3.1 and
get $\mathcal{D}=\\{(\boldsymbol{s},a,\boldsymbol{s}^{\prime},r)\\}$. Recall
that the current state $\boldsymbol{s}=(s^{1},s^{2},\dots,s^{\ell})$
represents the most recent interaction history, where $\ell\geq 1$ is a
hyperparameter for the fixed history length. The action $a$ is the next item
recommended to the user. The next state $\boldsymbol{s}^{\prime}$ is
transitioned according to (1). The reward $r$ is obtained by appropriate
shifting of the initial feedback. Details on the formulation of the reward are
deferred to Subsection 5.4.

Our method includes two stages. The first stage uses supervised learning to
train the policy on a next-item prediction task. The architecture of the
policy network is described in Figure 1. Specifically, we adopt DistilGPT2 as
the transformer part because of its capability of processing sequential data
as well as its computational advantage. Although GPTs and DistilGPTs were
initially designed for NLP tasks, their capability of processing sequential
data makes their architectures suitable for much broader applications. For
recommender systems, we treat each item as a word and a sequence of items with
which a user interacted as a sentence. In the first stage, the policy aims to
mimic the behavior observed in the dataset by minimizing the cross-entropy
loss defined below

$H(\boldsymbol{p}(s),a)=-\sum_{i\in\mathcal{I}}\boldsymbol{1}_{\\{i=a\\}}\log(p_{i})=-\log(p_{a}).$
(10)

In the above equation, we are evaluating the cross-entropy between the output
preference vector $\boldsymbol{p}(s)\in\Delta(\mathcal{I})$ and the true one-
hot vector $\\{\boldsymbol{1}_{\\{i=a\\}}\\}_{i\in\mathcal{I}}$. Since we want
to encourage recommendations with positive feedback, we only use the data with
$r>0$ for supervised training. We denote the learned policy from this step as
$\bar{\pi}$. The supervised learning step can be interpreted as a naive
behavior cloning that only focuses on the immediate next recommendation. We
have discovered that this is not enough to achieve long-term gain in the
system, since the goal is to output the next-item prediction without thinking
about the future.

Therefore, we construct a second stage in which we run offline RL with the
policy (actor) initialized exactly as the learned policy $\bar{\pi}$ from the
previous stage. The value network for the critic is similar to the actor,
except that the transformer is replaced by LSTM222Performance is similar for
LSTM and transformer as the critic. . The full architecture is plotted in
Figure 1. For computational efficiency, we extract the item embeddings from
the training of the first stage and freeze them during the second stage as we
discovered updating the embeddings did not help convergence. The first stage
of supervised learning benefits the second stage by providing a reasonable
starting point for RL. A good initialization not only stabilizes the
algorithmic trajectory but also improves learning efficiency. The second
stage, in turn, helps improve the myopic policy $\bar{\pi}$ trained from the
first stage, since the objective of RL is to maximize the cumulative reward
aggregated over future steps. We summarize our method in Algorithm 1. We will
discuss more on the choice of hyperparameters $b,\beta,\gamma$ and $\tau$ in
Subsection 5.4.

Data: $\mathcal{D}=\\{(s,a,s^{\prime},r)\\}$.

Result: Recommendation policy $\hat{\pi}$.

1 _First stage (supervised learning)_ \- train initial policy using next-item
prediction:
$\bar{\pi}=\text{argmin}_{\pi}\mathbb{E}_{(s,a,r)\sim\mathcal{D}}\left[H(\pi(s),a)\;|\;r>0\right].$

2 _Second stage (RL)_ \- run CRR with policy initialization
$\pi_{\phi}\leftarrow\bar{\pi}$. Randomly initialize critic $Q_{\theta}$.
Initialize target networks as:
$\pi_{\phi^{\prime}}\leftarrow\pi_{\phi},Q_{\theta^{\prime}}\leftarrow
Q_{\theta}$.

3 for  _$k=0$ to $K$_ do

4 Sample $\\{(s_{j},a_{j},s^{\prime}_{j},r_{j})\\}_{j=1}^{b}$ from
$\mathcal{D}$.

5 Calculate advantage estimator $\hat{A}$ according to (8).

6 Update actor $\phi$ with gradient
$-\nabla_{\phi}\frac{1}{b}\sum_{j}\log\pi_{\phi}(a_{j}|s_{j})\exp(\hat{A}(s_{j},a_{j})/\beta).$
Update critic $\theta$ with gradient
$\nabla_{\theta}\frac{1}{b}\sum_{j}\left[Q_{\theta}(s_{j},a_{j})-(r_{j}+\gamma\mathbb{E}_{a\sim\pi_{\phi^{\prime}}(s^{\prime}_{j})}Q_{\theta^{\prime}}(s_{j}^{\prime},a))\right]^{2}.$
Soft update target $\phi^{\prime}$ and $\theta^{\prime}$ with parameter
$\tau$:  $\displaystyle\phi^{\prime}$
$\displaystyle\leftarrow\tau\phi+(1-\tau)\phi^{\prime}$
$\displaystyle\theta^{\prime}$
$\displaystyle\leftarrow\tau\theta+(1-\tau)\theta^{\prime}.$

7 end for

Output final estimate $\hat{\pi}$.

Algorithm 1 CRR-Transformer |
---|---
Figure 1: Model architecture of the policy network (left) and value network
(right). For the policy network, the input is the state, which consists of the
most recent interaction history of length $\ell$. The output is a probability
distribution over the item space, which represents the probability of
recommending each item to the user. The item embeddings are vectors in
$\mathbb{R}^{d}$ and the state representation $\boldsymbol{z}$ is in
$\mathbb{R}^{I}$. MLP stands for multilayer perceptron, a fully connected
feedforward network for extracting state representations. For the value
network, the input is a state-action pair, and the output is an estimation of
the Q function. Item embeddings are directly from the training of the first
stage. Two layers of LSTM are added before MLP.

##  5 Experiments

In this section, we report the numerical results from implementing our
algorithm CRR-Transformer. We conduct experiments on two public datasets and
compare our method with other supervised learning algorithms.

###  5.1 Datasets

We conduct our experiments on MovieLens 25M [14] and Yoochoose [3]. The
MovieLens 25M (ML-25M) dataset contains 25 million 5-star movie ratings from
1995 to 2019. Each entry of rating records includes the user ID, movie ID,
rating (0.5–5.0), and timestamp. It also contains features of the movies,
computed via a machine learning algorithm based on user-contributed content
such as tags, ratings, and textual reviews. We will not use the features in
the dataset. Instead, we compute the features via the next-item prediction in
the first stage of our algorithm. We argue that our approach preserves
sequential information, which benefits learning in the second stage.
Additionally, the tag genome involves tag relevance scores for movies. As
described in [43], the tag genome was calculated via a machine learning
algorithm based on user-contributed content such as tags, ratings, and textual
reviews. Consequently, the tag genome encodes the level of certain properties
a movie exhibits, such as atmospheric, thought-provoking, realistic, etc.
Hence, we can directly use the tag relevance scores as item embeddings.
However, we discover that the item embeddings output by our supervised
learning step is much superior to the tag relevance scores. For ML-25M, our
training goal is to recommend movies for which the user gives high ratings.

The Yoochoose dataset contains six months of user activities for a large
e-commerce company that sells a variety of merchandise such as clothes, toys,
electronics, etc. The dataset includes both the clicks and purchases performed
by the users. Each entry, representing a click or purchase event, contains the
item ID, item price, quantity, session ID, and timestamp. Since our model only
focuses on recommending one item regardless of the price, we ignore the
quantity and price information in the implementation. For Yoochoose, we aim to
recommend items the user will click on and even purchase. We will see shortly
how the rewards are constructed for click and purchase. We provide some
statistics of the two datasets in Table 1.

Dataset | # of records | # of items
---|---|---
ML-25M | 25,000,095 | 62,423
Yoochoose | 1,177,769 | 52,739
Table 1: Dataset statistics.

In the ML-25M dataset, it is allowed to have multiple records for the same
user at the same time. However, this clearly violates our model assumption
under which only one item is recommended at a time. To mitigate this issue, we
delete the duplicate entries at the same timestamp and only keep the single
records. This conflict between the datasets and our modeling can be mostly
attributed to the fact that the timestamps are not sufficiently fine-grained.
For the ML-25M dataset, it is impossible for a person to watch and review two
movies at the same time. There must be a sequential order that is not
reflected by the current timestamp. We argue that a random shuffling of
simultaneous records would also work. We delete the overlapping entries for
efficiency and simplicity.

In terms of train-test split, there are two major ways. One is a chronological
split and the other is a random split by users. We discover that the first
chronological way typically leads to slightly worse performance, compared to
the second one. This phenomenon can be explained by different degrees of
distribution shift. When split based on users, the distribution shift from the
training set to the test set depends on how similar the users behave. If the
users are sampled in an i.i.d. way, then there is no distribution shift. On
the other hand, splitting chronologically forces us to do out-of-time
prediction, which is difficult due to the potential shift of user preference
over time. We decide to split the training set and test set according to the
timestamp, as this is the most natural way from the sequential perspective:
the training is done in the past and we try to predict recommendations in the
future. Specifically, for ML-25M, the earliest 99% of the data is used during
training. The next 0.2% in chronological order is the validation set and the
latest 0.8% is the test set. For Yoochoose, the training stage uses the early
98% of data. The next 0.2% is used for validation and the latest 1.8% is for
testing.

###  5.2 Evaluation Metrics

We discuss how to evaluate and compare different methods. We consider four
metrics in total, two of which are based on hit rate (HR) and the rest on
normalized discounted cumulative gain (NDCG).

HR is calculated by comparing the next recommendations with the true answer.
Since the true answer only exists when the reward is positive, we average over
all positive reward entries. Let $n^{+}$ denote the number of positive reward
entries in the validation or test set, namely
$n^{+}=\sum_{j\in[n]}\boldsymbol{1}_{\\{r_{j}>0\\}}$. From a candidate pool of
items, a prediction model selects the top $10$ items with the highest scores,
denoted by $\mathcal{I}_{j}=\\{i_{j}^{1},\cdots,i_{j}^{10}\\}$ for some data
entry $j$. The average HR@10 over data entries that have positive rewards is
defined as

$\operatorname{HR}@10=\frac{1}{n^{+}}\sum_{j:r_{j}>0}\boldsymbol{1}_{\\{a_{j}\in\mathcal{I}_{j}\\}},$
(11)

where $a_{j}$ is the true item liked by the user ($r_{j}>0$). For each sample,
the quality of the recommendation is determined by whether the true item is in
the top $10$ recommendations. The position of the true item in the
recommendation ranking does not matter. On the other hand, NDCG takes into
account the ranking of recommended items and therefore provides a more fine-
grained view of the recommender performance [20]. To understand how it works,
we first define the discounted cumulative gain in which the relevance of each
item is weighted based on its position

$\operatorname{DCG}@10=\frac{1}{n^{+}}\sum_{j:r_{j}>0}\sum_{h=1}^{10}\frac{\boldsymbol{1}_{\\{a_{j}=i_{j}^{h}\\}}}{\log_{2}(h+1)}.$

The higher the rank of the true item is, the larger DCG is. However, the
magnitude of DCG is dependent on the number of items recommended. To resolve
this issue, we can normalize DCG by the ideal DCG score when we recommend the
most relevant items first. Because there is only one relevant item in our
case, IDCG is binary and therefore NDCG coincides with DCG.

For each sample $j$, we construct two different action spaces:
$\mathcal{A}_{j}^{\text{rand}}$ and $\mathcal{A}_{j}^{\text{all}}$. To
decrease the space cardinality and thereby reduce the difficulty of
estimation, quite a few prior works [24, 47, 2] selected the action space
$\mathcal{A}_{j}^{\text{rand}}$ to be the union of the true answer
$\\{a_{j}\\}$ and a fixed number of randomly selected items unrated up till
this timestamp. In this paper, we set the number of randomly selected items to
be $100$. We refer to the HR@10 and NDCG@10 calculated by using
$\mathcal{A}_{j}^{\text{rand}}$ as HR@10-rand and NDCG@10-rand, respectively.
However, such metrics sometimes cannot truthfully reflect the performance of a
recommender system as the values can rely on the choices of random items in
the action space, especially when the size of the item universe is much larger
than the size of the random items. We believe a more accurate metric should
consider all unrated items to form the action space. Thus, we define
$\mathcal{A}_{j}^{\text{all}}$ as the candidate pool containing all the
previously unrated items. We use HR@10 and NDCG@10 to denote the metrics
obtained from using $\mathcal{A}_{j}^{\text{all}}$. Because
$\mathcal{A}_{j}^{\text{all}}$ can be significantly larger than
$\mathcal{A}_{j}^{\text{rand}}$, HR@10 and NDCG@10 are usually lower than
HR@10-rand and NDCG@10-rand, and also much harder to improve.

###  5.3 Baselines

Before we compare Algorithm 1 with other methods, we demonstrate the first
stage (supervised learning of policy) alone is already superior to several
state-of-the-art supervised learning methods for sequential recommendation. We
list the baseline methods we compare against as follows.

  * •

Caser [41] is a CNN-based approach to capture the order of interactions. A
sequence of recent items is embedded as an “image” in the time and latent
spaces. Sequential patterns can thus be learned with convolutional filters.

  * •

SASRec [21] is a self-attention based sequential model to allow both long-term
semantics and short-term preferences. It utilizes a point-wise feed-forward
network on top of the attention mechanism to incorporate non-linearity and to
consider interactions between different latent dimensions.

The learned policy from the first stage of Algorithm 1 is only an intermediate
result, but is already better than both Caser and SASRec (see Figure 2 and 3).
This gain is precisely due to the transformer’s remarkable ability to process
sequential data. As a result, we argue that combining RL with the transformer
architecture is far better than with other architectures such as CNN or RNN.
In the sequel, we refer to the first stage of Algorithm 1 as “Transformer-
only”, to distinguish it from the full two-stage method.

###  5.4 Experimental Setup

We first discuss the values of the hyperparameters. For both datasets, the
state sequence vector contains the most recent $30$ items. If the sequence
length is less than $30$, we pad the sequence from the beginning. We use the
Adam optimizer for all optimizations. The learning rate is chosen from
$\\{10^{-4},10^{-3},5\times 10^{-3}\\}$. The dropout ratio is chosen from
$\\{0,0.05,0.1,0.2\\}$. For the CRR algorithm in the RL stage, we use the
exponential filter with $\beta=1$ and clip the filter value with an upper
bound of $20$ for stability. The value network (critic) has two layers of LSTM
with hidden size of $1,024$. For advantage estimation (8), we sample $m=4$
actions and take the average333There are other ways of advantage estimation.
In our case, averaging works the best. We choose a small $m$ for efficiency
reasons. . Both the policy parameter and the Q parameter are updated in a soft
manner with $\tau=0.01$. The discount factor $\gamma$ is set as $0.6$ for
ML-25M and $0.9$ for Yoochoose, respectively. For more details, please refer
to Appedix A.

In terms of the immediate reward, we construct simple discrete reward
functions that effectively encourages convergence and stabilizes training. For
ML-25M, to encourage recommendations of relevant movies that receive high
ratings from the user, we give a positive reward for movies with ratings
greater than or equal to a threshold of $3.5$:

$r_{\text{ML}}=\begin{cases}1&\text{if rating}\geq 3.5\\\
0&\text{otherwise}.\\\ \end{cases}$ (12)

For Yoochoose, to promote purchase behaviors, we set a higher reward for
purchase events. Specifically, we have

$r_{\text{YC}}=\begin{cases}3&\text{if purchase}\\\ 1&\text{if
click}.\end{cases}$ (13)

In the first stage (supervised learning), we only train the policy and
evaluate the model performance on the validation set at the end of each epoch.
We then select the model at the epoch that achieves the highest validation
score (HR@10) and we pass the policy parameters on to the second stage as its
initialization. During the second stage, we evaluate the performance on the
validation set every $1,000$ batches and output the policy with the highest
validation score (HR@10). For ML-25M, the evaluation takes the next true movie
with a positive reward as the target. For Yoochoose, the evaluation takes the
next true item, whether clicked or purchased, as the target.

###  5.5 Results

In this subsection, we present and discuss the results.

####  5.5.1 Learning Curves

Figure 2: Learning curves for the first stage (supervised learning) and second
stage (RL) on ML-25M dataset. The evaluation metric is the HR@10 score
averaged over $3$ experiments. For the second stage, the policy is initialized
from the first stage checkpoint with the best validation score. Figure 3:
Learning curves for the first stage (supervised learning) and second stage
(RL) on the Yoochoose dataset. The evaluation metric is the HR@10 score
averaged over $3$ experiments. For the second stage, the policy is initialized
from the first stage checkpoint with the best validation score.

Figure 2 and 3 present the learning curves of the two stages in our CRR-
Transformer approach on validation data for ML-25M and Yoochoose,
respectively. We observe that for the first stage (supervised learning ),
Transformer-only reaches the best evaluation performance quickly. Even though
it suffers from overfitting later, the peak of Transformer-only’s HR@10 score
still outperforms baselines across all epochs, which justifies our choice of
Transformer-only as the initialization seed for the second stage. At the
second stage, the offline RL algorithm continues to improve the HR@10 score.

####  5.5.2 Test Results

Table 2: Test set results on HR@10, HR@10-rand, NDCG@10, NDCG@10-rand with the ML-25M dataset. We show the scores averaged over $3$ experiments as well as the standard deviations. Boldface highlights the highest average score.  | Caser | SASRec | Transformer-only | CRR-only | CRR-Transformer
---|---|---|---|---|---
HR@10 | $0.0275\pm 0.0020$ | $0.0270\pm 0.0012$ | $0.1303\pm 0.0146$ | $0.1627\pm 0.0035$ | $0.1840\pm 0.0010$
HR@10-rand | $0.3815\pm 0.0083$ | $0.4115\pm 0.0030$ | $0.8876\pm 0.0069$ | $0.8689\pm 0.0042$ | $0.8737\pm 0.0026$
NDCG@10 | $0.0128\pm 0.0005$ | $0.0131\pm 0.0007$ | $0.0676\pm 0.0078$ | $0.0889\pm 0.0022$ | $0.1019\pm 0.0008$
NDCG@10-rand | $0.2632\pm 0.0064$ | $0.2794\pm 0.0005$ | $0.6502\pm 0.0213$ | $0.6672\pm 0.0063$ | $0.6850\pm 0.0022$
Table 3: Test set results on HR@10, HR@10-rand, NDCG@10, NDCG@10-rand with the Yoochoose dataset. We show the scores averaged over $3$ experiments as well as the standard deviations. Boldface highlights the highest average score.  |  | Caser | SASRec | Transformer-only | CRR-only | CRR-Transformer
---|---|---|---|---|---|---
Purchase | HR@10 | $0.1733\pm 0.0036$ | $0.1948\pm 0.0051$ | $0.2274\pm 0.0110$ | $0.2424\pm 0.0110$ | $0.2559\pm 0.0065$
| HR@10-rand | $0.7659\pm 0.0117$ | $0.7202\pm 0.0130$ | $0.8216\pm 0.0093$ | $0.7479\pm 0.0099$ | $0.8253\pm 0.0061$
| NDCG@10 | $0.0906\pm 0.0015$ | $0.1083\pm 0.0027$ | $0.1199\pm 0.0059$ | $0.1288\pm 0.0055$ | $0.1374\pm 0.0041$
| NDCG@10-rand | $0.6333\pm 0.0133$ | $0.6003\pm 0.0081$ | $0.7114\pm 0.0108$ | $0.6455\pm 0.0106$ | $0.7237\pm 0.0032$
Click | HR@10 | $0.3236\pm 0.0035$ | $0.3801\pm 0.0015$ | $0.3792\pm 0.0054$ | $0.3386\pm 0.0052$ | $0.3973\pm 0.0063$
| HR@10-rand | $0.8258\pm 0.0020$ | $0.8117\pm 0.0084$ | $0.8553\pm 0.0081$ | $0.8065\pm 0.0050$ | $0.8580\pm 0.0006$
| NDCG@10 | $0.2013\pm 0.0017$ | $0.2350\pm 0.0024$ | $0.2371\pm 0.0068$ | $0.2034\pm 0.0025$ | $0.2541\pm 0.0047$
| NDCG@10-rand | $0.7252\pm 0.0046$ | $0.7291\pm 0.0056$ | $0.7715\pm 0.0074$ | $0.7127\pm 0.0034$ | $0.7775\pm 0.0021$

Table 2 and 3 summarizes the performance of all methods in terms of the four
evaluation metrics for the ML-25M and Yoochoose datasets, respectively. CRR-
only refers to running CRR without pre-trained transformer initialization,
i.e. we skip the first stage of Algorithm 1 and start the second stage with
random initialization for both the policy and the value networks. For a fair
comparison, we stop CRR-only at the same iteration as CRR-Transformer (first
stage and second stage combined). We observe that CRR-only underperforms CRR-
Transformer in both datasets. This could be caused by the slow convergence or
instability of RL training, which again demonstrates the necessity to
initialize the RL training with a pre-trained transformer that has effectively
learned the sequential information.

In Table 2 that shows the test results for ML-25M, we can see that
Transformer-only already outperforms both baselines across all scores, which
demonstrates the effectiveness of DistilGPT2 to model sequential information.
Based on the high-quality policy output by Transformer-only, our CRR-
Transformer approach is able to further improve upon it by taking long-term
interactions into account and differentiating between relevant and irrelevant
movies via the binary reward signal (12). The RL stage improves all scores
except for HR@10-rand. Since the action space $\mathcal{A}^{\text{all}}$ can
be significantly larger than $\mathcal{A}^{\text{rand}}$, predicting with
$\mathcal{A}^{\text{rand}}$ is much easier than with
$\mathcal{A}^{\text{all}}$. As a result, HR@10-rand may not accurately reflect
the quality of the recommendation. Moreover, CRR-Transformer achieves a higher
NDCG@10-rand average score than Transformer-only. This shows that even with
the same randomly selected universe, CRR-Transformer is more likely to
recommend relevant movies at top positions. Furthermore, CRR-Transformer also
results in much smaller standard deviations than Transformer-only for all
metrics, confirming the stability and robustness of our two-stage training
process.

For the Yoochoose dataset, the performance of all methods on both the purchase
and click tasks is shown in Table 3. Different from the learning curves, we
split the test set evaluation by event types (click or purchase). Transformer-
only achieves better average scores than the two baselines excluding HR@10 for
the click task, where SASRec performs slightly better over Transformer-only.
Nevertheless, CRR-Transformer achieves the highest average scores consistently
across all metrics. We argue that during the RL stage, the model learns to
focus more on purchase events that have higher rewards. We can see that CRR-
Transformer outperforms other methods by a larger margin on the purchase task,
which aligns with our recommendation goal, and also verifies that the agent
successfully learns from the additional long-term rewards.

In summary, the consistent improvement in both datasets demonstrates the
effectiveness and robustness of our approach across different recommendation
regimes and goals.

####  5.5.3 Sensitivity to Discount Factor

Figure 4: Sensitivity to the discount factor for ML-25M. The line and band
refer to the average and standard deviation of HR@10 and NDCG@10 over $3$
experiments. The discount factor $\gamma$ is chosen from
$\\{0.1,0.3,0.6,0.8,0.9,0.95\\}$.  Figure 5: Sensitivity to the discount
factor for Yoochoose. The blue line and band refer to the average and standard
deviation of HR@10 and NDCG@10 for the click task over $3$ experiments. The
green line and band refer to the average and standard deviation of HR@10 and
NDCG@10 for the purchase task over $3$ experiments. The discount factor
$\gamma$ is chosen from $\\{0.1,0.3,0.6,0.8,0.9,0.95\\}$.

Finally, we discuss the sensitivity of our results to the discount factor
$\gamma$. We focus on the test results of CRR-Transformer under different
values of $\gamma$. Figure 4 and 5 illustrate the sensitivity of HR@10 and
NDCG@10 to $\gamma$ for the ML-25M and Yoochoose datasets, respectively. The
tight range of metric scores shows that HR@10 and NDCG@10 are not overly
sensitive to the choice of $\gamma$. Furthermore, there are certain trends we
can observe. For ML-25M, both the average HR@10 and NDCG@10 improve when
$\gamma$ gradually increases from $0.1$ to $0.6$. However, the performance
saturates around $0.6$ and starts to drop as $\gamma$ continues to increase.
Not only does $\gamma=0.6$ yields the highest average score, it also results
in the smallest standard deviation, indicating that the algorithm is more
stable at $\gamma=0.6$. For Yoochoose, the click task and the purchase task
show different sensitivity to the discount factor. The purchase task has a
similar sensitivity trend as the ML-25M case: the score first increases and
then decreases. The only difference is that it achieves peak performance at
$\gamma=0.9$. The click task, on the other hand, gets better results when
$\gamma$ is relatively small. A small $\gamma$ indicates that the model
emphasizes more on the immediate reward rather than the long-term gains. Since
the purchase events usually involve longer interactions between the user and
items, compared to click events, this change in sensitivity trend indeed makes
sense: to achieve a successful purchase, the agent needs to be incentivized
with future rewards.

##  6 Discussion

We propose a fully offline RL algorithm for recommender systems, leveraging
the sequential benefit provided by the transformer architecture. Our results
demonstrate that our method is superior to several supervised learning
baselines, which confirms that RL combined with transformers is a powerful
tool. By combining offline RL and transformers, our model not only inherits
the sequential information learned by the transformers but also attains
substantially better long-term performance. However, there are still
limitations to the current approach: KL divergence style regularization might
be overly pessimistic when it comes to out-of-distribution actions. For future
work, we plan to find the balance between over-estimation (caused by optimism)
and under-estimation (caused by pessimism). To that end, we can improve the
offline RL algorithm by incorporating ideas from both online and offline RL
literature. For instance, we can borrow ideas from proximal policy
optimization for the policy updates. Furthermore, we can explore ways to
incorporate uncertainty into value updates. Another exciting direction is to
integrate RL with transformers in a more systematic way.

##  Appendix A More on experimental setup

In this appendix, we provide more details on the hyperparameters. For ML-25M,
the learning rates for Transformer-only, Caser, and SASRec are $0.0001$,
$0.0001$, $0.001$, respectively. The batch size is set as $128$. The item
embedding size is $50$ for the baselines and $768$ for DistilGPT2 as default.
For Caser, the model consists of two vertical convolution filters and two
horizontal filters with heights chosen from $\\{1,2,3,4,5\\}$. The dropout
ratio is $0.05$. For SASRec, following the original paper [21], we set the
self-attention as single-head. The number of self-attention blocks is set as
$2$. The item embedding size is $50$. The drop-out ratio is $0$. For
Yoochoose, the learning rates for Transformer-only, Caser, and SASRec are
$0.001$, $0.001$, $0.005$, respectively. The batch size is set as $128$ for
DistilGPT2 and $256$ for the baselines. The item embedding size is $64$ for
the baselines and $768$ for DistilGPT2 as default. For DistilGPT2, the
residual dropout is set as $0.2$. For Caser, we use one vertical convolution
filter and $16$ horizontal filters with heights chosen from $\\{2,3,4\\}$. The
dropout ratio is $0.1$. For SASRec, both the numbers of heads and self-
attention blocks are set as $1$. The drop-out ratio is $0.1$. For the CRR
algorithm in the RL stage for both datasets, the learning rate is set as
$0.0001$. We also use a cosine annealing scheduler with no learning rate
warmup. The minibatch size is set as $128$. The dropout ratio is set as $0.2$.

## References

  * [1] M. Mehdi Afsar, Trafford Crump, and Behrouz Far.  Reinforcement learning based recommender systems: A survey.  ACM Comput. Surv., jun 2022.
  * [2] Stefanos Antaris and Dimitrios Rafailidis.  Sequence adaptation via reinforcement learning in recommender systems.  In Fifteenth ACM Conference on Recommender Systems, RecSys ’21, page 714–718, New York, NY, USA, 2021. Association for Computing Machinery.
  * [3] David Ben-Shimon, Alexander Tsikinovsky, Michael Friedmann, Bracha Shapira, Lior Rokach, and Johannes Hoerle.  Recsys challenge 2015 and the yoochoose dataset.  In Proceedings of the 9th ACM Conference on Recommender Systems, RecSys ’15, page 357–358, New York, NY, USA, 2015. Association for Computing Machinery.
  * [4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * [5] Olivier Chapelle and Lihong Li.  An empirical evaluation of thompson sampling.  In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
  * [6] Haokun Chen, Xinyi Dai, Han Cai, Weinan Zhang, Xuejian Wang, Ruiming Tang, Yuzhou Zhang, and Yong Yu.  Large-scale interactive recommendation with tree-structured policy gradient.  In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019.
  * [7] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  Advances in neural information processing systems, 34:15084–15097, 2021.
  * [8] Minmin Chen, Can Xu, Vince Gatto, Devanshu Jain, Aviral Kumar, and Ed Chi.  Off-policy actor-critic for recommender systems.  In Proceedings of the 16th ACM Conference on Recommender Systems, pages 338–349, 2022.
  * [9] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder-decoder approaches.  arXiv preprint arXiv:1409.1259, 2014.
  * [10] S. Sathiya Devi and G. Parthasarathy.  A hybrid approach for movie recommendation system using feature engineering.  In 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT), pages 378–382, 2018.
  * [11] Mingsheng Fu, Anubha Agrawal, Athirai A. Irissappane, Jie Zhang, Liwei Huang, and Hong Qu.  Deep reinforcement learning framework for category-based item recommendation.  IEEE Transactions on Cybernetics, pages 1–14, 2021.
  * [12] Chengqian Gao, Ke Xu, Kuangqi Zhou, Lanqing Li, Xueqian Wang, Bo Yuan, and Peilin Zhao.  Value penalized q-learning for recommender systems.  In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 2008–2012, New York, NY, USA, 2022. Association for Computing Machinery.
  * [13] Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, and Yongfeng Zhang.  Towards long-term fairness in recommendation.  In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM ’21, page 445–453, New York, NY, USA, 2021. Association for Computing Machinery.
  * [14] F Maxwell Harper and Joseph A Konstan.  The movielens datasets: History and context.  Acm transactions on interactive intelligent systems (tiis), 5(4):1–19, 2015.
  * [15] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua.  Neural collaborative filtering.  Proceedings of the 26th International Conference on World Wide Web, Apr 2017.
  * [16] Balázs Hidasi and Domonkos Tikk.  Fast ALS-based tensor factorization for context-aware recommendation from implicit feedback.  In Machine Learning and Knowledge Discovery in Databases, pages 67–82, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.
  * [17] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk.  Session-based recommendations with recurrent neural networks, 2015.
  * [18] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network, 2015.
  * [19] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 9:1735–1780, 1997.
  * [20] Kalervo Järvelin and Jaana Kekäläinen.  Cumulated gain-based evaluation of ir techniques.  ACM Transactions on Information Systems (TOIS), 20(4):422–446, 2002\.
  * [21] Wang-Cheng Kang and Julian McAuley.  Self-attentive sequential recommendation.  In 2018 IEEE International Conference on Data Mining (ICDM), pages 197–206, 2018.
  * [22] Yehuda Koren, Robert Bell, and Chris Volinsky.  Matrix factorization techniques for recommender systems, 2009.
  * [23] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine.  Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction.  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * [24] Yu Lei and Wenjie Li.  Interactive recommendation with user-specific deep reinforcement learning.  ACM Trans. Knowl. Discov. Data, 13(6), oct 2019.
  * [25] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  arXiv preprint arXiv:2005.01643, 2020.
  * [26] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire.  A contextual-bandit approach to personalized news article recommendation.  Proceedings of the 19th international conference on World wide web - WWW ’10, 2010.
  * [27] Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, Yuzhou Zhang, and Xiuqiang He.  State representation modeling for deep reinforcement learning based recommendation.  Knowledge-Based Systems, 205:106170, 2020.
  * [28] Raymond J. Mooney and Loriene Roy.  Content-based book recommending using learning for text categorization.  In Proceedings of the Fifth ACM Conference on Digital Libraries, DL ’00, page 195–204, New York, NY, USA, 2000. Association for Computing Machinery.
  * [29] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans.  Trust-pcl: An off-policy trust region method for continuous control.  arXiv preprint arXiv:1707.01891, 2017.
  * [30] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine.  Awac: Accelerating online reinforcement learning with offline datasets, 2020.
  * [31] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al.  Stabilizing transformers for reinforcement learning.  In International conference on machine learning, pages 7487–7498. PMLR, 2020.
  * [32] Doina Precup.  Eligibility traces for off-policy policy evaluation.  Computer Science Department Faculty Publication Series, page 80, 2000.
  * [33] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [34] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  In NeurIPS EMC2 Workshop, 2019.
  * [35] J. Ben Schafer, Joseph Konstan, and John Riedl.  Recommender systems in e-commerce.  In Proceedings of the 1st ACM Conference on Electronic Commerce, EC ’99, page 158–166, New York, NY, USA, 1999. Association for Computing Machinery.
  * [36] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.  Mastering atari, go, chess and shogi by planning with a learned model.  Nature, 588(7839):604–609, 2020.
  * [37] Guy Shani, Ronen I. Brafman, and David Heckerman.  An mdp-based recommender system.  In Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, UAI’02, page 453–460, San Francisco, CA, USA, 2002\. Morgan Kaufmann Publishers Inc.
  * [38] Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller.  Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.  arXiv preprint arXiv:2002.08396, 2020.
  * [39] Aravind Srinivas, Michael Laskin, and Pieter Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning, 2020.
  * [40] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [41] Jiaxi Tang and Ke Wang.  Personalized top-n sequential recommendation via convolutional sequence embedding.  In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM ’18, page 565–573, New York, NY, USA, 2018\. Association for Computing Machinery.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [43] Jesse Vig, Shilad Sen, and John Riedl.  The tag genome: Encoding community knowledge to support novel interaction.  ACM Trans. Interact. Intell. Syst., 2(3), sep 2012.
  * [44] Jun Wang, Arjen P. de Vries, and Marcel J. T. Reinders.  Unifying user-based and item-based collaborative filtering approaches by similarity fusion.  SIGIR ’06, page 501–508, New York, NY, USA, 2006. Association for Computing Machinery.
  * [45] Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha.  Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation.  Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Jul 2018.
  * [46] Ziyu Wang, Alexander Novikov, Konrad Żołna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas.  Critic regularized regression.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  * [47] Liwei Wu, Shuqing Li, Cho-Jui Hsieh, and James Sharpnack.  Sse-pt: Sequential recommendation via personalized transformer.  In Fourteenth ACM Conference on Recommender Systems, RecSys ’20, page 328–337, New York, NY, USA, 2020. Association for Computing Machinery.
  * [48] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh.  Uncertainty weighted actor-critic for offline reinforcement learning.  In ICML, 2021.
  * [49] Teng Xiao and Donglin Wang.  A general offline reinforcement learning framework for interactive recommendation.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4512–4520, May 2021.
  * [50] Chunqiu Zeng, Qing Wang, Shekoofeh Mokhtari, and Tao Li.  Online context-aware recommendation with time varying multi-armed bandit.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 2025–2034, New York, NY, USA, 2016. Association for Computing Machinery.
  * [51] Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin.  Recommendations with negative feedback via pairwise deep reinforcement learning.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 1040–1048, New York, NY, USA, 2018. Association for Computing Machinery.