# Aligning GPTRec with Beyond-Accuracy Goals with Reinforcement Learning

Aleksandr V. Petrov  University of GlasgowUnited Kingdom
<EMAIL_ADDRESS>and  Craig Macdonald  University of
GlasgowUnited Kingdom<EMAIL_ADDRESS>
(2024)

###### Abstract.

Sequential recommender models work with chronologically ordered sequences of
interactions, to predict the next interaction in the sequence. Recently,
adaptations of Transformer models, such as BERT4Rec and SASRec, have achieved
state-of-the-art performance in the next-item prediction task according to
accuracy-based metrics, such as NDCG. These models treat items as tokens, and
then utilise a score-and-rank approach (_Top-K_ strategy), where the model
first computes item scores and then ranks them according to this score. While
this approach works well for accuracy-based metrics, it is hard to use it for
optimising more complex beyond-accuracy metrics such as diversity. Recently,
the GPTRec model, which uses a different (_Next-K_ strategy) has been proposed
as an alternative to the Top-K models. In contrast with traditional Top-K
recommendations, Next-K generates recommendations item-by-item and, therefore,
can account for complex item-to-item interdependencies important for the
beyond-accuracy measures. However, the original GPTRec paper focused only on
accuracy in experiments, and did not address how to optimise the model for
complex beyond-accuracy metrics. Indeed, training GPTRec for beyond-accuracy
goals is challenging, because the interaction training data available for
training recommender systems is typically not aligned with beyond-accuracy
recommendation goals. To solve the misalignment problem, we train GPTRec using
a 2-stage approach: in the first stage, we use a teacher-student approach to
train GPTRec, mimicking the behaviour of traditional Top-K models; at the
second stage, we use Reinforcement Learning to align the model for beyond-
accuracy goals. Our proposed training scheme, in principle, can align the
model with any measurable recommendation metric. In particular, we experiment
with increasing recommendation diversity and reducing popularity bias. Our
experiments on two datasets show that in 3 out of 4 cases, GPTRec’s Next-K
generation approach offers a better tradeoff between accuracy and secondary
metrics than classic greedy re-ranking techniques. For example, when compared
to BERT4Rec, it can simultaneously improve NDCG@10 by 8.8% while having 8.6%
lower popularity bias when fine-tuned to reduce popularity bias. ††margin:

††journalyear: 2024††copyright: acmlicensed††conference: The 2nd Workshop on
Recommendation with Generative Models; May 13–17, 2024; Singapore,
Singapore††booktitle: The 2nd Workshop on Recommendation with Generative
Models (GenRec@WWW ’24), May 13–17, 2024, Singapore, Singapore

##  1\. Introduction

†† Accepted for presentation at the 2nd Workshop on Recommendation with
Generative Models, in conjunction with the ACM Web Conference 2024, May 13 –
17, 2024,
Singapore, Singapore.
©Copyright held by the authors

Sequential recommendation models utilise ordered sequences of user-to-item
interactions to produce recommendations. State-of-the-art sequential models
usually employ a score-and-rank approach to generate recommendations (known as
the Top-K strategy (Petrov and Macdonald, 2023a)). The problem with this
method is that items are scored independently, and similar items are likely to
have similar scores, limiting the method’s ability to optimise metrics beyond
accuracy. For example, many researchers argue that recommendation diversity is
one of the key components of good recommender systems (Kunaver and Požrl,
2017; Antikacioglu et al., 2019; Zhou et al., 2010). However, under the Top-K
strategy, the models’ outputs are likely to be dominated by similar types of
items, compromising the user experience. A recent paper (Petrov and Macdonald,
2023a) proposed GPTRec model, which uses an alternative _Next-K strategy_ ,
where recommendations are generated item-by-item. In that case, when the model
generates a recommendation in position $i$, it already knows what items are
recommended in position $1..i-1$, and may adjust the result accordingly. The
main challenge with the Next-K strategy is model training. Indeed, standard
supervised learning algorithms do not work well for training a generative
model, as they require a source of _good_ recommendations, which isn’t
typically available. For example, to train a model to generate diverse and
relevant recommendations, we would need a training set of diverse and relevant
recommendations for a large number of users. However, to obtain such a set, we
would need a recommendation model with at least the same performance as the
generative model we intend to train, which we assume is unavailable – indeed,
the only available data training data in a typical recommendation scenario is
historic user-item interactions. A similar problem exists in the training of
language models, in that the available training data is not aligned with the
desired task. However, this problem has recently been addressed by the
Reinforcement Learning With Human Feedback (RLHF) approach (Ouyang et al.,
2022). Inspired by RLHF, we propose a reinforcement learning-based approach,
where we sample recommendations from the current version of the recommendation
model (”Actor”), use a separate model to estimate the quality of produced
recommendations (”Critic”), and then use the Critic’s output to improve the
main recommendation model. This approach can optimise for almost any metric
measured on a full recommendation list. In particular, we apply this approach
to optimise recommendation diversity and reduce popularity bias. ††margin:

In our proposed 2-stage training scheme, at the pre-training stage, the model
learns to mimic the teacher model, and at the fine-tuning stage, we align it
with beyond-accuracy goals using Reinforcement Learning. We use the pre-
training/fine-tuning approach with GPTRec on two datasets, namely MovieLens-1M
(Harper and Konstan, 2015) and Steam-2M (a smaller version of the Steam (Kang
and McAuley, 2018) dataset). Our experiments show that GPTRec in generative
mode matches the performance of state-of-the-art BERT4Rec when tuned for
accuracy only. However, when the model is tuned for more intricate goals, such
as increasing diversity or decreasing popularity bias, GPTRec provides a
better tradeoff between accuracy and one of these goals, compared to applying
BERT4Rec with greedy re-ranking techniques such as using Maximum Marginal
Relevance (MMR) (Carbonell and Goldstein, 1998) for improving diversity. For
example, on Steam-2M, while exhibiting the same amount of diversity (measured
by the Intra-List Distance (ILD) (Antikacioglu et al., 2019)), GPTRec achieves
10% higher NDCG@10 than BERT4Rec with MMR.

Following (Deffayet et al., 2023), for practical reasons, we limit the scope
of this paper to datasets with a few thousand items in the catalogue. While we
acknowledge the dataset size as a limitation of this research, we argue that
there are many industrial applications where the catalogue size does not
exceed this size. Examples of such applications include but are not limited to
single-brand online shops (e.g. apple.com), news agency websites (it only
makes sense to recommend news no older than a few days old), genre
recommendation in music (where the system recommends specific genres instead
of concrete songs) and many others. In practice, it is hard to train
transformer-based models for recommendations with more than a few hundred
thousand items due to memory and computational constraints (Petrov and
Macdonald, 2022a). Moreover, in our case, we have to keep several copies of
the model in memory as we use multiple asynchronous processes for training;
therefore, we limit the scope of this research to a smaller number of items to
be able to experiment quickly on limited hardware; Furthermore, the success of
InstructGPT (Ouyang et al., 2022), which used a similar reinforcement learning
approach to train a model with more than 100 billion parameters allows us to
argue that the same approach will work on a much larger scale, but that
requires access to a large amount of industry-grade GPUs, which is beyond our
available budget111While in this work, we experiment with the datasets with a
moderate number of items, we note that scaling is possible by using sub-item
representation. However, scaling transformer models for recommendation with
large catalogues is an orthogonal research direction, which has been addressed
in several recent works (Petrov and Macdonald, 2023a, 2024, b). In industrial
settings, these limitations can also be mitigated using engineering solutions,
such as storing the model in shared memory, but these optimisations are beyond
the scope of this paper..

In short, we summarise the contributions of this paper as follows: (i) we
propose a supervised pre-training/reinforcement fine-tuning approach for
training generative recommender models for complex recommendation goals; (ii)
we show that when optimised for accuracy only, it can match state-of-the-art
transformer-based models, such as BERT4Rec while generating recommendations
using the Next-K technique; (iii) we show that GPTRec can be optimised for
complex recommendation goals, such as increased diversity and decreased
popularity bias, and an optimised version of GPTRec provides a better tradeoff
between accuracy and secondary metrics than state-of-the-art BERT4Rec with
greedy re-ranking.

The rest of the paper is organised as follows: Section 2 covers related work
on alignment in language models and sequential recommendation models. Section
3 describes the proposed two-stage training process for GPTRec; Sections 4 and
5 contain the experimental evaluation of GPTRec; Section 6 contains final
remarks.

††margin:

##  2\. Related work

In this section we provide a brief overview of related work on alignment in
language models, as well as describe GPTRec (Petrov and Macdonald, 2023a)
model, which we use in our experiments.

###  2.1. The Misalignment Problem and Reinforce- ment Learning with Human
Feedback

One of the big problems of state-of-the-art generative language models is the
problem of _misalignment_ between training data and the actual desired
outcome. A typical desired use-case for generative models is to work in a
”chat-bot” mode and follow the instructions of the user. However, the training
data for such interactive communication and instruction-following is limited
and very expensive to gather. In contrast, ”regular” text data is abundant and
practically unlimited in the modern days: texts can be crawled from websites,
books, public datasets, etc. Earlier generative models, such as GPT-2 (Radford
et al., 2019) and T5 (Raffel et al., 2020), were trained only using ”regular”
training data and therefore had limited use for building interactive
applications.

Notably, Oyang et al. (Ouyang et al., 2022) proposed a solution for the
misalignment problem using the idea of ”Reinforcement Learning with Human
Feedback” (RLHF). The authors proposed a three-step solution for the problem,
which we now briefly describe step-by-step: (Step 1) train a language model
for the next token prediction task using a ”regular” training dataset and
fine-tune it using a small amount of available ”high-quality” dialogue data
that was generated with human labellers; (Step 2) use the model from Step (1)
to generate different possible outputs for the same model input and ask the
labellers to compare generated outputs. These labels are then used to train a
_reward_ model, which essentially associates a numerical value with the
generated output: the larger the reward, the better the output, according to
the model; (Step 3) use a reinforcement learning approach from step (1) to
fine-tune the language model using the Proximal Policy Optimisation (Schulman
et al., 2017) algorithm.

††margin:

We argue that a similar misalignment problem arises in the recommender systems
when the recommendation objectives include beyond-accuracy components, such as
diversity. Indeed, sequential recommendation datasets typically contain a
large number of user-item interaction sequences that can be used for training
sequential recommendation models, such as BERT4Rec or SASRec. Hence, as the
main goal of these models is to predict the next item in the sequence of
interactions, the training data is _aligned_ , with the target goal of
accurately predicting the next item, and these models can achieve state-of-
the-art results when trained appropriately using different training
objectives, such as sequence shifting, item masking or recency sampling of
sequences. However, the datasets typically do not contain ”perfect” model
outputs for beyond-accuracy goals, such as diversity, and, therefore,
supervised training for such goals is challenging due to data scarcity.
Usually, there is no source of high-quality training data for beyond-accuracy
metrics; the typical solution involves training a model for accuracy (for
which training data is abundant) and then re-rank using a heuristic approach,
such as MMR (Carbonell and Goldstein, 1998) or Serendipity Oriented Greedy
(Kotkov et al., 2020). While these heuristics somewhat allow to mitigate the
misalignment problem, there is no evidence that such heuristics lead to a
solution that may be close to optimal. Therefore, inspired by the success of
the RLHF for language models, we propose to apply a similar approach to
recommender systems. The difference between our approach and the RLHF is that
we assume that the target quality metric is known and measurable (in our
experiments, we use a linear combination between accuracy (measured by NDCG)
and a secondary objective, such as diversity (for example, measured by Intra-
List-Distance (Antikacioglu et al., 2019)). This simplifies our approach, as
we do not need human labellers to generate high-quality recommendations and
compare generated recommendations. Instead, we directly use a known quality
measure as a reward and train the model to maximise it using reinforcement
learning.

###  2.2. Transformers for sequential recommendation

Sequential Recommendation is usually cast as the _next item prediction task_.
Formally, given the sequence of historical user-item interactions
$\\{h_{1},h_{2}...h_{n}\\}$, the goal of the recommender system is to predict
the next item $h_{n+1}$. This problem is similar to the _next token
prediction_ task common in language models, and therefore, adaptations of
language models are frequently used for sequential recommendation. In
particular, Transformer (Vaswani et al., 2017) based model, such as SASRec
(Kang and McAuley, 2018), BERT4Rec (Sun et al., 2019) and gSASRec (Petrov and
Macdonald, 2023b) achieved state-of-the-art results for sequential
recommendation. These models employ the Top-K recommendation strategy, which,
as we argue in 1, is problematic for the beyond-accuracy metrics.

A recent paper (Petrov and Macdonald, 2023a) proposed GPTRec, a generative
sequential recommendation model that utilises the GPT-2 architecture (Radford
et al., 2019), which in turn is based on the Decoder part of the Transformer
model (Vaswani et al., 2017). In contrast with most existing Transformer-based
models, such as BERT4Rec, GPTRec uses the Next-K recommendation approach, i.e.
it produces recommendations iteratively (or _autoregressively_ , i.e. item-by-
item) and, therefore, potentially is more suitable for beyond-accuracy
optimisation.

Although GPTRec does not directly use pre-trained weights of its backbone
GPT-2 model, it still shares many similarities with GPT. In particular, as we
argued in Section 2.1, there is a misalignment problem: the data available for
training usually is not aligned well with the desired recommendation
objectives. In the next section, we describe how we solve this problem using
the 2-stage pre-training/fine-tuning approach.

##  3\. Training GPTRec for beyond-accuracy goals

As discussed in Section 2.2, training a version of GPTRec model for Next-K
generation that is aligned with the beyond-accuracy goals is a challenging
task, as training high-quality training samples are rarely available. In this
section, we describe a two-stage process that is capable of addressing this
challenge. To do this, we first describe the training objectives in Section
3.1. We then describe the 2-stage pre-training/fine-tuning approach we use to
train the model in Section 3.2. Finally, we describe an efficient process
decomposition for the reinforcement learning fine-tuning in Section 3.3.

††margin:

###  3.1. Training Objectives

In this section, we formalise the optimisation problem for training GPTRec for
beyond-accuracy goals. Consider a sequence of user-item interactions
$h=\\{h_{1},h_{2}..h_{n}\\};h_{i}\in I$, where $I$ is the catalogue of all
available items . The goal of a recommender system is to _generate_ a list of
recommendations $g=\\{g_{1},g_{2}...g_{K}\\};g_{i}\in I$, which maximises an
_effectiveness measure_ $R(h,g)$. $R$ may have a complex structure and depend
not only on accuracy but also on item popularity, diversity, etc. Note that
$R$ depends on the whole recommendation list, and therefore, it can not be
optimised using a pointwise loss functions, such as Binary Cross-Entropy. It
also is not required to be differentiable, limiting our ability to optimise
$R$ using gradient-based methods. Therefore, instead of relying on standard
supervised learning, we propose to optimise it using a reinforcement learning
approach, which we describe in Section 3.2.

Without loss of generality, we can say that metric $R$ can be decomposed into
the sum of _immediate rewards_ (a part of overall effectiveness measure $R$,
which can be calculated after generating each individual item in the
recommendation list):

(1) $\displaystyle
R(h,g)=\sum_{i=1}^{K}\mathrm{r}(h,g_{1},g_{2},...g_{i})=\sum_{i=1}^{K}\mathrm{r}(h,g^{(i)})$

where $g^{(i)}$ denotes a partially generated list of recommendations up to
position $i$: $g^{(i)}=\\{g_{1},g_{2},..g_{i}\\}$.

Even if the metric is only meaningful over the whole list of items, we can say
that the immediate reward is 0 for each step, except for the last item in the
list; this is also known as _delayed reward_ :

(2) $\displaystyle\mathrm{r}(h,g^{(i)})=\begin{cases}0;i\in 1..K-1\\\
R(h,g);i=K\end{cases}$

††margin:

However, in many cases, the immediate reward may have a non-zero value even
for a partially generated list. For example, if accuracy is part of the
overall metric, the immediate reward may be positive when the model generates
a relevant item; for decreasing popularity bias, the immediate reward may be
negative when generating a popular item. Optimising the model for immediate
reward is easier, as it requires the model to do less planning, which is
generally a challenging task (Mnih et al., 2013). We discuss examples of
possible metrics for $R$ and their decompositions into immediate rewards in
Section 4. We now discuss how GPTRec can be trained to optimise any given
effectiveness metric $R$.

Figure 1. GPTRec’s Pre-Training/Fine-Tuning scheme.  Pre-training (Step 1)
takes the form of using a Top-K model like BERT4Rec (teacher) to pre-train a
GPTRec model checkpoint (student). In Step 2 (Fine-tuning), the Policy model
$\pi$ is the GTPRec model itself initialised by the student model checkpoint
from Stage 1; the Value model is a Transformer Decoder-based model with a
regression head. The Transformer Decoder layer of the Value model is
initialised from the Transformer Decoder layer of the student model, and the
regression head is initialised randomly.

When GPTRec is used with the Next-K strategy, it outputs recommendations item-
by-item. Consider a partially generated list of recommendations $g^{(i)}$ for
the sequence $h$. As GPTRec is a probabilistic model, it estimates the
conditional probability of an item appearing next in the recommendation list:

(3) $\displaystyle\pi_{\Theta}(h,g^{(i)},g_{i+1})\approx P(g_{i+1}|g^{(i)};h)$

where $\Theta$ is the set of learnable model parameters. Here, we use the
symbol $\pi$ for the GPTRec model itself to highlight that this is used as a
policy model in the Reinforcement Learning setup. To understand which item to
put to the position $i+1$, the model obtains the conditional probability
distribution $\pi_{\Theta}$ and selects the item with the maximum probability:

(4) $\displaystyle g_{i+1}=\arg\max_{g_{i+1}\in
I}\pi_{\Theta}(h,g^{(i)},g_{i+1})$

Consider that we have a training set containing ”perfect” recommendations
$\hat{g}=\\{\hat{g}_{1},\hat{g}_{2},\hat{g}_{3},..\hat{g}_{k}\\}$ for the
sequences of historical interactions $h$ (the lists of recommendations, which
maximise a given effectiveness measure $R(h,g)$ given $h$:

(5) $\displaystyle\hat{g}=\operatorname*{arg\,max}_{g}R(h,g)$

Then, GPTRec can be optimised using the standard Language Modelling (LM) loss
(the same way, as GPT (Radford et al., 2019) is optimised given a large
dataset of texts):

(6) $\displaystyle\mathcal{L}_{LM}(\Theta)=-\sum_{i\in
1..k}\log\pi_{\Theta}(h,\hat{g}^{(i-1)},\hat{g}_{i})$

Unfortunately, the ”perfect” list of recommendations, $\hat{g}$, is usually
unknown. Indeed, given the complex structure of many possible quality metrics
for $R$, in the general case, we have to score every possible recommendation
list to find the perfect list $\hat{g}$ according to Equation (5), which is
not feasible due to the combinatorial size of the set of all possible
recommendation lists.

To solve this problem and inspired by the success of InstructGPT (Ouyang et
al., 2022) for solving a similar problem in the language processing domain, we
propose a two-stage pre-training/fine-tuning approach. Figure 1 illustrates
the overall idea of the two-stage approach. As the figure shows, at the first
stage, we use supervised pre-training to train the model to mimic the
behaviour of a traditional Top-K recommendation model (a _teacher_ model),
such as BERT4Rec (Sun et al., 2019). To do that, we first train the teacher
using the training set; then, for each training sequence $h_{j}$, we generate
a list of teacher recommendations $\tilde{g}_{j}$; we then use $\tilde{g}_{j}$
as the ”perfect” recommendation $\hat{g}_{j}$ in Equation (6) and optimise
GPTRec using the LM loss. This process drives the model to generate
recommendations that are as similar as possible to the ones generated by the
teacher model. The problem is that this learning goal is not necessarily
aligned with optimising the target metric $R$. Indeed, most traditional Top-K
models are trained to optimise accuracy. At the same time, $R$ may include
other components, such as diversity; therefore, a model pre-trained this way
is likely to be sub-optimal with respect to $R$.

Hence, in the second stage, as illustrated in Figure 1, we use a reinforcement
learning-based approach to align GPTRec with the target metric $R$. To achieve
this, following the InstructGPT paper (Ouyang et al., 2022), we use the
Proximal Policy Optimisation (PPO) algorithm (Schulman et al., 2017). PPO is
an Actor-Critic type of approach (Sutton and Barto, 2018, Ch. 15.1), which
jointly trains the main _policy model $\pi$_ (”Actor”) and an auxiliary _value
model $V$_ (”Critic”). Similar to other reinforcement learning methods, PPO is
a trial-and-error approach, where the Actor iteratively performs actions
(generates recommendations) with the goal of beating the expectations of the
Critic, and the Critic tries to predict the performance of the Actor in each
case.

In our case, we use GPTRec itself as the policy model $\pi$ and another
GPTRec-based value model as the value model $V$. The difference between the
policy and value models is that the policy generates a probability
distribution over all possible items in the catalogue, whereas the value model
generates a single number: a predicted recommendation effectiveness according
to metric $R$. ††margin:

###  3.2. Pre-Train/Fine Tune approach

Despite having different outputs, the input to both models has the same
structure (illustrated in Figure 6). Therefore, in both cases, we use the same
GPT-2-based Transformer Decoder as the model backbone; however, in the value
model, we use a simple feed-forward layer to predict a single number instead
of a standard GPT-2 token prediction head. Following best practices (Silver et
al., 2016), we initialise the policy model $\pi$ from the supervised
checkpoint trained in the first stage ($GPTRec_{sl}$ in Figue 2). We also
initialise the backbone Transformer Decoder of the value model $V$ model from
the backbone Transformer Decoder of the same pre-trained student model. We
then fine-tune both policy and value models using the PPO algorithm. We
provide a high-level overview of Reinforcement Learning in general and PPO in
particular in Appendix A and refer to the original PPO paper (Schulman et al.,
2017) for more details of the approach.

Figure 2. GPTRec fine-tuning processes diagram. Green boxes are processes;
purple boxes are data.

###  3.3. Efficient Asynchronous Decomposition of Reinforcement Fine-Tuning

As we discussed in Section 3.2, the PPO algorithm alternates between
generating recommendations using the current generation of the policy model
and improving the policy and value models using gradient descent. Improving
the policy and the value models using gradient descent can be efficiently
parallelised using hardware acceleration on a GPU and a deep learning
framework (we use Tensorflow (TensorFlow Developers, 2023))222The code for
this paper: https://github.com/asash/gptrec_rl; however, generating
recommendations with the Next-K strategy is an iterative process
(recommendation for position $i+1$ can be only generated after generating the
recommendation for position $i$). This iterative nature of generation limits
the efficiency of using the GPU for generation. In initial experiments, we
observed that directly alternating between generation and optimisation steps,
the GPU becomes under-utilised, and model training becomes too slow to be
feasible in practice (in our experiments, fine-tuning a single mode could take
up to a week). However, with PPO, we can make generation and optimisation
asynchronous. Therefore, to efficiently utilise available resources, we
decompose sample generation into a separate group of processes and perform the
generation using the Next-K strategy on the CPU. We also use a separate
validation process, which continuously evaluates the latest model checkpoint
on a validation dataset. The validation results are used to monitor metrics
during fine-tuning as well as in order to select the final model checkpoint.

Table 1. Reinforcment learning optimisation objectives we use in versions of GPTRec. $y_{i}$ corresponds to the ground truth relevance for an item generated at the position $i$, $\text{CosDist}(g_{i},g_{j})$ is the cosine distance between the genres of items generated at the positions $i$ and $j$, $\text{freq}(g_{i})$ is the frequency in the training data of the item generated at the position $i$ and $\lambda$ is the secondary metric weitgh Model | Learning Objective
---|---
GPTRec-Reinforcement-NDCG | $\sum_{i=1}^{K}\frac{y_{i}}{\log(i+1)}$
GPTRec-Reinforcement-Diversity | $\sum_{i=1}^{K}\left[\frac{y_{i}}{\log(i+1)}+\lambda\cdot\frac{1}{K\cdot(K-1)}\sum_{j=1}^{i-1}\text{CosDist}(g_{i},g_{j})\right]$
GPTRec-Reinforcement-pCOUNT | $\sum_{i=1}^{K}\left[\frac{y_{i}}{\log(i+1)}-\lambda\cdot\text{freq}(g_{i})\right]$
††margin:

Figure 2 summarises the processes involved in fine-tuning GPTRec. As can be
seen from the figure, to train GPTRec, we use 3 processes: (1) _A Generator_
process that continuously samples batches of user histories from the training
set and generates recommendations for these histories using the latest version
of the policy model. After generating the recommendation, it saves the batch
of histories and associated generated recommendations
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\langle
h,g\rangle}$ pairs into the generated recommendations cache. The
recommendation cache stores the last $M$ generated batches, where $M$ is a
hyperparameter. Multiple Generators can work simultaneously; (2) _An
Optimiser_ that continuously samples batches of
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\langle
h,g\rangle}$ pairs from the recommendations cache and performs optimisation
steps for policy and value models using these batches. It then saves
checkpoints of the Value and Policy models so the Generator and Validator
processes can use them; (3) _A Validator_ that continuously takes the latest
checkpoint and evaluates it using the validation dataset. This scheme allows
us to use both CPU and GPU system resources efficiently and reduces the
training time of a single model from a few days to a few hours.

This concludes the description of the proposed training scheme. We now to
experimental evaluation of GPTRec trained with it.

##  4\. Experimental Setup

We aim to answer the following research questions:

  1. RQ1

How effective is GPTRec trained as a supervised student compared to state-of-
the-art baselines?

  2. RQ2

What is the effect of the teacher-student pretraining scheme on the
effectiveness of the Next-K strategy at different ranking cutoffs $K$?

  3. RQ3

What is the effect of RL-based tuning of GPTRec with Next-K generation when
optimising for NDCG?

  4. RQ4

How does RL-based tuning affect beyond-accuracy metrics compared to greedy
reranking?

###  4.1. Implementation

We implement GPTRec using the GPT-2 architecture from HuggingFace Transformers
(Wolf et al., 2020) and the aprec333https://github.com/asash/bert4rec_repro
framework from a recent replicability study (Petrov and Macdonald, 2022b). We
follow the experimental setup described in the same replicability study
(Petrov and Macdonald, 2022b) and use the results of SASRec and BERT4Rec from
that paper as baselines. Compared to the original GPTRec model, we use a
slightly modified version of sequence structure, details of which we provide
in in Appendix B.

Table 2. Salient characteristics of experimental datasets

Dataset |  Users |  Items |  Interactions |  Mean Length |  Median Length |  Sparsity |  Genres
---|---|---|---|---|---|---|---
MovieLens-1M | 6040 | 3416 | 999611 | 165.49 | 96 | 0.95 | 18
Steam-2M | 201963 | 1000 | 2198260 | 10.88 | 7 | 0.99 | 318

Table 3. Evaluation results. The best results are highlighted in bold; second
best are underlined; * denotes significant difference with BERT4Rec
($pvalue<0.05$, Bonferroni multi-test correction), arrows indicate the metric
improvement direction

Model Type | Model |  Secondary Metric |  Secondary metric weight ($\lambda$) |  Recall@1 $\uparrow$ |  Recall@10 $\uparrow$ |  NDCG@10 $\uparrow$ |  Diversity (ILD@10) $\uparrow$ |  Popularity Bias (nPCOUNT@10) $\downarrow$
---|---|---|---|---|---|---|---|---
Baselines | Popularity |  |  |  0.0056< |  0.0363< |  0.0178< |  0.3222> |  1.0000>
MF-BPR |  |  |  0.0113< |  0.0719< |  0.0366< | 0.2745 |  0.2867<
SASRec |  |  |  0.0464< |  0.2482< |  0.1306< |  0.2703< | 0.2389<
GPTRec-Shifting |  |  | 0.0603 |  0.2647< |  0.1486< | 0.2749 |  0.2985<
BERT4Rec |  |  | 0.0608 | 0.2921 | 0.1617 | 0.2746 | 0.3222
Supervised | GPTRec-Supervised (MC Teacher) |  |  |  0.0791> |  0.2690< | 0.1638 |  0.2798> |  0.4470>
NDCG Tuned | GPTRec-Reinforcement-NDCG |  |  | 0.0829> | 0.2785 | 0.1682 |  0.2878> |  0.4205>
Diversity tuned | GPTRec-Reinforcement-Diversity-0.2 | ILD | 0.2 |  0.0798> |  0.2298< |  0.1499< | 0.3621> |  0.5212>
GPTRec-Reinforcement-Diversity-1.0 | ILD | 1.0 |  0.0795> |  0.1748< |  0.1272< | 0.4349> |  0.5016>
Tuned to decrease popularity bias | GPTRec-Reinforcement-pCOUNT-3.0 | PCOUNT | 3.0 | 0.0820> |  0.2733< | 0.1669 | 0.2774 |  0.3936>
GPTRec-Reinforcement-pCOUNT-6.0 | PCOUNT | 6.0 | 0.0710 |  0.2154< |  0.1392< |  0.2648< | 0.1742<

(a) MovieLens-1M

Model Type | Model |  Secondary Metric |  Secondary metric weight ($\lambda$) |  Recall@1 $\uparrow$ |  Recall@10 $\uparrow$ |  NDCG@10 $\uparrow$ |  Diversity (ILD@10) $\uparrow$ |  Popularity Bias (nPCOUNT@10) $\downarrow$
---|---|---|---|---|---|---|---|---
Baselines | Popularity |  |  |  0.0113< |  0.0725< |  0.0368< |  0.3282> |  1.0000>
MF-BPR |  |  |  0.0103< |  0.0570< |  0.0294< |  0.2923< | 0.1093<
SASRec |  |  |  0.0197< |  0.1139< |  0.0588< | 0.3135 |  0.4826<
GPTRec-Shifting |  |  | 0.0329 | 0.1719 | 0.0912 |  0.3114< |  0.4301<
BERT4Rec |  |  | 0.0331 | 0.1745 | 0.0923 | 0.3147 | 0.4973
Supervised | GPTRec-Supervised (MC Teacher) |  |  |  0.0187< |  0.1060< |  0.0546< |  0.3210> |  0.8790>
GPTRec-Supervised (BERT4Rec Teacher) |  |  | 0.0326 | 0.1699 | 0.0908 | 0.3149 |  0.5041>
NDCG Tuned | GPTRec-Reinforcement-NDCG |  |  | 0.0328 | 0.1709 | 0.0912 | 0.3148 |  0.5029>
Diversity tuned | GPTRec-Reinforcement-Diversity-1.0 | ILD | 1.0 | 0.0325 |  0.1530< |  0.0837< | 0.3636> |  0.4825<
GPTRec-Reinforcement-Diversity-3.0 | ILD | 3.0 | 0.0303 |  0.1174< |  0.0699< | 0.4205> |  0.4744<
Tuned to decrease popularity bias | GPTRec-Reinforcement-pCOUNT-3.0 | PCOUNT | 3.0 | 0.0346 |  0.1606< |  0.0878< | 0.3141 |  0.3355<
GPTRec-Reinforcement-pCOUNT-6.0 | PCOUNT | 6.0 |  0.0245< |  0.0964< |  0.0563< |  0.3182> | 0.1049<

(b) Steam-2M

††margin:

###  4.2. Datasets

As discussed in Section 1, the scope of this work is limited to the datasets
with a few thousand items in the catalogue. Indeed, there are solutions to
allow Transformer models to scale to large catalogues, for instance, by using
multiple tokens to represent each item (sub-item representation) (Petrov and
Macdonald, 2023a, 2024) – specifically, original GPTRec paper (Petrov and
Macdonald, 2023a) showed that GPTRec could be used with sub-item
representations. Adding sub-item representations is a further experimental
variable outside the scope of this paper.

Hence, we use the MovieLens-1M and Steam-2M datasets for our experiments,
which both have $<$ 4000 items. While there are some known issues with the
MovieLens-1M dataset (Harper and Konstan, 2015) (e.g. users don’t rate movies
in the dataset in the same order as they watch them), the dataset is used
consistently across many papers, allowing comparison with published results.
We use the version of the dataset from the SASRec (Kang and McAuley, 2018)
repository and do not apply any additional preprocessing. The same version was
used in the original GPTRec paper (Petrov and Macdonald, 2023a) as well as
many other recent publications (Kang and McAuley, 2018; Sun et al., 2019;
Petrov and Macdonald, 2022b, 2023b). We also use Steam-2M, a smaller version
of the Steam (Kang and McAuley, 2018) dataset. In this version, we only use
the 1000 most popular items. Additionally, we de-duplicate the interactions in
this dataset and remove users with less than 5 interactions. Overall, Steam-2M
has $2\times$ more interactions than Movielens-1M and $33\times$ more users,
allowing us to test the model in a larger setting. The salient characteristics
of both experiments are listed in Table 2.

###  4.3. Data Splitting

We employ a leave-one-out strategy for model testing, holding out each user’s
latest interaction in the test set. In addition, for 128 randomly selected
users, we hold out their second-to-last action in a separate validation set.
This validation set is used for the early stopping mechanism and to control
model quality during training, ensuring that our model generalises well to new
data.

Following common practice in recommendation (Kang and McAuley, 2018; Sun et
al., 2019; Petrov and Macdonald, 2022b), we use a relatively shallow
Transformer model with three Transformer blocks. We set the maximum sequence
length to 200 and truncate the sequence to the last 200 interactions if the
user interacted with more items. We use 256-dimensional item embeddings in our
experiments. At the supervised pre-training stage, we use a 200-epoch early
stopping mechanism to ensure model convergence. At the fine-tuning stage, we
tune the model for 64000 steps. We select the best model according to the
target metric $R$ on validation data.

††margin:

###  4.4. Effectiveness Metrics and Optimisation Goals

We use NDCG@10 as the main accuracy metric of the models. We use PCOUNT
(Borges and Stefanidis, 2021) (mean recommended item popularity) as the
popularity bias metric. To make PCOUNT independent of the dataset, we also use
nPCOUNT — a normalised version of the PCOUNT metric, in which we divide the
PCOUNT metric by the PCOUNT metric of the popularity models. This way, the
item popularity model (which has maximum possible popularity bias) always has
an nPCOUNT value of 1, and the smaller values nPCOUNT of nPCOUNT correspond to
the smaller popularity bias of the model (i.e., lower is better).
Additionally, we use the Intra-List Distance (ILD) as the metric of the
diversity of the recommended items (Antikacioglu et al., 2019). ILD is
measured as a sum of pair-wise distances of the recommended items, and as the
distance metric, we use the cosine distance between movie genre vectors for
MovieLens-1M and game genres for Steam-2M. As the immediate reward
$\mathrm{r}(h;g^{(i)})$ for NDCG@10 we use discounted relevance
($\frac{y_{i}}{\log(i+1)}$), where $y_{i}$ is the ground truth relevance or
$i^{th}$ recommendation $g_{i}$); for PCOUNT as an immediate reward we use the
negative item popularity and for diversity as an immediate reward we use the
sum of pairwise diversities $\frac{1}{K\cdot(K-1)}\sum_{j\in
1..i-1}\text{CosDist}(g_{i},g_{j})$. For each immediate reward, the sum over
all recommendation positions returns the value of the full metric. For PCOUNT,
the immediate reward is negative, as we want to minimise the metric instead of
maximising it. Table 1 also summaries reinforcement learning objectives we use
in versions of GPTRec

††margin:

###  4.5. Baselines

As the baseline models, we use two simple models: Item Popularity and BPR
(Rendle et al., 2009) (we use a version of BPR from the LightFM library).
Additionally, we use several Transformer-based models: (1) SASRec (Kang and
McAuley, 2018), the most cited transformer-based recommendation model. SASRec
is architecturally very similar to GPTRec (it is also Transformer Decoder-
based) but uses the Top-K recommendation strategy. In contrast with GPTRec, it
is trained using sequence-to-sequence shifting objectives. The loss function
is Binary Cross-Entropy;(2) BERT4Rec is a popular Transformer Encoder-based
baseline, which exhibits state-of-the-art effectiveness when fully converged
(Petrov and Macdonald, 2022b). It uses the Softmax Cross-Entropy as the loss
function; (3) GPTRec-Shifting is a GPTRec-based baseline, that has been
trained using a traditional training scheme rather than the proposed 2-stage
scheme. Similar to SASRec, it is trained using the sequence shifting
objective, but similarly to BERT4Rec, it uses the Softmax Cross-Entropy the
loss function instead of the Binary Cross-Entropy. Indeed, Softmax Cross-
Entropy has been shown to be more effective for sequential recommendation
(Petrov and Macdonald, 2023b). Similar to GPTRec, GPTRec-Shifting uses GPT-2
architecture as the backbone. The motivation for us to include GPTRec-Shifting
as a baseline is to decouple the effect of model architecture change from the
training scheme change when comparing GPTRec with the traditional models, such
as SASRec and BERT4Rec.

###  4.6. Teacher Models for Supervised Learning

For the first stage of the training process, we use two teacher models. First,
we use a simple Markov Chain-based (MC) model, which calculates the empirical
probability of item $a$ appearing in the sequence after item $b$ (immediately
or after several steps). It then computes the final score of the weighted sum
of empirical probabilities: $s(g_{i})=\sum_{j\in 1..n}\beta^{j}\log
p(g_{i}|h_{n-j+1})$, where $\beta$ is a decay hyperparameter, which we set to
0.6 (we find this to work the best experimentally). This teacher model is very
simple, and training it only takes a few seconds on both datasets. Second, for
the Steam-2M dataset, where using Markov Chain as a teacher resulted in weaker
performance viz. the baselines, we also use BERT4Rec as a teacher.

(a) MovieLens-1M

(b) Steam-2M

Figure 3. Models’ NDCG@K with Top-K and Next-K recommendation strategies when
varying ranking cutoff $K$. Red arrows demonstrate the effectiveness gap
between the Top-K and the Next-K inference strategies at a given ranking
cutoff $K$.

##  5\. Results

††margin:

###  5.1. RQ1 Supervised-Only training

We first analyse the effectiveness of GPTRec when it is trained only using the
first stage of the training process, without reinforcement learning. Table 3
contains the results of this comparison (see model types ”Supervised” and
”Baselines”) for (a) MovieLens-1M and (b) Steam-2M. As we can see from Table
3(a), for MovieLens-1M, the model with this simple teacher performs
surprisingly well, achieving similar accuracy metrics to BERT4Rec (NDCG@10
0.1638 for GPTRec, 0.1617 for BERT4Rec, difference not significant).
Interestingly, on this dataset GPTRec also achieves much higher Recall@1
compared to BERT4Rec (0.0791, +30.1%). We speculate that the improvement is
due to a better alignment of GPTRec’s training task with the next item
prediction task; see (Petrov and Macdonald, 2023b) for similar findings. On
the other hand, on the Steam-2M dataset (Table 3(b)), GPTRec with the Markov
Chain teacher has a lower effectiveness than BERT4Rec (NDCG@10 0.0546 vs.
0.0923, significant); however, it is still comparable with SASRec, which has a
slightly higher NDCG@10 of 0.0588. Due to the weaker effectiveness of GPTRec
compared to BERT4Rec on the Steam-2M dataset, we additionally train GPTRec
using BERT4Rec as a teacher. In this case, GPTRec achieves a similar NDCG@10
to BERT4Rec (1.6% difference, not significant). Hence, for steam dataset in
all following experiments we use the version of GPTRec pre-trained with the
BERT4Rec teacher.

Overall, in answer to RQ1, we say that GPTRec pre-trained as supervised
student can achieve similar accuracy (measured by NDCG@10) to the state-of-
the-art BERT4Rec model and outperforms other baselines. Importantly, it
achieves good effectiveness using the _generative_ Next-K strategy, meaning
that the models from the first training stage can serve as a good starting
point for the fine-tuning at the second stage.

††margin:

###  5.2. RQ2 Effect of Teacher-Student pertaining on the Next-K
recommendation.

In principle, the Next-K strategy can be used with any sequential
recommendation model, by iteratively adding a generated item to the end of the
input sequence. However, we hypothesise the model needs to be trained
specifically for Next-K generation – i.e., using a Top-K model checkpoint
leads to effectiveness degradation at large cutoffs $K$ for Next-K generation.
To empirically validate the hypothesis, we compare the NDCG@$K$ metric at
different cutoffs $K$ when using the Top-K and the Next-K inference. We
analyse the three baseline Transformer-based models (SASRec, BERT4Rec and
GPTRec-Shifting) trained using their classical training objectives and GPTRec
trained with the teacher-student scheme. We compare these baseline models
using both Top-K and Next-K generation inference strategies. For all four
models, we compare the same model checkpoint under different inference
strategies; the only change is switching from Top-K inference to Next-K
inference.

Figure 3 shows the results of this analysis on both experimental datasets. As
we can see from the figure, in all cases at cutoff $K=1$, both Top-K and
Next-K strategies always achieve the same NDCG@1. Indeed, when we are only
interested in one item, both strategies select the item with the maximum score
according to the model, and hence both strategies are equal in case $K=1$.
However, at deeper ranking cutoffs, we observe quality effectiveness
degradation when using the Next-K strategy with the baseline models. For
example, with $K=10$, SASRec shows 33.7% lower NDCG@10 when using the Next-K
strategy compared to the Top-K. In contrast, for GPTRec pre-trained with the
teacher-student, the difference between Top-K and Next-K is usually small and
sometimes positive. For example, when using ranking cutoff $K=8$, GPTRec with
Next-K strategy achieves a slightly higher NDCG@8 (+0.35%) on the Steam-2M
dataset while achieving slightly lower NDCG@8 on MovieLens-1M dataset
(-2.99%).  ††margin:  When comparing the plots of the GPTRec with GPTRec-
Shifting (which uses the same backbone architecture but classic sequence
shifting training), we observe markedly different trends. Indeed, on both
datasets, similarly to other baselines, GPTRec-Shifting exhibits effectiveness
degradations when switching from the Top-K to the Next-K inference strategy.
However, for GPTRec, Top-K and Next-K exhibit widely similar effectiveness
across $K$ values. For example, we observe a -16.14% NDCG@8 drop when
switching from Top-K to Next-K at ranking cutoff $K=8$ on Steam-2M, which has
a similar magnitude to the other baseline models (-15.59% for SASRec, -18.93%
for BERT4Rec). In contrast, for the same dataset and at the same cutoff $K=8$,
GPTRec exhibits a +0.35% improvement. This shows that the absence of
effectiveness degradation at deeper cutoffs $K$ when using the Next-K strategy
compared to Top-K in GPTRec is caused by the training scheme from sequence
shifting to the teacher-student pre-training and not by the architecture
(GPTRec & GPTRec-Shifting share the backbone architecture but perform
differently).

Overall, for RQ2, we conclude that a teacher-student training scheme is
necessary to achieve the same effectiveness, using the Next-K strategy, as it
is achievable with the Top-K strategy. Hence Next-K is a viable strategy for
recommendation, but requires an appropriate training scheme. In the next
sections, we empirically demonstrate its benefits compared Top-K for accuracy
and beyond-accuracy metrics as a result of applying reinforcement learning.

###  5.3. RQ3. Reinforcement Learning for Accuracy

We now analyse the effectiveness of the full 2-stage training scheme when our
goal is to optimise accuracy, i.e., using NDCG@10 as the effectiveness measure
$R$. For MovieLens-1M, we start from the checkpoint which was trained using
the Markov Chain teacher, and for Steam-2M, we use the checkpoint trained
using the BERT4Rec teacher; both these checkpoints achieve competitive results
to BERT4Rec even without fine-tuning. Table 3 summarises the results (see
”NDCG-Tuned” Model Type). As we can see from the table, fine-tuning increases
NDCG@10 in both cases, but this improvement is very small (and not
statistically significant). For example, on MovieLens-1M, NDCG@10 improved
from 0.1638 to 0.1682 (+2.6%, not significant), and on Steam-2M, it improved
from 0.0908 to 0.0912 (+0.4%, not significant). In both cases, the result is
on par with BERT4Rec (statistically indistinguishable). In summary, for RQ3,
we observe a small positive effect of reinforcement learning tuning for NDCG,
but this effect is small and not statistically significant. In practice, it is
enough to use only first-stage training if the goal is to optimise the model
for accuracy only (but not for more complex goals; see the next section).

(a) MovieLens-1M (b) Steam-2M

Figure 4. Accuracy (NDCG@10) / Diversity(ILD@10) tradeoff. Arrows represent
the direction of metric improvement, and horizontal and vertical lines
represent standard errors.

###  5.4. RQ4. Reinforcement Learning for Beyond-Accuracy Measures

††margin:

In our last research question, we analyse the effect of the 2nd-stage tuning
when the effectiveness metric $R$ includes additional components. In our
experiments, we use the compositional effectiveness metric
$R=NDCG+\lambda\cdot R_{Secondary}$, where $R_{Secondary}$ measures diversity
(using ILD) or popularity bias (using negative PCOUNT). In our experiments, we
vary $\lambda$ between 0 and 3 for ILD and between 0 and 6 for PCOUNT. As the
baseline for secondary metric optimisation, we use greedy reranking techniques
over the BERT4Rec results: Maximal Marginal Relevance (MMR) (Carbonell and
Goldstein, 1998) as the diversity baseline, and greedy re-ranking (Kaya, 2018)
for decreasing popularity bias. In both cases, we control the tradeoff between
primary and secondary metrics in the baselenies by varying the weight of the
secondary metric during the re-ranking stage. Figures 4 and 5 illustrate the
results of our experiments. As can be seen from the figures, the results
achieved by GPTRec are not worse compared to MMR for diversity and compared to
greedy re-ranking for PCOUNT. Indeed, in all cases, when placing most
optimisation emphasis on NDCG, the results are not distinguishable between
BERT4Rec and GPTRec (we already discussed that in RQ3). However, when we
increase the importance of the secondary metrics, GPTRec outperforms greedy
techniques over BERT4Rec in 3 out of 4 cases, with the only exception being
tuning for decreasing popularity bias on Steam-2M. For example, when GPTRec is
tuned to decrease popularity bias on MovieLens-1M, it achieves NDCG@10 of
0.139 and nPCOUNT of 0.1742. At the same time, BERT4Rec can only achieve lower
NDCG@10 of 0.1267 (-8.8 %, significant), with higher popularity bias (nPCOUNT
0.1893, +8.6%, significant).

††margin:

We also provide the quantitative results of tuning the models for beyond-
accuracy metrics in Table 3. As can be seen from the table, when the model is
tuned for a specific secondary metric, the model is able to achieve good
results according to this metric while retaining reasonably high NDCG@10. For
example, when we tune GPTRec for diversity on the Steam-2M dataset (Table
3(b)), with the weight of secondary metric $\lambda=6.0$, we obtain the best
ILD (0.4205, +33% over BERT4Rec, significant) while still retaining NDCG@10
comparable to SASRec.

Overall, in answer to RQ4, we conclude that fine-tuning allows us to optimise
GPTRec with the Next-K generation strategy for complex recommendation goals,
and the results are better (in $\frac{3}{4}$ cases) or statistically
indistinguishable ($\frac{1}{4}$ case) compared to greedy reranking of the
BERT4Rec results.

(a) MovieLens-1M (b) Steam-2M

Figure 5. Accuracy (NDCG@10) / Popularity Bias (nPCOUNT@10) tradeoff. Arrows
represent the direction of metric improvement, and horizontal and vertical
lines represent standard errors.

##  6\. Conclusions

In this paper, we proposed a 2-stage pre-training/fine-tuning approach to
align generative sequential models with beyond-arccuracy goals, where at the
pre-training stage, a generative model learns to mimic the behaviour of
traditional Top-K recommenders, and during the fine-tuning stage, it can be
optimised for any metric using reinforcement learning. We applied this
approach to the recently proposed GPTRec model, which uses the Next-K strategy
for recommendations. We demonstrated that the first stage (supervised pre-
training) is enough to achieve NDCG@10 comparable to the state-of-the-art
BERT4Rec model; however, fine-tuning can improve secondary objectives, such as
diversity and popularity bias. In 3 out of 4 experiments, GPTRec fine-tuned
for complex metrics, including diversity and popularity bias, outperformed the
greedy re-ranking over BERT4Rec results. In 1 out of 4, the results were
indistinguishable. For example, a fine-tuned version of GPTRec allowed us to
simultaneously achieve 8.8% better NDCG and 8.6% lower popularity bias
compared to greedy reranking over BERT4Rec results.

The methodology proposed in this allows direct optimisation of GPTRec (or,
indeed, any other Transformer Decoder model) for any effectiveness measure
without requiring the measure to be differentiable or computable at a single-
item level. We demonstrated the applicability of this methodology using such
beyond-accuracy objective targets as increasing diversity and decreasing
popularity bias; however, the methodology can be used with other metrics,
including variations of fairness measures, measures that balance the needs of
various stakeholders of recommender systems and so on.

There are a number of possible future research directions that can build upon
this work: Firstly, the scope of applicability of the methodology can be
extended to larger datasets by applying sub-item representations instead of
using atomic item IDs; Secondly, the quality of the generated rankings could
possibly be further be improved by enhancing the model architecture; Thirdly,
the recommendation goal can be used as the model input, allowing a single
model to generate different kinds of recommendations and providing the user
control over the desired output. Overall, exploring these future research
directions could further refine the effectiveness, efficiency and
applicability of recommendation systems in many scenarios.

## References

  * (1)
  * Afsar et al. (2023) M. Mehdi Afsar, Trafford Crump, and Behrouz Far. 2023\.  Reinforcement Learning Based Recommender Systems: A Survey.  _Comput. Surveys_ 55, 7 (2023), 1–38.
  * Antikacioglu et al. (2019) Arda Antikacioglu, Tanvi Bajpai, and R. Ravi. 2019.  A New System-Wide Diversity Measure for Recommendations with Efficient Algorithms.  _SIAM Journal on Mathematics of Data Science_ 1, 4 (2019), 759–779.
  * Bai et al. (2019) Xueying Bai, Jian Guan, and Hongning Wang. 2019.  A Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation. In _Proc. NeurIPS_. 10735–10746.
  * Borges and Stefanidis (2021) Rodrigo Borges and Kostas Stefanidis. 2021.  On Mitigating Popularity Bias in Recommendations via Variational Autoencoders. In _Proc. SAC_. 1383–1389.
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. In _Proc. SIGIR_. 335–336.
  * Deffayet et al. (2023) Romain Deffayet, Thibaut Thonet, Jean-Michel Renders, and Maarten De Rijke. 2023.  Generative Slate Recommendation with Reinforcement Learning. In _Proc. WSDM_. 580–588.
  * Harper and Konstan (2015) F. Maxwell Harper and Joseph A. Konstan. 2015.  The MovieLens Datasets: History and Context.  _ACM Transactions on Interactive Intelligent Systems (TiiS)_ 5, 4 (Dec. 2015), 19:1–19:19.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-Attentive Sequential Recommendation. In _Proc. ICDM_. 197–206.
  * Kaya (2018) Mesut Kaya. 2018\.  Accurate and Diverse Recommendations Using Item-Based SubProfiles. In _Proc. FLAIRS_.
  * Kotkov et al. (2020) Denis Kotkov, Jari Veijalainen, and Shuaiqiang Wang. 2020\.  How Does Serendipity Affect Diversity in Recommender Systems? A Serendipity-Oriented Greedy Algorithm.  _Computing_ 102, 2 (2020), 393–411.
  * Kunaver and Požrl (2017) Matevž Kunaver and Tomaž Požrl. 2017.  Diversity in recommender systems–A survey.  _Knowledge-based systems_ 123 (2017), 154–162.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing Atari with Deep Reinforcement Learning.  arXiv:1312.5602 [cs]
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022\.  Training Language Models to Follow Instructions with Human Feedback.  arXiv:2203.02155 [cs]
  * Petrov and Macdonald (2024) Aleksandr Petrov and Craig Macdonald. 2024.  RecJPQ: Training Large-Catalogue Sequential Recommenders. In _Proc. WSDM_.
  * Petrov and Macdonald (2022a) Aleksandr V. Petrov and Craig Macdonald. 2022a.  Effective and Efficient Training for Sequential Recommendation Using Recency Sampling. In _Proc. RecSys_. 81–91.
  * Petrov and Macdonald (2022b) Aleksandr V. Petrov and Craig Macdonald. 2022b.  A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation. In _Proc. RecSys_. 436–447.
  * Petrov and Macdonald (2023a) Aleksandr V. Petrov and Craig Macdonald. 2023a.  Generative Sequential Recommendation with GPTRec. In _Proc. Gen-IR@SIGIR_.
  * Petrov and Macdonald (2023b) Aleksandr V. Petrov and Craig Macdonald. 2023b.  gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling. In _Proc. RecSys_. 116–128.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models Are Unsupervised Multitask Learners.  _OpenAI blog_ (2019).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21, 140 (2020), 1–67.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian Personalized Ranking from Implicit Feedback. In _Proc. UAI_. 452–461.
  * Schulman et al. (2015) John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. 2015.  Trust Region Policy Optimization. In _Proc. ICML_. 1889–1897.
  * Schulman et al. (2016) John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. 2016\.  High-Dimensional Continuous Control Using Generalized Advantage Estimation. In _Proc. ICLR_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal Policy Optimization Algorithms.  arXiv:1707.06347 [cs]
  * Silver et al. (2016) David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016\.  Mastering the Game of Go with Deep Neural Networks and Tree Search.  _Nature_ 529, 7587 (2016), 484–489.
  * Stamenkovic et al. (2022) Dusan Stamenkovic, Alexandros Karatzoglou, Ioannis Arapakis, Xin Xin, and Kleomenis Katevas. 2022\.  Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learning. In _Proc. WSDM_. 957–965.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In _Proc. CIKM_. 1441–1450.
  * Sunehag et al. (2015) Peter Sunehag, Richard Evans, Gabriel Dulac-Arnold, Yori Zwols, Daniel Visentin, and Ben Coppin. 2015.  Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions.  arXiv:1512.01124 [cs]
  * Sutton and Barto (2018) Richard S. Sutton and Andrew G. Barto. 2018.  _Reinforcement Learning: An Introduction_ (second edition ed.).  The MIT Press.
  * TensorFlow Developers (2023) TensorFlow Developers. 2023\.  TensorFlow.  https://www.tensorflow.org/
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need. In _Proc. NeurIPS_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020\.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  arXiv:1910.03771 [cs]
  * Xin et al. (2020) Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M. Jose. 2020.  Self-Supervised Reinforcement Learning for Recommender Systems. In _Proc. SIGIR_. 931–940.
  * Zhang et al. (2022) Qihua Zhang, Junning Liu, Yuzhuo Dai, Yiyan Qi, Yifan Yuan, Kunlun Zheng, Fan Huang, and Xianfeng Tan. 2022\.  Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems. In _Proc. KDD_. 4510–4520.
  * Zhou et al. (2010) Tao Zhou, Zoltán Kuscsik, Jian-Guo Liu, Matúš Medo, Joseph Rushton Wakeling, and Yi-Cheng Zhang. 2010.  Solving the Apparent Diversity-Accuracy Dilemma of Recommender Systems.  _Proceedings of the National Academy of Sciences_ 107, 10 (2010), 4511–4515.
  * Zou et al. (2019) Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. 2019\.  Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems. In _Proc. KDD_. 2810–2818.

##  Appendix A Extra information on Reinforcement learning

###  A.1. Basics of Reinforcement Learning

In this section, we describe the basics of reinforcement learning necessary
for describing our methodology. Reinforcement Learning (Sutton and Barto,
2018) (RL) is a class of machine learning methods in which an _agent_
interacts with an _environment_ by performing _actions_. The environment
reacts to the actions by giving the agent _rewards_ and changing the agent’s
_state_. The agent learns a _policy_ of selecting actions depending on the
state that maximises the _long-term reward_ (reward accumulated over multiple
steps).

We now introduce some definitions and notations used across the paper to
describe the Reinforcement Learning problem setup; Typically, a Reinforcement
Learning problem is formalised using the framework of _Markov Decision
Process_ (MDP) (Sutton and Barto, 2018, Ch.3). While there are some variations
in the definition of MDP in the literature, we use the definitions and
notations from the TRPO paper (Schulman et al., 2015) that largely influenced
the Proximal Policy Algorithm (Schulman et al., 2017) – the main RL algorithm
we use in this paper. In particular, a Markov Decision Process is defined as a
tuple $MDP=\left<\mathcal{S},\mathcal{A},P,r,\rho_{0},\gamma\right>$ where $S$
is a finite set of states, $\mathcal{A}$ is a finite set of actions,
$P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ is the
distribution of transition probability, $r:\mathcal{S}\rightarrow\mathbb{R}$
is the reward function, $\rho_{0}:\mathcal{S}\rightarrow\mathbb{R}$ is the
distribution of initial state $s_{0}$, $\gamma\in[0,1]$ is the discount factor
that defines how much we prefer immediate award over the award at the next
step.  ††margin:  Given an $MDP$, the agent-environment interaction dynamics
can be described as follows:

  1. (1)

The initial state $s_{0}$ is drawn from the distribution $\rho_{0}$

  2. (2)

At each time step $i$, the agent observes the state $s_{i}\in\mathcal{S}$ and
selects an action $a_{i}\in\mathcal{A}$

  3. (3)

The environment draws a new state $s_{i+1}$ from the transition probability
distribution $P(s_{i+1}|s_{i},a_{i})$

  4. (4)

The agent receives a reward $r(s_{i+1})$

The ultimate goal of the agent is this setup to maximise the long-term
discounted reward $r_{0:\infty}=\sum_{i=0}^{\infty}\gamma^{i}r_{i}$. To
achieve its goal, the agent follows a _stochastic policy_ , which is defined
as a probability distribution
$\pi:\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$. When the agent
observes the state $s_{i}$, it selects the next action $a_{i}$ by drawing the
action from the conditional distribution $\pi(a_{i}|s_{i})$. In a typical
Reinforcement Learning scenario, the agent also adjusts its policy when
observing rewards and state transitions from the environment. In deep
learning, the policy is typically modelled using a neural network
$\pi_{\Theta}(a|s)$ where $\Theta$ is the set of network parameters. In that
case, the policy is adjusted by changing the set of parameters $\Theta$.

Note that in the general case, the parameters of the MDP, such as the
transition probability distribution $P(s_{i+1}|a_{i},s_{i})$ as well as the
reward function $r(s)$ are not known to the agent (however, an agent may have
_some_ prior knowledge about them); The agent has to _learn_ the structure of
these parameters through the interactions with the environment and reflect
this structure in the parameters of its policy. While, generally speaking, the
number of interactions that an agent performs with the environment may be
infinite, in practice, in many situations, there is usually a _terminal_
state, after which the MDP stops. The process then may restart again. A single
sequence of state actions that end in a terminal state is known as an
_episode_. The agent may use the knowledge obtained from interactions with the
interactions in the previous episode to maximise the reward in the next
episode.

Reinforcement Learning is an active area of research, and there are many ways
of learning optimal policies. For a comprehensive overview of the problem and
the methodology, we refer to the classic book (Sutton and Barto, 2018). In our
work, we use the _Proximal Policy Optimisation_ (PPO) algorithm as the main
algorithm for reinforcement learning, because this algorithm has recently
shown remarkable results in achieving goals similar to ours in the domain of
Language Modelling. We describe the details of the PPO algorithm with
application to our recommendation task in Section A.3.

We now briefly describe how Reinforcement learning has been applied to
Recommender Systems.

###  A.2. Reinforcment Learning for RecSys

The use of RL is appealing for recommender systems because it can optimise for
complex beyond-accuracy rewards , which are not necessarily differentiable
(e.g. diversity),  as well as optimise for long-term goals, which are hard to
optimise using traditional supervised learning. Indeed, we are not the first
to apply RL-based techniques for recommender systems (Zou et al., 2019; Zhang
et al., 2022; Stamenkovic et al., 2022; Deffayet et al., 2023; Afsar et al.,
2023; Bai et al., 2019).

Generally, existing approaches can be divided into two types based on what
these methods use as the action space $\mathcal{A}$. We now briefly describe
these existing types.

††margin:

#### T1: Item-based methods

In this type, the methods (e.g. (Stamenkovic et al., 2022; Xin et al., 2020)))
use possible items $i\in I$ as the actions and make the recommendation by
sorting the items according to the probability of selecting them as the next
action (i.e. using the Top-K strategy). However, as we argue in Section 1, it
is hard to optimise the models that use the Top-K strategy for complex
metrics, such as diversity, because these methods compute items’ scores
independently from each other.

#### T2: Slate-based methods

Methods (e.g. (Deffayet et al., 2023; Sunehag et al., 2015)) in this type use
all possible _slates_ (full lists of recommended items $g\in\mathcal{G}$) as
the action space and generates the slate using a single model inference. The
problem with this group of approaches is that the space of all possible
recommendation lists is very large (proportional to $|I|^{K}$, where $|I|$ is
the number of possible items, and $k$ is the size of the slate). It is
computationally infeasible to score all possible slates to select the best
one. Therefore, slate-based models usually require an existing source of high-
quality slates that they can use as training data, which is not always
available, or rely on some other heuristics, which limit the search space.
Both of these groups of methods use the historical user-item interactions as
the agent’s state, using which the agent selects actions.

#### T3. Generative

The method proposed in this paper does not fall into either of the existing
types and therefore, we put it in a separate group. Indeed, similar to the
methods from the slate-based method, it focuses on optimising listwise
metrics; however, similar to the methods from the first group, the actions in
our method correspond to items. We achieve this by changing the state space
$\mathcal{S}$: in our case, the state includes not only historical user-item
interactions but also a partially generated recommendations list. It allows
the method to take into account items already recommended to the user, as well
as plan for future actions ahead. In contrast to existing slate-based methods,
our approach does not require existing sources of high-quality slates.
Instead, it uses a 2-stage approach, where, in the first stage, the model
learns to mimic the behaviour of the teacher model, and in the second stage,
it aligns the model with the given effectiveness measure $R$.

The limitation of our method compared to the existing methods is that it
requires inferring the recommendation model $K$ to make a ranking of $K$
items, whereas both Top-K methods and slate-based methods produce the list in
just a single model inference; however, in practice, we find that our model is
still able to generate recommendations reasonably fast: generating a
recommendation on our hardware requires approximately 300 milliseconds, which
is enough for many practical applications. This is similar to interactive
language model applications, such as ChatGPT, which generate a few tokens per
second and still remain useful for a broad range of tasks.

###  A.3. Details of the PPO Algorithm

Proximal Policy Optimisation (PPO) is a state-of-the-art algorithm for many
Reinforcement Learning problems. As we describe in Section A.1, the goal of
Reinforcement Learning is to train an _agent_ (a recommender system in our
case) to perform _actions_ (choosing an item to recommend at position $i$)
given a _state_ (a user with history $h$ and a partially generated list
$g^{(i)}=[g_{1},g_{2},..g_{i-1}]$) to optimise the long-term _reward_ (metric
$R$ over the fully generated recommendation list $g$). PPO assumes that the
agent follows a stochastic _policy $\pi_{\Theta}$_ parametrised by a set of
parameters $\Theta$: the agent selects actions from a probability distribution
over the set of all possible actions, and PPO iteratively improves the policy
by updating its parameters $\Theta$.

In our case, the policy defined by the GPTRec model itself defines the policy
$\pi_{\Theta}$, and therefore PPO iteratively updates GPTRec’s parameters to
maximise effectiveness $R$.

PPO follows the Actor-Critic optimisation scheme: it uses an auxiliary model,
$V_{\Psi}$, parametrised by a set of parameters $\Psi$, which predicts the
expected discounted long-term reward given the current state. The idea is to
increase the probabilities of actions that increase our expected reward at
each training iteration and decrease the probabilities of actions that
decrease our expected reward. Therefore, in addition to training the main
policy model, we train the value model:

(7) $\displaystyle
V_{\Psi}(h,g^{(i)})\approx\mathbb{E}_{\pi_{\Theta}}\left[\sum_{j\in
0..K-i-1}\gamma^{j}\mathrm{r}(h;g^{(i+j)})\right]$

here, $0<\gamma\leq 1$ is the discount hyperparameter, which controls how much
the model prefers immediate reward over delayed reward, and
$\mathbb{E}_{\pi_{\Theta}}$ means expectation over all possible generated
recommendations with the GPTRec model. To optimise the value model, PPO uses a
standard mean squared loss:

(8)
$\displaystyle\mathcal{L}_{MSE}(\Psi)=-\sum_{i=1}^{K}\left(V_{\Psi}(h,g^{(i)})-\sum_{j=0}^{K-i-1}\gamma^{j}\mathrm{r}(h;g^{(i+j)})\right)^{2}$

PPO uses the value function to compute _advantages_ , which measures how much
better or worse our model selected an item at each generation step compared to
the expected value (according to metric $R$ and expected reward returned by
$V$). PPO uses the _Generalised Advantage Estimators_ (GAE) (Schulman et al.,
2016). To compute the GAE advantages, we first compute the temporal difference
(TD) residual. The TD residual, denoted as $\delta_{i}$, at position $i$ is
given by:

(9) $\delta_{i}=\mathrm{r}(h,g^{(i+1)})+\gamma
V_{\Psi}(h,g^{(i+1)})-V_{\Psi}(h,g^{(i)})$

where $\mathrm{r}(h,g^{(i+1)})$ is the immediate reward at position $i+1$
(Equation (1)), and $\gamma$ is the same discount factor as used in Equation
(7). The GAE advantage fo position $i$, denoted as $A_{i}$, computes the
advantage at each position $i$ using a decay factor $\lambda$ (which is also a
hyperparameter that similarly to $\gamma$ controls the model focus on
long/short term rewards) and TD residuals, is then by:

(10) $A_{i}=\sum_{l=0}^{K-i-1}(\gamma\lambda)^{l}\delta_{i+l}$ ††margin:

To drive the model towards better rewards, PPO optimises the surrogate
objective, which depends on the Ratio($\cdot$) function:

(11)
$\displaystyle\text{Ratio}(\Theta,h,g^{(i)})=\frac{\pi_{\Theta}(h,g^{(i-1)},g_{i})}{\pi_{\Theta_{old}}(h,g^{(i-1)},g_{i})}$

where $\Theta_{old}$ is the set of model parameters used at the time of
generating the recommendations list $g^{(i)}$. Note that PPO uses the same
generated recommendations for multiple optimisation steps, and therefore,
$\Theta_{old}$ is not equal to $\Theta$, and overall, the proportion in
Equation (11) is not equal to 1. The PPO surrogate objective (also known as
the CLIP objective) is then defined as:

$\displaystyle\mathcal{L}_{CLIP}(\Theta)$
$\displaystyle=-\sum_{i=1}^{K}\min\bigg{(}\text{Ratio}(\Theta,h,g^{(i)})A_{i},$
(12)
$\displaystyle\text{clip}\left(\text{Ratio}(\Theta,h,g^{(i)}),1-\epsilon,1+\epsilon\right)A_{i}\bigg{)}$

where the $\text{clip}(\cdot)$ function limits the minimum and maximum of the
first argument, and $\epsilon$ is a hyperparameter that controls maximum
divergence from old model parameters $\Theta_{old}$. Without clipping, the
Ratio function (Equation (11)) is known to diverge from 1 very quickly;
therefore, clipping is required to prevent an explosion of gradients of the
CLIP objective.

Overall, the loss function optimised by PPO can be written as:

(13)
$\displaystyle\mathcal{L}_{PPO}(\Psi,\Theta)=\mathcal{L}_{MSE}(\Psi)+\mathcal{L}_{CLIP}(\Theta)$

The original PPO paper also includes an entropy bonus in the loss function
(which encourages the model to perform exploration). However, in our initial
experiments, we observed no improvement in the overall quality of GPTRec when
using an entropy bonus; therefore, we omit it for training GPTRec. Overall,
PPO can be summarised as repeating the following steps: (Step 1) Sample a
batch of user histories $h_{1},h_{2}...h_{B}$; (Step 2) for each history $h$
in the batch, generate recommendations $g$ using the current version of GPTRec
$\pi_{\Theta}$. Note that at this step, we generate recommendations
stochastically, drawing items from the distribution stochastically rather than
selecting the item maximum score; (step 3) for each pair history-
recommendations pair $\langle h,g\rangle$, compute advantages
$A_{1},A_{2}\ldots,A_{K}$ using Equation (10); (step 4) perform multiple
optimisation steps with respect to value function parameters $\Psi$ and GPTRec
parameters $\Theta$, using gradient descent with $\mathcal{L}_{PPO}$ loss
(Equation (13)).

This concludes the overview of the PPO algorithm for GPTRec optimisation. For
more details, we refer to the original PPO and GAE papers (Schulman et al.,
2017, 2016). ††margin:

##  Appendix B Sequence strucutre

Similar to the original GPT-2 model, GPTRec generates recommendations
autoregressively using the Next-K recommendations strategy. This means model
outputs at the $i$-th recommendations stage become model inputs at the
$i+1$-th stage (alongside the original user-item interaction history). To help
GPTRec’s training, we propose a input sequence structure that consists of 4
parts:

Figure 6. Proposed input sequence structure

  1. (1)

optional padding to equalise lengths of all sequences;

  2. (2)

user-item historical interactions $h=\\{h_{1},h_{2}..h_{n}\\};h_{i}\in I$;

  3. (3)

separator symbol that mark the end of the ”history” part;

  4. (4)

generated recommendations. $g=\\{g_{1},g_{2}..g_{k}\\};g_{i}\in I$

Each position in the sequence consists of three integer numbers: (1) a token
ID (the item ID, or one of two special tokens for padding and separator); (2)
a position ID used by the Transformer architecture to preserve positional
information; and (3) an attention mask, which tells GPTRec the positions in
the sequence it can ignore.

Initially, the sequence consists of the user history $h$ followed by a
separator token; then, at each next generation step $i$, the sequence shifts
left, and a recommended item $g_{i}$ is added to the end of the sequence.
Because GPTRec shifts the same sequence multiple times during generation, the
same position may have different semantics. For example, the second last
position belongs to the ”history” part at the first generation step, after the
first sequence shift separator moves to the second to last place, and at the
third step, it is occupied by the first recommended item. Therefore, to help
the model distinguish between different parts of the sequence, we adopt the
following position ID scheme:
$\bullet$ The separator token always has position ID 0;
$\bullet$ The position IDs of historical user-item interactions are in reverse
chronological order. This way, the last action (which is the most important
for recommendation (Petrov and Macdonald, 2022a)) always has position ID 1;
the second last has position ID 2, etc.;
$\bullet$ The position IDs associated with recommended items are in ascending
order starting from position ID $n+1$ where $n$ is the maximum length of the
user-item interactions history; i.e. first recommended item has position ID
$n+1$, second item $n+2$ etc. This way, each position in the recommendation is
associated with a specific position ID that does not change during the
generating process;
$\bullet$ The model ignores item padding tokens, so their position IDs can be
anything.