# Recommendations by Concise User Profiles from Review Text

Ghazaleh H. Torbati<EMAIL_ADDRESS>Max Planck Institute for
InformaticsSaarbrückenGermany ,  Anna Tigunova<EMAIL_ADDRESS>Max
Planck Institute for InformaticsSaarbrückenGermany ,  Andrew Yates
<EMAIL_ADDRESS>University of AmsterdamAmsterdamNetherlands and  Gerhard
Weikum<EMAIL_ADDRESS>Max Planck Institute for
InformaticsSaarbrückenGermany

###### Abstract.

Recommender systems are most successful for popular items and users with ample
interactions (likes, ratings etc.). This work addresses the difficult and
underexplored case of supporting users who have very sparse interactions but
post informative review texts. Our experimental studies address two book
communities with these characteristics. We design a framework with
Transformer-based representation learning, covering user-item interactions,
item content, and user-provided reviews. To overcome interaction sparseness,
we devise techniques for selecting the most informative cues to construct
concise user profiles. Comprehensive experiments, with datasets from Amazon
and Goodreads, show that judicious selection of text snippets achieves the
best performance, even in comparison to ChatGPT-generated user profiles.

recommender, sparse data, user text, language model, transformer

##  1\. Introduction

###  1.1. Motivation

Recommender systems have become a mature technology and are ubiquitous in
applications (Ricci et al., 2022). Methodologies fall into two major families
or hybrid combinations: i) interaction-based recommenders that leverage binary
signals (e.g., membership in personal playlists or libraries) or numeric
ratings for user-item pairs, and ii) content-based recommenders that exploit
item features and user-provided content, ranging from meta-data attributes
(e.g., item categories) all the way to textually rich reviews. In many
settings, where user-item signals are easily collectable and abundant,
interaction-based recommenders achieve excellent performance and are hard to
beat. Some applications combine interaction data with informative item
features, again relying on massive data (e.g., (Steck et al., 2021)).

In contrast, the data-poor regime of long-tail users and long-tail items has
received less attention. Some works advocate countering data sparseness by
clustering long-tail items (e.g., (Park, 2013)). However, this is not
sufficient to gear up for users with few interactions but diverse tastes.
Approaches for cold-start and cross-domain recommendations and zero-shot
learning (e.g., (Li et al., 2019; Zang et al., 2021; Raziperchikolaei et al.,
2021; Liu et al., 2022)), are often focused on the item side only: handling
new items by transferring (latent) knowledge from existing ones with dense
data.

Note that sparseness is not just a lack of user-item interactions, but also
refers to the users and items themselves: users with merely a handful of items
do not exhibit a clear picture of their tastes, and items that are completely
unseen at training time are a challenging case. Moreover, even users with a
good number of items can be a major problem when the preferred items are
highly diverse, such as less known books from all kinds of genres. To address
these difficulties, the most promising approach is to leverage review texts by
users. In settings where users spend substantial time per item, such as books
or travel destinations (as opposed to items with short attention spans, like
music or restaurants), even data-poor users with few interactions may leave
rich texts that express their interests and tastes. The approach has been
studied before (e.g., (Zhang et al., 2019; Wu et al., 2021; Chen et al., 2018;
Liu et al., 2019)), but the performance is much lower than for settings with
ample interaction data. This paper aims to advance the state of the art on
this theme, by tackling both issues of data sparseness and diversity of tastes
via text-based concise user profiles.

###  1.2. Research Questions

Our approach constructs and encodes concise user profiles from reviews of
data-poor text-rich users, focusing on the book domain for concreteness (using
data from Goodreads and Amazon). Unlike movies or restaurants, books have a
much longer tail of popularity and exhibit huge diversity of user tastes,
making books a particularly challenging domain for data-poor recommendations.
Also, negative reviews are very rare (most have ratings like 4 or 5 stars), so
that we deal with binary data and have no negative samples. This approach
raises several research questions (RQs):

  * RQ1:

Learning with Language Models. How can we best incorporate language models,
like BERT, T5 or ChatGPT, for encoding the gist of a user’s reviews, enhanced
with world knowledge?

  * RQ2:

Negative Samples. How can we mitigate the absence of negative samples for
training? What are the consequences for evaluating models?

  * RQ3:

Long Text. Since neural architectures can consume only limited input size
(e.g., 512 tokens ), we have to select a subset of snippets. Even with a large
capacity, the computational and energy cost is typically quadratic in the
number of input tokens, so that being selective has benefits. What are the
most informative cues that should be fed into the model?

  * RQ4:

Review Aspects. User reviews express a mix of aspects, including personal
background (e.g., “I’m a retired teacher”), sentiment expressions (e.g., “what
a great story”), general emotions (e.g., “brings back lovely memories”), and
comments about the book contents. As sentiments do not add value over the
already known fact that the book was liked, only the content comments yield
cues towards new recommendations. How should this issue be considered in
constructing a concise user profile with informative cues?

The paper’s emphasis is on how to capture user reviews into concise profiles,
that is, RQ3 and RQ4.

###  1.3. Approach

We devise a general framework, called CUP, for constructing $\underline{\rm
C}$oncise $\underline{\rm U}$ser $\underline{\rm P}$rofiles, and encoding them
in a recommender system, which allows us to configure a variety of methods. We
propose a Tranformer-based architecture that supports end-to-end learning of
item and user encodings, making use of a large language model. Our choice for
the language model is BERT; alternatives such as T5 can be easily plugged in
but incur higher computational cost. The end-to-end learning operates on
sufficiently short, judiciously constructed profiles, and leverages the
available user-item interactions as well.

On top of the Tranformer, we place feed-forward layers for a supervised
classifier. The classifier and the representation learning are jointly trained
end-to-end. The prediction scores for user-item pairs yield the per-user
ranking of items. This architecture can take as input a spectrum of user and
item information including user reviews and item descriptions and metadata
(e.g., categories).

We specifically devise new techniques for coping with the case where users
have few reviews with long and noisy text (see RQ3 and RQ4), such that their
total length exceeds the number of tokens that the encoders can consume. Our
techniques are based on information-retrieval-style statistics (like idf
weights) for n-grams or sentences, or on neural similarity measures (using
SentenceBERT), or on running review texts through a generative language model,
like T5 or ChatGPT, for generating user profiles.

In experiments, we limit the number of tokens per user profile to 128, as a
stress-test. The rationale for setting this tight limit is twofold. First, the
user-item data in benchmarks are samples of much larger websites where the
total text length per user is much higher. So 128 tokens can be viewed as
capturing the same fraction that 512 would cover on the complete data. Second,
the computational cost of the Transformer, and thus also the environmental
footprint, are quadratic in the number of input tokens. So less is more, from
a cost and energy perspective.

###  1.4. Contributions

The salient contributions of this work are the following:

  * $\bullet$

a new framework, called CUP, for Transformer-based recommenders that leverage
language models and concise user profiles from reviews;

  * $\bullet$

judicious techniques for selecting and encoding informative cues from the long
and noisy text of user reviews, outperforming prior baselines as well as
ChatGPT-generated profiles;

  * $\bullet$

comprehensive experiments with focus on data-poor but text-rich users with
highly diverse preferences, and difficult predictions (e.g., recommending new
authors, not just new books by previously seen authors).

We make code and experimental data available to reviewers under an anonymous
Github page111https://github.com/anonymousresearcherrandom/CUP. This will
eventually be made public.

##  2\. Related Work

Exploiting Item Features: Content-based recommender systems incorporate item
tags, item-item similarity, and user-side features. Item-item similarity
typically maps the tag clouds of items into a latent space and computes
distances between the embedding points. This can be combined with interaction-
based methods that employ latent-space techniques including deep learning
(e.g., (Zhang et al., 2019; Qi et al., 2021; Wu et al., 2020; Steck et al.,
2021; Fazelnia et al., 2022)) or graph-based inference (e.g., (Nikolakopoulos
and Karypis, 2019; Nikolakopoulos et al., 2021)). These methods excel in
performance, but experimental results often benefit from a large fraction of
favorable test cases. For example, when the model is trained with books by
some author, predicting that the user also likes other books by the same
author is likely a (near-trivial) hit. In our experiments, we ensure that such
cases are excluded.

Exploiting User Reviews: The most important user features are reviews of
items, posted with binary likes or numeric ratings. Early works either mine
sentiments on item aspects or map all textual information to latent features
using topic models like LDA or static word embeddings like word2vec (see,
e.g., (Chen et al., 2015)).

Recent works have shifted to deep neural networks with attention mechanisms
(Zheng et al., 2017; Zhang et al., 2017; Chen et al., 2018; Liu et al., 2019;
Wu et al., 2019, 2021), or feed review text into latent-factor models (Hu et
al., 2019; Shalom et al., 2019; Peña et al., 2020). Some works augment
collaborative filtering (CF) models with user text, to mitigate data
sparseness (e.g., (Sar Shalom et al., 2018; Lu et al., 2018; Liu et al.,
2020)). However, pre-dating the advent of large language models, all these
methods rely on static word-level encodings such as word2vec, and are
inherently limited. As a salient representative, we include DeepCoNN (Zheng et
al., 2017) in the baselines of our experiments.

Transformer-based inference: Recent works leverage pre-trained language models
(LMs), mostly BERT, for recommender systems in different ways: i) encoding
item-user CF signals with transformer-based embeddings, ii) making inference
from rich representations of the input review texts, or iii) implicitly
incorporating the “world knowledge” that LMs have in latent form. An early
representative of the first line is BERT4Rec (Sun et al., 2019; Petrov and
Macdonald, 2022), which uses BERT to learn item representations for sequential
predictions based on item titles and user-item interaction histories, but does
not incorporate any text. The P5 method of (Geng et al., 2022) employs a suite
of prompt templates for the T5 language model, in a multi-task learning
framework covering direct as well as sequential recommendations along with
generating textual explanations. We include a text-enriched variant of the P5
method in our experiments.

A representative of the line of works based on an LM’s “world knowledge” is
(Penha and Hauff, 2020), which employs masked probing of BERT to elicit its
knowledge about movie, music and book genres. The resulting candidate items
are then fed into a conversational recommender for ranking.

Closest to our approach are the methods of (Pugoy and Kao, 2020, 2021), using
BERT to create representations for user and item text, aggregated by averaging
(Pugoy and Kao, 2020) or k-means clustering (Pugoy and Kao, 2021). The
resulting latent vectors are used for predicting item scores. A major
limitation is that the text encodings are for individual sentences only, which
tends to lose signals from user reviews where cues span multiple sentences.
Also, BERT itself is fixed, and the latent vectors for users and items are
pre-computed without awareness of the prediction task. Our experiments include
the method of (Pugoy and Kao, 2020), called BENEFICT, as a baseline.

##  3\. Methodology

###  3.1. System Architecture

The CUP framework is based on a two-tower architecture for representation
learning (one ”tower” for users, the other for items, following the prevalent
architecture in neural information retrieval with query and document/passage
encodings). The two towers are jointly trained, coupled by the shared loss
function and ground truth. Figure 1 shows a pictorial overview.

User reviews and items descriptions are fed into BERT followed by a feed-
forward network to learn latent representations. Downstream, the vectors are
simply compared by a dot product for scores that indicate whether the user
likes an item or not. Importantly, unlike prior works, we allow the top-most
layer of BERT itself to be fine-tuned as part of the end-to-end training
process.

The per-item text usually comprises book titles, tags like categories or genre
labels, which can be coarse (e.g., “thriller”) or fine-grained (e.g.,
“Scandinavian crime noir”), and a short description of the book contents. The
per-user text can comprise the titles and tags of her training-set books and
the entirety of her review texts, which vary widely in length and
informativeness, hence the need for smart text selection.

In the following, we present the CUP training procedure in Subsection 3.2, the
test/deployment-time inference in Subsection 3.3, the judicious selection of
input text snippets in Subsection 3.4, and the selection of negative training
samples in Subsection 3.5.

Figure 1. CUP Architecture

.

###  3.2. Training

The input to CUP consists of an item description (almost always short,
otherwise truncated) and a judiciously selected subset of the user-provided
text (which may total to a longer text). For user $u$, this a sequence of text
tokens $w^{u}_{1}\dots w^{u}_{b}$, where $b$ is the token budget by which the
input is limited (set to 128 in our experiments). The sequences are fed
through each of the two towers, consisting of BERT and a feed-forward network
(FFN), to obtain a user-representation vector $t^{u}$ (by averaging the per-
token vectors). The FFN has two layers with ReLU activation

$t^{u}=ReLU(t^{u}W^{u}_{1}+c^{u}_{1})W^{u}_{2}+c^{u}_{2}$

for user representation, and analogously for items.

Optionally, we enhance text-based representations with collaborative filtering
(CF) signals, by concatenating the learned vectors with latent per-user and
per-item vectors $p^{u}$ and $p^{i}$, respectively. These are precomputed by
factorizing the training-set user-item interaction matrix, using the method of
(Funk, 2006), with gradient descent for minimizing a cross-entropy loss.

Another optional mechanism is to process multiple chunks of bounded-length
text, which could be of interest for long user reviews. In this case, the per-
user representations are learned for each chunk, and there is an additional
max-pooling step between the FFN and the dot product, to combine the per-chunk
vectors. We use this option for comparison in experiments. As it incurs $m$
times higher computational and energy costs, with $m$ being the number of
chunks, we do not favor it.

A user-item pair is classified with score calculated by

$s^{ui}=\sigma(<t^{u},t^{i}>)$

with $<,>$ denoting dot product and $\sigma$ for the sigmoid function.

We use the Adam optimizer to minimize the binary cross-entropy loss between
predicted labels and the ground truth. During training we update the top-most
layer of BERT, which allows end-to-end training of all components. This is an
important difference to earlier works with similar architectures, which use
frozen BERT to encode user and item representations.

###  3.3. Inference

Prediction for Ranking. At test time, a prediction is made for user-item
pairs. We encode the item description by running it through the trained
network, and we compare it to the already learned user vector, which is based
on the user’s training-time reviews (none for the given test item). The scores
for different test items, computed by the final dot product, yield the ranking
of the candidate items. This is a very efficient computation, following
established practice in neural information retrieval (Lin et al., 2021).

Search-based Recommendation. In a deployed system (as opposed to lab
experiments with test samples), a typical usage mode would be search-based re-
ranking: a user provides context with a tag-based query or an example of a
specific liked item, which can be thought of as query-by-example. The user’s
expectation is to see a ranked list of recommended items that are similar to
her positive sample (as opposed to recommendations from all kinds of
categories). The system achieves this by first computing approximate matches
to the query item (i.e., similarity-search neighbors), and then re-ranking a
shortlist of say top-100 candidates. The CUP framework supports this mode, by
using a light-weight BM25 retrieval model, querying all unlabeled items with
the category and textual description of the positive point at hand, and
keeping the top-100 highest scoring matches.

###  3.4. Coping with Long and Noisy Texts

For constructing user profiles from text, the simplest idea would be to
concatenate all available reviews into a long token sequence. Two problems
arise, though. User reviews are a noisy mix of descriptive elements (e.g.,
“the unusual murder weapon”), sentiment expressions (e.g., “it was fun to
read”) and personal but irrelevant statements (e.g., “I read only on
weekends”). Only the first aspect is helpful for content-based profiling (as
the sentiment is already captured by the fact that the user likes the book).
Second, the entirety of user-provided text can be too long to be fully
digested by the Transformer. Even when it would fit into the token budget, the
computational and energy cost is quadratic in the number of input tokens.
Therefore, we tightly limit the tokens for each user’s text profile to 128,
and have devise a suite of light-weight techniques for judiciously selecting
the most informative pieces.

Alternatives include generating concise profiles by generative language models
upfront (e.g., using T5 or ChatGPT), or digesting multiple chunks of text with
downstream aggregation by max-pooling or averaging. We include these
techniques in our experimental comparisons, but emphasize that both of these
alternatives have much higher computational cost and climate footprint.

Our techniques for selecting the most informative parts of user reviews to
construct concise profiles, are as follows:

  * $\bullet$

Weighted Phrases: selected unigrams, bigrams or trigrams, ordered by
descending tf-idf weights, where tf is the frequency of the phrase in all of
the given user’s reviews, and idf is pre-computed on Google books n-grams to
capture informativeness.

  * $\bullet$

Weighted Sentences: selected sentences, ordered by descending idf weights,
where a sentence’s total weight is the sum of the per-word idf weights
normalized by sentence length.

  * $\bullet$

Similar Sentences: selected sentences, ordered by descending similarity scores
computed via Sentence-BERT (Reimers and Gurevych, 2019) for comparing the
user-review sentences against the sentences of the item description. To ensure
that the selected set is not dominated by a single item, the sentences are
picked from different items in a round-robin manner.

  * $\bullet$

ChatGPT-generated Profiles: feeding all reviews of a user, in large chunks,
into ChatGPT and instructing it to characterize the user’s book interests with
a few short keyphrases. The output size is limited per chunk with a total
budget of 128 tokens, to enforce concise profiles.

  * $\bullet$

T5-generated Keywords: using a T5 model fine-tuned for keyword generation, to
cast each user’s review text into a set of keywords, concatenated to create
the profile.

We give anecdotal examples of selected profile constructions for a Goodreads
and an Amazon user in the Supplementary Materials.

###  3.5. Coping with Unlabeled Data

A challenge in the data-poor regime is how to handle the extreme skew between
positive samples and unlabeled data points for sparse users. The crux in many
recommender applications is that there are extremely few, if any, explicitly
negative samples, such as books rated with low marks. This holds also for the
datasets in this work.

Therefore, we introduce and experiment with two different techniques to
construct negative samples from unlabeled data:

  * $\bullet$

Uniform random samples. Under the closed world assumption (CWA), aka Selected
Completely At Random, negative training points are sample uniformly from all
unlabeled data. This is the baseline technique, widely used in prior works.

  * $\bullet$

Weighted pos-neg samples. Prior works on PU learning (Bekker and Davis, 2020),
with positive and unlabeled data and without explicitly negative samples, is
largely based on treating unlabeled points as pairs of samples, one positive
and one negative with fractional weights. The weights can be based on (learned
estimates of) class priors, but the extreme skew in our data renders these
techniques ineffective. Instead we leverage the fact that relatedness measures
between item pairs can be derived from tagging features or interaction data.
We compute item-item relatedness via basic matrix factorization of the user-
item matrix for the entire dataset. The relatedness of two items is set to the
scalar product between their latent vectors, re-scaled for normalization
between 0 and 1. Then, each originally unlabeled sample is cloned, with one
instance positive with weight proportional to its average relatedness to the
user’s explicitly positive points. The negative clone’s weight is set to the
complement.

##  4\. Experimental Design

Table 1.  Statistics for datasets (only ratings $\geq$ 4, #books per user $\geq$ 3, train-test disjoint authors per user)  | #books |  avg $\pm$ stdv | #users |  avg $\pm$ stdv |  avg $\pm$ stdv | #authors |  avg $\pm$ stdv |  avg $\pm$ stdv
---|---|---|---|---|---|---|---|---
|  | #u per book |  | #b per user | #a per user |  | #u per auth | review len
GR | 1,573,290 |  6.54 $\pm$ 55.95 | 279,969 |  36.77 $\pm$ 93.25 |  26.83 $\pm$ 57.33 | 547,714 |  13.71 $\pm$ 151.35 |  178.07 $\pm$ 258.63
GR-10K-dense | 385,660 |  3.74 $\pm$ 18.96 | 10,000 |  144.16 $\pm$ 232.49 |  95.91 $\pm$ 137.85 | 145,065 |  6.61 $\pm$ 41.19 |  187.37 $\pm$ 258.78
GR-10K-sparse | 158,554 |  1.66 $\pm$ 3.09 | 10,000 |  26.25 $\pm$ 56.46 |  19.9 $\pm$ 35.23 | 73,748 |  2.7 $\pm$ 8.72 |  172.72 $\pm$ 248.22
GR-1K-rich | 45,412 |  1.17 $\pm$ 0.65 | 1000 |  53.32 $\pm$ 91.04 |  38.44 $\pm$ 55.49 | 24,641 |  1.56 $\pm$ 1.89 |  426.23 $\pm$ 383.91
AM | 2,281,866 |  12.75 $\pm$ 87.25 | 3,105,696 |  9.37 $\pm$ 26.55 |  7.76 $\pm$ 18.02 | 1,143,422 |  21.08 $\pm$ 209.04 |  120.72 $\pm$ 183.08
AM-10K-dense | 203,930 |  2.51 $\pm$ 7.3 | 10,000 |  51.19 $\pm$ 177.37 |  34.57 $\pm$ 109.76 | 98,638 |  3.5 $\pm$ 13.24 |  238.79 $\pm$ 280.94
AM-10K-sparse | 58,443 |  1.36 $\pm$ 1.6 | 10,000 |  7.93 $\pm$ 11.19 |  6.88 $\pm$ 8.8 | 39,117 |  1.76 $\pm$ 3.07 |  106.25 $\pm$ 168.94
AM-1K-rich | 15,753 |  1.07 $\pm$ 0.36 | 1000 |  16.79 $\pm$ 23.53 |  14.27 $\pm$ 19.18 | 12,228 |  1.17 $\pm$ 0.62 |  282.15 $\pm$ 264.46

###  4.1. Rationale

As a difficult and less explored application area for recommender systems, we
investigate the case of book recommendations in online communities. These come
with a long-tailed distribution of user activities, highly diverse user
interests, and demanding textual cues from user reviews and book descriptions.

Unlike in many prior works’ experiments, often on movies, restaurants or
mainstream products, the data in our experiments is much sparser regarding
user-item interactions. Since we aim to obtain insights particularly on
research questions RQ3 and RQ4 (see Subsection 1.2), we design our evaluation
as a stress-test experiment, with focus on text-rich but otherwise data-poor
users: users who liked relatively few items but wrote informative, yet noisy
reviews.

We further enforce the difficulty of predictions when items belong to groups
with high relatedness within a group, by constraining disjointedness of
authors per user in training and evaluation set. Thus, we rule out the near-
trivial case of predicting that a user likes a certain book given that another
book by the same author has been used for training.

###  4.2. Datasets

We use two book datasets, both from the UCSD recommender systems repository
(McAuley, 2022):

  * $\bullet$

GR (Wan and McAuley, 2018): a Goodreads sample with item-user interactions for
1.5M books and 280K users, incl. titles, genre tags, item descriptions,
ratings and textual reviews.

  * $\bullet$

AM (Ni et al., 2019): an Amazon crawl, filtered for the books domain with 2.3M
books and 3.1M users, incl. category tags, ratings and reviews.

While the GR data hardly appears in the literature, the AM-books data has been
used for experiments in prior works (e.g., (Wang et al., 2019)). However, the
data has been pre-processed in numerous variants, with differences in data
size, definition of positive and negative points, inclusion of features
(mostly disregarding reviews), and elimination of long-tail users and items.
The most widely used derivative is the 10-core variant where all users with
less than 10 items and all items with less than 10 users are eliminated. This
pre-processing clearly focuses on interaction-based predictions, whereas our
intention is to study the underexplored case of sparse interactions with
informative user reviews.

We view all book-user interactions with a rating of 4 or higher as positive,
and disregard the lower ratings as they are rare anyway. We further pre-
process the dataset by removing all the users with less than 3 books, as we
cannot split their interactions into the three sets (train, validation, test).

Table 1 gives statistics for the pre-processed GR and AM data. Our data pre-
processing is designed to evaluate text-based recsys performance with low
interaction density and text-rich users. We select 1K users from each of the
two datasets, based on descending order of average review length per book,
such that each user has reviewed at least 3 books. In Table 1, rows GR-1K-rich
and AM-1K-rich show the characteristics of these data slices. Both GR-1K-rich
and AM-1K-rich are extremely sparse in terms of users that share the same
items; so the emphasis is on leveraging text.

To investigate the influence of interaction density, we construct two further
slices from AM and GR, both covering 10K users in two variants: sparse (but
not as sparse as the 1K-rich slices) and dense. These are used only in Section
5.3 for sensitivity studies, and thus explained there.

###  4.3. Baselines

We compare our approach to several state-of-the-art baselines, which cover
different methods for recommendation, ranging from traditional collaborative
filtering approaches to text-centric neural models. We compare the following
methods:

  * •

CF: collaborative filtering operating on the user-item interaction matrix by
pre-computing per-user and per-item vectors via matrix factorization (Funk,
2006) (with 200 latent dimensions).

  * •

RecWalk (Nikolakopoulos and Karypis, 2019) makes predictions by random walks
over the interaction graph, incorporating item-item similarities to encourage
the exploration of long-tail items. We pre-compute the similarities by
factorizing the full-data matrix (i.e., over ca. 10M and 25M interactions in
GR and AM, resp.).

  * •

DeepCoNN (Zheng et al., 2017) is a salient representative of using
convolutional neural networks (CNN) over text inputs.

  * •

P5 (Geng et al., 2022): prompting the T5 language model (Raffel et al., 2020),
to provide a recommended item for a user, given their ids. Following (Geng et
al., 2022), we train P5 using the prompts for direct recommendation (e.g.
“Shall we recommend item item_id to user_id”) to generate a “yes” or “no”
answer. Pilot experiments show that the original method does not work well on
sparse data. Therefore, we extend P5 to leverage review texts and item
descriptions. Instead of ids, the prompts include item descriptions and
selected sentences from reviews, with the highest idf scores (i.e., one of our
own techniques).

  * •

BENEFICT (Pugoy and Kao, 2020) uses BERT to create representations for each
user review, which are averaged and concatenated to the item vectors.
Predictions are made by a feed-forward network on top. Following the original
paper, each review is truncated to its first 256 tokens.

  * •

BENEFICT-text: our own variant of BENEFICT where the averaging over all
reviews of a user is replaced by our idf-based selection of most informative
sentences, with the total length limited to 128 tokens (for comparability to
the CUP methods).

  * •

BERT5-text: using vanilla BERT (out of the box, without any fine-tuning) for
encoding user text and item descriptions, followed by a feed-forward network.
The text selection is 5 chunks of 128-tokens of the highest-idf sentences,
with max-pooling for aggregation.

All methods were run on NVIDIA Quadro RTX 8000 GPU with 48 GB memory, and we
implemented the models with PyTorch.

###  4.4. Performance Metrics

At test time, we present the trained system with each user’s withheld positive
items (20% of the user’s books, with authors disjoint from those of the user’s
training items), along with negative items, sampled from all non-positive
items, such that the ratio of positive to negative test points is 1:100. The
learned system scores and ranks these data points. We evaluate all methods in
two different modes of operation:

  * $\bullet$

Standard prediction: sampling the 100 negative test points uniformly at random
from all unlabeled data, and ranking the 100+1 test instances by the methods
under test.

  * $\bullet$

Search-based: given the positive test item, searching for the top-100
approximate matches to the item’s description, using the BM25 scoring model;
then ranking the 100+1 candidates by our methods.

Following the literature, our evaluation metrics are NDCG@5 (Normalized
Discounted Cumulative Gain) with binary 0-or-1 gain) and P@1 (precision at
rank 1). We compute these by micro-averaging (over all test items of all
users). We also experimented with macro-averaging over users; as the results
were not significantly different, we report only micro-average numbers in the
paper.

NDCG@5 reflects the observations that users care only about a short list of
top-N recommendations; P@1 is suitable for recommendations on mobile devices
(with limited UI). We also measured other metrics, like NDCG@k for higher k,
MRR and AUC. None of these provides any additional insight, so they are not
reported here.

###  4.5. Configurations

The CUP framework supports a variety of methods by specific configurations.
All variants use an input budget of 128 tokens, to construct a stress-test and
to emphasize that computational and environmental footprint is a major concern
as well. In the experiments, we focus on the following text-centric options
(see Subsection 3.4):

  * $\bullet$

CUPidf: sentences from reviews selected by idf scores.

  * $\bullet$

CUPsbert: sentences from reviews using Sentence-BERT similarity against an
item description.

  * $\bullet$

CUP1gram: single words selected by tf-idf scores.

  * $\bullet$

CUP3gram: word-level 3-grams selected by tf-idf scores.

  * $\bullet$

CUPGPT: a concise user profile generated by ChatGPT from all reviews of a
user.

  * $\bullet$

CUPkeywords: a user profile consisting of keywords that are generated by a
fine-tuned T5 model.222https://huggingface.co/ml6team/keyphrase-
generation-t5-small-inspec

When comparing against prior baselines, we employ CUPidf as a default
configuration. For comparison, we also configure more restricted variants that
use only category/genre tags for user text and title, category/genre, and
optionally description for item text. These are denoted as CUPbasic and
CUPexpanded, respectively.

To obtain refined insights into the performance for specific kinds of users
and items, we split the 1000 users and their items into the following groups,
reporting NDCG@5 for each group separately. Note that this refinement drills
down on the test outputs; the training is completely unaffected.

  * $\bullet$

Items are split into unseen (u) and seen (s) items. The former consist of all
test-time items that have not been seen at training time. The latter are those
items that appear as positive samples at test-time and are also among the
positive training items (for a different user)

  * $\bullet$

Users are split into three groups based on the #books-per-user distribution
(in train/dev/test points):

    * $\bullet$

Sporadic (s) users are the lowest 50% with the least numbers of books. For
GR-1K-rich, this threshold is 13 books per user; for AM-1K-rich it is 5 (with
means 6 and 3, resp.).

    * $\bullet$

Regular (r) users are those between the 50th percentile and 90th percentile,
which is between 13 and 71 books per user for GR-1K-rich, and between 5 and 20
for AM-1K-rich (with means 31 and 9, resp.).

    * $\bullet$

Bibliophilic (b) users are the highest 10%: above 75 books per user for
GR-1K-rich and above 20 for AM-1K-rich (with means 156 and 43, resp.).

We abbreviate these six groups as u-s, u-r, u-b, s-s, s-r and s-b.

Table 2. Standard Evaluation on both datasets. AM-1K-rich
---
| Train Uniform | Train Weighted
Method | NDCG@5 | P@1 | NDCG@5 | P@1
CF | 3.06 | 1.0 | 2.88 | 0.69
RecWalk | 1.17 | 0.55 | n/a | n/a
DeepCoNN | 3.0 | 0.8 | 3.01 | 0.83
P5text | 24.9 | 14.5* | n/a | n/a
BENEFICT | 9.4 | 3.53 | 14.4 | 5.74
BENEFICTtext | 24.38 | 12.98 | 24.66 | 12.81
BERT5text | 24.38 | 12.06 | 28.26* | 15.04*
CUPbasic | 25.42 | 13.64 | 26.95* | 14.27*
CUPexp | 27.31* | 14.99* | 28.83* | 16.05*
CUPtext |  29.21* | 15.82* |  31.09* |  17.71*
GR-1K-rich
CF | 4.44 | 2.69 | 3.83 | 2.26
RecWalk | 3.56 | 1.91 | n/a | n/a
DeepCoNN | 10.45 | 4.02 | 5.4 | 1.86
P5text | 28.01 | 14.46 | n/a | n/a
BENEFICT | 23.76 | 12.17 | 25.23 | 13.26
BENEFICTtext | 30.26 | 16.83 | 31.77 | 17.74
BERT5text | 33.24* | 18.64* | 36.12* | 20.56*
CUPbasic | 26.75 | 13.28 | 28.4 | 14.89
CUPexp | 30.92 | 16.3 | 33.28* | 17.83
CUPtext |  38.39* |  22.26* | 39.41* | 22.01*

##  5\. Experimental Results

###  5.1. Comparison of CUP against Baselines

Table 2 shows the results for the AM-1K-rich and GR-1K-rich data, comparing
our default configuration CUPtext (with idf-selected sentences) against all
baselines, for the two different ways of sampling negative training points.
Results with statistical significance over the BENEFICTtext baseline, by a
paired t-test with p-value $<$ 0.05, are marked with an asterisk. We make the
following key observations:

  * $\bullet$

The interaction-centric CF and RecWalk fail completely for this extremely
sparse data. The out-of-the-box baseline DeepCoNN also performs very poor, and
the BENEFICT method is only slightly better. At the same time, P5 and
BENEFICT, extended with our text-derived profiling, achieve decent
performance.

  * $\bullet$

The method with pre-computed, frozen BERT on 5-chunk input also performs
fairly well, particularly when harnessing weighted training. Note that the
selection of the text chunks already makes use of the CUP profiling by idf-
scored sentences.

  * $\bullet$

Between the three CUP configurations, we see a clear trend: titles and tags
alone ($basic$) are outperformed by adding item descriptions ($exp$), and the
profiling over user reviews ($text$) performs best. Remarkably, even CUPbasic
is better than all the baselines. CUPtext is ca. 5 percentage points better in
NDCG@5 than the baselines.

  * $\bullet$

In search-based evaluation, the absolute results are much lower, emphasizing
the difficulty of this realistic mode. Nonetheless, the relative comparisons
between methods are nearly identical to the observations with standard
evaluation. Again, CUPtext is the winner, with a clear margin.

  * $\bullet$

The absolute numbers on GR-1K-rich are generally higher, due to the different
data characteristics. The gains by CUPtext over the baselines and over the
simpler CUP configurations are even more pronounced (e.g., outperforming
BENEFICTtext by 8 percentage points with standard evaluation).

Table 3. Search-based Evaluation on both datasets. AM-1K-rich
---
| Train Uniform | Train Weighted
Method | NDCG@5 | P@1 | NDCG@5 | P@1
CF | 3.02 | 1.0 | 3.03 | 0.83
RecWalk | 1.25 | 0.75 | n/a | n/a
DeepCoNN | 0.05 | 0.0 | 0.05 | 0.0
P5text | 8.4* | 3.88* | n/a | n/a
BENEFICT | 2.45 | 0.66 | 3.69 | 1.29
BENEFICTtext | 6.49 | 2.33 | 8.11 | 3.53
BERT5text | 7.66* | 3.1* | 8.67 | 3.47
CUPbasic | 7.13 | 2.76 | 6.87 | 2.61
CUPexp | 7.41* | 2.93 | 7.0 | 2.84
CUPtext | 8.93* | 3.53* | 9.14* | 4.02
GR-1K-rich
CF | 3.45 | 1.72 | 3.05 | 1.5
RecWalk | 2.95 | 1.35 | n/a | n/a
DeepCoNN | 1.83 | 0.59 | 1.35 | 0.38
P5text | 8.69 | 2.94 | n/a | n/a
BENEFICT | 6.52 | 2.25 | 6.67 | 2.33
BENEFICTtext | 8.57 | 3.22 | 9.2 | 3.46
BERT5text | 9.86* | 3.7* | 10.86* | 4.33*
CUPbasic | 8.91 | 3.24 | 9.22 | 3.04
CUPexp | 10.23* | 4.06* | 10.57* | 3.64
CUPtext |  13.42* |  5.6* |  12.91* |  4.8*

Additionally, we ran a variant of the best performing model (CUPtext) using
increased input size of 256 tokens. We observed only slight improvements at
best, not significant enough to warrant the quadratic cost of doubling the
input length.

We also ran extended CUP configurations where pre-computed CF vectors were
incorporated (see Section 3.2). These achieved only very minor gains in some
of the settings, and there was even a detrimental effect on some cases. We
omit these hybrid methods here. We will revisit them in Subsection 5.3 for
sensitivity studies on data density. Another observation is that the weighted
training almost always improves performance. Therefore, the following
subsections focus on results with weighted training samples.

###  5.2. Comparison of CUP Configurations

We now turn to comparing all CUP configurations, with drilling down on user
and item groups. Table 4 shows the NDCG@5 results for the AM-1K-rich and
GR-1K-rich data. From here on, all presented results are obtained with
weighted training (as this outperforms uniform training) and with search-based
evaluation (corresponding results with standard evaluation are in the
Supplementary Materials). We focus on NDCG@5, as the P@1 metric strongly
correlates. The asterisk for statistical significance is by a paired t-test
with p $<$ 0.05 against CUPidf. We offer the following notable observations:

  * $\bullet$

Across all users, all CUP configurations are competitive. The overall
differences between them are relatively small. The winner, by a small margin,
is CUPkw, closely followed by the default configuration CUPidf and CUPsbert as
well as CUP1gram. None of the methods is able to extract the “perfect” gist
from the noisy review texts; but all of them do a decent job. Despite the fact
that CUPkw is slightly ahead of the others, the bottom line is that a
relatively simple configuration, like idf-selected sentences, is a very good
choice. Notably, even the seemingly most promising option CUPGPT mostly loses
against the simpler configurations.

  * $\bullet$

The T5-based CUPkw variant achieves its highest gains for the richer item/user
groups: seen items and regular or bibliophilic users. This provides T5 with
longer and more informative texts. A similar effect, but to a lesser and
noisier extent, can be observed for CUPGPT. Conversely, these methods perform
substantially worse on the sporadic-unseen group.

  * $\bullet$

On the GR-1K-rich data, the overall performance is higher than for AM across
all configurations. This can be attributed to the fact that GR has longer
reviews, and these texts tend to be more informative than the ones in the AM
data (which sometimes refer to packaging, shipment and other non-content
aspects).

Table 4. CUP Results for AM-1K-rich and GR-1K-rich by User/Item Groups (NDCG@5
with Search-based Evaluation). AM-1K-rich
---
Method | ALL | u-s | u-r | u-b | s-s | s-r | s-b
CUPidf | 9.14 | 5.93 | 7.09 | 12.0 | 8.6 | 11.71 | 12.78
CUPsbert | 9.0 | 5.19 | 7.32 | 11.46 | 9.44 | 14.53 | 14.23
CUP1gram | 9.08 | 6.24 | 7.14 | 11.21 | 9.76 | 17.0 | 13.25
CUP3gram | 8.98 | 5.54 | 7.01 | 11.81 | 8.2 | 13.97 | 10.99
CUPkw | 9.5 | 6.03 | 7.15 | 12.48 | 8.71 | 12.38 | 17.78
CUPGPT | 8.93 | 5.95 | 6.95 | 11.59 | 8.29 | 11.3 | 13.85
GR-1K-rich
CUPidf | 12.91 | 7.14 | 10.93 | 14.37 | 13.68 | 14.71 | 17.75
CUPsbert | 12.58 | 7.81 | 9.77 | 14.38 | 13.09 | 15.07 | 17.52
CUP1gram | 12.59 | 7.53 | 10.11 | 14.0 | 13.79 | 15.04 | 18.35
CUP3gram | 12.36 | 6.94 | 9.92 | 14.52 | 11.69 | 13.87 | 15.82
CUPkw | 12.72 | 6.98 | 9.83 | 14.24 | 14.0 | 15.71 | 19.82*
CUPGPT | 12.87 | 7.46 | 9.99 | 13.76 | 15.73 |  17.91* |  20.62*

###  5.3. Influence of Interaction Density

The slices AM-1K-rich and GR-1K-rich are constructed with a text-centric
stress-test in mind. Both are extremely sparse in terms of user-item
interactions. To study the influence of sparseness in isolation, we created
two larger samples of both datasets, one still sparse by design and the other
denser in terms of users sharing items. We refer to these as AM-10K-sparse and
AM-10K-dense, and analogously for GR. All these samples cover 10K users: 10x
larger than our previous text-rich slices, to allow more potential for
learning from interactions. Table 1 in Section 4.2 shows statistics.

Table 5.  NDCG@5 for AM-10K Sparse vs. Dense (Search-based Evaluation, by Item/User Groups)  Method | ALL | u-s | u-r | u-b | s-s | s-r | s-b
---|---|---|---|---|---|---|---
AM-10K-Sparse
CF | 5.06 | 0.0 | 0.01 | 0.01 | 18.79 | 17.16 | 12.39
CUPtext | 9.78 | 3.4 | 3.9 | 6.64 | 22.43 | 21.65 | 18.29
CUPtext+CF | 5.79 | 1.65 | 0.43 | 0.13 | 19.45 | 18.77 | 13.0
AM-10K-Dense
CF | 19.02 | 0.13 | 0.03 | 0.14 | 51.76 | 34.75 | 21.46
CUPtext | 20.24 | 1.23 | 2.44 | 5.81 | 49.98 | 34.4 | 21.12
CUPtext+CF | 19.12 | 0.38 | 0.04 | 0.27 | 51.7 | 35.04 | 21.49
Table 6.  NDCG@5 for GR-10K Sparse vs. Dense (Search-based Evaluation, by Item/User Groups)  Method | ALL | u-s | u-r | u-b | s-s | s-r | s-b
---|---|---|---|---|---|---|---
GR-10K-Sparse
CF | 14.08 | 0.0 | 0.0 | 0.0 | 41.65 | 32.67 | 25.21
CUPtext | 15.45 | 3.35 | 5.27 | 8.45 | 31.52 | 26.93 | 22.18
CUPtext+CF | 14.85 | 1.87 | 0.55 | 0.06 | 39.4 | 35.48 | 26.53
GR-10K-Dense
CF | 35.09 | 0.02 | 0.0 | 0.0 | 57.01 | 48.49 | 36.07
CUPtext | 26.03 | 3.33 | 3.76 | 6.48 | 39.4 | 33.74 | 26.72
CUPtext+CF | 36.17 | 0.27 | 0.04 | 0.03 | 57.3 | 49.46 | 38.19

The datasets are constructed as follows. To ensure connectivity in the
interaction graphs, we first select 500 users and sample 2000 books connected
to these users, both uniformly at random. This is the seed for constructing
two variants of data, based on the cumulative item degrees of users, that is,
the sum of the #users per book for all books that the user has in her
collection:

  * $\bullet$

10K-dense: We randomly select 10K (minus the initial 500) users from all users
in proportion to the users’ cumulative item degrees. This favors users with
many or popular books.

  * $\bullet$

10K-sparse: We select users inversely proportional to their cumulative item
degrees, thus favoring sparse users.

In this sensitivity study, we focus on comparing CF (based on matrix
factorization) against our default CUPtext configuration (with idf-selected
sentences) and the hybrid combination CUPtext \+ CF. Tables 5 and 6 show the
results for the AM-10K data and GR-10K data, comparing the sparse vs. the
dense variants with search-based evaluation (corresponding results with
standard evaluation are in the Supplementary Materials).

Key insights are:

  * $\bullet$

As the 10K-sparse data is already much denser than the stress-test 1K-rich
slices, CF can achieve decent results on the sparse data. Especially in
search-based mode, CF is almost on par with CUP. Note that these gains come
from the seen items alone, as CF is bound to fail on unseen items.

  * $\bullet$

CUPtext is still the clear winner on the sparse variant. The hybrid
configuration, with text and CF vectors combined, is inferior to learning from
text alone.

  * $\bullet$

For the 10K-dense data, CF alone performs well, as this is the traditional
regime for which CF has been invented. An interesting point arises for the
search-based mode and with dense data. Here, the negative test points are
closer to the positive sample (after the BM25 search), and pose higher
difficulty for text cues alone to discriminate them. Thus, CF becomes more
competitive. On GR, hybrid CUP+CF performs best.

##  6\. Conclusion

This work addressed text-centric recommender systems in data-poor situations,
specifically for the demanding case of book reviewing communities where many
users have only few items and exhibit diverse tastes, but post text-rich
comments. We presented a Transformer-based framework called CUP, with novel
techniques for constructing concise user profiles by selecting informative
pieces of user-side text. To mitigate the absence of explicitly negative
training points, we employed a technique of picking weighted negative samples
from unlabeled data. Our experiments, with both standard evaluation and a
search-based mode, show that leveraging user text is beneficial in this data-
poor regime, and that our CUP methods clearly outperform state-of-the-art
baselines like DeepCoNN, BENEFICT, P5 and a method based on frozen BERT.

Our main goal has been to shed light into research questions RQ3 and RQ4 (see
Subsection 1.2) about coping with long user text and how to mitigate their
noisy contents. For RQ3, we limited the number of input tokens for the
Transformer to 128, creating a stress-test but also aiming to minimize the
computational and energy cost at training time. Our experimental findings
confirmed that this tight length for profiling user interests is indeed
viable, yielding fairly good NDCG@5 and P@1 results (relative to the
difficulty of this data).

For RQ4, we devised various techniques for selecting informative cues from
user reviews. A key observation from the experiments is that a moderate degree
of sophistication, like idf-selected sentences, is largely sufficient. Our
default configuration CUPidf performs on par with more elaborated variants,
such as leveraging SentenceBERT or having T5-generated keywords for concise
profiles. Moreover, CUPidf outperforms even the profiles that were generated
by ChatGPT with the full review texts as input.

##  7\. Ethics Statement

Ethical concerns which are potentially relevant are about the privacy of the
users whose likes and reviews are kept in the data and the appropriateness of
the review contents. All data in our experiments was obtained from the public
repository at UCSD (McAuley, 2022). To the best of our knowledge, these
datasets were already sanitized and anonymized for public research. The
reviews on Amazon and Goodreads comply to the community guidelines of these
websites333https://www.amazon.com/gp/help/customer/display.html?nodeId=GLHXEX85MENUE4XF444https://www.goodreads.com/community/guidelines,
prohibiting hate speech, spam and otherwise offensive content.

## References

  * (1)
  * Bekker and Davis (2020) Jessa Bekker and Jesse Davis. 2020.  Learning from positive and unlabeled data: a survey.  _Mach. Learn._ 109, 4 (2020), 719–760.
  * Chen et al. (2018) Chong Chen, Min Zhang, Yiqun Liu, and Shaoping Ma. 2018\.  Neural Attentional Rating Regression with Review-level Explanations. In _2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, Apr 23-27, 2018_. ACM, 1583–1592.
  * Chen et al. (2015) Li Chen, Guanliang Chen, and Feng Wang. 2015.  Recommender systems based on user reviews: the state of the art.  _User Model. User Adapt. Interact._ 25, 2 (2015), 99–154.
  * Fazelnia et al. (2022) Ghazal Fazelnia, Eric Simon, Ian Anderson, Benjamin A. Carterette, and Mounia Lalmas. 2022\.  Variational User Modeling with Slow and Fast Features. In _WSDM ’22: The Fifteenth ACM International Conference on Web Search and Data Mining, Feb 21 - 25, 2022_. ACM, 271–279.
  * Funk (2006) Simon Funk. 2006\.  _Netflix Update: Try This at Home_.  https://sifter.org/~simon/journal/20061211.html, accessed on Sep 29, 2022.
  * Geng et al. (2022) Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022.  Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5). In _RecSys ’22: Sixteenth ACM Conference on Recommender Systems, Seattle, WA, USA, September 18 - 23, 2022_. ACM, 299–315.
  * Hu et al. (2019) Guangneng Hu, Yu Zhang, and Qiang Yang. 2019.  Transfer Meets Hybrid: A Synthetic Approach for Cross-Domain Collaborative Filtering with Text. In _The World Wide Web Conference, WWW 2019, May 13-17, 2019_. ACM, 2822–2829.
  * Li et al. (2019) Jingjing Li, Mengmeng Jing, Ke Lu, Lei Zhu, Yang Yang, and Zi Huang. 2019\.  From Zero-Shot Learning to Cold-Start Recommendation. In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, Jan 27 - Feb 1, 2019._ AAAI Press, 4189–4196.
  * Lin et al. (2021) Jimmy Lin, Rodrigo Frassetto Nogueira, and Andrew Yates. 2021.  _Pretrained Transformers for Text Ranking: BERT and Beyond_.  Morgan & Claypool Publishers.
  * Liu et al. (2022) Bulou Liu, Bing Bai, Weibang Xie, Yiwen Guo, and Hao Chen. 2022.  Task-optimized User Clustering based on Mobile App Usage for Cold-start Recommendations. In _KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Aug 14 - 18, 2022_. ACM, 3347–3356.
  * Liu et al. (2019) Donghua Liu, Jing Li, Bo Du, Jun Chang, and Rong Gao. 2019.  DAML: Dual Attention Mutual Learning between Ratings and Reviews for Item Recommendation. In _25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Aug 4-8, 2019_. ACM, 344–352.
  * Liu et al. (2020) Hongtao Liu, Yian Wang, Qiyao Peng, Fangzhao Wu, Lin Gan, Lin Pan, and Pengfei Jiao. 2020.  Hybrid neural recommendation with joint deep representation learning of ratings and reviews.  _Neurocomputing_ 374 (2020), 77–85.
  * Lu et al. (2018) Yichao Lu, Ruihai Dong, and Barry Smyth. 2018.  Coevolutionary recommendation model: Mutual learning between ratings and reviews. In _Proceedings of the 2018 World Wide Web Conference_. 773–782.
  * McAuley (2022) Julian McAuley. 2022\.  _Recommender Systems and Personalization Datasets_.  https://cseweb.ucsd.edu/~jmcauley/datasets.html, accessed on Sep 29, 2022.
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)_. 188–197.
  * Nikolakopoulos and Karypis (2019) Athanasios N. Nikolakopoulos and George Karypis. 2019.  RecWalk: Nearly Uncoupled Random Walks for Top-N Recommendation. In _Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, Feb 11-15, 2019_. ACM, 150–158.
  * Nikolakopoulos et al. (2021) Athanasios N. Nikolakopoulos, Xia Ning, Christian Desrosiers, and George Karypis. 2021.  Trust your neighbors: A comprehensive survey of neighborhood-based methods for recommender systems.  _CoRR_ abs/2109.04584 (2021).  arXiv:2109.04584
  * Park (2013) Yoon-Joo Park. 2013\.  The Adaptive Clustering Method for the Long Tail Problem of Recommender Systems.  _IEEE Trans. Knowl. Data Eng._ 25, 8 (2013), 1904–1915.
  * Peña et al. (2020) Francisco J. Peña, Diarmuid O’Reilly-Morgan, Elias Z. Tragos, Neil Hurley, Erika Duriakova, Barry Smyth, and Aonghus Lawlor. 2020\.  Combining Rating and Review Data by Initializing Latent Factor Models with Topic Models for Top-N Recommendation. In _RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Sep 22-26, 2020_. ACM, 438–443.
  * Penha and Hauff (2020) Gustavo Penha and Claudia Hauff. 2020.  What does BERT know about books, movies and music? Probing BERT for Conversational Recommendation. In _RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Sep 22-26, 2020_. ACM, 388–397.
  * Petrov and Macdonald (2022) Aleksandr Petrov and Craig Macdonald. 2022.  A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation. In _RecSys ’22: Sixteenth ACM Conference on Recommender Systems, Sep 18 - 23, 2022_. ACM, 436–447.
  * Pugoy and Kao (2020) Reinald Adrian Pugoy and Hung-Yu Kao. 2020.  BERT-Based Neural Collaborative Filtering and Fixed-Length Contiguous Tokens Explanation. In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_. Association for Computational Linguistics, Suzhou, China, 143–153.  https://aclanthology.org/2020.aacl-main.18
  * Pugoy and Kao (2021) Reinald Adrian Pugoy and Hung-Yu Kao. 2021.  Unsupervised extractive summarization-based representations for accurate and explainable collaborative filtering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 2981–2990.
  * Qi et al. (2021) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021\.  Personalized News Recommendation with Knowledge-aware Interactive Matching. In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Canada, July 11-15, 2021_. ACM, 61–70.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5485–5551.
  * Raziperchikolaei et al. (2021) Ramin Raziperchikolaei, Guannan Liang, and Young-joo Chung. 2021\.  Shared Neural Item Representations for Completely Cold Start Problem. In _RecSys ’21: Fifteenth ACM Conference on Recommender Systems, 27 Sep 2021 - 1 Oct 2021_. ACM, 422–431.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, Nov 3-7, 2019_. Association for Computational Linguistics, 3980–3990.
  * Ricci et al. (2022) Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). 2022\.  _Recommender Systems Handbook_.  Springer US.
  * Sar Shalom et al. (2018) Oren Sar Shalom, Guy Uziel, Alexandros Karatzoglou, and Amir Kantor. 2018.  A word is worth a thousand ratings: Augmenting ratings using reviews for collaborative filtering. In _Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval_. 11–18.
  * Shalom et al. (2019) Oren Sar Shalom, Guy Uziel, and Amir Kantor. 2019\.  A generative model for review-based recommendations. In _13th ACM Conference on Recommender Systems, RecSys 2019, Copenhagen, Denmark, Sep 16-20, 2019_. ACM, 353–357.
  * Steck et al. (2021) Harald Steck, Linas Baltrunas, Ehtsham Elahi, Dawen Liang, Yves Raimond, and Justin Basilico. 2021\.  Deep Learning for Recommender Systems: A Netflix Case Study.  _AI Mag._ 42, 3 (2021), 7–18.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In _28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Nov 3-7, 2019_. ACM, 1441–1450.
  * Wan and McAuley (2018) Mengting Wan and Julian McAuley. 2018.  Item recommendation on monotonic behavior chains. In _Proceedings of the 12th ACM conference on recommender systems_. 86–94.
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019.  Neural graph collaborative filtering. In _Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval_. 165–174.
  * Wu et al. (2019) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019\.  Neural news recommendation with multi-head self-attention. In _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)_. 6389–6394.
  * Wu et al. (2021) Le Wu, Xiangnan He, Xiang Wang, Kun Zhang, and Meng Wang. 2021.  A Survey on Neural Recommendation: From Collaborative Filtering to Content and Context Enriched Recommendation.  _CoRR_ abs/2104.13030 (2021).  arXiv:2104.13030
  * Wu et al. (2020) Shiwen Wu, Wentao Zhang, Fei Sun, and Bin Cui. 2020\.  Graph Neural Networks in Recommender Systems: A Survey.  _CoRR_ abs/2011.02260 (2020).  arXiv:2011.02260
  * Zang et al. (2021) Tianzi Zang, Yanmin Zhu, Haobing Liu, Ruohan Zhang, and Jiadi Yu. 2021.  A Survey on Cross-domain Recommendation: Taxonomies, Methods, and Future Directions.  _CoRR_ abs/2108.03357 (2021).  arXiv:2108.03357
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019\.  Deep Learning Based Recommender System: A Survey and New Perspectives.  _ACM Comput. Surv._ 52, 1 (2019), 5:1–5:38.
  * Zhang et al. (2017) Yongfeng Zhang, Qingyao Ai, Xu Chen, and W. Bruce Croft. 2017\.  Joint Representation Learning for Top-N Recommendation with Heterogeneous Information Sources. In _2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, Nov 06 - 10, 2017_. ACM, 1449–1458.
  * Zheng et al. (2017) Lei Zheng, Vahid Noroozi, and Philip S. Yu. 2017.  Joint Deep Modeling of Users and Items Using Reviews for Recommendation _(WSDM ’17)_. Association for Computing Machinery, New York, NY, USA, 425–434.

##  Appendix A Supplementary Material

###  A.1. Comparisons of CUP Configurations with Standard Evaluation

Table 7 shows the NDCG@5 results using standard evaluation for AM-1K-rich and
GR-1K-rich datasets.

Table 7. CUP Results for AM-1K-rich and GR-1K-rich by User/Item Groups (NDCG@5
with Standard Evaluation). AM-1K-rich
---
Method | ALL | u-s | u-r | u-b | s-s | s-r | s-b
CUPidf | 31.09 | 23.58 | 27.75 | 36.84 | 30.58 | 41.04 | 27.77
CUPsbert | 31.18 | 23.96 |  29.7* | 35.22 | 30.45 | 37.34 | 32.26
CUP1gram | 31.13 | 23.72 | 28.24 | 36.0 | 32.21 | 42.09 | 32.99
CUP3gram | 30.6 | 23.02 | 27.65 | 36.05 | 29.02 | 38.48 | 31.4
CUPkw | 31.55 |  25.98* | 29.04 | 35.97 | 25.81 | 38.38 | 34.22
CUPGPT | 30.82 | 23.52 | 28.59 | 35.21 | 30.95 | 38.67 | 34.8
GR-1K-rich
CUPidf | 39.41 | 30.09 | 36.72 | 41.95 | 38.22 | 40.51 | 46.65
CUPsbert | 39.82 | 30.87 | 37.08 | 42.71* | 37.09 | 39.56 | 46.82
CUP1gram | 39.24 | 29.74 | 36.44 | 41.34 | 38.82 | 40.85 | 48.82*
CUP3gram | 39.2 | 30.52 | 37.2 | 41.98 | 35.53 | 37.88 | 44.68
CUPkw | 39.64 | 30.1 | 36.49 | 42.08 | 36.73 | 42.47 | 48.67*
CUPGPT | 39.79 | 29.27 | 36.73 | 41.86 | 39.4 | 43.23* |  50.13*

###  A.2. Influence of Interaction Density with Standard Evaluation

Tables 8 and 9 contain NDCG@5 results with standard evaluation for the AM-10K
and GR-10K datasets, with sparse and dense variants.

Table 8.  NDCG@5 for AM-10K Sparse vs. Dense (Standard Evaluation, by Item/User Groups)  Method | ALL | u-s | u-r | u-b | s-s | s-r | s-b
---|---|---|---|---|---|---|---
AM-10K-Sparse
CF | 6.35 | 0.0 | 0.04 | 0.02 | 22.84 | 21.89 | 16.03
CUPtext | 32.53 | 19.06 | 25.56 | 36.05 | 39.75 | 44.72 | 45.8
CUPtext+CF | 13.26 | 11.66 | 6.28 | 1.47 | 35.29 | 31.55 | 20.77
AM-10K-Dense
CF | 33.32 | 0.08 | 0.04 | 0.2 | 68.76 | 56.25 | 43.05
CUPtext | 47.26 | 11.55 | 19.44 | 27.87 | 68.95 | 62.59 | 53.96
CUPtext+CF | 33.66 | 2.2 | 0.3 | 0.52 | 69.0 | 56.64 | 43.34
Table 9. NDCG@5 for GR-10K Sparse vs. Dense (Standard Evaluation, by Item/User Groups)  Method | ALL | u-s | u-r | u-b | s-s | s-r | s-b
---|---|---|---|---|---|---|---
GR-10K-Sparse
CF | 20.02 | 0.0 | 0.0 | 0.02 | 55.96 | 46.36 | 37.29
CUPtext | 44.62 | 24.48 | 32.59 | 38.31 | 56.32 | 58.01 | 53.98
CUPtext+CF | 26.92 | 18.58 | 11.25 | 2.63 | 60.98 | 57.24 | 41.46
GR-10K-Dense
CF | 47.36 | 0.01 | 0.01 | 0.0 | 73.9 | 64.76 | 50.4
CUPtext | 52.47 | 22.14 | 23.22 | 27.15 | 69.19 | 62.81 | 53.34
CUPtext+CF | 49.59 | 5.47 | 1.9 | 0.59 | 74.35 | 66.66 | 54.1

###  A.3. User Profile Examples

Table 10 shows anecdotal examples for the profiles generated for an Amazon
user and a Goodreads user, all cropped to the 128-token limit (or less if the
construction method does not fully utilize its budget).

| method | user profile
---|---|---
Amazon user | genres |  photography video, worship devotion, arts photography, religion spirituality
| ChatGPT |  amateur photography, everyday language, easy explanation, thorough explanations, covers many topics, beautiful photos, settings, down to earth, basic photographer, wealth of information, photo examples, exposure settings, inspirational, graduation gift,
| idf-sentence |  This was so inspirational. I love the way Bryan Peterson writes his books. It has sections for shutter speed, aperture, light as well as special techniques and filters. The photos are beautiful. He has also included so much information without cluttering the writing. I decided to purchase this book after seeing some info in a magazine. It even has a chapter on film vs. digital. Great gift idea. Great gift idea. I have given several away as graduation gifts. I have now been sharing it with my granddaughter as she is picking up the camera now. I say if you are an amateur photographer or helping someone else, it
| SBERT |  Great book on exposure. I have now been sharing it with my granddaughter as she is picking up the camera now. I have given several away as graduation gifts. So many times it is the exposure settings we can not get right and this book will definitely do the job. I have had this book for a while now and as a amateur photographer I refer back to it often. This was so inspirational. The way that ” Understanding Exposure ” is written, it does not do that. I say if you are an amateur photographer or helping someone else, it is well worth getting this book. The response I received from the graduate was also
| T5 keywords |  exposure, british photographer, beginner, professional, information overload, book, explanations. perfect book. reporter and amateur photographer, it is written in everyday language, family and friends, setting, reading. graduation gifts. great gift idea
Goodreads user |  genres |  non - fiction, history historical fiction biography, comics graphic, mystery thriller crime, fiction
| ChatGPT |  murder cases, assassination of john f. kennedy, minute by minute account, law enforcement investigation, graphic detail, true crime, serial killer, victimology, cuban missile crisis, nuclear war, leadership lessons, everyday americans, american civil war, scots - irish culture, jfk’s presidency, jfk’s assassination, enduring legacy, wealth gap, middle class, world war ii, pow, brutal captivity,
| idf-sentence |  Bugliosi leaves no stone unturned. In between, Bugliosi offers a narrative on the President’s assassination itself, the murder of Dallas PD Patrolman JD Tippet, the manhunt for the killer, the law enforcement investigation which occurs at break neck speed, the interrogations of Oswald, and the murder of Oswald by Jack Ruby. The book really excels when Sberna dives into the histories of Sowell’s 11 murder victims, along with the 3 women he assaulted but did not kill. Sabato then provides his own investigation of JFK’s assassination,
| SBERT |  Obviously there are thousands upon thousands of books that have been written about the American Civil War and its many different topics and issues. November 22, up through the moments when both JFK and assassin Lee Harvey Oswald are both buried in their respective graves on Mon. Sabato examines the remaining Presidents and the JFK legacy ; from Richard Nixon never fully escaping the Kennedy shadow, to Jimmy Carter’s feuds with Ted Kennedy, to Bill Clinton’s hero worship of Kennedy beginning when he was a teenager, to Barack Obama as the latest President to try and carry the torch as the Democratic party’s standard bearer. The authors break
| T5 keywords |  Hampton sides, book detailing the assassination of martin king man, hunt eric gal. hour by hour, narrative, account, cuban missile crisis, kuban revolutionaries, soviet missiles, american cities u. daily, regular americans, corporate americans, wall street politicians, reads the Assassins Gate book, middle class. Sherman american history, united states westward expansion. barlett and steele american politics, wealth gap, middle class, united states trade imbalance tax policies, 850 pages, cnn’s book, american civil war detailed accounts, union or confederate fighting units. book, former prosecutor paul smith
Table 10. Profiles of selected users constructed by various methods


# Fill in the  (a diffusion-based image inpainting pipeline)

Eyoel Gebre∗, Krishna Saxena∗, Timothy Tran∗
University of Washington
{eyoel23, ksaxena3<EMAIL_ADDRESS>
###### Abstract

Image inpainting is the process of taking an image and generating lost or
intentionally occluded portions. Inpainting has countless applications
including restoring previously damaged pictures, restoring the quality of
images that have been degraded due to compression, and removing unwanted
objects/text. Modern inpainting techniques have shown remarkable ability in
generating sensible completions for images with mask occlusions. In our paper,
an overview of the progress of inpainting techniques will be provided, along
with identifying current leading approaches, focusing on their strengths and
weaknesses. A critical gap in these existing models will be addressed,
focusing on the ability to prompt and control what exactly is generated. We
will additionally justify why we think this is the natural next progressive
step that inpainting models must take, and provide multiple approaches to
implementing this functionality. Finally, we will evaluate the results of our
approaches by qualitatively checking whether they generate high-quality images
that correctly inpaint regions with the objects that they are instructed to
produce.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

###  1.1 Motivation

Inpainting has been an important problem within the field of computer vision
for many decades. It is a functionality that is essential to many image
related applications such as object removal, image restoration, manipulation,
re-targeting, compositing, and image-based rendering. It is also safe to
believe that with the wide adoption of generative AI tools, functionalities
like inpainting could see a significant increase in use within creative
applications as well. As tools such as DALL-E, ChatGPT, Midjourney, and Sora
are proving that they can be of high utility to those whose career depends on
creative works, we are seeing for the first time how AI systems are
significantly changing a large industry (The U.S. Bureau of Economic Analysis
reports that arts and cultural production accounts for $1,016,249,142,000 and
4.4% of the U.S. economy, contributing 4,851,046 jobs [4]. Therefore,
improving functionalities such as inpainting and allowing them to have a more
robust range of capabilities can have a true economic impact.

###  1.2 Background

There are two primary general approaches to the problem of inpainting:
stochastic and deterministic. Stochastic methods output multiple sensible
inpainting results through a random sampling process, while deterministic
methods produce a single result. A common deterministic approach is to take an
image along with a binary mask representing the region to be inpainted and
pass these two images into a generator model such as a GAN trained to fill in
the missing region. The task of improving the generator’s ability to produce
acceptable inpaintings has been tackled from a wide range of approaches
including attention mechanisms, encoder-decoder connections, deep prior
guidance, and multi-scale aggregation [5]. Stochastic inpainting techniques,
such as flow-based and MLM-based methods, utilize generative models and
sequence prediction to reconstruct image structures and textures.
Additionally, many stochastic techniques take the approach of starting with a
noisy image and iteratively denoise it until reaching a sensible output, a
strategy commonly used by diffusion-based inpainting techniques which we will
focus on.

##  2 Related Work

As mentioned, many deterministic strategies for implementing inpainting take
an image dataset along with masks generated from some mask distribution and
then train a model to fill in these masks. An example of this is Large Mask
Inpainting (LaMa) [6]. During training, LaMa takes in sample images, and for
each image generates a mask from some fixed mask distribution. Then, it trains
a fast Fourier Convolutional Network to predict what is hidden behind the mask
for each image, which is able to achieve strong results. However, the downside
is that the model often struggles to generalize its abilities to masks which
are not within the distribution used to generate masks in the training
dataset.

The current leading effort in inpainting that we have been able to identify is
a paper from ETH Zurich called RePaint [3]. RePaint proposes a Markov Chain
based random walk to add Gaussian noise to an input image followed by a
reverse walk to denoise the masked portion of the image. This way, RePaint is
able to remove the dependency on the specific mask distribution used to train
the model by utilizing a pre-trained Denoising Diffusion Probabilistic Model
(DDPM) to denoise pixels inside an arbitrary mask alongside their context (the
unmasked portion of the original image with similar noise). Thus, RePaint gets
around the lack of mask generalization all without even needing a training
process by simply adjusting the input to an existing pre-trained DDPM. To add
more semantic variance to the model’s output, the RePaint authors proposed a
resampling step during the reversed random walk. In the resampling step,
Gaussian noise is added back to a partially denoised combination of the
learned masked region and noisy input unmasked region. This noisier
combination is autoregressive and is passed into the DDPM a fixed number of
times to extract the full benefit of the DDPM’s stochasticity, which is
explained in further detail in Section 3.3.

In short, RePaint emphasizes denoising on the masked region while using the
unmasked region mostly as a reference that provides context for the image.
This results in very natural looking inpainted images which were evaluated
higher than state-of-the-art models for a wide variety of mask distributions.
Uniquely, RePaint is more creative with the objects it is able to inpaint
while still allowing its final outputs to be semantically correct [3].

One unique feature of RePaint is that it often introduces random objects in
its generated regions. Although these objects still often look natural and
semantically correct, their randomness may be unwanted in many situations. For
example, a lizard is generated when a person wants to remove a human hand from
a portion of an image. We aim to allow RePaint to maintain its level of
performance and semantic correctness while allowing for more control over what
exactly gets generated, particularly in the masked region. RePaint currently
does not take into consideration any preferences for what should be inpainted.
There is also no clear interface that is best suited for providing the
context/information to the model on what our preferred inpaint will contain.

Thus, our goal is to extend the abilities of RePaint such that in addition to
taking in a mask and an image, it can also take in information regarding what
is preferred to be inpainted. There are countless options: providing a textual
description of what our target inpaint contains, an image of a preferred
object to be generated, a dataset of acceptable objects that will allow the
model to choose the most suitable option, etc.

One benefit of RePaint is that it is only modifying the denoising step of an
existing pre-trained diffusion model. This also means that we would not need
to label any data or train the model, making it quite straightforward to try a
large variety of these options for providing our preferred inpainted object
and evaluating each approach. We will initially focus on evaluating the
performance of RePaint when it is given just a single image containing what we
want to be inpainted into the masked region. This will be done by providing
the target image (i.e. a cutout of a dog) in the sampling (denoising) step of
the RePaint algorithm similar to how the current algorithm provides a
reference to the original unmasked region. In short, we will create a new
pipeline where given the scene background, mask, and a third image
representing a ”target” object, RePaint will inpaint the occluded portion of
the scene with an instance of the ”target” object in a sensible manner.

We will try various approaches to learning the borders of our target so that
the model generates transitions from scene to target which are sensible. Note
that one limitation of our model is that there is no ground truth to each
image inpainting, making it difficult to quickly analyze the performance. We
will instead evaluate the performance of this adjustment by manually comparing
it to the output of the original RePaint model and possibly other leading
inpainting models with the additional benchmark of whether it was able to
generate our preferred object while maintaining an equal level of quality and
semantic correctness. We expect this approach to work quite well as it does
not make any assumptions that the original RePaint algorithm doesn’t as well.

##  3 Method

Figure 1: Pipeline diagram of RePaint (without sampling or jumping) reproduced
from [Lugmayr et. al., 2022] for illustrative purposes. Figure 2: Core
pipeline without resampling or jumping.

Our pipeline builds on RePaint’s inference algorithm (Section 3.1 and Figure
1) by allowing the inpainting to be guided by a target image. To test our
pipeline, we created a small dataset comprising of scene and target images
alongside manually created binary masks for where the target will be inpainted
on the scene. These three images (scene, target, and mask) become the inputs
to our pipeline and get processed into one image as described in Section 3.2.

###  3.1 RePaint Pipeline Background

An in-depth overview of RePaint’s pipeline is necessary to understand our
contribution. Initially, the diffusion process transforms the masked starting
image $x_{0}$, into Gaussian white noise $x_{T}$ by adding iid Gaussian noise
with progressively greater variance to a scene image. At each time step $t$ of
the forward pass, the image undergoes transformations according to the
diffusion process, becoming noisier as $t$ increases until it reaches pure
noise at high $t$ values.

In the backward pass, the goal is to reconstruct the original image from the
white Gaussian noise. At each time step $t$, the noisy image is inputted into
a DDPM. The output of the masked-out portion of the image is extracted and
combined with the non-masked region obtained from the corresponding forward
pass step. This process gradually restores the original image by filling out
the masked-out portion as accurately as possible. (see Figure 1 for the
pipeline diagram of RePaint). RePaint also introduces jumping and sampling to
improve results, which will be explained in Section 3.3.

###  3.2 Core Pipeline for Inpainting Target Images

Our contribution is introducing the ability to specify an object to be
inpainted into the scene at a specific location. The pipeline is designed to
take in three input images: the background scenery, the target image, and a
binary mask representing the region of interest within the target image.
Notably, unlike RePaint’s pipeline, which solely processed the scene alongside
an arbitrary mask, our approach incorporates contextual information for more
target inpainting. In addition to denoising the scene during the forward pass,
our method also applies denoising to the target image using a similar
procedure. During the backward pass, the image at timestep $t$, $x_{t}$, is
passed to the DDPM as normal. This is described in equations 1-2, which were
inspired by equations 8a-c in [3]:

$\displaystyle x_{\text{scene},t-1}$
$\displaystyle\sim\mathcal{N}\left(\sqrt{\bar{\alpha}_{t}}x_{\text{scene},0},\left(1-\bar{\alpha}\right)I\right)$
(1a) $\displaystyle x_{\text{target},t-1}$
$\displaystyle\sim\mathcal{N}\left(\sqrt{\bar{\alpha}_{t}}x_{\text{target},0},\left(1-\bar{\alpha}\right)I\right)$
(1b) $\displaystyle x_{\text{repaint},t-1}$
$\displaystyle\sim\text{DDPM}\left(x_{t}\right)$ (2)

where $\bar{\alpha}_{t}:=\Pi_{i=1}^{T}(\beta_{i})$.

While the unmasked region of the scene is extracted from the corresponding
forward pass, the same as RePaint, a notable issue arises in handling the
masked target, since we have both the masked noised object from the forward
pass and the masked object from the DDPM. This discrepancy, which we call
“mask conflict”, arises from the coexistence of two masked versions of the
object: one generated during the forward pass and the other by the DDPM. To
resolve this conflict and produce the image at $x_{t-1}$, we take a convex
combination of the two masked target images using a parameter series
$\lambda_{t}$, described as

$\displaystyle x_{t-1}^{\text{unknown}}$ $\displaystyle=\lambda
x_{\text{repaint},t-1}+\left(1-\lambda\right)x_{\text{target},t-1}$ (3)

Finally, to achieve our image $x_{t-1}$ in the backward step pass, we take the
scene from the forward pass and the result of the ”mask conflict” using the
binary mask,

$\displaystyle x_{t-1}$ $\displaystyle=m\odot
x_{\text{scene},t-1}+\left(1-m\right)\odot x_{t-1}^{\text{unknown}}$ (4)

where $m$ represents the binary mask. This entire process is described in
Figure 2.

###  3.3 Resampling and Jumping

In the pipeline (Figure 2), each denoising (backward) step is a function of
only the noised scene image, noised target image, and previous noised
combination. During the linear combination to calculate the DDPM input, there
exists the possibility that the border between the target area and masked,
noised scene image has a sudden, unnatural change in color. The RePaint
authors noted a similar issue with the combination of the previously noised
combination and the masked scene and proposed resampling to address this
issue. Resampling is a trick to increase the diversity and smoothness of the
inpainted image by noising it and running it through the DDPM multiple times.
This allows the DDPM to predict the pixels on both sides of the mask’s
boundary which increases the quality of these regions (along with the added
benefit of increasing the variance of pixel values generated inside the mask).
For example, first, the linear combination of a noisy, masked scene and
generated image is passed into the DDPM. The result of this operation is a
slightly less noisy combination with jarring boundaries. Gaussian noise is
added to this result which is then passed back into the DDPM. This process is
repeated $r$ times, after which the boundary transition is smoother. Since
running the DDPM is time and computationally-intensive, the RePaint authors
propose a jumping schedule parameter $j$ to control when resampling occurs.
Every $j$ timesteps of the backward pass, $r$ resampling steps occur. When
$r<=j$, the pipeline has the same runtime and FLOPs as the original pipeline
($r,j=1$).

###  3.4 Initial Hyperparameter search

As discussed in Section 3.2, $\lambda_{t}$ controls the convex combination of
the generated inpainted target with the noised ground truth target at timestep
$t$. Note that with $\lambda_{t}=1$, our pipeline collapses into RePaint’s
original pipeline since there would be no contribution from the noised target
in the forward step target images. With this pipeline design, we had an
initial hyperparameter search over various values for
$\lambda_{t}\in[0.8,0.9,0.993,0.995,0.999,0.9999]$. Additionally, for the jump
length and jump size variables, we tried values in the range $[10,20,30,40]$.
Finally, for the timestep count, we tried values in the range
$[50,100,150,200,250]$.

As expected our result ends up looking more like random noise or takes on an
uninterpretable shape the closer our lambda value is to one. However, smaller
lambda values result in an output that looks like the target image was simply
cropped out and pasted on top of the scene background. We found that lambda
values in the range of 0.92-0.97 performed the best at keeping the content of
the target image without making it seem like it was an exact copy and paste.
Additionally, we expected to see that as our sampling parameter ($t$)
increases to a large value, our model’s output is less fuzzy/noisy. This makes
sense since this would put less weight on our forward pass noised target
images in the denoising steps.

Finally, we noticed that increasing our jump and jump length parameters
allowed for all-around higher quality outputs. This makes sense since the
whole purpose of jumping is to allow the model to generate output which takes
into account the surrounding scene’s context as much as possible when having
the DDPM generate output. In summary, we performed a standard grid search over
the ranges of all these parameters and identified the combination of optimal
values to be 40 for both jump length and jump size, 200 for timesteps, and
0.993 for $\lambda_{t}$.

(a) .8, 250, 20

(b) .993, 100, 30

(c) .997, 100, 40

(d) .999, 250, 40

(e) .999, 100, 40

(f) .80, 250, 20

Figure 3: Hyperparameter search ($\lambda$, $t$, $j$), where $t$ is the number
of noising timesteps in the forward pass and $j$ is both the number of jumps
and jump length.

During our experimentation, we identified two viable methods for defining the
target mask. The first approach involved creating an exact cutout of the
target object, allowing the model to generate a more detailed image. However,
this method often resulted in an unnatural-looking transition from the scene
to the target object, appearing as if the object was simply pasted onto the
image. Alternatively, we explored a more lenient approach by including
additional information surrounding the target object in the mask. While this
led to a more natural-looking transition between the scene and the target
object, the resulting image of the target tended to be less detailed and
smooth. Another issue with this approach was the dependency on the target
image being in a similar environment to the scene. For example, if the target
was a dog standing on gravel and the scene was a grass field, it would not
make sense to give the model the grey-colored gravel border to inpaint into
the final image. To address this, we investigated alternative modifications to
the exact mask approach, as outlined in Section 4.1.

##  4 Experiments

All though our initial results did not look like they were exact copy-pastes,
we were still disappointed with the lack of interaction that our target object
had with the background scene. An ideally inpainted object should interact
with elements of the background scene at least to some degree. For example, in
the case of the dog on the field, we would like to see the dog’s paws and tail
be hidden by blades of grass. However, our core pipeline cannot currently
achieve this. We propose several modifications to our original pipeline to
combat this issue.

###  4.1 Masking Alternatives

The biggest issue we found from our results was the unnatural-looking
boundaries in our unpainted images. Currently, with an exact binary mask, the
pipeline didn’t have context about how ”close” a pixel was to the boundary.
Thus, to help provide context and more flexibility to the DDPM in the boundary
error, we decided to explore some alternative masking methods.

####  4.1.1 Distance-Based ”Heated” Mask

To enhance boundary realism while still keeping the core target image, we
adjusted the backward pass to use a ”heated” buffer. Instead of using a binary
buffer where we would take the value of every pixel of the target, the values
of the mask were between 0 and 1. With this heated mask, we kept the forward
pass of the target for pixels that were far away from the boundary
(represented by values closer to 1) and allowed the DDPM the freedom and
flexibility to generate natural-looking borders for the pixels of the target
image close to the border.

For some integer $b$, the buffer size, the heated mask is a function of mask.

$\displaystyle m_{i,j}^{\text{HEATED}}=\max{\frac{d_{i,j}}{b},1}$ (5)

where $d_{i,j}$ is the Manhattan distance from coordinate $(i,j)$ to the
nearest black ($0$) pixel in the binary mask.

####  4.1.2 Scene Buffer

Secondly, to further improve boundary realism, we implemented a “scene buffer”
technique by adding a 4 pixel border around the target mask during the linear
combination. Here, an exact cutout of the mask is utilized, and during the
backward pass, the linear combination is computed to resolve the mask conflict
as previously described in 3. The key distinction lies in incorporating a
small border around the target from the DDPM output into the next backward
step, instead of taking from the forward pass scene. Note that for the rest of
the scene outside the small buffer, we take the forward pass from the
corresponding timestep of the scene as normal. This modification removes the
dependency for the target image to be in the same environment as the scene,
while still allowing the model to create a natural looking transition at the
border when using an exact cutout. This process is explained in the following
equations,

$\displaystyle m_{\text{ring}}$ $\displaystyle=m_{\text{ext}}-m$ (6)
$\displaystyle x^{\text{unknown, buf}}_{t-1}$
$\displaystyle=(cx_{\text{repaint},t-1}+(1-c)x_{\text{target},t-1})$ (7)
$\displaystyle x_{\text{scene, buf},t-1}$ $\displaystyle=m_{\text{ext}}\odot
x_{\text{scene},t-1}+m_{\text{ring}}\odot x^{\text{unknown}}_{t-1}$ (8)
$\displaystyle x_{t-1}$ $\displaystyle=m\odot x_{\text{scene,
buf},t-1}(1-m)\odot x^{\text{unknown, buf}}_{t-1}$ (9)

where $m_{\text{ext}}$ represents the extended mask and $c$ is a new constant
to replace $\lambda$, which is still used by $x^{\text{unknown}}_{t-1}$ from 4
for the buffer/ring area.

###  4.2 $\lambda$ Scheduling to Increase RePaint Integration

Additionaly, we explored increased integration with RePaint’s model to address
issues of lacking detailed features and producing smooth images with precise
masks or jarring boundaries. This involves relying less on the noised target
from the forward pass and more on the previous DDPM generation. Specifically,
for timesteps $t$ closer to 0 in 3, we set $\lambda_{t}=1$ so that the
pipeline can infer and create a more natural border around the inpainted
target portion for the remainder of the timesteps. While retaining the same
input structure (scenery, target, mask target), we modified the backward pass
to transition to RePaint’s model by scheduling $\lambda_{t}$ to be a linear
interpolation 4 from 0 to 1 for timesteps $T$ to $pT$, such that
$p\in\left[0,1\right]$ and 1 for all $t<pT$.

Figure 4: Schedule for $\lambda_{t}$ vs $t$

Based on our results from Section 3.4 (particularly note 5), we let the
hyperparameters $r,j=40$, reduce $T$ to $100$, solely to generate samples
quicker for experimentation and conduct a grid search for
$p\in{0.1,0.25,0.5,0.75,0.9}$.

We found that as $p$ increases, the inpainted image becomes less faithful to
the target and more blurry. This is because a high value of $p$ implies that
the pipeline is denoising without any direct contribution from the target
image for $pT$ denoising steps. For such denoising steps, the pipeline is
equivalent to RePaint, so the generated images will have high variance as
noted in Guided Diffusion [1] and RePaint [3].

We identify the best value for hyperparameter $p$ as $0.5$ for the inpainting
task. This setting allows for the inpainted image to be similar to the target
image while allowing the pipeline to seamlessly fill in the outer border of
the target subject. Additionally, this value allows the pipeline to retain the
semantic meaning from the target image while inpainting novel and semantically
consistent additions (e.g., the bow added to the dog in 5(c) and the grass
interacting with its paws)

(a) $p=0.1$

(b) $p=0.25$

(c) $p=0.5$

(d) $p=0.75$

(e) $p=0.9$

(f) $p=0.5$ (2)

Figure 5: Images generated with $\lambda$ schedule for varying values of $p$,
$T=100$, and $r,j=40$. Figure 6: Example scenes, masks, and targets along with
several sample outputs.

###  4.3 Failure Modes

As described previously, there is a high variance in the final inpainted when
running with $p>0.5$. The case when $p=0.5$ is particularly interesting
because much of the semantic information of the target image is kept but the
high number of denoising passes through the DDPM in this scenario makes the
generations creative.

For a run with $T=100,r,j=40,$ and $p=0.5$, a giraffe target image and
grassland scene (middle right image in Figure 6), the resulting image
maintains the shape and color of a giraffe due to the masking $\forall t$
denoising steps but the content inside of the mask resembles a flamingo-
giraffe as if it came from a Truffula forest scene from Dr. Seuss’ The Lorax.
After rerunning with these hyperparameters multiple times, we could not
produce a result that was so distinct from a giraffe. To mitigate this failure
mode in a production setting, we recommend producing at least two candidate
images per scene and target.

Another failure mode when $p>0.5$ is that this setting reveals the biases of
the DDPM train data. As the RePaint authors note, a DDPM model trained on
ImageNet, as is the one used to generate all images in this report, will be
biased towards denoising dogs [3]. For iterations $t$ of our pipeline such
that $\forall\tau<t,\lambda_{\tau}=1$, the autoregressive nature of the
denoising process means that all future steps in the denoising process are
biased towards high likelihood images in the DDPM dataset.

To address this failure mode, we recommend using a model like ResNet [2] to
find the class of the target image and using a DDPM trained on a dataset where
that image class is not underrepresented.

###  4.4 Final Pipeline

The authors present Fill in the  , a diffusion-based inpainting pipeline that
seamlessly inserts target images into scenes. Based on the current analyses,
the authors recommend the following hyperparameters: diffusion timesteps
$T=200$, jumping steps $j=40$, resampling steps $r=40$, and a piecewise linear
$\lambda$ schedule with $p=0.5$ (i.e., $\lambda_{t}=1$ if $t\leq 0.5T$ and a
linear interpolation from $(pT,1.0)$ to $(T,0)$ elsewhere.

##  5 Conclusion and Acknowledgements

###  5.1 Further Steps

Moving forward, there are several avenues for further exploration and
improvement in our research. First, we can delve deeper into mask
modifications, as described in Section 4.1. The majority of our
experimentation was with hand-made exact binary masks, but exploring
alternative masking techniques such as gradient-based or buffered masks may
allow the model to produce more realistic-looking inpaintings while still
keeping the details of the original target. Furthermore, while the
introduction of lambda scheduling in Section 4.2 improved the performance of
our core pipeline, the addition of a more dynamic-based lambda scheduling
could further enhance the adaptability and versatility of the pipeline. This
dynamic approach would involve adjusting the lambda values based on the
characteristics of different scene-target pairs, therefore optimizing the
inpainting process for each specific scenario.

In addition to exploring mask modifications and refining lambda scheduling,
expanding our testing to encompass a wider variety of images would be a huge
improvement. Currently, the manual creation of the target mask poses
scalability challenges. A solution could be to leverage segmentation
techniques that would automate mask generation, facilitating the creation and
efficient testing on a larger dataset. With a larger dataset, it would be much
easier to analyze a diverse range of scene-target pairs, leading to optimal
hyperparameters that generalize well across various scenarios.

Finally, our research aims to achieve a fully automated pipeline for
inpainting. This system would require minimal user input, consisting mainly of
providing a scene and target images and selecting the inpainted region while
automating the target position, mask creation, and choice of hyperparameters.

###  5.2 Conclusion

In this study, we aimed to improve image inpainting techniques by enhancing
control over what exactly gets generated. In particular, we identified that
current approaches did not allow a model to base its inpainted output on an
image of a target object as opposed to textual prompting or other ways of
specifying a desired inpainted object. Leveraging recent advancements in
generative AI and inpainting methodologies, we focused on a diffusion-based
approach where we modify the inputs given to a diffusion model on its
denoising steps to feed it the target image’s context. Through some
adjustments and refinements, we sought to mitigate challenges such as mask
conflicts and boundary realism, achieving some interesting results in the
quality and naturalness of inpainted images with target objects. Our
experiments provided valuable insights into the effectiveness of our approach
and the importance of various hyperparameters. Overall, our project represents
a modest yet meaningful step forward in the field of image inpainting.

###  5.3 Acknowledgements

We would like to kindly thank the CSE 493G staff for the amazing lecture
content and guidance that allowed us to complete this project. Additionally,
we express our appreciation to the RePaint paper and its authors, which was a
huge inspiration to our paper.

## References

  * [1] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis, 2021.
  * [2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [3] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models, 2022.
  * [4] National Assembly of State Arts Agencies.  Creative economy state profiles.  https://www.nasaa-arts.org, 2020.  Accessed: 2024-02-29.
  * [5] Weize Quan, Jiaxi Chen, Yanli Liu, Dong-Ming Yan, and Peter Wonka.  Deep learning-based image and video inpainting: A survey.  Received: date; Accepted: date.
  * [6] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.  Resolution-robust large mask inpainting with fourier convolutions, 2021.