11institutetext: University of Cagliari, Cagliari, Italy
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>22institutetext:
Polytechnic University of Turin, Turin, Italy
22email<EMAIL_ADDRESS>
# Knowledge is Power, Understanding is Impact: Utility and Beyond Goals,
Explanation Quality, and Fairness in Path Reasoning Recommendation

Giacomo Balloccu 11 0000-0002-6857-7709 Ludovico Boratto 11
0000-0002-6053-3015 Christian Cancedda 22 0000-0002-8206-3181 Gianni Fenu 11
0000-0003-4668-2476 Mirko Marras(✉) 11 0000-0003-1989-6057

###### Abstract

Path reasoning is a notable recommendation approach that models high-order
user-product relations, based on a Knowledge Graph (KG). This approach can
extract reasoning paths between recommended products and already experienced
products and, then, turn such paths into textual explanations for the user.
Unfortunately, evaluation protocols in this field appear heterogeneous and
limited, making it hard to contextualize the impact of the existing methods.
In this paper, we replicated three state-of-the-art relevant path reasoning
recommendation methods proposed in top-tier conferences. Under a common
evaluation protocol, based on two public data sets and in comparison with
other knowledge-aware methods, we then studied the extent to which they meet
recommendation utility and beyond objectives, explanation quality, and
consumer and provider fairness. Our study provides a picture of the progress
in this field, highlighting open issues and future directions. Source code:
https://github.com/giacoballoccu/rep-path-reasoning-recsys.

###### Keywords:

Recommender Systems, Knowledge Graphs, Replicability.

##  1 Introduction

Recommender systems (RS) are a popular strategy to enable personalized users’
experience [29]. Historical data (e.g., browsing activity and ratings) and
product characteristics (e.g., title and description) are well-recognized data
sources to train RSs. Product information is often augmented with _Knowledge
Graphs_ (KGs) [8, 27]. These KGs include _entities_ (e.g., users, movies,
actors) and _relations_ between entities (e.g., an actor starred a movie).
Integrating KGs within RSs has led to a gain in recommendation utility [36,
34], especially under sparse data and cold-start scenarios [18]. Their
inclusion is essential to make RS explainable and turn recommendation into a
more transparent social process [44, 33].

Notable recommendation methods based on KGs include _path reasoning methods_
[1, 37, 23, 38, 26, 25, 41, 31, 45]. To guide RS training, they rely on paths
that model high-order relations between users and products in the KG, and
identify those deemed as relevant between already experienced products and
products to recommend. Such paths are also used to create explanations,
through explanation templates or text generation. In the movie domain, the
path “user1 watched movie1 directed director1 directed-1 movie2” might lead to
the template-based explanation “movie2 is recommended to you because you
watched movie1 also directed by director1”. Path reasoning methods are in
contrast to regularization methods, which weight product characteristics based
on their importance for a given recommendation but do not provide any
explanation [43, 35, 19, 16, 34, 36].

An abundance of KGs were proposed for recommendation, along with path
reasoning methods, to produce both recommendations and explanations [5].
However, evaluation protocols were heterogeneous (e.g., different train-test
splits) and limited to a narrowed set of evaluation data sets and metrics.
Prior works often showed that a novel method led to a higher recommendation
utility, compared to (non) knowledge-aware baselines. None of the them deeply
analyzed beyond utility goals (e.g., coverage, serendipity) nor monitored
consumer (i.e., end users) and provider fairness. Hence, it remains unclear
whether path reasoning methods emphasize any trade-off between goals
unexplored so far. Being the landscape convoluted and polarized to utility,
there is a need for a common evaluation ground to understand how and when each
method can be adopted.

In this paper, we conduct a replicability study (different team and
experimental setup) on unexplored evaluation perspectives relevant to path
reasoning methods. In a first step, we scanned the proceedings of top-tier
conferences and journals, identifying seven relevant papers. We tried to
replicate the original methods based on the released source code, but only
three of them were replicable. In a second step, we defined a common
evaluation protocol, including two public data sets (movies; music), two
sensitive attributes (gender; age), and sixteen metrics pertaining to four
perspectives (recommendation utility; beyond utility goals; explanation
quality; fairness). We evaluated path reasoning methods under this protocol
and compared them against other knowledge-aware methods. Results reveal that,
despite of an often similar utility, path reasoning methods differ in the way
they meet other recommendation goals. Our study calls for a broader evaluation
of these methods and a more responsible adoption.

##  2 Research Methodology

In this section, we describe the collection process for path reasoning
methods, the steps for their replication, and the common evaluation protocol.

###  2.1 Papers Collection

To collect existing path reasoning methods, we systematically scanned the
recent proceedings of top-tier information retrieval events (CIKM, ECIR, ECML-
PKDD, FAccT, KDD, RecSys, SIGIR, WSDM, WWW, UMAP) and journals edited by top-
tier publishers (ACM, Elsevier, IEEE, Springer). The adopted keywords combined
a technical term between “ _path reasoning recommender systems_ ” and “
_explainable recommender system_ ” and a non-technical term between “
_explainable AI_ ” and “ _knowledge enabled AI_ ”. We marked a paper as
relevant if (a) it addressed recommendation, (b) it proposed a KG-based
method, and (c) the method could produce reasoning paths. Papers on other
domains or tasks, e.g., non-personalized rankings or mere entity prediction
tasks (w/o any recommendation) were excluded. We also excluded knowledge-aware
methods unable to yield reasoning paths, although we will use some
representatives of this class for comparison. Seven relevant papers were
selected for our study (Table 1).

We attempted to replicate the method of each relevant paper, relying as much
as possible on the original source code. To obtain it, we first tried to
search for the source code repository into the original paper and on the Web.
As a last resort, we sent an e-mail to the original authors. We considered a
method to be replicable in case a fully working version of the source code was
obtained and needed minor changes to accept another data set and extract
recommendations (and reasoning paths). Three out of the seven relevant papers
were replicable with a reasonable effort. As per the non-replicable ones,
three did not provide any source code111 Note that the source code of these
papers might appear soon online as an effect of our e-mails to the original
authors. We leave their replication as a future work. . The other one included
unavailable external dependencies [20].

Table 1: Path reasoning methods deemed as relevant in our study.

Method | Year | Status1 | Experimental Setting
---|---|---|---
|  |  | Data Sets2 | Split Size3 | Split Method4 | Recommendation5 | Explanation5
PGPR [41] | 2019 | $RE$ | $AZ$ |  $70$-$00$-$30$ | $Rand$ | NDCG, R, HR, P | -
EKAR [31] | 2019 | $\overline{RE}$ |  $ML$, $LFM$, $DB$ |  $60$-$20$-$20$ | $Rand$ | NDCG, HR | -
CAFE [42] | 2020 | $RE$ | $AZ$ |  $70$-$00$-$30$ | $Rand$ | NDCG, R, HR, P | -
UCPR [32] | 2021 | $RE$ |  $ML$, $AZ$ |  $60$-$20$-$20$ | $Rand$ | NDCG, R, HR, P | PPC
MLR [39] | 2022 | $R\overline{E}$ | $AZ$ |  $70$-$00$-$30$ | $Rand$ | NDCG, R, HR, P | -
PLM-Rec [13] | 2022 | $\overline{RE}$ | $AZ$ |  $60$-$20$-$20$ | $Time$ | NDCG, R, HR, P | -
TAPR [47] | 2022 | $\overline{RE}$ | $AZ$ |  $60$-$10$-$30$ | $Rand$ | NDCG, R, HR, P | -
1 Status $RE$ : Replicable and Extensible; $R\overline{E}$ : Replicable but
not Extensible; $\overline{RE}$ : Not Replicable nor Extensible.
2 Data Set $AZ$ : Amazon [22]; $ML$ : MovieLens 1M [15]; $LFM$ : LastFM [30];
$DB$ : DBbook2014 [9].
3 Split Size reports the percentage of data for training, validation, and
test, respectively.
4 Split Method. $Rand$ : Random based; $Time$ : Time based.
5 Metrics $R$ : Recall; $HR$ : Hit Ratio $P$ : Precision; $PPC$ : Path Pattern
Concentration

###  2.2 Methods Replication

For each relevant paper, we analyzed the rationale of the proposed method and
the characteristics of the experimental setting, as summarized in Table 1.

PGPR [41] (original source code: https://github.com/orcax/PGPR) was based on
the idea of training a reinforcement learning (RL) agent for finding paths.
During training, the agent starts from a user and learns to reach the correct
products, with high rewards. During inference, the agent directly walks to
correct products for recommendation, without enumerating all the paths between
users and products. The original experiments were done on four AZ data sets
[22] and on a KG built from product metadata and reviews.

EKAR [31] (original source code not available) modeled the task as a Markov
decision process on the user-item-entity graph and used deep RL to solve it.
The user-item-entity graph is treated as the environment, from which the agent
gets a sequence of visited nodes and edges. Based on the encoded state, a
policy network outputs the probability distribution over the action space.
Finally, a positive reward is given if the agent successfully finds those
products consumed by the target users in the training set. The novelty lays in
using an LSTM for the policy network and a reward function that makes training
stable and encourages agent exploration. Only this study included data sets
from three diverse domains: movies (ML1M), music (LFM), and books (DB).

CAFE [42] (original source code: https://github.com/orcax/CAFE) follows the
coarse-to-fine paradigm. Given the KG, a user profile is created to capture
user-centric patterns in the coarse stage. To conduct multi-hop path reasoning
guided by the user profile, the reasoner is decomposed into an inventory of
neural reasoning modules. Then, these modules are combined based on the user
profile, to efficiently perform path reasoning. Original experiments followed
the PGPR experimental setting (same data sets, data split, and evaluation
metrics).

UCPR [32] (original source code: https://github.com/johnnyjana730/UCPR/)
introduces a multi-view structure leveraging not only local sequence reasoning
information, but also a view of the user’s demand portfolio. The user demand
portfolio, built in a pre-processing phase and updated via a multi-step
refocusing, makes the path selection process adaptive and effective. The
original experimental setting covered the movie (ML1M) and e-commerce domains
(AZ). This study was the only one assessing an explanation quality property,
i.e., to what extent the KG relation type differs among the selected paths.

MLR [39] (source code shared by e-mail, but external dependencies missing) is
another RL framework that leverages both ontology-view and instance-view KGs
to model multi-level user interests. Through the Microsoft Concept Graph (MCG)
[20], the method creates various conceptual levels (e.g., Prada is an Italian
luxury fashion brand). The reasoning is then performed by navigating through
these multiple levels with an RL agent. The authors provided the source code,
but the MCG was no longer online and the provided KG dump referred only to the
originally used AZ data sets.

PLM-Rec [13] (original source code not available), given a KG, extracts
training path sequences under different hop constraints. By leveraging
augmentations of language features with semantics, the method obtains a series
of training data sequences. A transformer-based decoder is then used to train
an auto-regressive path language model. This method could limit previous
methods’ recall bias in terms of KG connectivity. Only AZ data sets were used
in the experiments.

TAPR [47] (original source code not available) proposed another path reasoning
approach based on RL, characterized by the incorporation of a temporal term in
the reward function. This temporal term guides a temporal-informed search for
the agent, to capture recent trends of user’s interests. Original results (on
AZ data sets) showed a gain in utility compared to PGPR, although the model
was evaluated using a random split, which is not ideal for time-aware models.

Table 2: Interaction and knowledge information for the two considered data
sets.

Interaction | ML1M | LFM1M
---|---|---
Users | 6,040 | 4,817
Products | 2,984 | 12,492
Interactions | 932,295 | 1,091,275
Density | 0.05 | 0.01
Gender (Age) Groups | 2 (7) | 2 (7)

Knowledge | ML1M | LFM1M
---|---|---
Entities (Types) | 13,804 (12) | 17,492 (5)
Relations (Types) | 193,089 (11) | 219,084 (4)
Sparsity | 0.0060 | 0.0035
Avg. Degree Overall | 28.07 | 25.05
Avg. Degree Products | 64.86 | 17.53

###  2.3 Evaluation Protocol

To ensure evaluation consistency and uniformity across methods, given the
heterogeneous original experimental settings, we mixed replication and
reproduction [7], but use only the term “replicability” for convenience
throughout this paper. Specifically, we relied on the source code provided by
the original authors to run their methods, and our own data and source code to
(a) pre-process the input data sets as per their requirements and (b) compute
evaluation metrics based on the recommendations and reasoning paths they
returned.

Data Collection. We conducted experiments on two data sets: MovieLens (ML1M)
[15] and LastFM (LFM1B) [30]. Given our interest in the fairness perspective,
we selected data sets that provide (or make it possible to collect) users and
providers’ demographic attributes. We therefore discarded other data sets,
such as the Amazon ones [22], where this was not reasonably possible. The
selected data sets are all public and vary in domain, extensiveness, and
sparsity, providing novel insights on the generalizability of the replicated
path reasoning methods under a common ground, with respect to their original
settings (see Table 1). For ML1M, we used the KG generated in [9] from
DBpedia, while we generated the KG from the Freebase dump extracted by [46]
for LFM1B.

Data Preprocessing. Concerning the ML1M data set, both gender and age
sensitive attributes for the consumers, but not for the providers, were
originally provided in [15]. Being directors considered as movie providers in
prior work [6], we relied on their sensitive attribute labels collected in
that study. In LFM1B, gender and age labels were attached only to a small
subset of end users. We therefore discarded all those users whose sensitive
attributes were not available. Given that the original papers included only
data sets far smaller than LFM1B and that our preliminary experiments
uncovered a low scalability for those methods222 Solving substantial
scalability issues goes beyond the scope of our replicability study., we then
uniformly sampled a subset of the filtered LFM1B, ensuring that users
(products) had at least 20 (10) interactions. We will refer to this data
subset as LFM1M throughout the paper and results. This sampling allowed us to
obtain a data set size comparable to ML1M and avoid cold-start scenarios,
which are not our focus. Since providers’ sensitive attributes were not
attached to the original data set (in music RS, artists are commonly
considered as providers), we crawled them from Freebase and released them with
our study.

Both KGs were pre-processed as performed in [2] to make triplets uniformly
formatted. More specifically, we only consider triplets composed of a product
as the entity head and an external entity as the entity tail, to obtain a
common ground data set for the analysis of both knowledge-aware and path-based
methods. Considering triplets having external entities or products as the head
and tail entities would have required to craft additional meta-paths (needed
for path-based methods) compared to the reproduced studies, going beyond the
scope of our work. In addition, to control sparsity, we removed relations
having a type represented in less than 3% of the total number of triplets.
Concerning the user-product interactions, we discarded products (and their
interactions) which are not present in the KG. Pre-processed data set
statistics are collected in Table 2.

Data Preparation and Split. For each data set, we first sorted the
interactions of each user chronologically. We then performed a training-
validation-test split, following a time-based hold-out strategy, with the 60%
oldest interactions in the training set, the following 20% for validation, and
the 20% most recent ones as the test set. The aforementioned pre-processed
data sets were used to train, optimize, and test each benchmarked model. This
allowed us to carry out the evaluation procedure in a realistic setting, in
which the trends that might determine interaction patterns are non-stationary
and evolve over time.

Comparative Knowledge-aware Models. Path reasoning methods belong to a
subclass of the knowledge-aware recommendation class. To better contextualize
our study, we therefore decided to provide comparisons (when interesting)
against two knowledge-aware models based on knowledge embeddings, namely CKE
[43] and CFKG [1], and a knowledge-aware model based on propagation, namely
KGAT [36]. These three models, unable to provide reasoning paths to users,
were replicated and evaluated under the same protocol333 For conciseness, we
did not include non-knowledge-aware methods (e.g., BPR), which were compared
against path reasoning methods under some metrics (e.g., NDCG) in studies like
[4]. Nevertheless, this is an important aspect for future work.. For
conciseness, we do not explain their replication in detail and refer the
reader to our repository.

Hyper-parameter Fine-tuning. Given a data set and a model, we selected the
best hyper-parameters setting via a grid search that involved those
hyperparameters (and their values) found to be sensitive in the original
papers. In certain cases, given our findings from preliminary experiments, we
extended the grid of values to better adhere to the characteristics of the
data set at hand. Full details on the hyper-parameters and their values in our
grid search are reported in our repository. Models obtained via different
hyper-parameter settings were evaluated on the validation set, selecting the
one achieving the highest NDCG.

Evaluation Metrics Computation. Given a model and a data set, we monitored
recommendation utility, beyond utility objectives, explanation quality, and
both consumer and provider fairness, on recommended lists with the well-known
size of $k=10$ (e.g., [7]), based on the corresponding test set. We describe
each metric in Table 3 and refer to the repository for implementation details.

Concerning recommendation utility for consumers, we monitored the Normalized
Discounted Cumulative Gain (NDCG) [40], using binary relevance scores and a
base-2 logarithm decay, and the Mean Reciprocal Rank (MRR) [11]. Differently
from recall and accuracy, NDCG takes into account the position of the relevant
products in the recommended list. MMR instead considers the position of the
first relevant product only, giving us a perspective different than NDCG.

In our work, we focused also on four well-known beyond utility goals [21]. We
monitored the extent to which the generated recommendations cover the catalog
of available products (coverage). High coverage may increase users’
satisfaction and the sales. Another goal, diversity, was found to be relevant
for human understanding [12] and content acceptance [10]. We computed it as
the percentage of distinct product categories in the recommended list.
Further, serendipity measures recommendation surprise[17]. Given our offline
setting, we compared the recommendations with those of a baseline model, i.e.,
a most popular recommender [24]. The more the recommendations differ between
the benchmarked and the baseline model, the higher the serendipity. Finally,
we estimated novelty as the inverse of product popularity (as per the received
ratings), assuming that products with low popularity are more likely to be
surprising [48].

Table 3: Evaluation metrics covered in our replicability study.

Perspective | Metric | Acronym | Range | Description
---|---|---|---|---
Consumers Utility |  Normalized Discounted Cumulative Gain | NDCG | [0, 1] | The extent to which the recommended products
|  |  | are useful for the user (1 means more useful).
| Mean Reciprocal Rank | MRR | [0, 1] | The extent to which the first recommended product
|  |  |  | is useful for the user (1 means more useful).
Consumers Beyond Utility | Coverage | COV | (0, 1) | The percentage of products overall
|  |  | recommended at least once (1 means high coverage).
Diversity | DIV | (0, 1] | The percentage of product categories covered
|  |  | in the recommended list (1 means high diversity).
| Novelty | NOV | (0, 1) | Inverse of the popularity of products recommended
|  |  |  | to a user (1 means low popularity, so high novelty).
| Serendipity | SER | [0, 1] | The percentage of the recommended products not
|  |  |  | suggested also by a baseline (1 means more unexpected).
Consumers Explanation Quality | Fidelity | FID | [0, 1] | The percentage of the recommended products that can
|  |  | be explained (1 means all products can be explained).
Linking Interaction Recency | LIR | [0, 1] | The recency of the past interaction in the paths
|  |  | accompanying recommended products (1 means recent).
Linking Interaction Diversity | LID | (0, 1] | The number of distinct past interactions in the paths
|  |  | accompanying recommended products (1 means different).
Shared Entity Popularity | SEP | [0, 1] | The popularity of the shared entity in the paths
|  |  | accompanying recommended products (1 means popular).
| Shared Entity Diversity | SED | (0, 1] | The number of distinct shared entities in the paths
|  |  |  | accompanying recommended products (1 means different).
| Path Type Diversity | PTD | (0, 1] | The percentage of distinct path types within paths
|  |  |  | accompanying recommended products (1 means different).
| Path Type Concentration | PTC | [0, 1] | The extent to which the distinct path types
|  |  |  | representation is equally balanced (1 means balanced).
Providers Utility | Exposure | EXP | [0, 1] |  Exposure of the items of a given provider in the recommended list (1 means high exposure).

With regard to explanation quality, we considered the proportion of
explainable products in a recommended list (fidelity) [28]. In addition, we
monitored reasoning paths properties concerning recency, popularity, and
diversity [3]. Recent linking interactions can help the user to better catch
the explanation. The linked interaction recency measures the recency of the
past interaction presented in the explanation path, whereas the linked
interaction diversity monitors how many distinct past interactions are
present. The second perspective is related to shared entities, assuming that
more popular shared entities have a higher chance of being familiar to the
user. The shared entity popularity measures the popularity (node degree in the
KG) of the shared entity in an explanation path. Conversely, the shared entity
diversity monitors the distinct shared entities. Finally, path type diversity
focuses on how many distinct path types are included. Path type concentration
monitors whether path types are equally balanced.

With the increasing importance received by fairness, we also assessed fairness
with respect to a notion of demographic parity [7, 14]. For consumer fairness,
given a metric, we computed the average value of that metric for each
demographic group and monitored the absolute pairwise difference between
groups444 Our data includes sensitive attributes pertaining to the gender
(Male, Female) and age (Under 18, 18-24, 25-34, 35-44, 45-49, 50-55, 56$+$),
as per the data set labels. . Concerning provider fairness, we computed the
average exposure given to products of providers in a given demographic group
[6]. Again, we finally computed the average absolute pairwise difference
between provider groups.

##  3 Experimental Results

Our study aimed to investigate multiple evaluation perspectives of path
reasoning methods, by answering to the following research questions:

  1. RQ1

Do path reasoning methods trade recommendation utility and/or beyond utility
objectives for explanation power?

  2. RQ2

To what extent can path reasoning methods produce explanations for all the
recommended products, depending on the recommended list size?

  3. RQ3

How does the quality of the selected paths vary among path reasoning methods,
based on the path type and characteristics?

###  3.1 Trading Recommendation Goals for Explanation Power (RQ1)

In a first analysis, we investigated whether there exists any substantial
difference in recommendation utility and beyond utility objectives between the
considered path reasoning methods (PGPR, CAFE, UCPR) and relevant knowledge-
aware but not explainable methods (KGAT, CKE, CFKG).  To assess statistical
significance, t-tests were carried out for each metric, considering the two
categories of methods as the two separate groups, under each data set. This
allowed us to discern behavior also with respect to the sparsity of the KG.
Although the total sample size (six methods) is rather small, this setup made
it possible to notice some preliminary characteristics of the two method
classes. Still, further studies on a broader set of methods should be run to
assess results generalizability more. P-values obtained for each test are
reported in Table 5.

Figure 1 depicts our evaluation results in terms of utility (NDCG, MMR) and
beyond utility goals (serendipity, diversity, novelty, and coverage). We also
report provider fairness estimates, whereas we will discuss consumer fairness
later on. Even though Figure 1 makes the comparison easier, we refer to Table
4 for specific values. Concerning recommendation utility, path reasoning
methods achieved comparable scores (0.26 to 0.28 NDCG; 0.18 to 0.21 MRR) to
knowledge-aware non-explainable methods (0.26 to 0.29 NDCG; 0.21 to 0.23 MRR)
in ML1M. These observations did not hold in LFM1M, where path reasoning
methods led to recommendations of lower utility (0.15 to 0.34 NDCG; 0.09 to
0.27 MMR) than knowledge-aware baselines (0.13 to 0.40 NDCG; 0.10 to 0.34
MMR).  However, in both cases and under both data sets, no statistical
differences were found in terms of recommendation utility between method
classes (all p-values were greater than $0.05$).

Figure 1: Comparison on recommendation utility and beyond utility goals [RQ1].
Table 4: Metric scores for recommendation utility and beyond utility goals
[RQ1].

Method | ML1M | LFM1M
---|---|---
| NDCG $\uparrow$ | MMR $\uparrow$ | SER $\uparrow$ | DIV $\uparrow$ | NOV $\uparrow$ | PF1 $\downarrow_{0}$ | COV $\uparrow$ | NDCG $\uparrow$ | MMR $\uparrow$ | SER $\uparrow$ | DIV $\uparrow$ | NOV $\uparrow$ | PF1 $\downarrow_{0}$ | COV $\uparrow$
CKE | 0.29 | 0.23 | 0.26 | 0.10 | 0.93 | 0.19 | 0.70 | 0.40 | 0.34 | 0.82 | 0.18 | 0.88 | 0.18 | 0.91
CFKG | 0.26 | 0.21 | 0.11 | 0.11 | 0.92 | 0.25 | 0.16 | 0.13 | 0.10 | 0.04 | 0.27 | 0.86 | 0.34 | 0.02
KGAT | 0.29 | 0.23 | 0.29 | 0.10 | 0.93 | 0.19 | 0.75 | 0.37 | 0.31 | 0.79 | 0.19 | 0.88 | 0.18 | 0.89
PGPR | 0.28 | 0.21 | 0.78 | 0.42 | 0.93 | 0.27 | 0.42 | 0.31 | 0.25 | 0.81 | 0.54 | 0.82 | 0.32 | 0.20
UCPR | 0.26 | 0.20 | 0.53 | 0.42 | 0.93 | 0.22 | 0.25 | 0.34 | 0.27 | 0.94 | 0.57 | 0.87 | 0.22 | 0.41
CAFE | 0.26 | 0.18 | 0.63 | 0.44 | 0.93 | 0.36 | 0.21 | 0.15 | 0.09 | 0.75 | 0.58 | 0.84 | 0.36 | 0.11
For each dataset: best result in bold, second-best result underlined. 1
Metrics $PF$: Provider Fairness.

Table 5: T-test p-values to assess statistically significant differences
between path-based and knowledge-aware methods across evaluation metrics
[RQ1].

Data Set1 | NDCG | MMR | SER | DIV | NOV | PF | COV
---|---|---|---|---|---|---|---
ML1M | 0.33 | 0.08 | 0.01 | 0 | 0.422 | 0.21 | 0.33
LFM1M | 0.77 | 0.65 | 0.38 | 0 | 0.16 | 0.38 | 0.34
P-values below $0.05$ are reported in bold.

Figure 2: Disparate impacts between groups (gender and age) on recommendation
utility, beyond utility objectives, and provider fairness. The lower it is,
the fairer [RQ1].

With regard to beyond utility objectives, path reasoning methods achieved
substantially higher serendipity (0.53 to 0.78 ML1M; 0.75 to 0.94 LFM1M) than
knowledge-aware non-explainable baselines (0.11 to 0.29 ML1M; 0.04 to 0.82
LFM1M). Interestingly, it was confirmed that path reasoning methods tend to
perform worse in LFM1M than ML1M.  Furthermore, our statistical tests show
that, on ML1M, the two method classes performed differently in terms of
serendipity (p-value equal to 0.011), with path reasoning methods showing
higher serendipity than the knowledge aware methods, on average. Similar
patterns were found for diversity. Path reasoning methods led to a higher
diversity on average (respectively for path-reasoning and knowledge-aware
methods, 0.56 and 0.21 on LFM1M; 0.43 and 0.10 on ML1M), on both ML1M and
LFM1M (p-value $\approx 0$).  Conversely, on coverage, the best path reasoning
method showed a decrease of 44% on ML1M and 54.9% on LFM1M than the best
knowledge-aware method. This might be due to the low number of paths in the KG
available to the path-reasoning methods. Except for CFKG, there is evidence
that knowledge-aware methods should be preferred in case someone aims to
optimize for coverage.  Finally, novelty scores were similar between the two
classes of methods.

From a provider fairness perspective, path reasoning methods led to a fairer
exposure of provider groups (0.22 to 0.36 ML1M; 0.22 to 0.36 LFM1M), compared
to the other family (0.19 to 0.25 ML1M; 0.18 to 0.34 LFM1M). Surprisingly,
CFKG reported the second best provider fairness score, despite of its low
recommendation utility. On the other hand, consumer fairness estimates
according to the considered evaluation metrics555 Differences between the
demographic groups achieving the best and worst score on avg. were all
statistically significant under t-test (if applicable) or h-test otherwise.
are collected in Figure 2. Being the patterns comparable across data sets and
demographic groups, we describe only the results obtained in ML1M and the
gender groups. For the latter, all the models presented some yet low levels of
unfairness in term of utility (both NDCG and MRR). Path reasoning methods
(PGPR, UCPR, CAFE) achieved higher levels of unfairness on coverage,
respectively 0.084, 0.045, 0.032, compared to baseline methods (KGAT reached
the highest coverage unfairness, with 0.06). For the other metrics, we
uncovered small yet comparable differences.

Findings RQ1.  Path reasoning methods trade recommendation utility and
coverage for explanation power, especially in LFM1M. Conversely, they resulted
in higher estimates on other beyond utility objectives and provider fairness
than knowledge-aware non-explainable baselines.

###  3.2 Producing Explanations for All Recommended Products (RQ2)

In a second analysis, we were interested in understanding the extent to which
path reasoning methods can produce explanations for all the recommended
products across recommended lists of different sizes. This property, fidelity,
is essential for a method which yields reasoning paths (and produces
explanations). Table 6 shows fidelity scores for the two data sets on lists of
size 10, 20, 50 and 100.

Concerning PGPR, paths were attached to almost every product of the
recommended list, until the size of 50. Under a size of 100, fidelity
remarkably decreased to 78%. This decay in fidelity was exacerbated in LFM1M.
In the latter data set, already with a size of 20, only 74% of recommended
products were explained. With regard to UCPR, we observed a similar but
reversed pattern, compared to PGPR, on the two considered data sets. UCPR was
challenged to produce explanations even under a size of 10 on ML1M (only 61%
of products were explained). Conversely, in LASTFM, the same model obtained a
higher fidelity than PGPR, with 99% of explained products under a size of 10.
Surprisingly, CAFE was able to provide a reasoning path for each recommended
product until a list of size 100. It should however be noted that, to make
this happen, the list size must be specified in advance during training. CAFE
indeed automatically adapts the size of the neighbourhood to search around,
according to the list size. Hence, this method would be the best choice when
the list size is known in advance, constant, or up to a certain limit.
Although fidelity could be controlled, doing this led to a smaller NDCG for
CAFE (see Section 3.1).

Findings RQ2.  Path reasoning methods show very different patterns in terms of
fidelity. CAFE’s fidelity is high and stable across data sets and recommended
list sizes. On the other hand, PGPR provides higher but rapidly decaying
fidelity in ML1M than LFM1M, viceversa for UCPR.

Table 6: Explanation fidelity analysis across cut-offs $k=\\{10,20,50,100\\}$
[RQ2].

Method | ML1M | LFM1M
---|---|---
| 10 | 20 | 50 | 100 | 10 | 20 | 50 | 100
PGPR | 1.00 | 0.99 | 0.99 | 0.78 | 0.98 | 0.74 | 0.31 | 0.15
CAFE | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
UCPR | 0.61 | 0.34 | 0.14 | 0.07 | 0.99 | 0.98 | 0.68 | 0.35

###  3.3 Differences on Explanation Quality (RQ3)

In a final analysis, we investigated how the quality of the selected paths
(and so of the resulting explanations) varied based on the path
characteristics. To this end, Table 7 collects seven explanation path quality
perspectives (LIR, LID, SEP, SED, PTD, PTC, PPC) for each reasoning path
method and data set.

Concerning the recency dimension, we did not observe any substantial
difference in linked interaction recency among the three methods, with the
maximum (minimum) value 0.44 (0.34) achieved by PGPR (CAFE) on ML1M (similarly
on LFM1M). Whereas, in terms of linked interaction diversity, it can be
interestingly noted that PGPR (0.84 ML1M; 0.77 LFM1M) and UCPR (0.82 ML1M;
0.84 LFM1M) led to higher diversity than CAFE.

Moving to the popularity perspective and, the shared entity popularity in
particular, CAFE was able to obtain the highest SEP in both data sets (0.75
ML1M; 0.77 LFM1M), meaning that it had the tendency to yield paths with more
popular shared entities. Compared to CAFE, PGPR and UCPR showed instead
substantially lower values. Estimates on SED were very high (0.92 to 1 ML1M;
0.78 to 0.98 LFM1M) for all the methods. These methods had hence the ability
to include a good variety of shared entities in their reasoning paths.

Patterns regarding path types were particularly interesting. Specifically,
both path type diversity and path type concentration were higher in CAFE and
UCPR than PGPR. This highlights that the explanations of the former were, on
average, richer in terms of path types (e.g., starred by, directed by), while
PGPR’s explanations were limited, on average, to a narrow set of different
path types.

Table 7: Explanation quality analysis [RQ3].

Model | ML1M | LFM1M
---|---|---
| LIR $\uparrow$ | LID $\uparrow$ | SEP $\uparrow$ | SED $\uparrow$ | PTD $\uparrow$ | PTC $\uparrow$ | PPC $\uparrow$ | LIR $\uparrow$ | LID $\uparrow$ | SEP $\uparrow$ | SED $\uparrow$ | PTD $\uparrow$ | PTC $\uparrow$ | PPC $\uparrow$
PGPR | 0.44 | 0.84 | 0.43 | 0.99 | 0.12 | 0.03 | 0.12 | 0.49 | 0.77 | 0.61 | 0.94 | 0.30 | 0.05 | 0.24
CAFE | 0.34 | 0.16 | 0.75 | 1.00 | 0.33 | 0.73 | 0.37 | 0.49 | 0.25 | 0.77 | 0.98 | 0.25 | 0.62 | 0.50
UCPR | 0.40 | 0.82 | 0.35 | 0.92 | 0.24 | 0.01 | 0.24 | 0.49 | 0.84 | 0.67 | 0.78 | 0.42 | 0.24 | 0.34
For each dataset: best result in bold, second-best result underlined.

Figure 3: Disparate impacts between groups (gender and age) on explanation
quality pertaining to recency, popularity, and diversity. The lower it is, the
fairer [RQ3].

Figure 3 depicts pairwise differences in the average score of a given metric
between demographic groups (consumer fairness). Again, under the same
conditions of RQ1, all the differences were statistically valid. For
conciseness, we discuss only the results for gender groups. PGPR and CAFE
showed very low unfairness estimates across all metrics, with all scores lower
than 0.01 in UCPR and 0.02 in PGPR on both data sets. Differently, UCPR
emphasized unfairness on PTD, SED and PPC. Remarkably, the strongest disparate
impact was reported on FID (0.12), PTD and PPC (0.05) for both data sets, and
SED (0.03).  We conjecture that these estimates of unfairness could be driven
by data imbalance across the different demographic groups.

Findings RQ3.  Path reasoning methods often yield substantially different
paths in terms of recency, popularity, and diversity. Although they exist, no
remarkable disparate impacts on explanation quality were found.

##  4 Discussion and Conclusion

In this section, we connect the main findings coming from the individual
experiments and present the implications and limitations of our replicability
study.

In the first analysis, we analyzed how methods able to produce explanations
(path reasoning) compared against knowledge-aware non-explainable baselines in
terms of utility, beyond utility objectives, and provider fairness. Results
show that these two classes of methods diverge slightly in terms of utility,
although explanations may have a persuasive effect which could not be captured
offline. Further studies should investigate how explanations impact user
decisions and consequently utility. Considering beyond accuracy objectives, we
observed that path reasoning methods, due to their internal mechanics, tend to
favor serendipity and diversity. At the same time, the methodological
decisions made to produce explanations make the methods more sensible to the
KG structure, consequently resulting in low product coverage for the
benchmarked methods. Results also show that all methods (including baselines)
emphasize some levels of unfairness in almost all perspectives, especially
utility. Our study calls for debiasing methods that consider multiple
perspectives in the knowledge-aware setup.

In the second analysis, we analyzed whether path reasoning methods can produce
explanations across various recommended list sizes. What emerged is that some
models (e.g., UCPR) are more sensible to the data and KG composition, which
influence their capability of producing reasoning paths even under short
recommended lists.  This limitation could be avoided by making specific model
design choices. For example, CAFE operates with a pre-defined search space for
each user to deliver reasoning paths for each recommended product (although
this design choice might affect recommendation utility).

In the last analysis, we went beyond the ability of just producing reasoning
paths, focusing on their quality. Several studies highlighted the benefits of
explanations [33]. Recent studies also showed that path properties (e.g.,
recency, popularity, and diversity) can influence the user perception of
explanations [3]. Results show that not all of these goals can be met at the
same time. For instance, PGPR fails to produce diverse explanations in ML1M,
whereas CAFE yields explanations based on a tiny set of past user
interactions. Future studies should address this aspect through in- and post-
processing methods and look at other explanation perspectives (e.g.,
persuasiveness, trust, and efficiency).

Overall, our analyses showed that replicating research in this area is still a
challenging task. In future work, we plan to explore in detail the impact of
KG characteristics on the considered perspectives, as well as devise novel
path reasoning methods robust to the KG structure and effective on multiple
objectives.

## References

  * [1] Ai, Q., Azizi, V., Chen, X., Zhang, Y.: Learning heterogeneous knowledge base embeddings for explainable recommendation. Algorithms 11(9), 137 (2018)
  * [2] Balloccu, G., Boratto, L., Fenu, G., Marras, M.: Hands on explainable recommender systems with knowledge graphs. In: Proceedings of the 16th ACM Conference on Recommender Systems. p. 710–713. RecSys ’22, Association for Computing Machinery, New York, NY, USA (2022)
  * [3] Balloccu, G., Boratto, L., Fenu, G., Marras, M.: Post processing recommender systems with knowledge graphs for recency, popularity, and diversity of explanations. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 646–656. SIGIR ’22, Association for Computing Machinery, New York, NY, USA (2022)
  * [4] Balloccu, G., Boratto, L., Fenu, G., Marras, M.: Reinforcement recommendation reasoning through knowledge graphs for explanation path quality. Knowledge-Based Systems 260, 110098 (2023)
  * [5] Barredo Arrieta, A., Díaz Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., Herrera, F.: Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020)
  * [6] Boratto, L., Fenu, G., Marras, M.: Interplay between upsampling and regularization for provider fairness in recommender systems. User Model. User Adapt. Interact. 31(3), 421–455 (2021)
  * [7] Boratto, L., Fenu, G., Marras, M., Medda, G.: Consumer fairness in recommender systems: Contextualizing definitions and mitigations. In: Hagen, M., Verberne, S., Macdonald, C., Seifert, C., Balog, K., Nørvåg, K., Setty, V. (eds.) Advances in Information Retrieval. pp. 552–566. Springer, Cham (2022)
  * [8] Cao, Y., Hou, L., Li, J., Liu, Z.: Neural collective entity linking. In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 675–686. Association for Computational Linguistics, Santa Fe, New Mexico, USA (Aug 2018)
  * [9] Cao, Y., Wang, X., He, X., Hu, Z., Tat-seng, C.: Unifying knowledge graph learning and recommendation: Towards a better understanding of user preference. In: WWW (2019)
  * [10] Carbonell, J., Goldstein, J.: The use of mmr, diversity-based reranking for reordering documents and producing summaries. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 335–336. SIGIR ’98, Association for Computing Machinery, New York, NY, USA (1998)
  * [11] Craswell, N.: Mean Reciprocal Rank, pp. 1703–1703. Springer US (2009)
  * [12] Gedikli, F., Jannach, D., Ge, M.: How should i explain? a comparison of different explanation types for recommender systems. International Journal of Human-Computer Studies 72(4), 367–382 (2014)
  * [13] Geng, S., Fu, Z., Tan, J., Ge, Y., de Melo, G., Zhang, Y.: Path language modeling over knowledge graphsfor explainable recommendation. In: Proceedings of the ACM Web Conference 2022. p. 946–955. WWW ’22, Association for Computing Machinery, New York, NY, USA (2022)
  * [14] Gómez, E., Zhang, C.S., Boratto, L., Salamó, M., Marras, M.: The winner takes it all: Geographic imbalance and provider (un)fairness in educational recommender systems. In: SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021. pp. 1808–1812. ACM (2021)
  * [15] Harper, F.M., Konstan, J.A.: The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. 5(4) (dec 2015)
  * [16] He, G., Li, J., Zhao, W.X., Liu, P., Wen, J.R.: Mining implicit entity preference from user-item interaction data for knowledge graph completion via adversarial learning. In: Proceedings of The Web Conference 2020. pp. 740–751 (2020)
  * [17] Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (jan 2004)
  * [18] Huang, C., Gan, Z., Ye, F., Wang, P., Zhang, M.: KNCR: knowledge-aware neural collaborative ranking for recommender systems. In: IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress, DASC/PiCom/CBDCom/CyberSciTech 2020, Calgary, AB, Canada, August 17-22, 2020. pp. 339–344. IEEE (2020)
  * [19] Huang, J., Zhao, W.X., Dou, H., Wen, J.R., Chang, E.Y.: Improving sequential recommendation with knowledge-enhanced memory networks. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. p. 505–514. SIGIR ’18, Association for Computing Machinery, New York, NY, USA (2018)
  * [20] Ji, L., Wang, Y., Shi, B., Zhang, D., Wang, Z., Yan, J.: Microsoft Concept Graph: Mining Semantic Concepts for Short Text Understanding. Data Intelligence 1(3), 238–270 (06 2019)
  * [21] Kaminskas, M., Bridge, D.: Diversity, serendipity, novelty, and coverage: A survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Trans. Interact. Intell. Syst. 7(1) (dec 2016)
  * [22] Linden, G., Smith, B., York, J.: Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Comput. 7(1), 76–80 (2003)
  * [23] Ma, W., Zhang, M., Cao, Y., Jin, W., Wang, C., Liu, Y., Ma, S., Ren, X.: Jointly learning explainable rules for recommendation with knowledge graph. In: The world wide web conference. pp. 1210–1221 (2019)
  * [24] Murakami, T., Mori, K., Orihara, R.: Metrics for evaluating the serendipity of recommendation lists. In: Satoh, K., Inokuchi, A., Nagao, K., Kawamura, T. (eds.) New Frontiers in Artificial Intelligence. pp. 40–46. Springer Berlin Heidelberg, Berlin, Heidelberg (2008)
  * [25] Musto, C., de Gemmis, M., Lops, P., Semeraro, G.: Generating post hoc review-based natural language justifications for recommender systems. User Modeling and User-Adapted Interaction 31(3), 629–673 (2021)
  * [26] Ni, J., Li, J., McAuley, J.J.: Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. pp. 188–197. Association for Computational Linguistics (2019)
  * [27] Oramas, S., Ostuni, V.C., Di Noia, T., Serra, X., Di Sciascio, E.: Sound and music recommendation with knowledge graphs. ACM Trans. Intell. Syst. Technol. 8(2), 21:1–21:21 (2017)
  * [28] Peake, G., Wang, J.: Explanation mining: Post hoc interpretability of latent factor models for recommendation systems. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. p. 2060–2069. KDD ’18, Association for Computing Machinery, New York, NY, USA (2018)
  * [29] Ricci, F., Rokach, L., Shapira, B.: Recommender Systems Handbook, vol. 1-35, pp. 1–35. Springer (10 2010)
  * [30] Schedl, M.: The lfm-1b dataset for music retrieval and recommendation. In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval. p. 103–110. ICMR ’16, ACM, New York, NY, USA (2016)
  * [31] Song, W., Duan, Z., Yang, Z., Zhu, H., Zhang, M., Tang, J.: Ekar: An explainable method for knowledge aware recommendation. CoRR abs/1906.09506 (2022)
  * [32] Tai, C.Y., Huang, L.Y., Huang, C.K., Ku, L.W.: User-Centric Path Reasoning towards Explainable Recommendation, p. 879–889. Association for Computing Machinery, New York, NY, USA (2021)
  * [33] Tintarev, N., Masthoff, J.: A survey of explanations in recommender systems. In: Proceedings of the 23rd International Conference on Data Engineering Workshops, ICDE 2007. pp. 801–810. IEEE Computer Society (2007)
  * [34] Wang, H., Zhang, F., Wang, J., Zhao, M., Li, W., Xie, X., Guo, M.: Ripplenet: Propagating user preferences on the knowledge graph for recommender systems. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. p. 417–426. CIKM ’18, Association for Computing Machinery, New York, NY, USA (2018)
  * [35] Wang, H., Zhang, F., Xie, X., Guo, M.: Dkn: Deep knowledge-aware network for news recommendation (2018)
  * [36] Wang, X., He, X., Cao, Y., Liu, M., Chua, T.S.: Kgat: Knowledge graph attention network for recommendation. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. p. 950–958. KDD ’19, Association for Computing Machinery, New York, NY, USA (2019)
  * [37] Wang, X., Wang, D., Xu, C., He, X., Cao, Y., Chua, T.S.: Explainable reasoning over knowledge graphs for recommendation. In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI Press (2019)
  * [38] Wang, X., Wang, D., Xu, C., He, X., Cao, Y., Chua, T.S.: Explainable reasoning over knowledge graphs for recommendation. In: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI Press (2019)
  * [39] Wang, X., Liu, K., Wang, D., Wu, L., Fu, Y., Xie, X.: Multi-level recommendation reasoning over knowledge graphs with reinforcement learning. In: Proceedings of the ACM Web Conference 2022. p. 2098–2108. WWW ’22, Association for Computing Machinery, New York, NY, USA (2022)
  * [40] Wang, Y., Wang, L., Li, Y., He, D., Liu, T.: A theoretical analysis of NDCG type ranking measures. In: COLT 2013 - The 26th Annual Conference on Learning Theory, June 12-14, 2013, Princeton University, NJ, USA. JMLR Workshop and Conference Proceedings, vol. 30, pp. 25–54. JMLR.org (2013)
  * [41] Xian, Y., Fu, Z., Muthukrishnan, S., de Melo, G., Zhang, Y.: Reinforcement knowledge graph reasoning for explainable recommendation. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 285–294. SIGIR’19, Association for Computing Machinery, New York, NY, USA (2019)
  * [42] Xian, Y., Fu, Z., Zhao, H., Ge, Y., Chen, X., Huang, Q., Geng, S., Qin, Z., de Melo, G., Muthukrishnan, S., Zhang, Y.: Cafe: Coarse-to-fine neural symbolic reasoning for explainable recommendation. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. p. 1645–1654. CIKM ’20, Association for Computing Machinery, New York, NY, USA (2020)
  * [43] Zhang, F., Yuan, N.J., Lian, D., Xie, X., Ma, W.Y.: Collaborative knowledge base embedding for recommender systems. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. p. 353–362. KDD ’16, Association for Computing Machinery, New York, NY, USA (2016)
  * [44] Zhang, Y., Chen, X.: Explainable recommendation: A survey and new perspectives. Foundations and Trends® in Information Retrieval 14(1), 1–101 (2020)
  * [45] Zhao, K., Wang, X., Zhang, Y., Zhao, L., Liu, Z., Xing, C., Xie, X.: Leveraging Demonstrations for Reinforcement Recommendation Reasoning over Knowledge Graphs, p. 239–248. ACM, New York, NY, USA (2020)
  * [46] Zhao, W.X., He, G., Yang, K., Dou, H., Huang, J., Ouyang, S., Wen, J.R.: KB4Rec: A Data Set for Linking Knowledge Bases with Recommender Systems. Data Intelligence 1(2), 121–136 (04 2019)
  * [47] Zhao, Y., Wang, X., Chen, J., Wang, Y., Tang, W., He, X., Xie, H.: Time-aware path reasoning on knowledge graph for recommendation. ACM Trans. Inf. Syst. (apr 2022), just Accepted
  * [48] Zhou, T., Kuscsik, Z., Liu, J., Medo, M., Wakeling, J.R., Zhang, Y.C.: Solving the apparent diversity-accuracy dilemma of recommender systems. Proceedings of the National Academy of Sciences 107, 4511 – 4515 (2010)