11institutetext: Juniper Networks, Sunnyvale CA, USA
11email<EMAIL_ADDRESS>
# A Comprehensive Survey of Evaluation Techniques for Recommendation Systems

Aryan Jadon 11 0000-0002-2991-9913 Avinash Patil 11 0009-0002-6004-370X

###### Abstract

The effectiveness of recommendation systems is pivotal to user engagement and
satisfaction in online platforms. As these recommendation systems increasingly
influence user choices, their evaluation transcends mere technical performance
and becomes central to business success. This paper addresses the multifaceted
nature of recommendation system evaluation by introducing a comprehensive
suite of metrics, each tailored to capture a distinct aspect of system
performance. We discuss

  * •

Similarity Metrics: to quantify the precision of content-based filtering
mechanisms and assess the accuracy of collaborative filtering techniques.

  * •

Candidate Generation Metrics: to evaluate how effectively the system
identifies a broad yet relevant range of items.

  * •

Predictive Metrics: to assess the accuracy of forecasted user preferences.

  * •

Ranking Metrics: to evaluate the effectiveness of the order in which
recommendations are presented.

  * •

Business Metrics: to align the performance of the recommendation system with
economic objectives.

Our approach emphasizes the contextual application of these metrics and their
interdependencies. In this paper, we identify the strengths and limitations of
current evaluation practices and highlight the nuanced trade-offs that emerge
when optimizing recommendation systems across different metrics.

The paper concludes by proposing a framework for selecting and interpreting
these metrics to not only improve system performance but also to advance
business goals. This work is to aid researchers and practitioners in
critically assessing recommendation systems and fosters the development of
more nuanced, effective, and economically viable personalization strategies.
Our code is available at GitHub.

###### Keywords:

Business Metrics, Candidate Generation Metrics, Recommendation Systems,
Predictive Metrics, Ranking Metrics, Similarity Metrics

##  1 Introduction

Recommendation systems have become an integral component of the digital
landscape, influencing the way we discover products, content, and even social
connections. From e-commerce to online streaming, these systems underpin user
experience by personalizing content and suggesting items that align with
individual preferences. The proliferation of such systems has catalyzed a need
for robust evaluation methods, as the efficiency of a recommendation system is
pivotal to user satisfaction and business success.

While recommendation systems have become increasingly complex and
sophisticated, evaluating their performance remains a challenge. Previous
research [3] [10] has focused on specific subsets of performance metrics,
often tailored to the domain where the recommendation system is applied[33].
However, a unified approach to performance evaluation, considering the
multifaceted aspects of these systems, is largely missing from the literature.

The performance of recommendation systems is multi-dimensional and cannot be
encapsulated by a single metric[15]. To comprehensively assess these systems,
one must consider a variety of metrics, each offering unique insights into
different aspects of system performance. This paper introduces five key types
of metrics that collectively provide a holistic evaluation framework:
similarity metrics, candidate generation metrics, predictive metrics, ranking
metrics, and business metrics.

Similarity metrics[29] are the cornerstone of content-based and collaborative
filtering methods, offering a quantitative measure of how closely items or
user preferences align. Candidate generation metrics[16] ensure a balanced
recommendation spectrum, avoiding the pitfalls of overly narrow or excessively
broad selections. Predictive metrics[41] go a step further, providing an
assessment of a system’s ability to accurately forecast user ratings or
preferences. Ranking metrics [24] are critical when the sequence of
recommendations is pivotal, evaluating the order in which items are presented
to the user. Lastly, business metrics[31] connect system performance with
tangible business outcomes, such as sales conversion rates or customer
engagement levels, ensuring the recommendation system aligns with overarching
business objectives.

In deploying these metrics, one must navigate a landscape rife with trade-offs
and complementary relationships, as the improvement in one metric could
potentially lead to the detriment of another[20]. Therefore, the selection and
interpretation of these metrics must be approached with a nuanced
understanding of the recommendation system’s goals, context, and the
characteristics of the dataset being used.

Through the subsequent sections, this paper will delve into each metric type,
elucidating their definitions, applications, and significance in evaluating
the efficacy of recommendation systems. By dissecting these metrics, we aim to
provide a framework that academics, practitioners, and stakeholders can adopt
to gauge the success of their recommendation systems, thereby enabling the
continuous advancement of personalized user experiences in the digital domain.

##  2 Evaluation Metrics

###  2.1 Similarity Metrics

Similarity metrics are used to measure the likeness or similarity between
items, users, or any relevant entities in a recommendation system. These
metrics help in identifying items that are similar to each other, which is
crucial for various recommendation techniques like content-based filtering and
collaborative filtering. Some of the key Similarity Metrics used in
recommendation systems are:

  1. 1.

Cosine Similarity

  2. 2.

Euclidean Distance

  3. 3.

Jaccard Index

  4. 4.

Hamming Distance

  5. 5.

Manhattan Distance

  6. 6.

Chebyshev Distance

  7. 7.

Adjusted Cosine Similarity

  8. 8.

Pearson Correlation Coefficient

  9. 9.

Spearman Rank Order Correlation Coefficient

####  2.1.1 Cosine Similarity

Cosine similarity[28] is a measure used to determine the similarity between
two non-zero vectors in an n-dimensional space, capturing how closely related
they are in orientation. It is calculated as the dot product of the vectors
divided by the product of their magnitudes. The formula for cosine similarity
is:

$\text{cos}(\theta)=\frac{\sum_{i=1}^{n}A_{i}B_{i}}{\sqrt{\sum_{i=1}^{n}A_{i}^{2}}\times\sqrt{\sum_{i=1}^{n}B_{i}^{2}}}$

This metric ranges from -1 (exactly opposite) to 1 (the same), with 0
typically indicating orthogonality or no similarity.

####  2.1.2 Euclidean Distance

Euclidean distance[37] is a widely used distance metric that gauges the
straight-line distance between two points in Euclidean space. It’s calculated
by taking the square root of the sum of the squared differences between
corresponding elements of the vectors. The formula for the Euclidean distance
between two points, $A$ and $B$, in an n-dimensional space is:

$d(A,B)=\sqrt{\sum_{i=1}^{n}(A_{i}-B_{i})^{2}}$

This metric ranges from 0 to infinity, where 0 indicates that the points are
identical.

####  2.1.3 Jaccard Index

The Jaccard Index [38], also known as the Jaccard similarity coefficient, is a
statistic used for gauging the similarity and diversity of sample sets. It’s
defined as the size of the intersection divided by the size of the union of
two sets. For two sets $A$ and $B$, the Jaccard Index $J$ is given by the
formula:

$J(A,B)=\frac{|A\cap B|}{|A\cup B|}$

Here, $|A\cap B|$ represents the number of elements common to both sets, while
$|A\cup B|$ is the total number of distinct elements present in either set.
The Jaccard Index ranges from 0 to 1, where 0 means there is no overlap
between the sets and 1 indicates that the sets are identical.

####  2.1.4 Hamming Distance

Hamming Distance [32] is a metric for comparing two strings of equal length,
quantifying the number of positions at which the corresponding symbols are
different. It measures the minimum number of substitutions required to change
one string into the other, which also corresponds to the minimum number of
errors that could have transformed one string into the other. In the context
of recommendation systems, it can be particularly useful for comparing user
preferences or item characteristics that are represented as binary vectors.

The Hamming Distance, $d_{H}$, between two strings (or binary vectors) $A$ and
$B$ is calculated as:

$d_{H}(A,B)=\sum_{i=1}^{n}\left[A_{i}\neq B_{i}\right]$

where $n$ is the length of the strings, and the notation $\left[A_{i}\neq
B_{i}\right]$ is an indicator function equal to 1 if $A_{i}$ and $B_{i}$ are
different, and 0 if they are the same.

####  2.1.5 Manhattan Distance

Manhattan Distance [44], also known as City Block Distance, is a distance
metric that measures the sum of the absolute differences between the
coordinates of a pair of objects. It is mathematically defined as:

$D_{\text{Manhattan}}(A,B)=\sum_{i=1}^{n}|A_{i}-B_{i}|$

where $A$ and $B$ are two vectors in $n$-dimensional space, and
$|A_{i}-B_{i}|$ denotes the absolute difference between the $i$-th components
of $A$ and $B$.

####  2.1.6 Chebyshev Distance

Chebyshev Distance[14], named after Pafnuty Chebyshev, is a distance metric
used in multi-dimensional spaces, often in data science and game theory. It
represents the maximum difference along any single dimension between two
points. For two points $\mathbf{p}=(p_{1},p_{2},\ldots,p_{n})$ and
$\mathbf{q}=(q_{1},q_{2},\ldots,q_{n})$ in an n-dimensional space, the
Chebyshev Distance is defined as:

$D_{\text{Chebyshev}}(\mathbf{p},\mathbf{q})=\max_{i}\left|p_{i}-q_{i}\right|$

Here, $\max_{i}$ signifies taking the maximum of the absolute differences
along each dimension. This metric is especially useful in scenarios where the
greatest single difference is more significant than the sum of all
differences, such as in chess, to calculate the minimum moves for a king, or
in clustering and classification tasks in data analytics.

####  2.1.7 Adjusted Cosine Similarity

Adjusted Cosine Similarity [48] is a variation of the traditional cosine
similarity that accounts for user rating biases. It adjusts the rating vectors
for each user by subtracting the user’s average rating before applying the
cosine similarity formula. This can enhance recommendation system performance
by normalizing user ratings and centering them around zero. The formula for
Adjusted Cosine Similarity is:

$\text{AC}(\theta)=\frac{\sum_{i=1}^{n}(R_{u,i}-\bar{R}_{u})\times(R_{v,i}-\bar{R}_{v})}{\sqrt{\sum_{i=1}^{n}(R_{u,i}-\bar{R}_{u})^{2}}\times\sqrt{\sum_{i=1}^{n}(R_{v,i}-\bar{R}_{v})^{2}}}$

where $R_{u,i}$ and $R_{v,i}$ are the ratings given by user $u$ and user $v$
to item $i$, respectively, and $\bar{R}_{u}$ and $\bar{R}_{v}$ are the average
ratings of user $u$ and user $v$, respectively.

####  2.1.8 Pearson Correlation Coefficient

The Pearson Correlation Coefficient (PCC)[42], denoted as $r$, is a measure of
linear correlation between two sets of data, yielding a value between -1 and
1. A value of 1 implies a perfect positive correlation, -1 a perfect negative
correlation, and 0 no correlation at all. For two vectors $X$ and $Y$, each
with $n$ elements, PCC is calculated as:

$r=\frac{\sum_{i=1}^{n}(X_{i}-\overline{X})(Y_{i}-\overline{Y})}{\sqrt{\sum_{i=1}^{n}(X_{i}-\overline{X})^{2}}\sqrt{\sum_{i=1}^{n}(Y_{i}-\overline{Y})^{2}}}$

where $\overline{X}$ and $\overline{Y}$ are the means of the $X$ and $Y$
vectors, respectively.

In Table 1, we delineate specific scenarios for the application of various
Similarity Metrics. This table serves as a concise guide, assisting in
selecting the appropriate metric in alignment with the particular
characteristics and requirements of each use case. Through this structured
presentation, we aim to provide clarity and ease in the decision-making
process for choosing the most suitable Similarity Metric, tailored to the
needs of distinct scenarios.

Table 1: Summary of Similarity Metrics. Metric |  Use Cases
---|---
Cosine Similarity |  Use cosine similarity when evaluating the orientation, not magnitude, of user or item vectors in recommendation systems, making it ideal for text-based or attribute-rich data where the pattern of interest, rather than the absolute value, is indicative of user preferences.
Euclidean Distance |  Use Euclidean Distance in evaluating recommendation systems when comparing profiles in a feature space with numerical attributes to gauge similarity based on the ’straight-line’ distance between points (user-item pairs).
Jaccard Index |  Use the Jaccard Index when evaluating the similarity between users’ or items’ sets of preferences or choices, particularly when the data is binary and the size of the intersection relative to the union of datasets is of interest.
Hamming Distance |  Use Hamming Distance in evaluating recommendation systems when comparing binary vectors of user preferences or item attributes, typically for assessing similarity or diversity in content-based filtering or collaborative filtering with binary datasets.
Manhattan Distance |  Use Manhattan Distance in recommendation systems when dealing with high-dimensional, sparse data sets, as it can be more robust to outliers than Euclidean distance and better at capturing differences when multiple dimensions contribute to the dissimilarity.
Chebyshev Distance |  Use Chebyshev Distance in recommendation systems when capturing the maximum disparity across dimensions is crucial, especially in high-dimensional spaces where the most significant difference between items is the most informative for recommendations.
Adjusted Cosine Similarity |  Use Adjusted Cosine Similarity when evaluating item-based collaborative filtering systems to account for varying user ratings and reduce bias by subtracting the user’s average rating from each of their ratings before computing similarity.
Pearson Correlation Coefficient |  Use Pearson Correlation in evaluating recommendation systems when assessing the linear relationship between users’ ratings, especially when the scale of ratings is important and you assume a normal distribution of the underlying data.
Spearman Rank Order Correlation Coefficient |  Use Spearman Rank Order Correlation Coefficient in recommendation systems when assessing the strength and direction of a monotonic relationship between ranked variables, particularly useful in scenarios with non-parametric, ordinal data or when evaluating the ranking quality of recommendations.

###  2.2 Candidate Generation Metrics

Candidate Generation Metrics play a pivotal role in the efficacy of
recommendation systems, acting as the backbone for filtering and presenting
the most relevant options to users. At their core, these metrics are
algorithms designed to sift through vast datasets, identifying potential items
or services that align closely with a user’s preferences, search history, and
behavioral patterns.

This initial step is crucial as it directly influences the quality and
relevance of recommendations presented to the user. By efficiently narrowing
down the pool of candidates from potentially millions to a manageable few,
these metrics not only enhance the user experience by providing targeted and
personalized recommendations but also significantly improve computational
efficiency. Furthermore, well-calibrated Candidate Generation Metrics help
avoid information overload, ensuring that users are not overwhelmed by too
many choices, which can lead to decision paralysis.

In essence, these metrics are indispensable for creating a tailored, user-
centric approach in recommendation systems, leading to increased user
engagement, satisfaction, and, ultimately, retention. Some of the key
Candidate Generation Metrics used in recommendation systems are :

  1. 1.

Novelty

  2. 2.

Diversity

  3. 3.

Serendipity

  4. 4.

Catalog Coverage

  5. 5.

Distributional Coverage

####  2.2.1 Novelty

Novelty[49]in recommendation systems measures how unexpected the recommended
items are to users, focusing on less-known items. Mathematically, for a set of
recommended items $R_{u}$ to user $u$, novelty is:

$\text{Novelty}(R_{u})=\frac{1}{|R_{u}|}\sum_{i\in
R_{u}}(1-\text{popularity\\_score}(i))$

Here, $\text{popularity\\_score}(i)$ is the normalized popularity of item $i$,
calculated as the ratio of users who interacted with $i$ to the maximum
popularity in the catalog. This approach inversely relates the popularity of
items to novelty, promoting less popular items to enhance user discovery and
exploration. It’s an essential metric for ensuring a diverse and engaging
recommendation experience.

####  2.2.2 Diversity

Diversity[27] in recommendation systems, crucial for maintaining user
engagement, is quantified using metrics like Intra-List Diversity (ILD). ILD
measures the average dissimilarity between all pairs of items in a
recommendation list. Mathematically, for a set of recommended items
$R=\\{r_{1},r_{2},\ldots,r_{n}\\}$, ILD is defined as:

$ILD(R)=\frac{2}{n\cdot(n-1)}\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}dissimilarity(r_{i},r_{j})$

Here, $n$ is the number of items in $R$, and $dissimilarity(r_{i},r_{j})$
computes the dissimilarity between items $r_{i}$ and $r_{j}$. ILD ranges from
0 (no diversity) to 1 (maximum diversity), reflecting the variety in
recommendations and ensuring that users are exposed to a broad range of
options.

####  2.2.3 Serendipity

Serendipity[25] is a metric used in recommendation systems to quantify the
degree to which the recommendations are both unexpected and useful to the
user. This concept is crucial in evaluating the effectiveness of a
recommendation system, especially in its ability to introduce users to items
they might not have discovered otherwise, but find surprisingly relevant and
enjoyable.

Mathematically, serendipity can be defined in the context of user-item
interactions. Let’s consider the following notations:

\- $U$ is the set of users. \- $I$ is the set of items. \- $R(u)$ is the set
of items recommended to user $u$. \- $L(u)$ is the set of items liked by user
$u$. \- $D(u)$ is the set of items discovered by user $u$ through the
recommendation system.

The serendipity of the system for user $u$ can be calculated as:

$\text{Serendipity}(u)=\frac{|\\{i\in R(u)\cap L(u)\cap D(u)\\}|}{|R(u)|}$

This formula calculates the proportion of recommended items that are both
liked and discovered by the user, indicating the element of surprise and
relevance in the recommendations.

To get the overall serendipity of the system across all users, we average this
value over all users:

$\text{Serendipity}=\frac{1}{|U|}\sum_{u\in U}\text{Serendipity}(u)$

Here, $|U|$ denotes the number of users in the system. This overall
serendipity score gives an indication of how well the recommendation system
performs in terms of introducing relevant yet unexpected items to its users.
Higher scores indicate a system’s stronger ability to provide serendipitous
recommendations.

####  2.2.4 Catalog Coverage

Catalog Coverage[11] is a vital metric in evaluating the breadth of a
recommendation system’s reach. It measures the proportion of items in the
entire catalog that are actually recommended to users, providing insight into
the diversity of the system’s suggestions. The formula for Catalog Coverage
is:

$\text{Catalog Coverage}=\frac{|\text{Unique Items Recommended}|}{|\text{Total
Items in Catalog}|}\times 100\%$

Here, $|\text{Unique Items Recommended}|$ is the count of distinct items the
system has recommended, and $|\text{Total Items in Catalog}|$ represents the
total number of unique items available in the catalog. To compute this, one
must tally all unique items recommended over a certain period, count the total
items in the catalog, and then divide the former by the latter, expressing the
result as a percentage.

This metric is crucial for gauging how well a recommendation system explores
and utilizes the full range of available items. A high Catalog Coverage
indicates a system that suggests a wide variety of items, potentially
appealing to a diverse user base. On the contrary, low Catalog Coverage might
indicate a tendency to focus on a limited set of popular items, which could
neglect users with unique or niche interests. Therefore, monitoring and
optimizing Catalog Coverage is essential for maintaining a balanced and
inclusive recommendation system.

####  2.2.5 Distributional Coverage

Distributional coverage[7] is a crucial metric in recommendation systems,
focusing on the diversity of recommendations across the entire item catalog.
It ensures that a system is not biased towards a few popular items but instead
promotes a broader range of choices. To quantify this, distributional coverage
is often calculated using entropy, a measure of unpredictability or diversity.
For a catalog with $N$ items, where $p(i)$ represents the probability of
recommending item $i$, the distributional coverage ($DC$) can be expressed as:

$DC=-\sum_{i=1}^{N}p(i)\log_{2}p(i)$

Here, the sum is over all catalog items, and $p(i)$ is estimated from the
frequency of item $i$’s appearance in recommendation lists. A higher $DC$
value indicates a more diverse recommendation pattern, implying a wide array
of items being recommended, while a lower value suggests a concentration on
fewer items. Balancing this metric with others, such as personalization and
relevance, is vital to maintain the effectiveness of the recommendation system
while ensuring variety.

Table 2 presents a comprehensive overview of the appropriate application
scenarios for various Candidate Generation Metrics. This table serves as a
guide, delineating which metrics are most suitable for specific use cases in
recommendation systems.

Table 2: Summary of Candidate Generation Metrics. Metric |  Use Cases
---|---
Novelty |  Use novelty in evaluating recommendation systems when assessing the system’s ability to suggest unexpected, lesser-known, or new items, thereby enhancing user experience by introducing diversity and reducing the filter bubble effect prevalent in personalized recommendations.
Diversity |  Use Diversity in recommendation systems when the goal is to broaden user exposure beyond familiar items, avoid echo chambers, and enhance user engagement by presenting a varied range of options, thereby catering to a wider spectrum of user interests and preferences.
Serendipity |  Use serendipity in evaluating recommendation systems when assessing their ability to offer novel, unexpected, yet relevant items, enhancing user experience by introducing diversity beyond typical or predictable suggestions.
Catalog Coverage |  Use Catalog Coverage to evaluate a recommendation system when assessing its ability to suggest a wide variety of items, particularly important in scenarios where exposing users to a broader selection of the catalog is crucial for enhancing user experience and business objectives.
Distributional Coverage |  Distributional Coverage is used when evaluating the diversity of recommendations in a system, ensuring a wide range of items are suggested, rather than repeatedly offering popular or similar items, thus enhancing user discovery and experience.

###  2.3 Predictive Metrics

Predictive metrics are used to assess the predictive accuracy of a
recommendation system. These metrics evaluate how well the system predicts
user preferences or ratings for items. Some of the key Rating Metrics used in
recommendation systems are :

  1. 1.

Root Mean Squared Error (RMSE)

  2. 2.

Mean Absolute Error (MAE)

  3. 3.

Mean Squared Error (MSE)

  4. 4.

Mean Absolute Percentage Error (MAPE)

  5. 5.

$R^{2}$

  6. 6.

Explained Variance

####  2.3.1 Root Mean Squared Error (RMSE)

Root Mean Squared Error (RMSE)[23] is a standard way to measure the error of a
model in predicting quantitative data. Formally, it represents the square root
of the average of the squares of the differences between predicted and
observed values.

In the context of recommendation systems, it quantifies the differences
between the ratings predicted by the model and the actual ratings given by the
users. The formula for RMSE is given by:

$RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2}}$

where $N$ is the number of observations, $y_{i}$ is the actual value of an
observation, and $\hat{y}_{i}$ is the predicted value.

####  2.3.2 Mean Absolute Error (MAE)

Mean Absolute Error (MAE)[6] is a metric used to evaluate the accuracy of a
prediction model. It calculates the average magnitude of errors in a set of
predictions, without considering their direction. The MAE is given by the
formula:

$\text{MAE}=\frac{1}{n}\sum_{i=1}^{n}|y_{i}-\hat{y}_{i}|$

where $n$ is the number of predictions, $y_{i}$ is the actual value, and
$\hat{y}_{i}$ is the predicted value. The absolute difference between the
actual and predicted values indicates the error magnitude, and the MAE
aggregates these errors across all predictions.

####  2.3.3 Mean Squared Error (MSE)

Mean Squared Error (MSE)[2] is a widely used measure of prediction accuracy in
recommendation systems, quantifying the difference between predicted and
actual values. The MSE is computed by averaging the squares of the errors,
i.e., the differences between predicted ($\hat{y}_{i}$) and observed ($y_{i}$)
values over $n$ predictions. The formula for MSE is:

$\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_{i}-y_{i})^{2}$

The squaring of the errors ensures that larger errors are more prominently
reflected in the total, emphasizing the cost of significant deviations and
ensuring that the result is always non-negative.

####  2.3.4 Mean Absolute Percentage Error (MAPE)

Mean Absolute Percentage Error (MAPE)[5] is a statistical measure used to
assess the accuracy of forecasting models. It represents the average absolute
percent error for each data point, omitting the direction of the error. The
formula for MAPE is given as:

$\text{MAPE}=\left(\frac{100\%}{n}\right)\sum_{i=1}^{n}\left|\frac{y_{i}-\hat{y}_{i}}{y_{i}}\right|$

Where: \- $n$ is the number of observations, \- $y_{i}$ is the actual value,
\- $\hat{y}_{i}$ is the forecasted value.

The MAPE is useful because it provides a quick, intuitive percentage error,
allowing for comparison across different datasets or models. However, its
interpretability can be compromised when dealing with zero or very small
actual values.

####  2.3.5 $R^{2}$

The coefficient of determination, denoted as $R^{2}$, is a crucial metric in
evaluating the predictive accuracy of models, including those in
recommendation systems. It quantifies the proportion of variance in the
dependent variable that is predictable from the independent variables. The
formula for $R^{2}$ is:

$R^{2}=1-\frac{SS_{\text{res}}}{SS_{\text{tot}}}$

where $SS_{\text{res}}$ is the residual sum of squares
($\sum(y_{i}-\hat{y}_{i})^{2}$), representing the unexplained variance by the
model, and $SS_{\text{tot}}$ is the total sum of squares
($\sum(y_{i}-\bar{y})^{2}$), reflecting the total variance in the observed
data. Here, $y_{i}$ are the observed values, $\hat{y}_{i}$ are the predicted
values, and $\bar{y}$ is the mean of observed data.

$R^{2}$ ranges from 0 to 1, where 0 indicates no explanatory power and 1
indicates perfect prediction. In recommendation systems, a higher $R^{2}$
suggests that the model accurately predicts user ratings or preferences,
making it an essential tool for assessing the performance of these systems. It
complements other metrics like similarity, classification, and business
metrics, offering a comprehensive view of the system’s effectiveness and
efficiency in personalizing user experiences.

####  2.3.6 Explained Variance

Explained Variance is a key statistical measure in predictive modeling,
crucial for evaluating recommendation systems. It quantifies the proportion of
variance in the dependent variable (like user ratings) explained by the
independent variables in the model. The formula for Explained Variance is:

$\text{Explained Variance}=1-\frac{\text{Var}(e)}{\text{Var}(Y)}$

or in a more detailed form:

$\text{Explained
Variance}=1-\frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}$

Here, $\text{Var}(e)$ represents the variance of the model errors,
$\text{Var}(Y)$ is the total variance of the dependent variable, $y_{i}$ are
the actual values, $\hat{y}_{i}$ the predicted values, $\bar{y}$ the mean of
actual values, and $n$ the number of observations. While closely related to
$R^{2}$, the coefficient of determination, Explained Variance offers a nuanced
understanding of a model’s ability to capture data variability. This metric is
vital for researchers in developing efficient recommendation systems as it
helps assess the accuracy and reliability of predictions.

In Table 3, we present a comprehensive guide outlining specific scenarios for
the application of various Predictive Metrics. This table serves as an
essential reference for determining the appropriate metric to employ in
distinct contexts, thereby optimizing the effectiveness of predictive
analysis.

Table 3: Summary of Predictive Metrics Metric |  Use Cases
---|---
Root Mean Squared Error (RMSE) |  Use Root Mean Squared Error (RMSE) to measure the magnitude of prediction errors, giving more weight to larger errors, which is ideal for scenarios where large deviations from actual values are particularly undesirable.
Mean Absolute Error (MAE)) |  Use Mean Absolute Error (MAE) to measure the average magnitude of errors in predictions, prioritizing equal weighting to all deviations regardless of direction, useful for scenarios where all errors are equally important.
Mean Squared Error (MSE) |  Use Mean Squared Error (MSE) when evaluating a recommendation system’s predictive accuracy for continuous output, as it penalizes larger errors more severely, ensuring the model’s robustness by emphasizing the minimization of large prediction errors.
Mean Absolute Percentage Error (MAPE) |  Use Mean Absolute Percentage Error (MAPE) to measure forecast accuracy as a percentage, which is useful when you need to understand error magnitude relative to actual values, particularly in inventory or capacity planning where proportional errors are more meaningful.
$R^{2}$ |  Use $R^{2}$ (coefficient of determination) in evaluating recommendation systems when assessing the proportion of variance in the observed outcomes (e.g., ratings) that can be predicted from the input variables, providing a measure of how well unseen samples are likely to be predicted.
Explained Variance |  Use Explained Variance in evaluating recommendation systems when you aim to measure the proportion of user preference or rating variance that the model successfully captures, indicating the system’s overall effectiveness in predicting user behavior accurately.

###  2.4 Ranking Based Metrics

Ranking-based metrics are specifically designed to evaluate the quality of the
item ranking produced by a recommendation system. These metrics focus on how
well the system orders items to maximize user satisfaction.

  1. 1.

Mean Reciprocal Rank (MRR)

  2. 2.

ARHR@k

  3. 3.

Normalized Discounted Cumulative Gain (nDCG@K)

  4. 4.

Precision@k

  5. 5.

Recall@k

  6. 6.

F1@K

  7. 7.

Average Recall@k

  8. 8.

Average Precision@k

  9. 9.

MAP

####  2.4.1 Mean Reciprocal Rank (MRR)

Mean Reciprocal Rank (MRR)[43] is a statistical measure used to evaluate the
performance of recommendation systems or information retrieval systems,
focusing on the rank of the first correct answer. For a set of queries, MRR is
the average of the reciprocal ranks of results for the queries. The reciprocal
rank of a query response is the inverse of the rank of the first correct item.
If the correct item is at rank $k$, the reciprocal rank is $\frac{1}{k}$. For
a set of $Q$ queries, MRR is calculated as:

$MRR=\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{1}{\text{rank}_{i}}$

where $|Q|$ is the number of queries and $\text{rank}_{i}$ is the position of
the first relevant item for the $i$-th query.

####  2.4.2 Average Reciprocal Hit-Rank at K (ARHR@k)

The Average Reciprocal Hit-Rank at K (ARHR@k)[9] is a crucial metric in
evaluating recommendation systems, particularly emphasizing the ranking
efficiency of the first relevant recommendation within the top-K items. It’s
defined by the formula:

$\text{ARHR@k}=\frac{1}{|U|}\sum_{u\in U}\sum_{i=1}^{k}\frac{\delta(i,u)}{i}$

Here, $|U|$ represents the total number of users, $k$ is the cut-off rank for
top recommendations, and $\delta(i,u)$ is an indicator function that equals 1
if the item at rank $i$ is relevant to user $u$, and 0 otherwise. This metric
calculates the average of the reciprocal ranks of the first ’hit’ or relevant
item for each user but only if it appears within the top-K suggestions. A
higher ARHR@k score indicates that the system not only accurately identifies
relevant items but also ranks them highly, enhancing user experience and
satisfaction. This metric is particularly valuable in scenarios where the
prominence of recommendations significantly impacts user engagement.

####  2.4.3 Normalized Discounted Cumulative Gain (nDCG)

Normalized Discounted Cumulative Gain (nDCG)[4] is a measure of ranking
quality that captures the performance of ranking algorithms in recommendation
systems. It evaluates how well the predicted ranking of items corresponds to
the ideal ranking, considering the relevance of each item. The DCG is computed
as:

$\text{DCG}_{p}=\sum_{i=1}^{p}\frac{2^{rel_{i}}-1}{\log_{2}(i+1)}$

where $rel_{i}$ is the relevance score of the item at position $i$ and $p$ is
the number of ranked items. nDCG is obtained by normalizing DCG with the ideal
DCG (iDCG), which is the DCG score obtained by the perfect ranking:

$\text{nDCG}_{p}=\frac{\text{DCG}_{p}}{\text{iDCG}_{p}}$

A perfect ranking would result in an nDCG of 1, while any deviation from the
ideal ranking results in an nDCG less than 1.

####  2.4.4 Precision@k

Precision@k is a performance metric that evaluates the relevance of a list of
recommended items. It measures the proportion of recommended items in the
top-k set that are relevant to the user. The formula for Precision@k is given
by:

$\text{Precision@k}=\frac{\text{Number of relevant items in top-k}}{k}$

Here, ”relevant items” are those that are deemed to be of interest to the user
based on some ground truth, such as past user behavior or explicit ratings.
The metric provides a straightforward indication of recommendation quality at
a fixed list size $k$.

####  2.4.5 Recall@k

Recall@k is a metric used to evaluate recommendation systems based on how many
relevant items are selected out of all possible relevant items. Specifically,
for a set of queries, it measures the proportion of relevant items found in
the top-k recommendations. The formula for Recall@k is given by:

$\text{Recall@k}=\frac{|\\{\text{Relevant items}\\}\cap\\{\text{Top-k
recommended items}\\}|}{|\\{\text{Relevant items}\\}|}$

Here, $|\\{\text{Relevant items}\\}|$ is the number of relevant items, and
$|\\{\text{Relevant items}\\}\cap\\{\text{Top-k recommended items}\\}|$ is the
number of relevant items that are in the top-k recommendations.

####  2.4.6 F1@K (F1 score at K)

The F1 score is the harmonic mean of precision and recall. At a specific cut-
off point K, it is calculated as:

$F1@K=2\times\frac{\text{Precision@K}\times\text{Recall@K}}{\text{Precision@K}+\text{Recall@K}}$

where Precision@K and Recall@K are the precision and recall calculated at the
cut-off K.

####  2.4.7 Average Recall@K

Recall@K for a single user is the proportion of relevant items that are in the
top K recommendations. The Average Recall@K across all users is:

$\text{Average Recall@K}=\frac{1}{U}\sum_{u=1}^{U}\frac{|\text{Relevant
Items}_{u}\cap\text{Recommended Items}_{u}@K|}{|\text{Relevant Items}_{u}|}$

where U is the total number of users, Relevant Items u is the set of relevant
items for user u, and Recommended Items u@K is the set of top K recommended
items for user u.

####  2.4.8 Average Precision@K

Precision@K for a single user is the proportion of recommended items in the
top K that are relevant. The Average Precision@K is:

$\text{Average Precision@K}=\frac{1}{U}\sum_{u=1}^{U}\frac{|\text{Relevant
Items}_{u}\cap\text{Recommended Items}_{u}@K|}{K}$

####  2.4.9 Mean Average Precision (MAP)

MAP considers the order of recommendations. It is the mean of the Average
Precision at each point a relevant item is retrieved, averaged over all users:

$\text{MAP}=\frac{1}{U}\sum_{u=1}^{U}\left(\frac{1}{|\text{Relevant
Items}_{u}|}\sum_{k=1}^{|\text{Recommended
Items}_{u}|}\text{Precision@k}\times\text{rel}_{u}(k)\right)$

where rel u(k) is an indicator function that is 1 if the item at rank k is
relevant to user u and 0 otherwise.

Table 4 enumerates the specific scenarios for applying Ranking Metrics.

Table 4: Summary of Ranking Metrics Metric |  Use Cases
---|---
Mean Reciprocal Rank (MRR) |  Use Mean Reciprocal Rank (MRR) when evaluating the effectiveness of a recommendation system at returning the first relevant item in a ranked list of results, emphasizing the importance of the top-most recommendation.
ARHR@k (Average Reciprocal Hit Rank at k) |  Use ARHR@k (Average Reciprocal Hit Rank at k) to measure the average quality of recommendations by considering the reciprocal of the rank at which relevant items are found in the top k positions. It’s useful for scenarios where ranking accuracy and the order of recommendations are crucial factors.
Normalized Discounted Cumulative Gain (nDCG) |  Use nDCG when ranking relevance matters, and you need to evaluate the quality of the recommendations in a list, considering the position of highly relevant items.
Mean Reciprocal Rank (MRR) |  Use Mean Reciprocal Rank (MRR) when evaluating the effectiveness of a recommendation system at returning the first relevant item in a ranked list of results, emphasizing the importance of the top-most recommendation.
Precision@k |  Use Precision@k in recommendation systems when evaluating the proportion of relevant items among the top-k recommendations, particularly when the cost of false positives is high and the focus is on the quality of the top few recommendations.
Recall@k |  Use Recall@k in recommendation systems when prioritizing the model’s ability to capture all relevant items in the top-k suggestions is more important than avoiding irrelevant ones, especially when the cost of missing a relevant recommendation is high.
F1@K |  Use F1@K (F1 score at K) for scenarios where the focus is on the precision of the top K recommendations, such as content curation or ranking tasks, to ensure that the most relevant items are presented to users within a limited set.
Average Recall@K |  Use Average Recall@K when you need to assess the proportion of relevant items that a recommendation system successfully retrieves within the top K recommendations, prioritizing comprehensive coverage of relevant content over precision.
Average Precision@K |  Use Average Precision@k when evaluating the effectiveness of a recommendation system for scenarios where the order of recommended items matters, such as search engine result rankings or content playlists, as it quantifies the precision of the top k recommendations.
Mean Average Precision (MAP) |  Use Mean Average Precision (MAP) when evaluating the quality of a ranked list of items in recommendation systems particularly when relevance varies across items, as it provides a comprehensive measure of precision at different points in the ranking, emphasizing the importance of higher-ranked items.

###  2.5 Business Metrics

Business metrics play a crucial role in assessing the performance and impact
of recommendation systems (RS) in various domains[8]. One essential metric is
Click-through Rate (CTR), which measures the number of clicks generated by
recommendations. Higher CTR indicates that recommendations are more relevant,
making it a popular metric in the news recommendation domain. Platforms like
Google News and Forbes utilize CTR to gauge the effectiveness of their
recommendations. Personalized suggestions based on CTR have been shown to
increase clicks by up to 38% compared to popularity-based systems,
underscoring its importance.

Adoption and conversion metrics provide deeper insights into user behavior.
While CTR indicates clicks, it doesn’t determine if those clicks led to
conversions or purchases. Platforms like YouTube and Netflix use alternative
adoption measures such as ”Long CTR” (counting clicks when users watch a
specific percentage of a video) and ”Take rate” (counting views after
recommendations) to assess user engagement and conversion. In cases where an
item cannot be viewed directly, domain-specific measures, such as the number
of contacts made with an employer after a job offer recommendation on
LinkedIn, become crucial.

Sales and revenue metrics are ultimately what matter for businesses. Although
CTR and adoption metrics are informative, changes in sales and revenue reflect
the actual impact on the bottom line. However, attributing improvements solely
to RS can be challenging, as users may have made purchases anyway, making it
necessary to consider the broader business context.

Measuring the effects on sales distribution is another critical aspect. This
metric directly compares sales before and after the introduction of RS.
However, it requires an understanding of how the shifts in sales distribution
impact diversity at the individual level. Efforts to maintain diversity may be
needed to prevent unintended consequences.

User behavior and engagement metrics highlight the impact of RS on user
activity and retention. Recommendations often increase user engagement, and a
positive correlation between customer engagement and retention is observed in
various domains, such as Spotify. However, measuring this can be challenging,
especially when churn rates are low. These metrics collectively provide a
comprehensive view of RS performance, helping businesses make data-driven
decisions to enhance recommendation systems.

##  3 Experiments and Results

In this study, we conducted experiments on three different MovieLens datasets,
namely MovieLens 100k, MovieLens 1m, and MovieLens 10m, to evaluate the
performance of our recommendation system. We aimed to assess various metrics
to gain insights into the quality and effectiveness of our recommendation
algorithms. The results of these experiments are summarized in Tables 5 and 6
below.

We present the experimental results obtained from the analysis of two
datasets: Amazon Electronics Dataset and Amazon Movies and TV Dataset. The
experiments were conducted to evaluate the performance of various similarity
metrics in the context of recommendation systems. The results are summarized
in the following table:

Table 5: Similarity Metrics Experiments on Amazon Electronics Dataset Similarity Measure | EPOC | Loss | AUC
---|---|---|---
Adjusted Cosine Similarity | 4 | 0.7289 | 0.9531
Chebyshev Distance Similarity | 4 | 0.7105 | 0.2423
Cosine Similarity | 4 | 0.7128 | 0.9531
Euclidean Distance Similarity | 4 | 0.7107 | 0.1898
Hamming Distance | 4 | 0.7054 | 0.2740
Jaccard Index Similarity | 4 | 0.7167 | 0.9060
Manhattan Distance Similarity | 4 | 0.7098 | 0.1849
Pearson Correlation Coefficient Similarity | 4 | 0.7130 | 0.9158
Spearman Rank Order Correlation Coefficient Similarity | 4 | 0.7111 | -0.5373

These results provide valuable insights into the performance of different
similarity measures when applied to recommendation systems, with AUC serving
as a key metric for evaluating the quality of recommendations. The experiments
were conducted with a consistent number of epochs and batch size (2800) across
all similarity measures for a fair comparison.

Table 6: Similarity Metrics Experiments on Amazon Movies and TV Dataset Similarity Measure | EPOC | Loss | AUC
---|---|---|---
Adjusted Cosine Similarity | 4 | 0.6177 | 0.9444
Chebyshev Distance Similarity | 4 | 0.6051 | 0.2364
Cosine Similarity | 4 | 0.6123 | 0.9449
Euclidean Distance Similarity | 4 | 0.6044 | 0.1738
Hamming Distance | 4 | 0.6047 | 0.2507
Jaccard Index Similarity | 4 | 0.6040 | 0.8765
Manhattan Distance Similarity | 4 | 0.6046 | 0.1690
Pearson Correlation Coefficient Similarity | 4 | 0.6088 | 0.9181
Spearman Rank Order Correlation Coefficient Similarity | 4 | 0.6049 | -0.5654

Table 7 presents the catalog coverage, distributional coverage, novelty,
diversity, and serendipity metrics for each dataset. These metrics provide
valuable insights into the recommendation system’s ability to cover a wide
range of items, recommend items not previously seen by users, introduce novel
items, maintain diversity, and offer serendipitous recommendations. The
results indicate that as the dataset size increases, the recommendation
system’s performance in terms of these metrics generally improves.

Table 7: Candidate Generation Metrics on MovieLens Datasets Dataset | Catalog Coverage | Distributional Coverage | Novelty | Diversity | Serendipity
---|---|---|---|---|---
100k | 0.3951 | 8.2069 | 11.5816 | 0.8801 | 0.8750
1m | 0.2438 | 7.6453 | 15.2664 | 0.9594 | 0.9365
10m | 0.1570 | 7.4687 | 20.0912 | 0.9708 | 0.9809

We also evaluated two collaborative filtering algorithms, ALS (Alternating
Least Squares) and SVD (Singular Value Decomposition), on each dataset using a
fixed value of K (number of recommendations). The results of these experiments
are summarized in the following table:

Table 8 provides an overview of the performance metrics for each combination
of dataset and algorithm, including training time, predicting time, Root Mean
Square Error (RMSE), Mean Absolute Error (MAE), Mean Squared Error (MSE), Mean
Absolute Percentage Error (MAPE), R-squared (R2), and Explained Variance.
These metrics allow us to assess the effectiveness of ALS and SVD in providing
recommendations on different scales of MovieLens datasets.

Table 8: Predictive Metrics on MovieLens Datasets Dataset | 100k | 100k | 1m | 1m | 10m | 10m
---|---|---|---|---|---|---
Algorithm | ALS | SVD | ALS | SVD | ALS | SVD
K | 10 | 10 | 10 | 10 | 10 | 10
Train Time (s) | 5.1556 | 0.5020 | 3.7938 | 5.7888 | 19.0998 | 56.7193
Predicting Time (s) | 0.0442 | 0.1992 | 0.0165 | 0.7162 | 0.0168 | 8.9013
RMSE | 0.9659 | 0.9425 | 0.8583 | 0.8841 | 0.7984 | 0.8077
MAE | 0.7521 | 0.7448 | 0.6761 | 0.6961 | 0.6184 | 0.6211
MSE | 0.9329 | 0.8883 | 0.7367 | 0.7817 | 0.6375 | 0.6524
MAPE | 28.8383 | 29.9627 | 25.2899 | 27.0609 | 24.8207 | 25.8009
R2 | 0.2685 | 0.2999 | 0.4159 | 0.3761 | 0.4393 | 0.4201
Explained Variance | 0.2648 | 0.2999 | 0.4120 | 0.3762 | 0.4333 | 0.4201

We evaluated the performance of seven different recommendation algorithms,
namely ALS[36], SAR [12], SVD [13], NCF[40], BPR [39], BiVAE [47], and
LightGCN[18], across both datasets. The following table summarizes the key
metrics for each algorithm on the MovieLens 100k dataset.

Table 9 provides a comprehensive overview of the algorithm performance in
terms of various evaluation metrics, including Mean Average Precision (MAP),
normalized Discounted Cumulative Gain (nDCG@k), Precision@k, Recall@k, F1@k,
Mean Reciprocal Rank (MRR), Average Rank-based Half-life Reciprocal (ARHR@k),
Average Recall@k, and Average Precision@k. These metrics offer valuable
insights into the recommendation quality and efficiency of each algorithm on
the MovieLens 100k dataset.

Table 9: Ranking Metrics on MovieLens 100k Dataset Dataset | 100k | 100k | 100k | 100k | 100k | 100k | 100k
---|---|---|---|---|---|---|---
Algorithm | ALS | SAR | SVD | NCF | BPR | BiVAE | LightGCN
K | 10 | 10 | 10 | 10 | 10 | 10 | 10
Train time (s) | 4.7152 | 0.2113 | 0.3912 | 15.9303 | 3.5350 | 35.8652 | 20.4386
Recommending time (s) | 0.0744 | 0.1555 | 6.2471 | 3.4046 | 0.6358 | 0.6011 | 0.0313
MAP | 0.0035 | 0.1140 | 0.0121 | 0.0945 | 0.1340 | 0.1467 | 0.1227
nDCG@k | 0.0342 | 0.3938 | 0.0944 | 0.3687 | 0.4450 | 0.4753 | 0.4216
Precision@k | 0.0398 | 0.3406 | 0.0891 | 0.3288 | 0.3887 | 0.4152 | 0.3644
Recall@k | 0.0144 | 0.1854 | 0.0301 | 0.1640 | 0.2166 | 0.2254 | 0.1980
F1@k | 0.0212 | 0.2401 | 0.0450 | 0.2189 | 0.2781 | 0.2921 | 0.2565
MRR | - | 0.7180 | 0.4487 | 0.6649 | 0.7273 | 0.7644 | 0.7208
ARHR@k | - | 0.3466 | 0.3183 | 0.3355 | 0.3401 | 0.3412 | 0.3474
Average Recall@k | - | 0.2958 | 0.0936 | 0.3082 | 0.3144 | 0.3505 | 0.2903
Average Precision@k | - | 0.4144 | 0.1510 | 0.3868 | 0.4428 | 0.4854 | 0.4343

Table 10 provides a comprehensive overview of the algorithm performance on the
MovieLens 1M dataset.

Table 10: Ranking Metrics on MovieLens 1M Dataset Dataset | 1M | 1M | 1M | 1M | 1M | 1M | 1M
---|---|---|---|---|---|---|---
Algorithm | ALS | SAR | SVD | BPR | BiVAE | LightGCN | NCF
K | 10 | 10 | 10 | 10 | 10 | 10 | 10
Train time (s) | 11.3532 | 1.7349 | 4.7138 | 50.0558 | 569.2301 | 1958.8973 | 331.4241
Recommending time (s) | 0.1666 | 3.2854 | 90.2653 | 11.9814 | 28.6078 | 0.6001 | 54.6452
MAP | 0.0022 | 0.0650 | 0.0088 | 0.0838 | 0.0898 | 0.0889 | 0.0667
nDCG@k | 0.0283 | 0.3121 | 0.0902 | 0.3931 | 0.4247 | 0.4241 | 0.3630
Precision@k | 0.0358 | 0.2803 | 0.0841 | 0.3603 | 0.3890 | 0.3864 | 0.3337
Recall@k | 0.0105 | 0.1106 | 0.0224 | 0.1415 | 0.1465 | 0.1468 | 0.1144
F1@k | 0.0163 | 0.1587 | 0.0354 | 0.2032 | 0.2129 | 0.2128 | 0.1704
MRR | - | 0.6407 | 0.4512 | 0.6776 | 0.7074 | 0.7092 | 0.6729
ARHR@k | - | 0.3405 | 0.3276 | 0.3292 | 0.3305 | 0.3334 | 0.3320
Average Recall@k | - | 0.1842 | 0.0714 | 0.2223 | 0.2474 | 0.2432 | 0.2083
Average Precision@k | - | 0.3418 | 0.1455 | 0.4009 | 0.4434 | 0.4460 | 0.3983

##  4 Conclusion

Evaluating the efficacy of recommendation systems in business contexts is
multifaceted, requiring more than just accuracy-related metrics. Although
rank-aware accuracy metrics serve as a good starting point for generating
candidate items via machine learning techniques, they often overlook crucial
aspects like diversity and novelty, which are key to customer satisfaction.
The experiments conducted on various datasets underscore the limitations of
accuracy-focused assessments.

Importantly, real-world business value and user satisfaction emerge not solely
from these ML metrics but significantly from user feedback. This is evident in
the enhancements observed in click-through rates, sales, revenue, and related
derivatives when A/B testing is incorporated. Consequently, A/B testing
emerges as an essential tool, bridging the gap between ML model predictions
and tangible business outcomes. By aligning business strategies with machine
learning models through comprehensive evaluation, including user feedback,
recommendation systems can be fine-tuned to achieve harmony between technical
performance and business success.

## References

  * [1] Agarwal, D.K., Chen, B.C.: Statistical methods for recommender systems. Cambridge University Press (2016)
  * [2] Airen, S., Agrawal, J.: Movie recommender system using k-nearest neighbors variants. National Academy Science Letters 45(1), 75–82 (2022)
  * [3] Beel, J., Langer, S.: A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In: Research and Advanced Technology for Digital Libraries: 19th International Conference on Theory and Practice of Digital Libraries, TPDL 2015, Poznań, Poland, September 14-18, 2015, Proceedings 19. pp. 153–168. Springer (2015)
  * [4] Busa-Fekete, R., Szarvas, G., Elteto, T., Kégl, B.: An apple-to-apple comparison of learning-to-rank algorithms in terms of normalized discounted cumulative gain. In: ECAI 2012-20th European Conference on Artificial Intelligence: Preference Learning: Problems and Applications in AI Workshop. vol. 242. Ios Press (2012)
  * [5] Chicco, D., Warrens, M.J., Jurman, G.: The coefficient of determination r-squared is more informative than smape, mae, mape, mse and rmse in regression analysis evaluation. PeerJ Computer Science 7, e623 (2021)
  * [6] Cleger-Tamayo, S., Fernández-Luna, J.M., Huete, J.F.: On the use of weighted mean absolute error in recommender systems. In: RUE@ RecSys. pp. 24–26 (2012)
  * [7] Codina, V., Ricci, F., Ceccaroni, L.: Distributional semantic pre-filtering in context-aware recommender systems. User Modeling and User-Adapted Interaction 26, 1–32 (2016)
  * [8] Dijkman, R., Dumas, M., Van Dongen, B., Käärik, R., Mendling, J.: Similarity of business process models: Metrics and evaluation. Information Systems 36(2), 498–516 (2011)
  * [9] ElKorany, A.: Semantic link-based model for user recommendation in online community. International Journal of Computers and Technology 11(8), 2928–2938 (2013)
  * [10] Fkih, F.: Similarity measures for collaborative filtering-based recommender systems: Review and experimental comparison. Journal of King Saud University - Computer and Information Sciences 34(9), 7645–7669 (2022). https://doi.org/https://doi.org/10.1016/j.jksuci.2021.09.014, https://www.sciencedirect.com/science/article/pii/S1319157821002652
  * [11] Ge, M., Delgado-Battenfeld, C., Jannach, D.: Beyond accuracy: evaluating recommender systems by coverage and serendipity. In: Proceedings of the fourth ACM conference on Recommender systems. pp. 257–260 (2010)
  * [12] Gong, S.: A collaborative filtering recommendation algorithm based on user clustering and item clustering. J. Softw. 5(7), 745–752 (2010)
  * [13] Graham, S., Min, J.K., Wu, T.: Microsoft recommenders: tools to accelerate developing recommender systems. In: Proceedings of the 13th ACM Conference on Recommender Systems. pp. 542–543 (2019)
  * [14] Gultom, S., Sriadhi, S., Martiano, M., Simarmata, J.: Comparison analysis of k-means and k-medoid with ecluidience distance algorithm, chanberra distance, and chebyshev distance for big data clustering. In: IOP Conference Series: Materials Science and Engineering. vol. 420, p. 012092. IOP Publishing (2018)
  * [15] Gunawardana, A., Shani, G.: A survey of accuracy evaluation metrics of recommendation tasks. Journal of Machine Learning Research 10(12) (2009)
  * [16] Hayes, J.H., Dekhtyar, A., Sundaram, S.K.: Advancing candidate link generation for requirements tracing: The study of methods. IEEE Transactions on Software Engineering 32(1), 4–19 (2006)
  * [17] He, R., McAuley, J.: Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In: Proceedings of the 25th International Conference on World Wide Web. WWW ’16, International World Wide Web Conferences Steering Committee (Apr 2016). https://doi.org/10.1145/2872427.2883037, http://dx.doi.org/10.1145/2872427.2883037
  * [18] He, X., Deng, K., Wang, X., Li, Y., Zhang, Y., Wang, M.: Lightgcn: Simplifying and powering graph convolution network for recommendation. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pp. 639–648 (2020)
  * [19] Jadon, A., Kumar, S.: Leveraging generative ai models for synthetic data generation in healthcare: Balancing research and privacy. In: 2023 International Conference on Smart Applications, Communications and Networking (SmartNets). pp. 1–4 (2023). https://doi.org/10.1109/SmartNets58706.2023.10215825
  * [20] Jadon, A., Patil, A., Jadon, S.: A comprehensive survey of regression based loss functions for time series forecasting (2022)
  * [21] Jadon, S., Jadon, A.: Metaforecast: Harnessing model-agnostic meta-learning approach to predict key metrics of interconnected network topologies. In: 2023 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT). pp. 239–243 (2023). https://doi.org/10.1109/IAICT59002.2023.10205730
  * [22] Jadon, S., Jadon, A.: An overview of deep learning architectures in few-shot learning domain (2023)
  * [23] Jahrer, M., Töscher, A., Legenstein, R.: Combining predictions for accurate recommender systems. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 693–702 (2010)
  * [24] Karatzoglou, A., Baltrunas, L., Shi, Y.: Learning to rank for recommender systems. In: Proceedings of the 7th ACM Conference on Recommender Systems. pp. 493–494 (2013)
  * [25] Kotkov, D., Wang, S., Veijalainen, J.: A survey of serendipity in recommender systems. Knowledge-Based Systems 111, 180–192 (2016)
  * [26] Kumar, S., Jadon, A., Sharma, S.: Global message ordering using distributed kafka clusters (2023)
  * [27] Kunaver, M., Požrl, T.: Diversity in recommender systems–a survey. Knowledge-based systems 123, 154–162 (2017)
  * [28] Lahitani, A.R., Permanasari, A.E., Setiawan, N.A.: Cosine similarity to determine similarity measure: Study case in online essay assessment. In: 2016 4th International Conference on Cyber and IT Service Management. pp. 1–6 (2016). https://doi.org/10.1109/CITSM.2016.7577578
  * [29] Li, M., Chen, X., Li, X., Ma, B., Vitányi, P.M.: The similarity metric. IEEE transactions on Information Theory 50(12), 3250–3264 (2004)
  * [30] McAuley, J., Targett, C., Shi, Q., van den Hengel, A.: Image-based recommendations on styles and substitutes (2015)
  * [31] Melović, B., Dabić, M., Vukčević, M., Ćirović, D., Backović, T.: Strategic business decision making: the use and relevance of marketing metrics and knowledge management. Journal of Knowledge Management 25(11), 175–202 (2021)
  * [32] Norouzi, M., Fleet, D.J., Salakhutdinov, R.R.: Hamming distance metric learning. Advances in neural information processing systems 25 (2012)
  * [33] Paraschakis, D., Nilsson, B.J., Holländer, J.: Comparative evaluation of top-n recommenders in e-commerce: An industrial perspective. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). pp. 1024–1031. IEEE (2015)
  * [34] Patil, A., Han, K., Jadon, A.: A comparative study of text embedding models for semantic text similarity in bug reports (2023)
  * [35] Patil, A., Jadon, A.: Auto-labelling of bug report using natural language processing. In: 2023 IEEE 8th International Conference for Convergence in Technology (I2CT). pp. 1–7 (2023). https://doi.org/10.1109/I2CT57861.2023.10126470
  * [36] Pilászy, I., Zibriczky, D., Tikk, D.: Fast als-based matrix factorization for explicit and implicit feedback datasets. In: Proceedings of the fourth ACM conference on Recommender systems. pp. 71–78 (2010)
  * [37] Qian, G., Sural, S., Gu, Y., Pramanik, S.: Similarity between euclidean and cosine angle distance for nearest neighbor queries. In: Proceedings of the 2004 ACM symposium on Applied computing. pp. 1232–1237 (2004)
  * [38] Real, R., Vargas, J.M.: The probabilistic basis of jaccard’s index of similarity. Systematic biology 45(3), 380–385 (1996)
  * [39] Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 (2012)
  * [40] Sang, L., Xu, M., Qian, S., Wu, X.: Knowledge graph enhanced neural collaborative recommendation. Expert systems with applications 164, 113992 (2021)
  * [41] Schein, A.I., Popescul, A., Ungar, L.H., Pennock, D.M.: Methods and metrics for cold-start recommendations. In: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval. pp. 253–260 (2002)
  * [42] Sheugh, L., Alizadeh, S.H.: A note on pearson correlation coefficient as a metric of similarity in recommender system. In: 2015 AI & Robotics (IRANOPEN). pp. 1–6. IEEE (2015)
  * [43] Shi, Y., Karatzoglou, A., Baltrunas, L., Larson, M., Oliver, N., Hanjalic, A.: Climf: learning to maximize reciprocal rank with collaborative less-is-more filtering. In: Proceedings of the sixth ACM conference on Recommender systems. pp. 139–146 (2012)
  * [44] Strauss, T., von Maltitz, M.J.: Generalising ward’s method for use with manhattan distances. PloS one 12(1), e0168288 (2017)
  * [45] Su, X., Khoshgoftaar, T.M.: A survey of collaborative filtering techniques. Advances in artificial intelligence 2009 (2009)
  * [46] Takács, G., Pilászy, I., Németh, B., Tikk, D.: Investigation of various matrix factorization methods for large recommender systems. In: Proceedings of the 2nd KDD Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition. pp. 1–8 (2008)
  * [47] Truong, Q.T., Salah, A., Lauw, H.W.: Bilateral variational autoencoder for collaborative filtering. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. pp. 292–300 (2021)
  * [48] Xia, P., Zhang, L., Li, F.: Learning similarity with cosine similarity ensemble. Information sciences 307, 39–52 (2015)
  * [49] Zhang, L.: The definition of novelty in recommendation system. Journal of Engineering Science & Technology Review 6(3) (2013)