11institutetext:  Intelligent Systems Group, University of Siegen, Germany
11email: {lukas.wegmeth,tobias.vente}<EMAIL_ADDRESS>freiburg.de

# Revealing the Hidden Impact of Top-N Metrics on Optimization in Recommender
Systems

Lukas Wegmeth  0000-0001-8848-9434 Tobias Vente  0009-0003-8881-2379 Lennart
Purucker  0009-0001-1181-0549

###### Abstract

The hyperparameters of recommender systems for top-n predictions are typically
optimized to enhance the predictive performance of algorithms. Thereby, the
optimization algorithm, e.g., grid search or random search, searches for the
best hyperparameter configuration according to an optimization-target metric,
like _nDCG_ or _Precision_. In contrast, the optimized algorithm, e.g.,
_Alternating Least Squares Matrix Factorization_ or _Bayesian Personalized
Ranking_ , internally optimizes a different loss function during training,
like _squared error_ or _cross-entropy_. To tackle this discrepancy, recent
work focused on generating loss functions better suited for recommender
systems. Yet, when evaluating an algorithm using a top-n metric during
optimization, another discrepancy between the optimization-target metric and
the training loss has so far been ignored. During optimization, the top-n
items are selected for computing a top-n metric; ignoring that the top-n items
are selected from the recommendations of a model trained with an entirely
different loss function. Item recommendations suitable for optimization-target
metrics could be outside the top-n recommended items; hiddenly impacting the
optimization performance. Therefore, we were motivated to analyze whether the
top-n items are optimal for optimization-target top-n metrics. In pursuit of
an answer, we exhaustively evaluate the predictive performance of 250
_selection strategies_ besides selecting the top-n. We extensively evaluate
each _selection strategy_ over twelve implicit feedback and eight explicit
feedback data sets with eleven recommender systems algorithms. Our results
show that there exist _selection strategies_ other than top-n that increase
predictive performance for various algorithms and recommendation domains.
However, the performance of the top $\sim 43\%$ of _selection strategies_ is
not significantly different. We discuss the impact of our findings on
optimization and re-ranking in recommender systems and feasible solutions. The
implementation of our study is publicly available.

###### Keywords:

recommender systems re-ranking optimization autorecsys hyperparameter top-n
evaluation

##  1 Introduction

Top-n recommendations, i.e., recommending ranked item lists, are probably the
most common task for recommender systems nowadays. To tackle this task,
recommender systems developers often apply machine learning algorithms, e.g.,
nearest neighbor or matrix factorization approaches [37]. The models produced
by such algorithms are then used to predict personalized ranked lists. These
recommendations are then commonly evaluated in terms of predictive performance
with optimization-target metrics like the _nDCG_ or _Precision_ [53, 19, 54,
20]. This performance is influenced by the data set, the algorithm, and its
hyperparameters. Hyperparameter optimization techniques like grid search,
random search, or Bayesian optimization are commonly applied to improve
recommendation performance by determining the best hyperparameter values for
an algorithm [12, 56, 55, 49, 30, 3, 42, 39, 21].

No matter which optimization technique is applied, it is vital to correctly
approximate the predictive accuracy of a set of hyperparameters, as this
influences future decisions during the optimization or when deploying the
model in production. However, the predictive performance according to
optimization-target metrics is not necessarily optimized during the training
of recommender systems. The research community knows this discrepancy between
training loss metrics, e.g., _squared error_ , and optimization-target
metrics, like _nDCG_. Their problem is finding a training loss metric that
accurately represents the optimization-target metric such that the trained
weights are optimized correctly. To this end, recent work focused on
empirically proven loss functions or learning better loss functions [48, 38,
25].

In theory, the aforementioned discrepancy also exists in evaluation metrics,
e.g., _nDCG@10_. By definition, _nDCG@10_ is the result of calculating the
_nDCG_ of the top 10 predicted items, ignoring all items ranked below the top
10. Metrics like _nDCG@10_ strongly differ from training loss functions that
evaluate model performance without the concept of a ranking or relevance
threshold. Consequently, we hypothesize that, due to the aforementioned
discrepancy between the metrics, evaluating only the top-n selection is
insufficient to optimize for the highest predictive accuracy of the
recommender system. In other words, evaluating only the top-n predicted items
but ignoring all others might not always yield the highest possible predictive
accuracy of a trained model.

Therefore, we aim to determine whether there are cases where selecting items
other than the top-n results in higher predictive accuracy. If there were such
cases, it would imply that there is a hidden impact of top-n metrics on
optimization performance due to the assumption that evaluating the top-n
recommended items results in the highest predictive accuracy w.r.t. the
evaluated hyperparameters. The aforementioned implication motivated us to
conduct the exploratory study that we present in this paper. In detail, our
study aims to answer the following questions:

  1. RQ1

Does the selection of items other than the top-n during the evaluation of
recommender systems yield improved predictive accuracy for specific
algorithms, domains, or data sets?

  2. RQ2

If there are cases where selecting items other than the top-n improves
predictive accuracy, is there a _significant_ impact of top-n metrics on
optimization?

To answer the research questions, we conduct a study of the performance of
nine recommendation algorithms on twelve implicit feedback and eight explicit
feedback data sets. Our contribution is the first large-scale study and
discussion of the, so far, hidden impact of top-n metrics on optimization in
recommender systems. Our results prove that there are cases where selecting
items other than the top-n yields increased performance, answering RQ1.
However, we indicate that the impact is likely insignificant, answering RQ2.
Therefore, we reveal the hidden impact of top-n metrics on optimization in
recommender systems and provide evidence that researchers do not have to worry
about it being a confounding factor in the evaluation of recommender systems
using traditional collaborative filtering algorithms. Our long-term goal is to
clear doubts about hidden problems in evaluating recommender systems and to
raise awareness in the community.

Our implementation is publicly available on our GitHub
repository111https://code.isg.beel.org/scoring-optimizer and contains
documentation for the reproducibility of our experiments.

##  2 Related Work

To the best of our knowledge, there exists no analysis of the quality of the
top-n selection of ranked lists. However, our work is related to and motivated
by recent work on hyperparameter optimization, training loss metrics, and re-
ranking.

The optimization of hyperparameters requires some score that approximates the
highest predictive accuracy of a model. If the score fails to do that, the
optimization strategy runs into the risk of either optimizing for undesired
criteria, e.g. in Bayesian optimization, or the results can not be interpreted
correctly, e.g. in grid search and random search. Recommender systems research
that reports evaluation scores usually obtains these scores on optimized
hyperparameters [50, 9, 44]. Additionally, there are efforts to transfer
automated machine learning techniques to recommender systems [45, 2, 57, 27,
23, 8], and one of the core problems of automated machine learning is
automated hyperparameter optimization [13].

Related to optimization, the discrepancy between training loss and
optimization-target metrics has been explored by the recommender systems
community [47, 6]. The problem can be tackled either by engineering loss
metrics that fit well to the desired accuracy metric [48, 38] or by learning
the loss metric automatically [25]. In contrast, we thoroughly analyze the
validity of selecting only the top-n elements for evaluation.

To achieve secondary optimization goals that are different from increasing
predictive accuracy, e.g., removing popularity bias, re-ranking techniques are
commonly applied [15, 1, 28, 34, 22]. Re-ranking techniques assume that the
underlying ranked list is predicted by an implicit feedback recommender system
optimized for predictive accuracy. However, our analysis focuses on re-ranking
predicted items during the optimization of a model to better approximate the
predictive accuracy of a set of hyperparameters. Our approach is more similar
to works that analyze strategies that randomly sample recommendations from the
top predicted items [24], or define a relevance cutoff for recommended items
[5].

The Probability Ranking Principle (PRP) [40] assumes that an optimal
recommender algorithm ranks items in order of probability of relevance to the
user [33]. In contrast, our work investigates whether a global ranking order
of items is optimal on average for all users. Hence, if the best possible
global ranking order of an algorithm’s predictions contains exactly the top-n
predicted items, then the PRP holds on average for all users. However, our
work aims to research the impact of top-n metrics on optimization.

##  3 Method

At the core of this paper is the evaluation of different _selection
strategies_. A recommender system predicts a ranked list of items $P$ for each
user. We define a _selection strategy_ as an approach to select any $n$ items
from $P$, with $n\leq|P|$, for the evaluation of the recommender system’s
accuracy with a threshold-based metric like _nDCG@n_. Hence, the number of
possible _selection strategies_ is $\binom{|P|}{n}$. In the traditional
evaluation of recommender systems, e.g. with _nDCG@10_ (i.e., $n=10$), the
commonly used _selection strategy_ is choosing the top 10 items from $P$. We
call this _selection strategy_ the _top-n selection strategy_. To illustrate
an alternative to the _top-n selection strategy_ , assume we randomly select
10 items from $P$ and use this _non-top-n selection strategy_ to compute the
_nDCG@10_. Such an approach was, for example, proposed in research on the
effectiveness of randomly sampling ranked items [24].

Our work focuses on finding a _non-top-n selection strategy_ with a higher
_nDCG@n_ than the _top-n selection strategy_. Additionally, we define the
subset $K$, with $K\subseteq P$, as the subset that contains the top-$|K|$
elements of $P$, which we need later.

The experiments presented in this paper were executed in a highly parallel
manner on a cluster where each node has 256 GB RAM and a total of 64 cores
from two AMD EPYC 7452 CPUs.

###  3.1 Selection Strategy Design Decisions

For our exploratory study, we set $|K|=10$ and $n=5$ to investigate the
resulting 252 distinct _selection strategies_. In this paragraph, we provide
our motivation for this choice. For context, the _top-n selection strategy_ in
this setting is choosing the top 5 predicted items from $K$. To answer RQ1,
technically, we only need to show that there exists at least one _non-top-n
selection strategy_ with higher performance than the _top-n selection
strategy_ for any combination of $|K|$ and $n$. However, to sufficiently
answer RQ2, we want to acquire more exhaustive results than for, e.g. $|K|=2$
and $n=1$, or $|K|=11$ and $n=10$. Further, evaluating the lowest-ranked items
does not provide additional information, e.g., when $|K|$ is close to or equal
to $|P|$. Moreover, if _selection strategies_ that choose the lowest-ranked
items perform well, the evaluated algorithm likely fails to learn correctly
from the data. Additionally, the number of possible _selection strategies_
increases exponentially with $|K|$ if it significantly differs to $n$. In the
complete study, including data preprocessing, fitting, and predicting, the
parameters $|K|$ and $n$ have the most considerable influence on computational
resource requirements for our exploratory study. With $|K|=10$ and $n=5$,
analyzing one data set requires about one CPU year on average with our
evaluation setup. Furthermore, $n=5$ is commonly chosen for recommender
systems evaluations. Considering all the above points, we selected $|K|=10$
and $n=5$ for this study.

###  3.2 Data Sets and Algorithms

We analyzed twelve implicit and eight explicit feedback data sets from six
different domains with nine recommendation algorithms plus two baselines from
two recommender systems libraries. Four of the recommendation domains,
_shopping_ , _music_ , _movies_ , and _articles_ , are represented by three or
more data sets each. The remaining data sets are from the domains _social_ and
_locations_. The implicit feedback data sets are: _Adressa One Week_ [17],
_Citeulike-a_ [46], _Cosmetics-Shop_ 222https://rees46.com/, _Globo_ [41, 31],
_Gowalla_ [10], _Hetrec-Lastfm_ [7], _Nowplaying-rs_ [36], _Retailrocket_
333https://www.kaggle.com/datasets/retailrocket/ecommerce-dataset, _Sketchfab_
444https://github.com/EthanRosenthal/rec-a-sketch, _Spotify-Playlists_ [35],
_Yelp_ 555https://www.yelp.com/dataset, and _Yoochoose_
666https://www.kaggle.com/datasets/chadgostopp/recsys-challenge-2015. The
explicit feedback data sets are: _Amazon CDs &Vinyl_ [32], _Amazon Musical
Instruments_ [32], _Amazon Video Games_ [32], _CiaoDVD_
777https://guoguibing.github.io/librec/datasets.html, _Jester3_ [16],
_MovieLens-1M_ [18], _MovieLens-100k_ [18], and _MovieTweetings_ [11]. For
these explicit feedback data sets, we treat a rating that is $>60\%$ of the
maximum rating as an interaction according to standard practice [4, 29, 26].
Furthermore, we prune all data sets such that all users and items have at
least five interactions, commonly called five-core pruning [43, 51, 52]. We do
this to reduce the impact of cold start cases since the used algorithms can
not predict cold start scenarios. Table 1 contains the data set statistics for
the preprocessed data sets. We used the libraries _Implicit_ [14] and
_LensKit_ [12] for their implementation of the recommender algorithms. We used
all algorithms from these libraries that natively support implicit feedback
for a total of nine algorithms plus two baselines. The algorithms from
_Implicit_ are _Alternating Least Squares_ , _Logistic Matrix Factorization_ ,
_Bayesian Personalized Ranking_ , and _Item-Item Nearest Neighbors_ with
distance metrics _Cosine Similarity_ , _TF-IDF_ , and _BM25_. The algorithms
from _LensKit_ are _Implicit Matrix Factorization_ , _User-User Nearest
Neighbors_ , _Item-Item Nearest Neighbors_ , _Most Popular_ , and _Random_.

###  3.3 Experimental Pipeline

We perform five-fold cross-validation by randomly splitting the interactions
of each user into three separate sets with fixed sizes: training (60%),
validation (20%), and test (20%). All recommenders were optimized on the
validation set with random search for two hours. The range of hyperparameter
values in the configuration space is set to be around the default values given
by the library. Finally, we exhaustively evaluate all 252 _selection
strategies_ for both the validation and test data. We focus on and present the
results from evaluating _selection strategies_ on the test data. However, we
also obtain the results from evaluating _selection strategies_ on the
validation data to analyze the generalization capabilities of _selection
strategies_.

Table 1: Data set statistics after five-core pruning. Split between the
implicit (first part) and explicit (second part) feedback data sets.

Name | #Interactions | #Users | #Items | Avg.#Int./User | Avg.#Int./Item | Sparsity | Domain
---|---|---|---|---|---|---|---
Adressa One Week | 2,020,328 | 146,635 | 2,441 | 13.78 | 827.66 | 99.44% | Articles
Citeulike-a | 200,180 | 5,536 | 15,429 | 36.16 | 12.97 | 99.77% | Articles
Cosmetics-Shop | 4,949,482 | 230,248 | 44,009 | 21.5 | 112.47 | 99.95% | Shopping
Globo | 2,482,163 | 157,926 | 11,832 | 15.72 | 209.78 | 99.87% | Articles
Gowalla | 2,018,421 | 64,115 | 164,532 | 31.48 | 12.27 | 99.98% | Locations
Hetrec-Lastfm | 71,355 | 1,859 | 2,823 | 38.38 | 25.28 | 98.64% | Music
Nowplaying-rs | 2,447,318 | 64,392 | 95,277 | 38.01 | 25.69 | 99.96% | Music
Retailrocket | 240,938 | 22,178 | 17,803 | 10.86 | 13.53 | 99.94% | Shopping
Sketchfab | 547,477 | 25,655 | 15,274 | 21.34 | 35.84 | 99.86% | Social
Spotify-Playlists | 8,718,742 | 15,146 | 337,256 | 575.65 | 25.85 | 99.83% | Music
Yelp | 3,999,684 | 268,658 | 109,340 | 14.89 | 36.58 | 99.99% | Locations
Yoochoose | 10,195,058 | 1,283,296 | 27,995 | 7.94 | 364.17 | 99.97% | Shopping
Amazon CDs&Vinyl | 1,241,336 | 98,228 | 66,979 | 12.64 | 18.53 | 99.98% | Shopping
Amazon Musical Instruments | 176,631 | 21,420 | 8,642 | 8.25 | 20.44 | 99.9% | Shopping
Amazon Video Games | 362,996 | 40,793 | 14,497 | 8.9 | 25.04 | 99.94% | Shopping
CiaoDVD | 23,467 | 1,582 | 1,788 | 14.83 | 13.12 | 99.17% | Movies
Jester3 | 777,118 | 34,884 | 140 | 22.28 | 5,550.84 | 84.09% | Social
MovieLens-1M | 835,789 | 6,038 | 3,307 | 138.42 | 252.73 | 95.81% | Movies
MovieLens-100k | 81,697 | 943 | 1,203 | 86.64 | 67.91 | 92.8% | Movies
MovieTweetings | 563,309 | 20,643 | 8,810 | 27.29 | 63.94 | 99.69% | Movies

##  4 Results

Our results are presented in the following order. We first show aggregated
evaluation results of all twelve implicit and eight explicit data sets with
all eleven algorithms. Next, we present a closer look using a domain-specific
subset of the results. Zooming further into this subset, we provide a more
detailed, data set-specific, view of the behavior of different _selection
strategies_. Moreover, we analyze the generalization capabilities of
_selection strategies_. Finally, we perform statistical tests on the
significance of _selection strategies_ to understand the impact of potential
solutions. All results are split between implicit and explicit feedback data
sets to compare them.

Though we aggregated results as much as possible, we can only show exemplary
results for some parts of the analysis in this paper due to space constraints.
The results we can not show in this paper do not provide any additional
insight and do not contradict the presented results. For example, all data
presented in this paper feature results exclusively on the _nDCG_ metric.
However, we also performed the same evaluation procedure on the _Precision_
metric and found no difference compared to the results of the _nDGC_ metric.
This applies to algorithms, domains, and data sets as well. To confirm, the
interested reader may refer to the complete set of results of the whole
evaluation procedure, which are stored in our public repository. Any
additional material is therefore made available only for completeness and
reproducibility purposes.

#### Aggregated Results

Figure 1 provides an aggregated view of our results on all twelve implicit and
eight explicit data sets split between implicit and explicit feedback with all
eleven algorithms. We observe that there are some data sets and algorithms for
which the best _nDCG_ performance is not achieved with the _top-n selection
strategy_. However, the distribution of the data points also shows that the
_top-n selection strategy_ is _the best on average_. Furthermore, with the
baseline plot, we observe that the random and popularity baselines behave as
expected. When recommending random elements, there is a high probability that
the _top-n selection strategy_ is sub-optimal due to the amount of other
possible _selection strategies_. Recommending items based on popularity using
the _top-n selection strategy_ is also sub-optimal in most cases, simply
confirming that the most popular recommendations are not automatically the
best.

In all cases where a _non-top-n selection strategy_ performs better than the
_top-n selection strategy_ , we could search for the best _non-top-n selection
strategy_ to increase performance. The tested recommendation algorithms appear
to be stable in terms of relative performance since the difference in
performance in both directions is marginal at less than 0.4% for implicit
feedback and less than 1.5% for explicit feedback, indicating that finding the
best _selection strategy_ only has a marginal performance impact.

(a) Non-baseline algorithms for implicit feedback.

(b) Baseline algorithms for implicit feedback.

(c) Non-baseline algorithms for explicit feedback.

(d) Baseline algorithms for explicit feedback.

Figure 1: The relative _nDCG_ performance of the best _non-top-n selection
strategy_ versus the _top-n selection strategy_ evaluated on the test set. A
point to the right of the red line indicates that the best _non-top-n
selection strategy_ is better than the _top-n selection strategy_.

#### Domain-Specific Results

To further analyze the observations, we focus on the _articles_ and _movies_
domains, which contain only implicit and explicit data sets, respectively, in
Figure 2. Notably, there are cases with different data sets in these domains
where the best _non-top-n selection strategy_ is better than the _top-n
selection strategy_. Therefore, the aggregated and domain-specific plots
indicate that the relative performance of the best _selection strategies_ may
be specific to an algorithm or data set rather than a recommendation domain.
Additionally, the average difference in the relative performance of data sets
changes for different recommender system algorithms.

(a) _Articles_ recommendation domain (3 data sets, implicit feedback).

(b) _Movies_ recommendation domain (4 data sets, explicit feedback).

Figure 2: This Figure is a focused view on evaluating data sets that represent
the _articles_ (2(a)) and _movies_ (2(b)) recommendation domains. The shown
data points are a subset of Figures 1(a) and 1(c), respectively, but with a
focus on the data sets. The data sets are indicated with markers and colors to
distinguish them better.

#### Data Set-Specific Results

To better understand the scope of the previous observations, we have to
analyze not only the best _non-top-n selection strategy_ but all of them. We,
therefore, take an exemplary look at the exhaustive evaluation of _selection
strategies_ for the _Adressa One Week_ data set from the _articles_ domain on
the _Alternating Least Squares_ algorithm for implicit feedback. Furthermore,
we do the same for the _MovieLens-100k_ data set from the _movies_ domain on
the _Item-Item Nearest Neighbors_ algorithm for explicit feedback.

Figure 3 visualizes the results of the exhaustive evaluation through box
plots. If the algorithm ranks the items correctly, we expect a consistent
movement toward a lower median for lower-ranked elements, but there is no such
trend. In fact, the interquartile range shows that the model only appears to
be consistently performing well for the first predicted item. However, there
are items for which the plot shows increased average performance compared to a
higher-ranked element. Additionally, we already know from Figures 2(a) and
2(b), specifically for these examples, that there is at least one _non-top-n
selection strategy_ that improves over the _top-n selection strategy_. Figure
3 reveals the elements chosen in these strategies, indicated by the points to
the right of the red vertical line, e.g., the line indicating the performance
of the _top-n selection strategy_.

(a) _Adressa One Week_ data set.

(b) _MovieLens-100k_ data set.

Figure 3: The exhaustive evaluation of 252 _selection strategies_ with the
_nDCG_ metric on the data set _Adressa One Week_ with the _Logistic Matrix
Factorization_ algorithm (3(a)) and the data set _MovieLens-100k_ with the
_Alternating Least Squares_ algorithm (3(b)). Each _selection strategy_ picks
a different combination of 5 items from the top 10 predicted items. The
performance of a _selection strategy_ is averaged over all users in the test
data set. The y-axis refers to the index of the items in the predicted ranked
list, e.g., element 0 is the highest-ranked and the most relevant item. A
black dot refers to a _selection strategy_ that contains said item, e.g., all
_selection strategies_ represented by black dots in a row contain the item
according to the index stated on the y-axis.

#### Generalization Capabilities

Since we can show that there are _selection strategies_ that are better than
the _top-n selection strategy_ , we would need a way to find them reliably.
All of the results so far are evaluated on the test set, e.g., the data split
that we did not know during training. As a result, if we want to find the best
_selection strategy_ in a real-world scenario, we would need to search for the
best _selection strategy_ on the validation set and hope that it generalizes
to the test set. To that end, we performed the same exhaustive evaluation on
the validation set to observe the generalization capability of the _selection
strategies_. Continuing with the previously chosen example data set and
recommendation algorithm, we show the generalization capability in Figure 4.
With the generalization plot, we already see a trend regarding the
generalization capability of _selection strategies_. To measure the
generalization capability directly, we additionally calculated the Pearson
correlation coefficient of the performance of _selection strategies_ over the
validation and the test. A higher Pearson correlation coefficient indicates
higher generalization capability. The Pearson correlation coefficient averaged
over all data sets per algorithm is shown in Table 2.

(a) _Adressa One Week_

(b) _MovieLens-100k_

Figure 4: The relative performance of _selection strategies_ on the validation
and test set for the data sets _Adressa One Week_ (4(a)) and _MovieLens-100k_
(4(b)). If a point is on the identity line, the validation and test set have
equal relative performance and, therefore, generalize. Table 2: The Pearson
correlation coefficients of the performance of _selection strategies_ of the
validation set compared to the test set separated by implicit and explicit
feedback sets. A Pearson correlation coefficient of one would mean that
_selection strategies_ perfectly generalize from validation to test.

Algorithm Pearson Corr. Coeff. Implicit Pearson Corr. Coeff. Explicit Implicit
MF 0.998 0.988 User-based kNN 0.999 0.990 Item-based kNN 0.998 0.994
Alternating Least Squares 0.999 0.986 Bayesian Personalized Ranking 0.995
0.966 Logistic MF 0.995 0.983 Item-based kNN Cosine Sim. 0.999 0.990 Item-
based kNN TF-IDF Sim. 0.998 0.992 Item-based kNN BM25 Sim. 0.999 0.992 Random
0.004 -0.064 Popularity 0.994 0.995

#### Statistical Significance

Finally, we tested for the statistical difference in the performance of
_selection strategies_ over all data sets but per algorithm. We applied the
Friedman test to confirm that different _selection strategies_ result in
statistically significant performance differences ($p<0.05$). Then, we applied
the Nemenyi post hoc test to obtain the critical difference. The result shows
that for all non-baseline algorithms, the best $\sim 43\%$ _selection
strategies_ are not significantly different in performance. We note, however,
that the theory behind the Nemenyi test may not hold up due to the high number
of compared methods. Nevertheless, we believe it to be a worthwhile indicator
to answer our research questions and beyond.

##  5 Discussion

First, we consider our →research questions. To answer RQ1, we evaluated all
possible _selection strategies_ that choose 5 out of the top 10 predicted
items. We accept RQ1 since we found that $\sim 0.01\%$ of _non-top-n selection
strategies_ result in higher performance than the _top-n selection strategy_.
However, we can not identify specific criteria that lead to this effect, e.g.
it occurred in different types of algorithms and data sets from different
domains. To answer RQ2, we additionally tested the statistical significance of
_selection strategies_ based on their performance. The tests indicate that
most _selection strategies_ are not significantly different. Moreover, the
maximum performance gain is marginal at less than 0.4% for implicit feedback
data sets and less than 1.5% for explicit feedback data sets, as shown in the
aggregated results (Figure 1).

#### Searching for the best selection strategy

The average Pearson correlation coefficient of $>0.99$ for implicit feedback
data sets and $>0.96$ for explicit feedback sets for all non-baseline
algorithms shows that _selection strategies_ generalize their performance from
validation to test. Therefore, it is possible to reliably search and find the
best _selection strategy_ on the validation set. However, the search strategy
we used for the evaluation, e.g., exhaustively evaluating all possible
options, is not cost-efficient. We have also shown that the _top-n selection
strategy_ is _the best on average_. Therefore, searching for the best
_selection strategy_ in practice is only feasible with a highly efficient
search strategy or when the potential for a minor improvement outweighs the
cost of the search. In future work, finding the best _selection strategy_
could be accomplished with an efficient search algorithm, e.g., a greedy
search. Alternatively, our results indicate that we do not have to observe
items too far away from the top-n, e.g., we searched _selection strategies_
with 5 items out of a sample range of 10 items, but 8 items may have already
been enough to find the optimum. As a result, an exhaustive search on fewer
combinations may be feasible.

#### The impact on hyperparameter optimization

Hyperparameter optimization techniques rely on the developer to correctly
approximate the predictive accuracy of a model to find the best
hyperparameters. If the _top-n selection strategy_ does not achieve that, due
to being sub-optimal in terms of performance, we should avoid using it.
However, our statistical tests indicate that optimizing the _selection
strategy_ likely has no practical impact since the evaluated performance with
the majority of _selection strategies_ is not significantly different. In
turn, this means we may use most _selection strategies_ that choose 5 out of
10 items in practice.

#### The impact on re-ranking

Re-ranking algorithms change which items are recommended, e.g. they apply a
different _selection strategy_. Often, this means that predictive accuracy
performance, e.g. _nDCG_ performance, is sacrificed to gain increased
performance in secondary metrics, e.g. _diversity_ or _fairness_. Our results
show that, when the re-ranking algorithm is confined to elements close to the
top-n, the re-ranked predictions may actually not be significantly different
in terms of predictive accuracy.

#### Conclusion

We reveal the hidden impact of top-n metrics on optimization in recommender
systems and show that it is insignificant. As a result, there is no practical
benefit in optimizing _selection strategies_. Conclusively, our exploratory
study cleared any doubts on this confounding factor for the evaluation and
reproducibility of traditional collaborative filtering algorithms.

## Acknowledgements

The OMNI cluster of the University of Siegen was used to compute the results
presented in this paper.

## References

  * [1] Abdollahpouri, H., Burke, R., Mobasher, B.: Managing popularity bias in recommender systems with personalized re-ranking. CoRR abs/1901.07555 (2019), http://arxiv.org/abs/1901.07555
  * [2] Anand, R., Beel, J.: Auto-surprise: An automated recommender-system (autorecsys) library with tree of parzens estimator (tpe) optimization. In: Proceedings of the 14th ACM Conference on Recommender Systems. p. 585–587. RecSys ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3383313.3411467, https://doi.org/10.1145/3383313.3411467
  * [3] Anelli, V.W., Bellogín, A., Ferrara, A., Malitesta, D., Merra, F.A., Pomo, C., Donini, F.M., Noia, T.D.: Elliot: A comprehensive and rigorous framework for reproducible recommender systems evaluation. In: Diaz, F., Shah, C., Suel, T., Castells, P., Jones, R., Sakai, T. (eds.) SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021. pp. 2405–2414. ACM (2021). https://doi.org/10.1145/3404835.3463245, https://doi.org/10.1145/3404835.3463245
  * [4] Barkan, O., Hirsch, R., Katz, O., Caciularu, A., Koenigstein, N.: Anchor-based collaborative filtering. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. p. 2877–2881. CIKM ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3459637.3482056, https://doi.org/10.1145/3459637.3482056
  * [5] Beel, J., Dinesh, S.: Real-world recommender systems for academia: The pain and gain in building, operating, and researching them. In: Mayr, P., Frommholz, I., Cabanac, G. (eds.) Proceedings of the Fifth Workshop on Bibliometric-enhanced Information Retrieval (BIR) co-located with the 39th European Conference on Information Retrieval (ECIR 2017), Aberdeen, UK, April 9th, 2017. CEUR Workshop Proceedings, vol. 1823, pp. 6–17. CEUR-WS.org (2017), https://ceur-ws.org/Vol-1823/paper1.pdf
  * [6] Bruch, S., Wang, X., Bendersky, M., Najork, M.: An analysis of the softmax cross entropy loss for learning-to-rank with binary relevance. In: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. p. 75–78. ICTIR ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3341981.3344221, https://doi.org/10.1145/3341981.3344221
  * [7] Cantador, I., Brusilovsky, P., Kuflik, T.: Second workshop on information heterogeneity and fusion in recommender systems (hetrec2011). In: Proceedings of the Fifth ACM Conference on Recommender Systems. p. 387–388. RecSys ’11, Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/2043932.2044016, https://doi.org/10.1145/2043932.2044016
  * [8] Chen, B., Zhao, X., Wang, Y., Fan, W., Guo, H., Tang, R.: A comprehensive survey on automated machine learning for recommendations. ACM Trans. Recomm. Syst. (oct 2023). https://doi.org/10.1145/3630104, https://doi.org/10.1145/3630104, just Accepted
  * [9] Chen, H., Lin, Y., Pan, M., Wang, L., Yeh, C.C.M., Li, X., Zheng, Y., Wang, F., Yang, H.: Denoising self-attentive sequential recommendation. In: Proceedings of the 16th ACM Conference on Recommender Systems. p. 92–101. RecSys ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3523227.3546788, https://doi.org/10.1145/3523227.3546788
  * [10] Cho, E., Myers, S.A., Leskovec, J.: Friendship and mobility: User movement in location-based social networks. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. p. 1082–1090. KDD ’11, Association for Computing Machinery, New York, NY, USA (2011). https://doi.org/10.1145/2020408.2020579, https://doi.org/10.1145/2020408.2020579
  * [11] Dooms, Simon and De Pessemier, Toon and Martens, Luc: MovieTweetings: a movie rating dataset collected from twitter. In: Workshop on Crowdsourcing and Human Computation for Recommender Systems, held in conjunction with the 7th ACM Conference on Recommender Systems. p. 2 (2013)
  * [12] Ekstrand, M.D.: Lenskit for python: Next-generation software for recommender systems experiments. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management. p. 2999–3006. CIKM ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3340531.3412778, https://doi.org/10.1145/3340531.3412778
  * [13] Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., Hutter, F.: Efficient and robust automated machine learning. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 28. Curran Associates, Inc. (2015)
  * [14] Frederickson, B.: Fast python collaborative filtering for implicit datasets. URL https://github. com/benfred/implicit (2018)
  * [15] Ge, Y., Zhao, S., Zhou, H., Pei, C., Sun, F., Ou, W., Zhang, Y.: Understanding echo chambers in e-commerce recommender systems. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 2261–2270. SIGIR ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3397271.3401431, https://doi.org/10.1145/3397271.3401431
  * [16] Goldberg, K., Roeder, T., Gupta, D., Perkins, C.: Eigentaste: A constant time collaborative filtering algorithm. Inf. Retr. 4(2), 133–151 (jul 2001). https://doi.org/10.1023/A:1011419012209, https://doi.org/10.1023/A:1011419012209
  * [17] Gulla, J.A., Zhang, L., Liu, P., Özgöbek, O., Su, X.: The adressa dataset for news recommendation. In: Proceedings of the International Conference on Web Intelligence. p. 1042–1048. WI ’17, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3106426.3109436, https://doi.org/10.1145/3106426.3109436
  * [18] Harper, F.M., Konstan, J.A.: The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. 5(4) (dec 2015). https://doi.org/10.1145/2827872, https://doi.org/10.1145/2827872
  * [19] Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (jan 2004). https://doi.org/10.1145/963770.963772, https://doi.org/10.1145/963770.963772
  * [20] Hernández del Olmo, F., Gaudioso, E.: Evaluation of recommender systems: A new approach. Expert Systems with Applications 35(3), 790–804 (2008). https://doi.org/https://doi.org/10.1016/j.eswa.2007.07.047, https://www.sciencedirect.com/science/article/pii/S0957417407002928
  * [21] Jankiewicz, P., Kyrashchuk, L., Sienkowski, P., Wójcik, M.: Boosting algorithms for a session-based, context-aware recommender system in an online travel domain. In: Proceedings of the Workshop on ACM Recommender Systems Challenge. RecSys Challenge ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3359555.3359557, https://doi.org/10.1145/3359555.3359557
  * [22] Jannach, D., Adomavicius, G.: Price and profit awareness in recommender systems. CoRR abs/1707.08029 (2017), http://arxiv.org/abs/1707.08029
  * [23] Joglekar, M.R., Li, C., Chen, M., Xu, T., Wang, X., Adams, J.K., Khaitan, P., Liu, J., Le, Q.V.: Neural input search for large scale recommendation models. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. p. 2387–2397. KDD ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3394486.3403288, https://doi.org/10.1145/3394486.3403288
  * [24] Langer, S., Beel, J.: Apache lucene as content-based-filtering recommender system: 3 lessons learned. In: Mayr, P., Frommholz, I., Cabanac, G. (eds.) Proceedings of the Fifth Workshop on Bibliometric-enhanced Information Retrieval (BIR) co-located with the 39th European Conference on Information Retrieval (ECIR 2017), Aberdeen, UK, April 9th, 2017. CEUR Workshop Proceedings, vol. 1823, pp. 85–92. CEUR-WS.org (2017), https://ceur-ws.org/Vol-1823/paper8.pdf
  * [25] Li, Z., Ji, J., Ge, Y., Zhang, Y.: Autolossgen: Automatic loss function generation for recommender systems. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 1304–1315. SIGIR ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3477495.3531941, https://doi.org/10.1145/3477495.3531941
  * [26] Liang, D., Krishnan, R.G., Hoffman, M.D., Jebara, T.: Variational autoencoders for collaborative filtering. In: Proceedings of the 2018 World Wide Web Conference. p. 689–698. WWW ’18, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE (2018). https://doi.org/10.1145/3178876.3186150, https://doi.org/10.1145/3178876.3186150
  * [27] Liu, H., Zhao, X., Wang, C., Liu, X., Tang, J.: Automated embedding size search in deep recommender systems. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 2307–2316. SIGIR ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3397271.3401436, https://doi.org/10.1145/3397271.3401436
  * [28] Liu, W., Xi, Y., Qin, J., Sun, F., Chen, B., Zhang, W., Zhang, R., Tang, R.: Neural re-ranking in multi-stage recommender systems: A review. In: Raedt, L.D. (ed.) Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. pp. 5512–5520. International Joint Conferences on Artificial Intelligence Organization (7 2022). https://doi.org/10.24963/ijcai.2022/771, https://doi.org/10.24963/ijcai.2022/771, survey Track
  * [29] Melchiorre, A.B., Rekabsaz, N., Ganhör, C., Schedl, M.: Protomf: Prototype-based matrix factorization for effective and explainable recommendations. In: Proceedings of the 16th ACM Conference on Recommender Systems. p. 246–256. RecSys ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3523227.3546756, https://doi.org/10.1145/3523227.3546756
  * [30] Michiels, L., Verachtert, R., Goethals, B.: Recpack: An(other) experimentation toolkit for top-n recommendation using implicit feedback data. In: Proceedings of the 16th ACM Conference on Recommender Systems. p. 648–651. RecSys ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3523227.3551472, https://doi.org/10.1145/3523227.3551472
  * [31] Moreira, G.D.S.P., Jannach, D., Cunha, A.M.D.: Contextual hybrid session-based news recommendation with recurrent neural networks. IEEE Access 7, 169185–169203 (2019). https://doi.org/10.1109/access.2019.2954957, https://doi.org/10.1109%2Faccess.2019.2954957
  * [32] Ni, J., Li, J., McAuley, J.: Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 188–197. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1018, https://aclanthology.org/D19-1018
  * [33] Pang, L., Ai, Q., Xu, J.: Beyond probability ranking principle: Modeling the dependencies among documents. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. p. 1137–1140. WSDM ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3437963.3441662, https://doi.org/10.1145/3437963.3441662
  * [34] Pei, C., Zhang, Y., Zhang, Y., Sun, F., Lin, X., Sun, H., Wu, J., Jiang, P., Ge, J., Ou, W., Pei, D.: Personalized re-ranking for recommendation. In: Proceedings of the 13th ACM Conference on Recommender Systems. p. 3–11. RecSys ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3298689.3347000, https://doi.org/10.1145/3298689.3347000
  * [35] Pichl, M., Zangerle, E., Specht, G.: Towards a context-aware music recommendation approach: What is hidden in the playlist name? In: 2015 IEEE International Conference on Data Mining Workshop (ICDMW). pp. 1360–1365 (2015). https://doi.org/10.1109/ICDMW.2015.145
  * [36] Poddar, A., Zangerle, E., Yang, Y.H.: #nowplaying-rs: A new benchmark dataset for building context-aware music recommender systems. In: Proceedings of the 15th Sound & Music Computing Conference. Limassol, Cyprus (2018), http://mac.citi.sinica.edu.tw/~yang/pub/poddar18smc.pdf, code at https://github.com/asmitapoddar/nowplaying-RS-Music-Reco-FM
  * [37] Portugal, I., Alencar, P., Cowan, D.: The use of machine learning algorithms in recommender systems: A systematic review. Expert Systems with Applications 97, 205–227 (2018). https://doi.org/https://doi.org/10.1016/j.eswa.2017.12.020, https://www.sciencedirect.com/science/article/pii/S0957417417308333
  * [38] Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: Bpr: Bayesian personalized ranking from implicit feedback. In: Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence. p. 452–461. UAI ’09, AUAI Press, Arlington, Virginia, USA (2009)
  * [39] Rendle, S., Krichene, W., Zhang, L., Koren, Y.: Revisiting the performance of ials on item recommendation benchmarks. In: Proceedings of the 16th ACM Conference on Recommender Systems. p. 427–435. RecSys ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3523227.3548486, https://doi.org/10.1145/3523227.3548486
  * [40] ROBERTSON, S.: The probability ranking principle in ir. Journal of Documentation 33(4), 294–304 (Apr 1977). https://doi.org/10.1108/eb026647, http://dx.doi.org/10.1108/eb026647
  * [41] de Souza Pereira Moreira, G., Ferreira, F., da Cunha, A.M.: News session-based recommendations using deep neural networks. In: Proceedings of the 3rd Workshop on Deep Learning for Recommender Systems. ACM (oct 2018). https://doi.org/10.1145/3270323.3270328, https://doi.org/10.1145%2F3270323.3270328
  * [42] Sun, B., Wu, D., Shang, M., He, Y.: Toward auto-learning hyperparameters for deep learning-based recommender systems. In: Bhattacharya, A., Lee Mong Li, J., Agrawal, D., Reddy, P.K., Mohania, M., Mondal, A., Goyal, V., Uday Kiran, R. (eds.) Database Systems for Advanced Applications. pp. 323–331. Springer International Publishing, Cham (2022)
  * [43] Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., Jiang, P.: Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management. p. 1441–1450. CIKM ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3357384.3357895, https://doi.org/10.1145/3357384.3357895
  * [44] Tang, H., Liu, J., Zhao, M., Gong, X.: Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In: Proceedings of the 14th ACM Conference on Recommender Systems. p. 269–278. RecSys ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3383313.3412236, https://doi.org/10.1145/3383313.3412236
  * [45] Vente, T., Ekstrand, M., Beel, J.: Introducing lenskit-auto, an experimental automated recommender system (autorecsys) toolkit. In: Proceedings of the 17th ACM Conference on Recommender Systems. p. 1212–1216. RecSys ’23, Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3604915.3610656, https://doi.org/10.1145/3604915.3610656
  * [46] Wang, H., Chen, B., Li, W.J.: Collaborative topic regression with social regularization for tag recommendation. In: Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence. p. 2719–2725. IJCAI ’13, AAAI Press (2013)
  * [47] Wang, X., Li, C., Golbandi, N., Bendersky, M., Najork, M.: The lambdaloss framework for ranking metric optimization. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. p. 1313–1322. CIKM ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3269206.3271784, https://doi.org/10.1145/3269206.3271784
  * [48] Weston, J., Yee, H., Weiss, R.J.: Learning to rank recommendations with the k-order statistic loss. In: Proceedings of the 7th ACM Conference on Recommender Systems. p. 245–248. RecSys ’13, Association for Computing Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2507157.2507210, https://doi.org/10.1145/2507157.2507210
  * [49] Xu, L., Tian, Z., Zhang, G., Wang, L., Zhang, J., Zheng, B., Li, Y., Hou, Y., Pan, X., Chen, Y., Zhao, W.X., Chen, X., Wen, J.R.: Recent Advances in RecBole: Extensions with more Practical Considerations. arXiv e-prints arXiv:2211.15148 (Nov 2022). https://doi.org/10.48550/arXiv.2211.15148
  * [50] Yang, C., Hou, Y., Song, Y., Zhang, T., Wen, J.R., Zhao, W.X.: Modeling two-way selection preference for person-job fit. In: Proceedings of the 16th ACM Conference on Recommender Systems. p. 102–112. RecSys ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3523227.3546752, https://doi.org/10.1145/3523227.3546752
  * [51] Yue, Z., He, Z., Zeng, H., McAuley, J.: Black-box attacks on sequential recommenders via data-free model extraction. In: Proceedings of the 15th ACM Conference on Recommender Systems. p. 44–54. RecSys ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3460231.3474275, https://doi.org/10.1145/3460231.3474275
  * [52] Yue, Z., Zeng, H., Kou, Z., Shang, L., Wang, D.: Defending substitution-based profile pollution attacks on sequential recommenders. In: Proceedings of the 16th ACM Conference on Recommender Systems. p. 59–70. RecSys ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3523227.3546770, https://doi.org/10.1145/3523227.3546770
  * [53] Zangerle, E., Bauer, C.: Evaluating recommender systems: Survey and framework. ACM Comput. Surv. 55(8) (dec 2022). https://doi.org/10.1145/3556536, https://doi.org/10.1145/3556536
  * [54] Zhang, R., Bao, H., Sun, H., Wang, Y., Liu, X.: Recommender systems based on ranking performance optimization. Frontiers of Computer Science 10(2), 270–280 (Jul 2015). https://doi.org/10.1007/s11704-015-4584-1, http://dx.doi.org/10.1007/s11704-015-4584-1
  * [55] Zhao, W.X., Hou, Y., Pan, X., Yang, C., Zhang, Z., Lin, Z., Zhang, J., Bian, S., Tang, J., Sun, W., et al.: Recbole 2.0: Towards a more up-to-date recommendation library. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management. pp. 4722–4726 (2022)
  * [56] Zhao, W.X., Mu, S., Hou, Y., Lin, Z., Chen, Y., Pan, X., Li, K., Lu, Y., Wang, H., Tian, C., Min, Y., Feng, Z., Fan, X., Chen, X., Wang, P., Ji, W., Li, Y., Wang, X., Wen, J.: Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In: CIKM. pp. 4653–4664. ACM (2021)
  * [57] Zheng, R., Qu, L., Cui, B., Shi, Y., Yin, H.: Automl for deep recommender systems: A survey. ACM Trans. Inf. Syst. 41(4) (mar 2023). https://doi.org/10.1145/3579355, https://doi.org/10.1145/3579355