# Understanding metric-related pitfalls in image analysis validation

Annika Reinke<EMAIL_ADDRESS>German Cancer Research Center (DKFZ)
Heidelberg, Division of Intelligent Medical Systems and HI Helmholtz
ImagingGermany Faculty of Mathematics and Computer Science, Heidelberg
UniversityHeidelberg, Germany ,  Minu D. Tizabi  German Cancer Research Center
(DKFZ) Heidelberg, Division of Intelligent Medical SystemsGermany National
Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ
and University Medical Center HeidelbergGermany ,  Michael Baumgartner  German
Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image
ComputingGermany ,  Matthias Eisenmann  German Cancer Research Center (DKFZ)
Heidelberg, Division of Intelligent Medical SystemsGermany ,  Doreen Heckmann-
Nötzel  German Cancer Research Center (DKFZ) Heidelberg, Division of
Intelligent Medical SystemsGermany National Center for Tumor Diseases (NCT),
NCT Heidelberg, a partnership between DKFZ and University Medical Center
HeidelbergGermany ,  A. Emre Kavur  HI Applied Computer Vision Lab, Division
of Medical Image Computing; German Cancer Research Center (DKFZ) Heidelberg,
Division of Intelligent Medical SystemsGermany ,  Tim Rädsch  German Cancer
Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems and
HI Helmholtz ImagingGermany ,  Carole H. Sudre  MRC Unit for Lifelong Health
and Ageing at UCL and Centre for Medical Image Computing, Department of
Computer Science, University College LondonLondon, UK School of Biomedical
Engineering and Imaging Science, King’s College LondonLondon, UK ,  Laura
Acion  Instituto de Cálculo, CONICET – Universidad de Buenos AiresBuenos
Aires, Argentina ,  Michela Antonelli  School of Biomedical Engineering and
Imaging Science, King’s College LondonLondon, UK Centre for Medical Image
Computing, University College LondonLondon, UK ,  Tal Arbel  Centre for
Intelligent Machines and MILA (Quebec Artificial Intelligence Institute),
McGill UniversityMontreal, Canada ,  Spyridon Bakas  Division of Computational
Pathology, Dept of Pathology & Laboratory Medicine, Indiana University School
of Medicine, IU Health Information and Translational Sciences
BuildingIndianapolis, USA Center for Biomedical Image Computing and Analytics
(CBICA), University of Pennsylvania, Richards Medical Research Laboratories
FL7Philadelphia, PA, USA ,  Arriel Benis  Department of Digital Medical
Technologies, Holon Institute of TechnologyHolon, Israel European Federation
for Medical InformaticsLe Mont-sur-Lausanne, Switzerland ,  Matthew B.
Blaschko  Center for Processing Speech and Images, Department of Electrical
Engineering, KU LeuvenLeuven, Belgium ,  Florian Buettner  German Cancer
Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ
and UCT Frankfurt-MarburgGermany German Cancer Research Center (DKFZ)
HeidelbergGermany Goethe University Frankfurt, Department of MedicineGermany
Goethe University Frankfurt, Department of InformaticsGermany Frankfurt Cancer
InsitituteGermany ,  M. Jorge Cardoso  School of Biomedical Engineering and
Imaging Science, King’s College LondonLondon, UK ,  Veronika Cheplygina
Department of Computer Science, IT University of CopenhagenCopenhagen, Denmark
,  Jianxu Chen  Leibniz-Institut für Analytische Wissenschaften – ISAS –
e.V.Dortmund, Germany ,  Evangelia Christodoulou  German Cancer Research
Center (DKFZ) Heidelberg, Division of Intelligent Medical SystemsGermany ,
Beth A. Cimini  Imaging Platform, Broad Institute of MIT and HarvardCambridge,
Massachusetts, USA ,  Gary S. Collins  Centre for Statistics in Medicine,
University of OxfordOxford, UK ,  Keyvan Farahani  Center for Biomedical
Informatics and Information Technology, National Cancer InstituteBethesda, MD,
USA ,  Luciana Ferrer  Instituto de Investigación en Ciencias de la
Computación (ICC), CONICET-UBACiudad Universitaria, Ciudad Autónoma de Buenos
Aires, Argentina ,  Adrian Galdran  Universitat Pompeu FabraBarcelona, Spain
University of AdelaideAdelaide, Australia ,  Bram van Ginneken  Fraunhofer
MEVISBremen, Germany Radboud Institute for Health Sciences, Radboud University
Medical CenterNijmegen, The Netherlands ,  Ben Glocker  Department of
Computing, Imperial College LondonLondon, UK ,  Patrick Godau  German Cancer
Research Center (DKFZ) Heidelberg, Division of Intelligent Medical
SystemsGermany Faculty of Mathematics and Computer Science, Heidelberg
UniversityHeidelberg, Germany National Center for Tumor Diseases (NCT), NCT
Heidelberg, a partnership between DKFZ and University Medical Center
HeidelbergGermany ,  Robert Haase  Now with: Center for Scalable Data
Analytics and Artificial Intelligence (ScaDS.AI), Leipzig UniversityLeipzig,
Germany DFG Cluster of Excellence ”Physics of Life”, Technische Universität
(TU) DresdenDresden, Germany Center for Systems Biology Dresden, Germany ,
Daniel A. Hashimoto  Department of Surgery, Perelman School of
MedicinePhiladelphia, PA, USA General Robotics Automation Sensing and
Perception Laboratory, School of Engineering and Applied Science, University
of PennsylvaniaPhiladelphia, PA, USA ,  Michael M. Hoffman  Princess Margaret
Cancer Centre, University Health NetworkToronto, Canada Department of Medical
Biophysics, University of TorontoToronto, Canada Department of Computer
Science, University of TorontoToronto, Canada Vector Institute for Artificial
IntelligenceToronto, Canada ,  Merel Huisman  Department of Radiology and
Nuclear Medicine, Radboud University Medical CenterNijmegen, The Netherlands ,
Fabian Isensee  German Cancer Research Center (DKFZ) Heidelberg, Division of
Medical Image Computing and HI Applied Computer Vision LabGermany ,  Pierre
Jannin  Laboratoire Traitement du Signal et de l’Image – UMR_S 1099,
Université de Rennes 1Rennes, France INSERMParis Cedex, France ,  Charles E.
Kahn  Department of Radiology and Institute for Biomedical Informatics,
University of PennsylvaniaPhiladelphia, PA, USA ,  Dagmar Kainmueller  Max-
Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC),
Biomedical Image Analysis and HI Helmholtz ImagingBerlin, Germany University
of Potsdam, Digital Engineering FacultyPotsdam, Germany ,  Bernhard Kainz
Department of Computing, Faculty of Engineering, Imperial College
LondonLondon, UK Department AIBE, Friedrich-Alexander-Universität
(FAU)Erlangen-Nürnberg, Germany ,  Alexandros Karargyris  IHU
StrasbourgStrasbourg, France ,  Alan Karthikesalingam  Google Health
DeepMindLondon, UK ,  Hannes Kenngott  Department of General, Visceral and
Transplantation Surgery, Heidelberg University HospitalHeidelberg, Germany ,
Jens Kleesiek  Translational Image-guided Oncology (TIO), Institute for AI in
Medicine (IKIM), University Medicine EssenEssen, Germany ,  Florian Kofler
Helmholtz AIMünchen, Germany ,  Thijs Kooi  LunitSeoul, South Korea ,  Annette
Kopp-Schneider  German Cancer Research Center (DKFZ) Heidelberg, Division of
BiostatisticsGermany ,  Michal Kozubek  Centre for Biomedical Image Analysis
and Faculty of Informatics, Masaryk UniversityBrno, Czech Republic ,  Anna
Kreshuk  Cell Biology and Biophysics Unit, European Molecular Biology
Laboratory (EMBL)Heidelberg, Germany ,  Tahsin Kurc  Department of Biomedical
Informatics, Stony Brook UniversityStony Brook, NY, USA ,  Bennett A. Landman
Electrical Engineering, Vanderbilt UniversityNashville, TN, USA ,  Geert
Litjens  Department of Pathology, Radboud University Medical CenterNijmegen,
The Netherlands ,  Amin Madani  Department of Surgery, University Health
NetworkPhiladelphia, PA, Canada ,  Klaus Maier-Hein  German Cancer Research
Center (DKFZ) Heidelberg, Division of Medical Image Computing and HI Helmholtz
ImagingGermany Pattern Analysis and Learning Group, Department of Radiation
Oncology, Heidelberg University HospitalHeidelberg, Germany ,  Anne L. Martel
Physical Sciences, Sunnybrook Research InstituteToronto, Canada Department of
Medical Biophysics, University of TorontoToronto, ON, Canada ,  Peter Mattson
GoogleMountain View, USA ,  Erik Meijering  School of Computer Science and
Engineering, University of New South WalesSydney, Australia ,  Bjoern Menze
Department of Quantitative Biomedicine, University of ZurichZurich,
Switzerland ,  Karel G.M. Moons  Julius Center for Health Sciences and Primary
Care, UMC Utrecht, Utrecht UniversityUtrecht, The Netherlands ,  Henning
Müller  Information Systems Institute, University of Applied Sciences Western
Switzerland (HES-SO)Sierre, Switzerland Medical Faculty, University of
GenevaGeneva, Switzerland ,  Brennan Nichyporuk  MILA (Quebec Artificial
Intelligence Institute)Montréal, Canada ,  Felix Nickel  Department of
General, Visceral and Thoracic Surgery, University Medical Center Hamburg-
EppendorfHamburg, Germany ,  Jens Petersen  German Cancer Research Center
(DKFZ) Heidelberg, Division of Medical Image ComputingGermany ,  Susanne M.
Rafelski  Allen Institute for Cell ScienceSeattle, WA, USA ,  Nasir Rajpoot
Tissue Image Analytics Laboratory, Department of Computer Science, University
of WarwickTissue Image Analytics Laboratory, Department of Computer
ScienceCoventry, UK ,  Mauricio Reyes  ARTORG Center for Biomedical
Engineering Research, University of BernBern, Switzerland Department of
Radiation Oncology, University Hospital Bern, University of BernBern,
Switzerland ,  Michael A. Riegler  Simula Metropolitan Center for Digital
EngineeringOslo, Norway UiT The Arctic University of NorwayTromsø, Norway ,
Nicola Rieke  NVIDIA GmbHMünchen, Germany ,  Julio Saez-Rodriguez  Institute
for Computational Biomedicine, Heidelberg UniversityHeidelberg. Germany
Faculty of Medicine, Heidelberg University HospitalHeidelberg, Germany ,
Clara I. Sánchez  Informatics Institute, Faculty of Science, University of
AmsterdamAmsterdam, The Netherlands ,  Shravya Shetty  Google Health,
GoogleCA, USA ,  Ronald M. Summers  National Institutes of Health Clinical
CenterBethesda, MD, USA ,  Abdel A. Taha  Institute of Information Systems
Engineering, TU WienVienna, Austria ,  Aleksei Tiulpin  Research Unit of
Health Sciences and Technology, Faculty of Medicine, University of OuluOulu,
Finland Neurocenter Oulu, Oulu University HospitalOulu, Finland ,  Sotirios A.
Tsaftaris  School of Engineering, The University of EdinburghEdinburgh,
Scotland ,  Ben Van Calster  Department of Development and Regeneration and
EPI-centre, KU LeuvenLeuven, Belgium Department of Biomedical Data Sciences,
Leiden University Medical CenterLeiden, The Netherlands ,  Gaël Varoquaux
Parietal project team, INRIA Saclay-Île de FrancePalaiseau, France ,  Ziv R.
Yaniv  National Institute of Allergy and Infectious Diseases, National
Institutes of HealthBethesda, MD, USA ,  Paul F. Jäger  German Cancer Research
Center (DKFZ) Heidelberg, Interactive Machine Learning Group and HI Helmholtz
ImagingGermany and  Lena Maier-Hein  German Cancer Research Center (DKFZ)
Heidelberg, Division of Intelligent Medical Systems and HI Helmholtz
ImagingGermany Faculty of Mathematics and Computer Science and Medical
Faculty, Heidelberg UniversityHeidelberg, Germany National Center for Tumor
Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University
Medical Center HeidelbergGermany

###### Abstract.

Abstract: Validation metrics are key for the reliable tracking of scientific
progress and for bridging the current chasm between artificial intelligence
(AI) research and its translation into practice. However, increasing evidence
shows that particularly in image analysis, metrics are often chosen
inadequately in relation to the underlying research problem. This could be
attributed to a lack of accessibility of metric-related knowledge: While
taking into account the individual strengths, weaknesses, and limitations of
validation metrics is a critical prerequisite to making educated choices, the
relevant knowledge is currently scattered and poorly accessible to individual
researchers. Based on a multi-stage Delphi process conducted by a
multidisciplinary expert consortium as well as extensive community feedback,
the present work provides the first reliable and comprehensive common point of
access to information on pitfalls related to validation metrics in image
analysis. Focusing on biomedical image analysis but with the potential of
transfer to other fields, the addressed pitfalls generalize across application
domains and are categorized according to a newly created, domain-agnostic
taxonomy. To facilitate comprehension, illustrations and specific examples
accompany each pitfall. As a structured body of information accessible to
researchers of all levels of expertise, this work enhances global
comprehension of a key topic in image analysis validation.

Validation, Evaluation, Pitfalls, Metrics, Good Scientific Practice,
Biomedical Image Processing, Challenges, Computer Vision, Classification,
Segmentation, Instance Segmentation, Semantic Segmentation, Detection,
Localization, Medical Imaging, Biological Imaging

## Main

Measuring performance and progress in any given field critically depends on
the availability of meaningful outcome metrics. In a field such as athletics,
this process is straightforward because the performance measurements (e.g.,
the time it takes an athlete to run a given distance) exactly reflect the
underlying interest (e.g., which athlete runs a given distance the fastest?).
In image analysis, the situation is much more complex as, depending on the
underlying research question, vastly different aspects of an algorithm’s
performance might be of interest (Fig. 1) and meaningful in determining its
future practical, for example clinical, applicability. If the performance of
an image analysis algorithm is not measured according to relevant validation
metrics, no reliable statement can be made about the suitability of this
algorithm in solving the proposed task, and the algorithm is unlikely to ever
reach the stage of real-life application. Moreover, unsuitable algorithms
could be wrongly regarded as the best-performing ones, sparking entirely
futile resource investment and follow-up research while obscuring true
scientific advancements. In determining new state-of-the-art methods and
informing future directions, the use of validation metrics actively shapes the
evolution of research. In summary, validation metrics are the key for both
measuring and informing scientific progress, as well as bridging the current
chasm between image analysis research and its translation into practice.

In image analysis, while for some applications it might, for instance, be
sufficient to draw a box around the structure of interest (e.g., a polyp in
colonoscopic polyp detection), other applications (e.g., tumor volume
delineation for radiotherapy planning) could require determining the exact
structure boundaries. The suitability of any individual validation metric thus
depends crucially on the properties of the driving image analysis problem. As
a result, numerous metrics have so far been proposed in the field of image
processing. In our previous work, we analyzed all biomedical image analysis
competitions conducted within a period of about 15 years [Maier-Hein et al.,
2018]. We found a total of 97 different metrics reported in the field of
biomedicine alone, each with its own individual strengths, weaknesses, and
limitations, and hence varying degrees of suitability for meaningfully
measuring algorithm performance on any given research problem. Such a vast
lake of options makes tracking all related information impossible for any
individual researcher and consequently renders the process of metric selection
error-prone. Thus, the frequent reliance on flawed, historically grown
validation practices in current literature comes as no surprise. To make
matters worse, there is currently no comprehensive resource that can provide
an overview of the relevant definitions, (mathematical) properties,
limitations, and pitfalls pertaining to a metric of interest. While taking
into account the individual properties and limitations of metrics is
imperative for choosing adequate validation metrics, the required knowledge is
thus largely inaccessible.

Figure 1. Examples of metric-related pitfalls in image analysis validation.
(A) Medical image analysis example: Voxel-based metrics are not appropriate
for detection problems. Measuring the voxel-level performance of a prediction
yields a near-perfect Sensitivity. However, the Sensitivity at the instance
level reveals that lesions are actually missed by the algorithm. (B)
Biological image analysis example: The task of predicting fibrillarin in the
dense fibrillary component of the nucleolus should be phrased as a
segmentation task, for which segmentation metrics reveal the low quality of
the prediction. Phrasing the task as image reconstruction instead and
validating it using metrics such as the Person Correlation Coefficient yields
misleadingly high metric scores [Roberts et al., 2017, Chen et al., 2020,
Viana et al., 2023, Ounkomol et al., 2018, Viana et al., 2023].

As a result, numerous flaws and pitfalls are prevalent in image analysis
validation, with researchers often being unaware of them due to a lack of
knowledge of intricate metric properties and limitations. Accordingly,
increasing evidence shows that metrics are often selected inadequately in
image analysis (e.g., [Kofler et al., 2021, Gooding et al., 2018, Vaassen et
al., 2020]). In the absence of a central information resource, it is common
for researchers to resort to popular validation metrics, which, however, can
be entirely unsuitable, for instance due to a mismatch of the metric’s
inherent mathematical properties with the underlying research question and
specifications of the data set at hand (see Fig. 1).

The present work addresses this important roadblock in image analysis research
with a crowdsourcing-based approach that involved both a Delphi process
undergone by a multidisciplinary expert consortium as well as a social media
campaign. It represents the first comprehensive collection, visualization, and
detailed discussion of pitfalls, drawbacks, and limitations regarding
validation metrics commonly used in image analysis. Our work provides
researchers with a reliable, single point of access to this critical and yet,
until now, poorly retrievable or outright unavailable information. Owing to
the enormous complexity of the matter, the metric properties and pitfalls are
discussed in the specific context of classification problems, i.e., image
analysis problems that can be considered classification tasks at either the
image, object, or pixel level. Specifically, these encompass the four problem
categories of image-level classification, semantic segmentation, object
detection, and instance segmentation. Our contribution includes a dedicated
profile for each metric (Suppl. Note 3) as well as the creation of a new
common taxonomy that categorizes pitfalls in a domain-agnostic manner (Fig.
2). Depicted for individual metrics in tables provided in this paper (see
Extended Data Tabs. Extended Data-5), the taxonomy enables researchers to
quickly grasp whether using a certain metric comes with pitfalls in a given
use case. While our work grew out of image analysis research and practice in
the field of biomedicine, a field of high complexity and particularly high
stakes due to its direct impact on human health, we believe the identified
pitfalls to be transferable to other application areas of imaging research. It
should be noted that this work focuses on identifying, categorizing, and
illustrating metric pitfalls, while the sister publication of this work gives
specific recommendations on which metrics to apply under which circumstances
[Maier-Hein et al., 2022].

## Results

### Information on metric pitfalls is largely inaccessible

Researchers and algorithm developers seeking to validate image analysis
algorithms frequently face the problem of choosing adequate validation metrics
while at the same time navigating a range of potential pitfalls. Following
common practice is often not the best option, as evidenced by a number of
recent publications [Maier-Hein et al., 2018, Kofler et al., 2021, Gooding et
al., 2018, Vaassen et al., 2020]. Making an educated choice from a vast array
of possibilities requires a researcher to be aware of not only the definitions
and mathematical properties of different metrics but also their strengths and
weaknesses, as well as limitations related to their use under certain
conditions. The endeavor is notably complicated by the absence of any
comprehensive databases or reviews covering the topic and thus the lack of a
central resource for reliable information on validation metrics.

This lack of accessibility is considered by experts to be a major bottleneck
in image analysis validation [Maier-Hein et al., 2018]. To illustrate this
point, we searched the literature for available information on commonly used
validation metrics. The search was conducted on the platform Google Scholar
using search strings that combined different notations of the metric name,
including synonyms and acronyms, with search terms indicating problems, such
as “pitfall” or “limitation”. The mean and median number of hits for the
metrics addressed in the present work were 159,329 and 22,100, respectively,
and ranged between 49 for centerline Dice Similarity Coefficient (clDice) and
962,000 for Sensitivity. Moreover, despite valuable literature on individual
relevant aspects (e.g., [Taha et al., 2014, Taha and Hanbury, 2015, Grandini
et al., 2020, Kofler et al., 2021, Vaassen et al., 2020, Chicco et al., 2021,
Chicco and Jurman, 2020]), we did not find a common point of entry to metric-
related pitfalls in image analysis in the form of a review paper or other
credible source. It is thus unfeasible for any individual researcher to,
within reasonable time and effort, retrieve comprehensive information on
properties and pitfalls pertaining to one or multiple metrics of interest from
the current body of research literature. We conclude that the key knowledge
required for making educated decisions and avoiding pitfalls related to the
use of validation metrics is highly scattered and not accessible by
individuals.

### Historically grown practices are not always justified

To obtain an initial insight into current common practice regarding validation
metrics, we prospectively captured the designs of challenges organized by the
IEEE Society of the International Symposium of Biomedical Imaging (ISBI), the
Medical Image Computing and Computer Assisted Interventions (MICCAI) Society
and the Medical Imaging with Deep Learning (MIDL) foundation. The organizers
of the respective competitions were asked to provide a rationale for the
choice of metrics in their competition. An analysis of a total of 138
competitions conducted between 2018 and 2022 revealed that metrics are
frequently (in $24\%$ of the competitions) based on common practice in the
community. We found, however, that common practices are often not well-
justified, and poor practices may even be propagated from one generation to
the next.

One remarkable example for this issue is the widespread adoption of an
incorrect naming and inconsistent mathematical formulation of a metric
proposed for cell instance segmentation. The term ” mean Average Precision
(mAP)” usually refers to one of the most common metrics in object detection
(object-level classification) [Lin et al., 2014, Reinke et al., 2021b]. Here,
Precision denotes the Positive Predictive Value (PPV), which is ”averaged”
over varying thresholds on the predicted class scores of an object detection
algorithm. The ”mean” Average Precision (AP) is then obtained by taking the
mean over classes [Everingham et al., 2010, Reinke et al., 2021b]. Despite the
popularity of mAP, a widely known challenge on cell instance
segmentation111https://www.kaggle.com/competitions/data-science-
bowl-2018/overview/evaluation introduced a new ”Mean Average Precision” in
2018. Although the task matches the task of the original ”mean” AP, object
detection, all terms in the newly proposed metric (mean, average, and
precision) refer to entirely different concepts. For instance, the common
definition of Precision from literature
$\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})$ was altered to
$\mathrm{TP}/(\mathrm{TP}+\mathrm{FP}+\mathrm{FN})$, where TP, FP, and FN
refer to the cardinalities of the confusion matrix (i.e., the true/false
positives/negatives). The latter formula actually defines the Intersection
over Union (IoU) metric. Despite these problems, the terminology was adopted
by subsequent influential works [Schmidt et al., 2018, Stringer et al., 2021,
Kaggle, 2021], indicating widespread propagation and usage within the
community.

### A multidisciplinary Delphi process reveals numerous pitfalls in biomedical
image analysis validation

With the aim of creating a comprehensive, reliable collection and future point
of access to biomedical image analysis metric definitions and limitations, we
formed an international multidisciplinary consortium of 62 experts from
various biomedical image analysis-related fields that engaged in a multi-stage
Delphi process [Brown, 1968] for consensus building. Further pitfalls were
crowdsourced through the publication of a dynamic preprint of this work
[Reinke et al., 2021b] as well as a social media campaign, both of which asked
the scientific community for contributions. This approach allowed us to
integrate distributed, cross-domain knowledge on metric-related pitfalls
within a single resource. In total, the process revealed 37 distinct sources
of pitfalls (see Fig. 2). Notably, these pitfall sources (e.g., class
imbalances, uncertainties in the reference, or poor image resolution) can
occur irrespective of a specific imaging modality or application. As a result,
many pitfalls generalize across different problem categories in image
processing (image-level classification, semantic segmentation, object
detection, and instance segmentation), as well as imaging modalities and
domains. A detailed discussion of all pitfalls can be found in Suppl. Note 2.

### A common taxonomy enables domain-agnostic categorization of pitfalls

One of our key objectives was to facilitate information retrieval and provide
structure within this vast topic. Specifically, we wanted to enable
researchers to identify at a glance which metrics are affected by which types
of pitfalls. To this end, we created a comprehensive taxonomy that categorizes
the different pitfalls in a semantic fashion. The taxonomy was created in a
domain-agnostic manner to reflect the generalization of pitfalls across
different imaging domains and modalities. An overview of the taxonomy is
presented in Fig. 2, and the relations between the pitfall categories and
individual metrics can be found in Extended Data Tabs. Extended Data-5. We
distinguish the following three main categories:

Figure 2. Overview of the taxonomy for metric-related pitfalls. Pitfalls can
be grouped into three main categories: [P1] Pitfalls related to the inadequate
choice of the problem category, [P2] pitfalls related to poor metric
selection, and [P3] pitfalls related to poor metric application. [P2] and [P3]
are further split into subcategories. For all categories, pitfall sources are
presented (green), with references to corresponding illustrations of
representative examples. Note that the order in which the pitfall sources are
presented does not correlate with importance.

##### [P1] Pitfalls related to the inadequate choice of the problem category.

A common pitfall lies in the use of metrics for a problem category they are
not suited for because they fail to fulfill crucial requirements of that
problem category, and hence do not reflect the domain interest (Fig. 1). For
instance, popular voxel-based metrics, such as the Dice Similarity Coefficient
(DSC) or Sensitivity, are widely used in image analysis problems, although
they do not fulfill the critical requirement of detecting all objects in a
data set. In a cancer monitoring application they fail to measure instance
progress, i.e., the potential increase in number of lesions (Fig. 1), which
can have serious consequences for the patient. For some problems, there may
even be a lack of matching problem category (Fig. SN 2.4), rendering common
metrics inadequate. We present further examples of pitfalls in this category
in Suppl. Note 2.1.

##### [P2] Pitfalls related to poor metric selection.

Pitfalls of this category occur when a validation metric is selected while
disregarding specific properties of the given research problem or method used
that make this metric unsuitable in the particular context. [P2] can be
further divided into the following four subcategories:

##### [P2.1] Disregard of the domain interest

Commonly, several requirements arise from the domain interest of the
underlying research problem that may clash with particular metric limitations.
For example, if there is particular interest in the structure boundaries, it
is important to know that overlap-based metrics such as the DSC do not take
the correctness of an object’s boundaries into account, as shown in Fig. 4(a).
Similar issues may arise if the structure volume (Fig. SN 2.6) or center(line)
(Fig. SN 2.7) are of particular interest. Other domain interest-related
properties may include an unequal severity of class confusions. This may be
important in an ordinal grading use case, in which the severity of a disease
is categorized by different scores. Predicting a low severity for a patient
that actually suffers from a severe disease should be substantially penalized.
Common classification metrics do not fulfill this requirement. An example is
provided in Fig. 4(b). On pixel level, this property relates to an unequal
severity of over- vs. undersegmentation. In applications such as radiotherapy,
it may be highly relevant whether an algorithm tends to over- or undersegment
the target structure. Common overlap-based metrics, however, do not represent
over- and undersegmentation equally [Yeghiazaryan and Voiculescu, 2018].
Further pitfalls may occur if confidence awareness (Fig. SN 2.8),
comparability across data sets (Fig. SN 2.9), or a cost-benefit analysis (Fig.
SN 2.11) are of particular importance, as illustrated in Suppl. Note 2.2.1.

##### [P2.2] Disregard of the properties of the target structures

For problems that require capturing local properties (object detection,
semantic or instance segmentation), the properties of the target structures to
be localized and/or segmented may have important implications for the choice
of metrics. Here, we distinguish between size-related and shape- and topology-
related pitfalls. Common metrics, for example, are sensitive to structure
sizes, such that single-pixel differences may hugely impact the metric scores,
as shown in Extended Data Fig. 1(a). Shape- and topology-related pitfalls may
relate to the fact that common metrics disregard complex shapes (Extended Data
Fig. 1(b)) or that bounding boxes do not capture the disconnectedness of
structures (Fig. SN 2.16). A high variability of structure sizes (Fig. SN
2.13) and overlapping or touching structures (Fig. SN 2.15) may also influence
metric values. We present further examples of [P2.2] pitfalls in Suppl. Note
2.2.2.

Figure 3. [P1] Pitfalls related to the inadequate choice of the problem
category. Wrong choice of problem category. Effect of using segmentation
metrics for object detection problems. The pixel-level Dice Similarity
Coefficient (DSC) of a prediction recognizing every structure (Prediction 2)
is lower than that of a prediction that only recognizes one of the three
structures (Prediction 1).

##### [P2.3] Disregard of the properties of the data set

Various properties of the data set such as class imbalances (Fig. 5(a)), small
sample sizes (Fig. 5(b)), or the quality of the reference annotations, may
directly affect metric values. Common metrics such as the Balanced Accuracy
(BA), for instance, may yield a very high score for a model that predicts many
False Positive (FP) samples in an imbalanced setting (see Fig. 5(a)). When
only small test data sets are used, common calibration metrics (which are
typically biased estimators) either underestimate or overestimate the true
calibration error of a model (Fig. 5(b)) [Gruber and Buettner, 2022]. On the
other hand, metric values may be impacted by reference annotations (Fig. SN
2.19). Spatial outliers in the reference may have a huge impact on distance-
based metrics such as the Hausdorff Distance (HD) (Fig. 5(c)). Additional
pitfalls may arise from the occurrence of cases with an empty reference
(Extended Data Fig. 2(b)), causing division by zero errors. We present further
examples of [P2.3] pitfalls in Suppl. Note 2.2.3.

##### [P2.4] Disregard of the properties of the algorithm output

Reference-based metrics compare the algorithm output to a reference annotation
to compute a metric score. Thus, the content and format of the prediction are
of high importance when considering metric choice. Overlapping predictions in
segmentation problems, for instance, may return misleading results. In
Extended Data Fig. 2(a), the predictions only overlap to a certain extent, not
representing that the reference instances actually overlap substantially. This
is not detected by common metrics. Another example are empty predictions that
may cause division by zero errors in metric calculations, as illustrated in
Extended Data Fig. 2(b), or the lack of predicted class scores (Fig. SN 2.22).
We present further examples of [P2.4] pitfalls in Suppl. Note 2.2.3.

Figure 4. [P2.1] Disregard of the domain interest. (a) Importance of structure
boundaries. The predictions of two algorithms (Prediction 1/2) capture the
boundary of the given structure substantially differently, but lead to the
exact same Dice Similarity Coefficient (DSC), due to its boundary unawareness.
This pitfall is also relevant for other overlap-based metrics such as
centerline Dice Similarity Coefficient (clDice), pixel-level Fβ Score, and
Intersection over Union (IoU), as well as localization criteria such as
Box/Approx/Mask IoU, Center Distance, Mask IoU ¿ 0, Point inside
Mask/Box/Approx, and Intersection over Reference (IoR). (b) Unequal severity
of class confusions. When predicting the severity of a disease for three
patients in an ordinal classification problem, Prediction 1 assumes a much
lower severity for Patient 3 than actually observed. This critical issue is
overlooked by common metrics (here: Accuracy), which measure no difference to
Prediction 2, which assesses the severity much better. Metrics with pre-
defined weights (here: Expected Cost (EC)) correctly penalize Prediction 1
much more than Prediction 2. This pitfall is also relevant for other counting
metrics, such as Balanced Accuracy (BA), Fβ Score, Positive Likelihood Ratio
(LR+), Matthews Correlation Coefficient (MCC), Net Benefit (NB), Negative
Predictive Value (NPV), Positive Predictive Value (PPV), Sensitivity, and
Specificity. Figure 5. [P2.3] Disregard of the properties of the data set. (a)
High class imbalance. In the case of underrepresented classes, common metrics
may yield misleading values. In the given example, Accuracy and Balanced
Accuracy (BA) have a high score despite the high amount of False Positive (FP)
samples. The class imbalance is only uncovered by metrics considering
predictive values (here: Matthews Correlation Coefficient (MCC)). This pitfall
is also relevant for other counting and multi-threshold metrics such as Area
under the Receiver Operating Characteristic Curve (AUROC), Expected Cost (EC)
(depending on the chosen costs), Positive Likelihood Ratio (LR+), Net Benefit
(NB), Sensitivity, Specificity, and Weighted Cohen’s Kappa (WCK). (b) Small
test set size. The values of the Expected Calibration Error (ECE) depend on
the sample size. Even for a simulated perfectly calibrated model, the ECE will
be substantially greater than zero for small sample sizes [Gruber and
Buettner, 2022]. (c) Imperfect reference standard. A single erroneously
annotated pixel may lead to a large decrease in performance, especially in the
case of the Hausdorff Distance (HD) when applied to small structures. The
Hausdorff Distance 95th Percentile (HD95), on the other hand, was designed to
deal with spatial outliers. This pitfall is also relevant for localization
criteria such as Box/Approx Intersection over Union (IoU) and Point inside
Box/Approx. Further abbreviations: True Positive (TP), False Negative (FN),
True Negative (TN).

##### [P3] Pitfalls related to poor metric application.

Once selected, the metrics need to be applied to an image or an entire data
set. This step is not straightforward and comes with several pitfalls. For
instance, when aggregating metric values over multiple images or patients, a
common mistake is to ignore the hierarchical data structure, such as data from
several hospitals or a varied number of images per patient. We present three
examples of [P3] pitfalls in Fig. 6; for more pitfalls in this category,
please refer to Suppl. Note 2.3. [P3] can further be divided into five
subcategories that are presented in the following paragraphs.

##### [P3.1] Inadequate metric implementation

Metric implementation is, unfortunately, not standardized. As shown by
[Gooding et al., 2022], different researchers typically employ various
different implementations for the same metric, which may yield a substantial
variation in the metric scores. While some metrics are straightforward to
implement, others require more advanced techniques and offer different
possibilities. In the following, we provide some examples for inadequate
metric implementation:

  * •

The method of how identical confidence scores are handled in the computation
of the AP metric may lead to substantial differences in the metric scores.
Microsoft Common Objects in Context (COCO) [Lin et al., 2014], for instance,
processes each prediction individually, while CityScapes [Cordts et al., 2015]
processes all predictions with the same score in one joint step. Fig. 6(a)
provides an example with two predictions having the same confidence score, in
which the final metric scores differ depending on the chosen handling strategy
for identical confidence scores. Similar issues may arise with other curve-
based metrics, such as AUROC, AP, or Free-Response Receiver Operating
Characteristic (FROC) scores (see e.g., [Muschelli, 2020]).

  * •

Metric implementation may be subject to discretization issues such as the
chosen discretization of continuous variables, which may cause differences in
the metric scores, as exemplary illustrated in Fig. SN 2.24.

  * •

For metrics assessing structure boundaries, such as the Average Symmetric
Surface Distance (ASSD), the exact boundary extraction method is not
standardized. Thus, for example, the boundary extraction method implemented by
the Liver Tumor Segmentation (LiTS) challenge [Bilic et al., 2023] and that
implemented by Google DeepMind222https://github.com/deepmind/surface-distance
may produce different metric scores for the ASSD. This is especially critical
for metrics that are sensitive to small contour changes, such as the HD.

  * •

Suboptimal choices of hyperparameters may also lead to metric scores that do
not reflect the domain interest. For example, the choice of a threshold on a
localization criterion (see Fig. SN 2.25) or the chosen hyperparameter for the
Fβ Score will heavily influence the subsequent metric scores [Tran et al.,
2022].

More [P3.1] pitfalls can be found in Suppl. Note 2.3.1.

##### [P3.2] Inadequate metric aggregation

A common pitfall with respect to metric application is to simply aggregate
metric values over the entire data set and/or all classes. As detailed in Fig.
6(b) and Suppl. Note 2.3.2, important information may get lost in this
process, and metric results can be misleading. For example, the popular
TorchMetrics framework calculates the DSC metric by default as a global
average over all pixels in the data set without considering their image or
class of
origin333https://torchmetrics.readthedocs.io/en/stable/classification/dice.html?highlight=dice.
Such a calculation eliminates the possibility of interpreting the final metric
score with respect to individual images and classes. For example, errors in
small structures may be suppressed by correctly segmented larger structures in
other images (see e.g., Fig. SN 2.28). An adequate aggregation scheme is also
crucial for handling hierarchical class structure (Fig. SN 2.29), missing
values (Fig. SN 2.31), and potential biases (Fig. SN 2.30) of the algorithm.
Further [P3.2] pitfalls are shown in Suppl. Note 2.3.2.

##### [P3.3] Inadequate ranking scheme

Rankings are often created to compare algorithm performances. In this context,
several pitfalls pertain to either metric relationships or ranking
uncertainty. For example, to assess different properties of an algorithm, it
is advisable to select multiple metrics and determine their values. However,
the chosen metrics should assess complementary properties and should not be
mathematically related. For example, the DSC and IoU are closely related, so
using both in combination would not provide any additional information over
using either of them individually (Fig. SN 2.32). Note in this context that
unawareness of metric synonyms can equally mislead. Metrics can be known under
different names; for instance, Sensitivity and Recall refer to the same
mathematical formula. Despite this fact potentially appearing trivial, an
analysis of 138 biomedical image analysis challenges [Maier-Hein et al., 2022]
found three challenges that unknowingly used two versions of the same metric
to calculate their rankings. Moreover, rankings themselves may be unstable
(Fig. SN 2.33). [Maier-Hein et al., 2018] and [Wiesenfarth et al., 2021]
demonstrated that rankings are highly sensitive to altering the metric
aggregation operators, the underlying data set, or the general ranking method.
Thus, if the robustness of rankings is disregarded, the winning algorithm may
be identified by chance rather than true superiority.

##### [P3.4] Inadequate metric reporting

A thorough reporting of metric values and aggregates is important both in
terms of transparency and interpretability. However, several pitfalls are to
be avoided in this regard. Notably, different types of visualization may vary
substantially in terms of interpretability, as shown in Figs 6(c). For
example, while a box plot provides basic information, it does not depict the
distribution of metric values. This may conceal important information, such as
specific images on which an algorithm performed poorly. Other pitfalls in this
category relate to the non-determinism of algorithms, which introduces a
natural variability to the results of a neural network, even with fixed seeds
(Fig. 6). This issue is aggravated by inadequate reporting, for instance,
reporting solely the results from the best run instead of proper cross-
validation and reporting of the variability across different runs. Generally,
shortcomings in reporting, such as providing no standard deviation or
confidence intervals in the presented results, are common. Concrete examples
of [P3.4] pitfalls can be found in Suppl. Note 2.3.4.

##### [P3.5] Inadequate interpretation of metric values

Interpreting metric scores and aggregates is an important step for the
analysis of algorithm performances. However, several pitfalls can arise from
the interpretation. In rankings, for example, minor differences in metric
scores may not be relevant from an application perspective but may still yield
better ranks (Fig. SN 2.38). Furthermore, some metrics do not have upper or
lower bounds, or the theoretical bounds may not be achievable in practice,
rendering interpretation difficult (Fig. SN 2.37). More information on
interpretation-based pitfalls can be found in Suppl. Note 2.3.5.

Figure 6. [P3] Pitfalls related to poor metric application. (a) Non-
standardized metric implementation. In the case of the Average Precision (AP)
metric and the construction of the Precision-Recall (PR)-curve, the strategy
of how identical scores (here: confidence score of 0.80 is present twice) are
treated has a substantial impact on the metric scores. Microsoft Common
Objects in Context (COCO) [Lin et al., 2014] and CityScapes [Cordts et al.,
2015] are used as examples. (b) Non-independence of test cases. The number of
images taken from Patient 1 is much higher compared to that acquired from
Patients 2-5. Averaging over all Dice Similarity Coefficient (DSC) values,
denoted by $\varnothing$, results in a high averaged score. Aggregating metric
values per patient reveals much higher scores for Patient 1 compared to the
others, which would have been hidden by simple aggregation. (c) Uninformative
visualization. A single box plot (left) does not give sufficient information
about the raw metric value distribution. Adding the raw metric values as
jittered dots on top (right) adds important information (here: on clusters).
In the case of non-independent validation data, color/shape-coding helps
reveal data clusters.

### The first illustrated common access point to metric definitions and
pitfalls

To underline the importance of a common access point to metric pitfalls, we
conducted a search for individual metric-related pitfalls on the platforms
Google Scholar and Google, with the purpose of determining how many of the
pitfalls identified through our work could be located in existing resources.
We were only able to locate a portion of the pitfalls identified by our
approach in existing research literature (68%) or online resources such as
blog posts (11%; 8% were found in both). Only 27% of the located pitfalls were
presented visually.

Our work now provides this key resource in a highly structured and easily
understandable form. Suppl. Note 2, contains a dedicated illustration for each
of the pitfalls discussed, thus facilitating reader comprehension and making
the information accessible to everyone regardless of their level of expertise.
A further core contribution of our work are the metric profiles presented in
Suppl. Note 2, which, for each metric, summarize the most important
information deemed of particular relevance by the Metrics Reloaded consortium
of the sister work to this publication [Maier-Hein et al., 2022]. The profiles
provide the reader with a compact, at-a-glance overview of each metric and an
enumeration of the limitations and pitfalls identified in the Delphi process
conducted for this work.

## Discussion

Flaws in the validation of biomedical image analysis algorithms significantly
impede the translation of methods into (clinical) practice and undermine the
assessment of scientific progress in the field [Lennerz et al., 2022]. They
are frequently caused by poor choices due to disregarding the specific
properties and limitations of individual validation metrics. The present work
represents the first comprehensive collection of pitfalls and limitations to
be taken into account when using validation metrics in image-level
classification, semantic segmentation, instance segmentation, and object
detection tasks. Our work enables researchers to gain a deep understanding of
and familiarity with both the overall topic and individual metrics by
providing a common access point to previously largely scattered and
inaccessible information — key knowledge they can resort to when conducting
validation of image analysis algorithms. This way, our work aims to disrupt
the current common practice of choosing metrics based on their popularity
rather than their suitability to the underlying research problem. This
practice, which, for instance, often manifests itself in the unreflected and
inadequate use of the DSC, is concerningly prevalent even among prestigious,
high-quality biomedical image analysis competitions (challenges) [Honauer et
al., 2015, Correia and Pereira, 2006, Kofler et al., 2021, Gooding et al.,
2018, Vaassen et al., 2020, Konukoglu et al., 2012, Margolin et al., 2014,
Maier-Hein et al., 2018]. The educational aspect of our work is complemented
by dedicated ’metric profiles’ which detail the definitions and properties of
all metrics discussed. Notably, our work pioneers the examination of AI
validation pitfalls in the biomedical domain, a domain in which they are
arguably more critical than in many others as flaws in biomedical algorithm
validation can directly affect patient wellbeing and safety.

We posited that shortcomings in current common practice are marked by the low
accessibility of information on the pitfalls and limitations of commonly used
validation metrics. A literature search conducted from the point of view of a
researcher seeking information on individual metrics confirmed that the number
of search results far exceeds any amount that could be overseen within
reasonable time and effort, as well as the lack of a common point of entry to
reliable metric information. Even when knowing the specific pitfalls and
related keywords uncovered by our consortium, only a fraction of those
pitfalls could be found in existing literature, indicating the novelty and
added value of our work.

For transparency, several constraints regarding our literature search must be
noted. First, it must be acknowledged that the remarkably high search result
numbers inevitably include duplicates of papers (e.g., the same work in a
conference paper and on arXiv) as well as results that are out of scope (e.g.,
[Carbonell et al., 2006], [Di Sabatino and Corazza, 2012]), in the cited
examples for instance due to a metric acronym (AUC) simultaneously being an
acronym for another entity (a trinucleotide) in a different domain, or the
word ”sensitivity” being used in its common, non-metric meaning. Moreover,
common words used to describe pitfalls such as “problem” or “issue” are by
nature present in many publications discussing any kind of research, rendering
them unusable for a dedicated search, which could, in turn, account for
missing publications that do discuss pitfalls in these terms. Similarly, when
searching for specific pitfalls, many of the returned results containing the
appropriate keywords did not actually refer to metrics or algorithm validation
but to other parts of a model or biomedical problem (e.g., the need for
stratification is commonly discussed with regard to the design of clinical
studies but not with regard to their validation). Character limits in the
Google Scholar search bar further complicate or prevent the use of
comprehensive search strings. Finally, it is both possible and probable that
our literature search did not retrieve all publications or non-peer-reviewed
online resources that mention a particular pitfall, since even extensive
search strings might not cover the particular words used for a pitfall
description.

None of these observations, however, detracts from our hypothesis. In fact,
all of the above observations reinforce our finding that, for any individual
researcher, retrieving information on metrics of interest is difficult to
impossible. In many cases, finding information on pitfalls only appears
feasible if the specific pitfall and its related keywords are exactly known,
which, of course, is not the situation most researchers realistically find
themselves in. Overall accessibility of such vital information, therefore,
currently leaves much to be desired.

Compiling this information through a multi-stage Delphi process allowed us to
leverage distributed knowledge from experts across different biomedical
imaging domains and thus ensure that the resulting illustrated collection of
metric pitfalls and limitations turned out to be both comprehensive and of
maximum practical relevance. Continued proximity of our work to issues
occurring in practical application was achieved through sharing the first
results of this process as a dynamic preprint [Reinke et al., 2021a] with
dedicated calls for feedback, as well as crowdsourcing further suggestions on
social media.

Although their severity and practical consequences might differ between
applications, we found that the pitfalls generalize across different imaging
modalities and application domains. By categorizing them solely according to
their underlying sources, we were able to create an overarching taxonomy that
goes beyond domain-specific concerns and thus enjoys broad applicability.
Given the large number of identified pitfalls, our taxonomy crucially
establishes structure in the topic. Moreover, by relating types of pitfalls to
the respective metrics they apply to and illustrating them, it enables
researchers to gain a deeper, systemic understanding of the causes of metric
failure.

Our complementary Metrics Reloaded recommendation framework, which guides
researchers towards the selection of appropriate validation metrics for their
specific tasks and is introduced in a sister publication to this work [Maier-
Hein et al., 2022], shares the same principle of domain independence. Its
recommendations are based on the creation of a ’problem fingerprint’ that
abstracts from specific domain knowledge and, informed by the pitfalls
discussed here, captures all properties relevant to metric selection for a
specific biomedical problem. In this sister publication, we present
recommendations to avoid the pitfalls presented in this work. Importantly, the
finding that pitfalls generalize and can be categorized in a domain-
independent manner opens up avenues for future expansion of our work to other
fields of ML-based imaging, such as general computer vision (see below), thus
freeing it from its major constraint of exclusively focusing on biomedical
problems.

It is worth mentioning that we only examined pitfalls related to the tasks of
image-level classification, semantic segmentation, instance segmentation, and
object detection, as these can all be considered classification tasks at
different levels (image/object/pixel) and hence share similarities in their
validation. While including a wider range of biomedical problems not
considered classification tasks, such as regression or registration, would
have gone beyond the scope of the present work, we envision this expansion in
future work. Moreover, our work focused on pitfalls related to reference-based
metrics. Including pitfalls pertaining to non-reference-based metrics, such as
metrics that assess speed, memory consumption, or carbon footprint, could be a
future direction to take. Finally, while we aspired to be as comprehensive as
possible in our compilation, we cannot exclude that there are further pitfalls
to be taken into account that the consortium and the participating community
have so far failed to recognize. Should this be the case, our dynamic Metrics
Reloaded online platform, which is currently under development and will
continuously be updated after release, will allow us to easily and
transparently append missed pitfalls. This way, our work can remain a reliable
point of access, reflecting the state of the art at any given moment in the
future. In this context, we note that we explicitly welcome feedback and
further suggestions from the readership of Nature Methods.

The expert consortium was primarily compiled in a way to cover the required
expertise from various fields but also consisted of researchers of different
countries, (academic) ages, roles, and backgrounds (details can be found in
the Methods). It mainly focused on biomedical applications. The pitfalls
presented here are therefore of the highest relevance for biological and
clinical use cases. Their clear generalization across different biomedical
imaging domains, however, indicates broader generalizability to fields such as
general computer vision. Future work could thus see a major expansion of our
scope to AI validation well beyond biomedical research. Regardless of this
possibility, we strongly believe that by raising awareness of metric-related
pitfalls, our work will kick off a necessary scientific debate. Specifically,
we see its potential in inducing the scientific communities in other areas of
AI research to follow suit and investigate pitfalls and common practices
impairing progress in their specific domains.

In conclusion, our work presents the first comprehensive and illustrated
access point to information on validation metric properties and their
pitfalls. We envision it to not only impact the quality of algorithm
validation in biomedical imaging and ultimately catalyze faster translation
into practice, but to raise awareness on common issues and call into question
flawed AI validation practice far beyond the boundaries of the field.

## Methods

### Literature search

The literature search of metric pitfalls and limitations was conducted on the
platform Google Scholar. The checkbox ”include patents” was activated and the
checkbox ”include citations” was deactivated; other default settings were left
unchanged. For each metric, a specific search string using the Boolean
operators OR and AND was generated as follows:

  * •

(Different notations of the metric name, including synonyms and acronyms,
enclosed in quotation marks, respectively, and combined with OR)

  * •

AND ”metric”

  * •

AND (different expressions pertaining to the concept of pitfalls, limitations
and flaws, enclosed in quotation marks, respectively, and combined with OR)

For example, the following search string was used for the literature search of
DSC pitfalls: ("DSC" OR "Dice Similarity Coefficient" OR "Sørensen–Dice
coefficient" OR "F1 score" OR "DCE") AND "metric" AND ("pitfall" OR
"limitation" OR "caveat" OR "drawback" OR "shortcoming" OR "weakness" OR
"flaw" OR "disadvantage” OR "suffer").

A second literature search dedicated to the pitfalls collected during the
Delphi process was conducted on the platforms Google Scholar and Google. This
search served the purpose of determining how many of the proposed pitfalls
could be found in either existing research literature or online resources such
as blogs, assuming that the issue is already roughly known to the person
conducting the search. We further determined whether or not a found pitfall
was presented in a visual manner. We analyzed the first three results pages
(corresponding to thirty results) from each search platform and excluded our
own previous work on metric pitfalls from the analysis.

### Delphi process

The collection of pitfalls was achieved via a multi-stage Delphi process
conducted among an international expert consortium comprised of more than 60
biomedical image analysis experts, as well as community feedback. A Delphi
process is a structured group communication process that serves to pool
opinions from an expert panel via a series of individual interrogations,
usually in the form of questionnaires, interspersed with feedback from the
respondents [Brown, 1968]. The technique is widely used for building consensus
among experts in medicine, particularly in the development of best practices
in areas where evidence may be limited, conflicting, or absent [Nasa et al.,
2021]. Expert selection was initially based on membership in major relevant
societies such as the Biomedical Image Analysis ChallengeS (BIAS) initiative,
the Medical Open Network for Artificial Intelligence (MONAI) Working Group for
Evaluation, Reproducibility and Benchmarks, and the MICCAI Special Interest
Group for Challenges (previously MICCAI board working group), as well as a
track record of expertise in the areas of metrics, challenges and/or best
practices. To reflect as broad a range of application areas and metric
pitfalls as possible, the number of consortium members was increased
throughout the process to a final number of 62 members. The Delphi process
comprised four surveys. Each survey was developed by the coordinating team of
the process and sent out to the remaining members of the consortium. Upon
completion, the coordinating team then analyzed the results and iteratively
refined the list of pitfalls. The main stages of the compilation and consensus
building process are detailed in the following:

  1. (1)

Compilation of pitfall sources: The primary purpose of the first survey was
obtaining agreement on sources of pitfalls.

  2. (2)

Collection of pitfalls: The following survey specifically asked for concrete
pitfalls in the presence of those problem characteristics.

  3. (3)

Community feedback: The proposed list of pitfalls was further complemented by
social media-based feedback from the general scientific community.

  4. (4)

Final agreement on pitfalls: The subsequent survey served to obtain consensus
agreement on which pitfalls to include. For each pitfall, it asked whether the
pitfall should be included. In addition, the experts were given the
opportunity to provide feedback on each pitfall and to suggest further
pitfalls. The final collection of pitfalls was illustrated and all metric
values were verified by two independent observers.

  5. (5)

Creation of taxonomy: The collected pitfalls were analyzed and a taxonomy was
created. In the final survey, approval of the consortium for the structure and
phrasing of the taxonomy and the assignment of specific pitfalls to the
taxonomy was obtained.

### Expert consortium

The expert consortium consisted of a total of 70 researchers (70% male, 30%
female) from a total of 65 institutions. The majority of experts (50%) were
professors, followed by postdoctoral researchers (39%). The median h-index of
the consortium was 31.5 (mean: 36; minimum: 6; maximum: 113) and the median
academic age was 18 years (mean: 19; minimum: 3; max: 42). Experts were from
19 countries and 5 continents. 60% of experts had a technical, 6% a clinical,
3% a biological, and 23% a mixed background. Of the 65 institutions, we could
identify the number of employees for 89%. Of those, the majority of
institutions had a size between 1,000 and 10,000 employees (57%), followed by
even larger institutions between 10,000 and 100,000 employees (22%), and
smaller institutions below 1,000 employees (20%). Only a small portion of
institutions were above 100,000 employees (2%).

## Extended Data

Extended Data Fig. 1. [P2.2] Disregard of the properties of the target
structures. (a) Small structure sizes. The predictions of two algorithms
(Prediction 1/2) differ in only a single pixel. In the case of the small
structure (bottom row), this has a substantial effect on the corresponding
Dice Similarity Coefficient (DSC) metric value (similar for the Intersection
over Union (IoU)). This pitfall is also relevant for other overlap-based
metrics such as the centerline Dice Similarity Coefficient (clDice), and
localization criteria such as Box/Approx/Mask IoU and Intersection over
Reference (IoR). (b) Complex structure shapes. Common overlap-based metrics
(here: DSC) are unaware of complex structure shapes and treat Predictions 1
and 2 equally. The clDice uncovers the fact that Prediction 1 misses the fine-
granular branches of the reference and favors Prediction 2, which focuses on
the center line of the object. This pitfall is also relevant for other
overlap-based such as metrics IoU and pixel-level Fβ Score as well as
localization criteria such as Box/Approx/Mask IoU, Center Distance, Mask IoU ¿
0, Point inside Mask/Box/Approx, and IoR.

Extended Data Fig. 2. [P2.4] Disregard of the properties of the algorithm
output. (a) Possibility of overlapping predictions. If multiple structures of
the same type can be seen within the same image (here: reference objects R1
and R2), it is generally advisable to phrase the problem as instance
segmentation (InS; right) rather than semantic segmentation (SemS; left). This
way, issues with boundary-based metrics resulting from comparing a given
structure boundary to the boundary of the wrong instance in the reference can
be avoided. In the provided example, the distance of the red boundary pixel to
the reference, as measured by a boundary-based metric in SemS problems, would
be zero, because different instances of the same structure cannot be
distinguished. This problem is overcome by phrasing the problem as InS. In
this case, (only) the boundary of the matched instance (here: R2) is
considered for distance computation. (b) Possibility of empty prediction or
reference. Each column represents a potential scenario for per-image
validation of objects, categorized by whether True Positives (TPs), False
Negatives (FNs), and False Positives (FPs) are present (n ¿ 0) or not (n = 0)
after matching/assignment. The sketches on the top showcase each scenario when
setting ”n ¿ 0” to ”n = 1”. For each scenario, Sensitivity, Positive
Predictive Value (PPV), and the F${}_{\text{1}}$ Score are calculated. Some
scenarios yield undefined values (Not a Number (NaN)).

Extended Data Tab. 1. Overview of pitfall sources for image-level
classification metrics ((a): counting metrics, (b): multi-threshold metrics)
related to poor metric selection [P2]. A warning sign indicates a potential
pitfall for the metric in the corresponding column, in case the property
represented by the respective row holds true. Comprehensive illustrations of
pitfalls are available in Suppl. Note 2. A comprehensive list of pitfalls is
provided separately for each metrics in the metrics cheat sheets (Suppl. Note
3). Note that we only list sources of pitfalls relevant to the considered
metrics. Other sources of pitfalls are neglected for this table. (a) Counting
metrics. Considered metrics: Accuracy (Fig. SN 3.40), Balanced Accuracy (BA)
(Fig. SN 3.41), Expected Cost (EC) (Fig. SN 3.44), Fβ Score (Fig.SN 3.45),
Matthews Correlation Coefficient (MCC) (Fig. SN 3.48), Net Benefit (NB) (Fig.
SN 3.49), Negative Predictive Value (NPV) (Fig. SN 3.50), Positive Likelihood
Ratio (LR+) (Fig. SN 3.52), Positive Predictive Value (PPV) (Fig. SN 3.53),
Sensitivity (Sens) (Fig. SN 3.54), Specificity (Spec) (Fig. SN 3.55), Weighted
Cohen’s Kappa (WCK) (Fig. SN 3.56).

Source of potential pitfall |  Accuracy |  BA |  EC |  Fβ Score |  LR+ |  MCC |  NB |  PPV/ NPV |  Sens/ Spec |  WCK
---|---|---|---|---|---|---|---|---|---|---
Importance of confidence awareness |  * |  * |  * |  * |  * |  * |  * |  * |  * |  *
Importance of comparability across data sets |  (Fig. SN 2.9) |  |  ** (Fig. SN 2.9) |  (Fig. SN 2.9) |  |  (Fig. SN 2.9) |  (Fig. SN 2.9) |  (Fig. SN 2.9) |  |  (Fig. SN 2.9)
Unequal severity of class confusions |  (Fig. 4b) |  (Fig. 4b) |  |  *** (Fig. 4b) |  (Fig. 4b) |  (Fig. 4b) |  |  (Fig. 4b) |  (Fig. 4b) |
Importance of cost-benefit analysis |  (Fig. SN 2.11) |  (Fig. SN 2.11) |  |  *** (Fig. SN 2.11) |  (Fig. SN 2.11) |  (Fig. SN 2.11) |  |  (Fig. SN 2.11) |  (Fig. SN 2.11) |
High class imbalance |  (Figs. 5a, SN 2.17) |  (Fig. 5a) |  ** (Fig. 5a) |  |  (Fig. 5a) |  |  (Figs. 5a, SN 2.17) |  NPV: (Figs. 5a, SN 2.17) |  (Sens: Fig. 5a, Spec: Figs. 5a, SN 2.17) |  (Figs. 5a, SN 2.17)
Small test set size |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18)
* Discrimination metrics do not assess whether the predicted class scores reflect the confidence of the classifier. This is typically achieved with
additional calibration metrics, which come with their own pitfalls (see Figs.
SN 2.8 and SN 2.24, Extended Data Fig. 1b and the metric profiles in Suppl.
Note 3.2).
** The weights in EC can be adjusted to avoid this pitfall.
*** The hyperparameter $\beta$ can be used as a penalty for class confusions
in the binary case. This property is not applicable to multi-class problems.
|  |  |  |  |  |  |  |  |  |

(b) Multi-threshold metrics. Considered metrics: Area under the Receiver Operating Characteristic Curve (AUROC) (Fig. SN 3.57) and Average Precision (AP) (Fig. SN 3.58). Source of potential pitfall |  AP |  AUROC
---|---|---
Importance of confidence awareness |  * |  *
Importance of comparability across data sets |  (Fig. SN 2.9) |
High class imbalance |  |  (Fig. 5a)
Small test set size |  (Fig. SN 2.18) |  (Fig. SN 2.18)
Lack of predicted class scores |  (Fig. SN 2.22) |  (Fig. SN 2.22)
* Discrimination metrics do not assess whether the predicted class scores reflect the confidence of the classifier. This is typically achieved with
additional calibration metrics, which come with their own pitfalls (see Figs.
SN 2.8 and SN 2.24, Extended Data Fig. 1b and the metric profiles in Suppl.
Note 3.2).

Extended Data Tab. 2. Overview of pitfall sources for semantic segmentation
metrics ((a): overlap-based metrics, (b): boundary-based metrics) related to
poor metric selection [P2]. A warning sign indicates a potential pitfall for
the metric in the corresponding column, in case the property represented by
the respective row holds true. Comprehensive illustrations of pitfalls are
available in Suppl. Note 2. A comprehensive list of pitfalls is provided
separately for each metrics in the metrics cheat sheets (Suppl. Note 3). Note
that we only list sources of pitfalls relevant to the considered metrics.
Other sources of pitfalls are neglected for this table.

(a) Overlap-based metrics. Considered metrics: Considered metrics: centerline Dice Similarity Coefficient (clDice) (Fig. SN 3.42), Dice Similarity Coefficient (DSC) (Fig. SN 3.43), Fβ Score (Fig. SN 3.45), Intersection over Union (IoU) (Fig. SN 3.47). Source of potential pitfall |  clDice |  DSC/IoU |  Fβ Score
---|---|---|---
Importance of structure boundaries |  (Fig. 4a) |  (Fig. 4a) |  (Fig. 4a)
Importance of structure center(line) |  |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b)
Unequal severity of class confusions |  (Fig. SN 2.10) |  (Fig. SN 2.10) |
Small structure sizes |  (Fig. SN 2.12 , Extended Data Fig. 1a) |  (Fig. SN 2.12 , Extended Data Fig. 1a) |  (Fig. SN 2.12 , Extended Data Fig. 1a)
High variability of structure sizes |  (Fig. SN 2.13) |  (Fig. SN 2.13) |  (Fig. SN 2.13)
Complex structure shapes |  |  (Fig. SN 2.14) |  (Fig. SN 2.14)
Occurrence of overlapping or touching structures |  (Fig. SN 2.15) |  (Fig. SN 2.15) |  (Fig. SN 2.15)
Imperfect reference standard |  |  (Fig. SN 2.19) |  (Fig. SN 2.19)
Occurrence of cases with an empty reference |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20))
Possibility of empty prediction |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20))
Possibility of overlapping predictions |  (Fig. SN 2.21, Extended Data Fig. 2a) |  (Fig. SN 2.21, Extended Data Fig. 2a) |  (Fig. SN 2.21, Extended Data Fig. 2a)
|  |  |

(b) Boundary-based metrics. Considered metrics: Average Symmetric Surface Distance (ASSD) (Fig. SN 3.60), Boundary Intersection over Union (Boundary IoU) (Fig. SN 3.61), Hausdorff Distance (HD) (Fig. SN 3.62), Hausdorff Distance 95th Percentile (HD95) (Fig. SN 3.65), Mean Average Surface Distance (MASD) (Fig. SN 3.63), Normalized Surface Distance (NSD) (Fig. SN 3.64). Source of potential pitfall |  ASSD |  Boundary IoU |  HD |  HD95 |  MASD |  NSD
---|---|---|---|---|---|---
Importance of structure volume |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6)
Importance of structure center(line) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b)
Occurrence of overlapping or touching structures |  (Fig. SN 2.15) |  (Fig. SN 2.15) |  (Fig. SN 2.15) |  (Fig. SN 2.15) |  (Fig. SN 2.15) |  (Fig. SN 2.15)
Imperfect reference standard |  (Figs. 5c, SN 2.17) |  (Figs. 5c, SN 2.17) |  (Figs. 5c, SN 2.17) |  (Figs. 5c*, SN 2.17) |  (Figs. 5c, SN 2.17) |
Occurrence of cases with an empty reference |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20))
Possibility of empty prediction |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20)) |  (Fig. SN 2.20))
Possibility of overlapping predictions |  (Fig. SN 2.21, Extended Data Fig. 2a) |  (Fig. SN 2.21, Extended Data Fig. 2a) |  (Fig. SN 2.21, Extended Data Fig. 2a) |  (Fig. SN 2.21, Extended Data Fig. 2a) |  (Fig. SN 2.21, Extended Data Fig. 2a) |  (Fig. SN 2.21, Extended Data Fig. 2a)
* Can be mitigated by the choice of the percentile.

Extended Data Tab. 3. Overview of sources of pitfalls for object detection
metrics ((a): detection metrics, (b): localization criteria) related to poor
metric selection [P2]. A warning sign indicates a potential pitfall for the
metric in the corresponding column, in case the property represented by the
respective row holds true. Comprehensive illustrations of pitfalls are
available in Suppl. Note 2. A comprehensive list of pitfalls is provided
separately for each metrics in the metrics cheat sheets (Suppl. Note 3). Note
that we only list sources of pitfalls relevant to the considered metrics.
Other sources of pitfalls are neglected for this table.

(a) Detection metrics. Considered counting metrics: Fβ Score (Fig. SN 3.45), Positive Predictive Value (PPV) (Fig. SN 3.53), Sensitivity (Sens) (Fig. SN 3.54). Considered multi-threshold metrics: Average Precision (AP) (Fig. SN 3.58) and Free-Response Receiver Operating Characteristic (FROC) (Fig. SN 3.59). Source of potential pitfall |  Fβ Score |  PPV |  Sens |  AP |  FROC Score
---|---|---|---|---|---
Unequal severity of class confusions |  * (Fig. 4b) |  (Fig. 4b) |  (Fig. 4b) |  (Fig. 4b) |  (Fig. 4b)
High class imbalance |  |  |  (Fig. 5a) |  |
Small test set size |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18)
Occurrence of cases with an empty reference |  (Fig. SN 2.20, Extended Data Fig. 2b) |  (Fig. SN 2.20, Extended Data Fig. 2b) |  (Fig. SN 2.20, Extended Data Fig. 2b) |  (Fig. SN 2.20, Extended Data Fig. 2b) |  (Fig. SN 2.20, Extended Data Fig. 2b)
Possibility of empty prediction |  (Fig. SN 2.20, Extended Data Fig. 2b) |  (Fig. SN 2.20, Extended Data Fig. 2b) |  (Fig. SN 2.20, Extended Data Fig. 2b) |  (Fig. SN 2.20, Extended Data Fig. 2b) |  (Fig. SN 2.20, Extended Data Fig. 2b)
Lack of predicted class scores |  |  |  |  (Fig. SN 2.22) |  (Fig. SN 2.22)
* The hyperparameter $\beta$ can be used as a penalty for class confusions in the binary case.
This property is not applicable to multi-class problems.
|  |  |  |  |

(b) Localization criteria. Considered localization criteria: Box/Approx Intersection over Union (IoU) (Fig. SN 3.76), Center Distance (Fig. SN 3.74), Mask IoU ¿ 0 (Fig. SN 3.77), and Point inside Mask/ Box/ Approx (Fig. SN 3.78). Source of potential pitfall |  Box/ Approx IoU |  Center Distance |  Mask IoU ¿ 0 |  Point inside Mask/ Box/ Approx
---|---|---|---|---
Importance of structure boundaries |  (Fig. 5a) |  (Fig. 5a) |  (Fig. 5a) |  (Fig. 5a)
Importance of structure volume |  |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6)
Importance of structure center(line) |  (Fig.SN 2.7, Extended Data Fig. 1b) |  |  (Fig.SN 2.7, Extended Data Fig. 1b) |  (Fig.SN 2.7, Extended Data Fig. 1b)
Unequal severity of class confusions |  (Fig. SN 2.10) |  (Fig. SN 2.10)* |  (Fig. SN 2.10) |  (Fig. SN 2.10)*
Small structure sizes |  (Fig. SN 2.12, Extended Data Fig. 1a) |  |  |
Complex structure shapes |  (Figs. SN 2.13, SN 2.16) |  (Fig. SN 2.13) |  (Fig. SN 2.13) |  (Fig. SN 2.13)
Occurrence of disconnected structures |  (Fig. SN 2.16) |  |  |  Point inside Box:  (Fig. SN 2.16)
Imperfect reference standard |  (Fig. 5c) |  |  |
* Criterion implies point prediction, thus overlap assessment is not applicable.

Extended Data Tab. 4. Overview of sources of pitfalls for instance
segmentation metrics (Part 1) ((a): detection metrics, (b): localization
criteria) related to poor metric selection [P2]. A warning sign indicates a
potential pitfall for the metric in the corresponding column, in case the
property represented by the respective row holds true. Comprehensive
illustrations of pitfalls are available in Suppl. Note 2. A comprehensive list
of pitfalls is provided separately for each metrics in the metrics cheat
sheets (Suppl. Note 3). Note that we only list sources of pitfalls relevant to
the considered metrics. Other sources of pitfalls are neglected for this
table.

(a) Detection metrics. Considered counting metrics: Fβ Score (Fig. SN 3.45), Positive Predictive Value (PPV) (Fig. SN 3.53), Panoptic Quality (PQ) (Fig. SN 3.51), Sensitivity (Sens) (Fig. SN 3.54). Considered multi-threshold metrics: Average Precision (AP) (Fig. SN 3.58) and Free-Response Receiver Operating Characteristic (FROC) (Fig. SN 3.59). Source of potential pitfall |  Fβ Score |  PPV |  PQ |  Sens |  AP |  FROC Score
---|---|---|---|---|---|---
Unequal severity of class confusions |  * (Fig. 4b) |  (Fig. 4b) |  (Fig. 4b) |  (Fig. 4b) |  |
High class imbalance |  |  |  |  (Fig. 5a) |  |
Small test set size |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18) |  (Fig. SN 2.18)
Lack of predicted class scores |  |  |  |  |  (Fig. SN 2.22) |  (Fig. SN 2.22)
* The hyperparameter $\beta$ can be used as a penalty for class confusions in the binary case.
This property is not applicable to multi-class problems.
|  |  |  |  |  |

(b) Localization criteria. Considered localization criteria: Boundary Intersection over Union (Boundary IoU) (Fig. SN 3.73), Intersection over Reference (IoR) (Fig. SN 3.75), Mask IoU (Fig. SN 3.47). Source of potential pitfall |  Boundary IoU |  IoR |  Mask IoU
---|---|---|---
Importance of structure boundaries |  |  (Fig. 4a) |  (Fig. 4a)
Importance of structure volume |  (Fig. SN 2.4) |  |
Importance of structure center(line) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b)
Unequal severity of class confusions |  (Fig. SN 2.10) |  (Fig. SN 2.10) |  (Fig. SN 2.10)
Small structure sizes |  |  (Fig. SN 2.12 , Extended Data Fig. 1a) |  (Fig. SN 2.12 , Extended Data Fig. 1a)
Complex structure shapes |  |  (Fig. SN 2.14) |  (Fig. SN 2.12)
Imperfect reference standard |  (Fig. SN 2.19) |  (Fig. SN 2.19) |  (Fig. SN 2.19)
|  |  |

Extended Data Tab. 5. Overview of sources of pitfalls for instance
segmentation metrics (Part 2) ((a) per instance segmentation overlap-based
metrics, (b) per instance segmentation boundary-based metrics) related to poor
metric selection [P2]. A warning sign indicates a potential pitfall for the
metric in the corresponding column, in case the property represented by the
respective row holds true. Comprehensive illustrations of pitfalls are
available in Suppl. Note 2. Note that we only list sources of pitfalls
relevant to the considered metrics. Other sources of pitfalls are neglected
for this table.

(a) Per instance segmentation overlap-based metrics. Considered metrics: centerline Dice Similarity Coefficient (clDice) (Fig. SN 3.42), Dice Similarity Coefficient (DSC) (Fig. SN 3.43), Fβ Score (Fig. SN 3.45), Intersection over Union (IoU) (Fig. SN 3.47). Source of potential pitfall |  clDice |  DSC/IoU |  Fβ Score
---|---|---|---
Importance of structure boundaries |  (Fig. 4a) |  (Fig. 4a) |  (Fig. 4a)
Importance of structure center(line) |  |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b)
Unequal severity of class confusions |  (Fig. SN 2.10) |  (Fig. SN 2.10) |
Small structure sizes |  (Fig. SN 2.12 , Extended Data Fig. 1a) |  (Fig. SN 2.12 , Extended Data Fig. 1a) |  (Fig. SN 2.12 , Extended Data Fig. 1a)
Complex structure shapes |  |  (Fig. SN 2.14) |  (Fig. SN 2.14)
Imperfect reference standard |  |  (Fig. SN 2.19) |  (Fig. SN 2.19)
|  |  |

(b) Per instance segmentation boundary-based metrics. Considered metrics: Average Symmetric Surface Distance (ASSD) (Fig. SN 3.60), Boundary Intersection over Union (Boundary IoU) (Fig. SN 3.61), Hausdorff Distance (HD) (Fig. SN 3.62), Hausdorff Distance 95th Percentile (HD95) (Fig. SN 3.65), Mean Average Surface Distance (MASD) (Fig. SN 3.63), Normalized Surface Distance (NSD) (Fig. SN 3.64). Source of potential pitfall |  ASSD |  Boundary IoU |  HD |  HD95 |  MASD |  NSD
---|---|---|---|---|---|---
Importance of structure volume |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6) |  (Fig. SN 2.6)
Importance of structure center(line) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b) |  (Fig. SN 2.7, Extended Data Fig. 1b)
Imperfect reference standard |  (Figs. 5c, SN 2.19) |  (Figs. 5c, SN 2.19) |  (Figs. 5c, SN 2.19) |  (Figs. 5c*, SN 2.19) |  (Figs. 5c, SN 2.19) |
* Can be mitigated by the choice of the percentile.

## Code Availability Statement

We provide reference implementations for all Metrics Reloaded metrics within
the MONAI open-source framework. They are accessible at
https://github.com/Project-MONAI/MetricsReloaded.

## Acknowledgements

This work was initiated by the Helmholtz Association of German Research
Centers in the scope of the Helmholtz Imaging Incubator (HI), the MICCAI
Special Interest Group for biomedical image analysis challenges, and the
benchmarking working group of the MONAI initiative. It has received funding
from the European Research Council (ERC) under the European Union’s Horizon
2020 research and innovation programme (grant agreement No. [101002198],
NEURAL SPICING) and the Surgical Oncology Program of the National Center for
Tumor Diseases (NCT) Heidelberg. It was further supported in part by the
Intramural Research Program of the National Institutes of Health Clinical
Center as well as by the National Cancer Institute (NCI) and the National
Institute of Neurological Disorders and Stroke (NINDS) of the National
Institutes of Health (NIH), under award numbers NCI:U01CA242871 and
NINDS:R01NS042645. The content of this publication is solely the
responsibility of the authors and does not represent the official views of the
NIH. T.A. acknowledges the Canada Institute for Advanced Research (CIFAR) AI
Chairs program, the Natural Sciences and Engineering Research Council of
Canada. F.B. was co-funded by the European Union (ERC, TAIPO, 101088594).
Views and opinions expressed are however those of the authors only and do not
necessarily reflect those of the European Union or the European Research
Council. Neither the European Union nor the granting authority can be held
responsible for them. M.J.C. acknowledges funding from Wellcome/EPSRC Centre
for Medical Engineering (WT203148/Z/16/Z), the Wellcome Trust
(WT213038/Z/18/Z), and the InnovateUK funded London AI Centre for Value-Based
Healthcare. J.C. is supported by the Federal Ministry of Education and
Research (BMBF) under the funding reference 161L0272. V.C. acknowledges
funding from NovoNordisk Foundation (NNF21OC0068816) and Independent Research
Council Denmark (1134-00017B). B.A.C. was supported by NIH grant P41 GM135019
and grant 2020-225720 from the Chan Zuckerberg Initiative DAF, an advised fund
of the Silicon Valley Community Foundation. G.S.C. was supported by Cancer
Research UK (programme grant: C49297/A27294). M.M.H. is supported by the
Natural Sciences and Engineering Research Council of Canada
(RGPIN-2022-05134). A.Ka. is supported by French State Funds managed by the
“Agence Nationale de la Recherche (ANR)” - “Investissements d’Avenir”
(Investments for the Future), Grant ANR-10-IAHU-02 (IHU Strasbourg). M.K. was
funded by the Ministry of Education, Youth and Sports of the Czech Republic
(Project LM2018129). Ta.K. was supported in part by 4UH3-CA225021-03,
1U24CA180924-01A1, 3U24CA215109-02, and 1UG3-CA225-021-01 grants from the
National Institutes of Health. G.L. receives research funding from the Dutch
Research Council, the Dutch Cancer Association, HealthHolland, the European
Research Council, the European Union, and the Innovative Medicine Initiative.
S.M.R. wishes to acknowledge the Allen Institute for Cell Science founder Paul
G. Allen for his vision, encouragement and support. M.R is supported by
Innosuisse grant number 31274.1 and Swiss National Science Foundation Grant
Number 205320_212939. C.H.S. is supported by an Alzheimer’s Society Junior
Fellowship (AS-JF-17-011). R.M.S. is supported by the Intramural Research
Program of the NIH Clinical Center. A.T. acknowledges support from Academy of
Finland (Profi6 336449 funding program), University of Oulu strategic funding,
Finnish Foundation for Cardiovascular Research, Wellbeing Services County of
North Ostrobothnia (VTR project K62716), and Terttu foundation. S.A.T.
acknowledges the support of Canon Medical and the Royal Academy of Engineering
and the Research Chairs and Senior Research Fellowships scheme (grant
RCSRF1819\8\25). B.V.C. was supported by Research Foundation Flanders (FWO
grant G097322N) and Internal Funds KU Leuven (grant C24M/20/064).

We would like to thank Peter Bankhead, Gary S. Collins, Robert Haase, Fred
Hamprecht, Alan Karthikesalingam, Hannes Kenngott, Peter Mattson, David Moher,
Bram Stieltjes, and Manuel Wiesenfarth for the fruitful discussions on this
work.

We would like to thank Sandy Engelhardt, Sven Koehler, M. Alican Noyan, Gorkem
Polat, Hassan Rivaz, Julian Schroeter, Anindo Saha, Lalith Sharan, Peter
Hirsch, and Matheus Viana for suggesting additional illustrations that can be
found in [Reinke et al., 2021a].

## Competing Interests

The authors declare the following competing interests: F.B. is an employee of
Siemens AG (Munich, Germany). B.v.G. is a shareholder of Thirona (Nijmegen,
NL). B.G. is an employee of HeartFlow Inc (California, USA) and Kheiron
Medical Technologies Ltd (London, UK). M.M.H. received an Nvidia GPU Grant.
Th. K. is an employee of Lunit (Seoul, South Korea). G.L. is on the advisory
board of Canon Healthcare IT (Minnetonka, USA) and is a shareholder of Aiosyn
BV (Nijmegen, NL). Na.R. is the founder and CSO of Histofy (New York, USA).
Ni.R. is an employee of Nvidia GmbH (Munich, Germany). J.S.-R. reports funding
from GSK (Heidelberg, Germany), Pfizer (New York, USA) and Sanofi (Paris,
France) and fees from Travere Therapeutics (California, USA), Stadapharm (Bad
Vilbel, Germany), Astex Therapeutics (Cambridge, UK), Pfizer (New York, USA),
and Grunenthal (Aachen, Germany). R.M.S. receives patent royalties from iCAD
(New Hampshire, USA), ScanMed (Nebraska, USA), Philips (Amsterdam, NL),
Translation Holdings (Alabama, USA) and PingAn (Shenzhen, China); his lab
received research support from PingAn through a Cooperative Research and
Development Agreement. S.A.T. receives financial support from Canon Medical
Research Europe (Edinburgh, Scotland).

## References

  * Asgari Taghanaki et al. [2021] Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh.  Deep semantic segmentation of natural and medical images: a review.  _Artificial Intelligence Review_ , 54(1):137–178, 2021.
  * Attia [2003] John Attia.  Moving beyond sensitivity and specificity: using likelihood ratios to help interpret diagnostic tests.  _Australian prescriber_ , 26(5):111–113, 2003.
  * Bai and Urtasun [2017] Min Bai and Raquel Urtasun.  Deep watershed transform for instance segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5221–5229, 2017.
  * Bamira and Picard [2018] D Bamira and MH Picard.  Imaging: Echocardiology—assessment of cardiac structure and function.  _Elsevier_ , 2018.
  * Bandos et al. [2009] Andriy I Bandos, Howard E Rockette, Tao Song, and David Gur.  Area under the free-response roc curve (froc) and a related summary index.  _Biometrics_ , 65(1):247–256, 2009.
  * Beneš and Zitová [2015] Miroslav Beneš and Barbara Zitová.  Performance evaluation of image segmentation algorithms on microscopic image data.  _Journal of microscopy_ , 257(1):65–85, 2015.
  * Bilic et al. [2023] Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al.  The liver tumor segmentation benchmark (lits).  _Medical Image Analysis_ , 84:102680, 2023.
  * Bishop and Nasrabadi [2006] Christopher M Bishop and Nasser M Nasrabadi.  _Pattern recognition and machine learning_ , volume 4.  Springer, 2006.
  * Brown [1968] Bernice B Brown.  Delphi process: a methodology used for the elicitation of opinions of experts.  Technical report, Rand Corp Santa Monica CA, 1968.
  * Cao et al. [2020] Chang Cao, Davide Chicco, and Michael M Hoffman.  The mcc-f1 curve: a performance evaluation technique for binary classification.  _arXiv preprint arXiv:2006.11278_ , 2020.
  * Carbonell et al. [2006] Alberto Carbonell, Marcos De la Pena, Ricardo Flores, and Selma Gago.  Effects of the trinucleotide preceding the self-cleavage site on eggplant latent viroid hammerheads: differences in co-and post-transcriptional self-cleavage may explain the lack of trinucleotide auc in most natural hammerheads.  _Nucleic acids research_ , 34(19):5613–5622, 2006.
  * Chen et al. [2020] Jianxu Chen, Liya Ding, Matheus P Viana, HyeonWoo Lee, M Filip Sluezwski, Benjamin Morris, Melissa C Hendershott, Ruian Yang, Irina A Mueller, and Susanne M Rafelski.  The allen cell and structure segmenter: a new open source toolkit for segmenting 3d intracellular structures in fluorescence microscopy images.  _BioRxiv_ , page 491035, 2020.
  * Cheng et al. [2021] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov.  Boundary iou: Improving object-centric image segmentation evaluation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15334–15342, 2021.
  * Chicco and Jurman [2020] Davide Chicco and Giuseppe Jurman.  The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.  _BMC genomics_ , 21(1):1–13, 2020.
  * Chicco et al. [2021] Davide Chicco, Niklas Tötsch, and Giuseppe Jurman.  The matthews correlation coefficient (mcc) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation.  _BioData mining_ , 14(1):1–22, 2021.  The manuscript addresses the challenge of evaluating binary classifications. It compares MCC to other metrics, explaining their mathematical relationships and providing use cases where MCC offers more informative results.
  * Chinchor [1992] Nancy Chinchor.  Muc-4 evaluation metrics.  In _Proceedings of the 4th Conference on Message Understanding_ , MUC4 ’92, page 22–29, USA, 1992. Association for Computational Linguistics.  ISBN 1558602739.  doi: 10.3115/1072064.1072067.  URL https://doi.org/10.3115/1072064.1072067.
  * Cohen [1960] Jacob Cohen.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ , 20(1):37–46, 1960.
  * Cordts et al. [2015] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset.  In _CVPR Workshop on The Future of Datasets in Vision_ , 2015.
  * Correia and Pereira [2006] Paulo Correia and Fernando Pereira.  Video object relevance metrics for overall segmentation quality evaluation.  _EURASIP Journal on Advances in Signal Processing_ , 2006:1–11, 2006.
  * Cybenko et al. [1998] George Cybenko, Dianne P O’Leary, and Jorma Rissanen.  _The Mathematics of Information Coding, Extraction and Distribution_ , volume 107.  Springer Science & Business Media, 1998.
  * Davis and Goadrich [2006] Jesse Davis and Mark Goadrich.  The relationship between precision-recall and roc curves.  In _Proceedings of the 23rd international conference on Machine learning_ , pages 233–240, 2006.
  * De Brabandere et al. [2017] Bert De Brabandere, Davy Neven, and Luc Van Gool.  Semantic instance segmentation with a discriminative loss function.  _arXiv preprint arXiv:1708.02551_ , 2017.
  * De Fauw et al. [2018] Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O’Donoghue, Daniel Visentin, et al.  Clinically applicable deep learning for diagnosis and referral in retinal disease.  _Nature medicine_ , 24(9):1342–1350, 2018.
  * Delgado and Tibau [2019] Rosario Delgado and Xavier-Andoni Tibau.  Why cohen’s kappa should be avoided as performance measure in classification.  _PloS one_ , 14(9):e0222916, 2019.
  * DeLong et al. [1988] Elizabeth R DeLong, David M DeLong, and Daniel L Clarke-Pearson.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.  _Biometrics_ , pages 837–845, 1988.
  * Di Sabatino and Corazza [2012] Antonio Di Sabatino and Gino Roberto Corazza.  Nonceliac gluten sensitivity: sense or sensibility?, 2012.
  * Dice [1945] Lee R Dice.  Measures of the amount of ecologic association between species.  _Ecology_ , 26(3):297–302, 1945.
  * Everingham et al. [2006] Mark Everingham, Andrew Zisserman, Christopher KI Williams, Luc Van Gool, Moray Allan, Christopher M Bishop, Olivier Chapelle, Navneet Dalal, Thomas Deselaers, Gyuri Dorkó, et al.  The 2005 pascal visual object classes challenge.  In _Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers_ , pages 117–176. Springer, 2006.
  * Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88(2):303–338, 2010.
  * Everingham et al. [2015] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  _International journal of computer vision_ , 111(1):98–136, 2015.
  * Ferrer [2022] Luciana Ferrer.  Analysis and comparison of classification metrics.  _arXiv preprint arXiv:2209.05355_ , 2022.
  * Gao et al. [2019] Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang.  Ssap: Single-shot instance segmentation with affinity pyramid.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 642–651, 2019.
  * Gneiting and Raftery [2007] Tilmann Gneiting and Adrian E Raftery.  Strictly proper scoring rules, prediction, and estimation.  _Journal of the American statistical Association_ , 102(477):359–378, 2007.
  * Gooding et al. [2018] Mark J Gooding, Annamarie J Smith, Maira Tariq, Paul Aljabar, Devis Peressutti, Judith van der Stoep, Bart Reymen, Daisy Emans, Djoya Hattu, Judith van Loon, et al.  Comparative evaluation of autocontouring in clinical practice: a practical method using the turing test.  _Medical physics_ , 45(11):5105–5115, 2018.
  * Gooding et al. [2022] Mark J Gooding, Djamal Boukerroui, Eliana Vasquez Osorio, René Monshouwer, and Ellen Brunenberg.  Multicenter comparison of measures for quantitative evaluation of contouring in radiotherapy.  _Physics and Imaging in Radiation Oncology_ , 24:152–158, 2022.
  * Grandini et al. [2020] Margherita Grandini, Enrico Bagli, and Giorgio Visani.  Metrics for multi-class classification: an overview.  _arXiv preprint arXiv:2008.05756_ , 2020.
  * Gruber and Buettner [2022] Sebastian Gruber and Florian Buettner.  Trustworthy deep learning via proper calibration errors: A unifying approach for quantifying the reliability of predictive uncertainty.  _arXiv preprint arXiv:2203.07835_ , 2022.
  * Guo et al. [2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On Calibration of Modern Neural Networks.  _ICML_ , page 10, 2017.
  * Gurcan et al. [2010] Metin N Gurcan, Anant Madabhushi, and Nasir Rajpoot.  Pattern recognition in histopathological images: An icpr 2010 contest.  In _International Conference on Pattern Recognition_ , pages 226–234. Springer, 2010.
  * Hanley and McNeil [1982] James A Hanley and Barbara J McNeil.  The meaning and use of the area under a receiver operating characteristic (roc) curve.  _Radiology_ , 143(1):29–36, 1982.
  * Hastie et al. [2009] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman.  _The elements of statistical learning: data mining, inference, and prediction_ , volume 2.  Springer, 2009.
  * Hirsch et al. [2020] Peter Hirsch, Lisa Mais, and Dagmar Kainmueller.  Patchperpix for instance segmentation.  _arXiv preprint arXiv:2001.07626_ , 2020.
  * Honauer et al. [2015] Katrin Honauer, Lena Maier-Hein, and Daniel Kondermann.  The hci stereo metrics: Geometry-aware performance analysis of stereo algorithms.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2120–2128, 2015.
  * Huttenlocher et al. [1993] Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge.  Comparing images using the hausdorff distance.  _IEEE Transactions on pattern analysis and machine intelligence_ , 15(9):850–863, 1993.
  * Jaccard [1912] Paul Jaccard.  The distribution of the flora in the alpine zone. 1.  _New phytologist_ , 11(2):37–50, 1912.
  * Kaggle [2021] Kaggle.  Satorius Cell Instance Segmentation 2021.  https://www.kaggle.com/c/sartorius-cell-instance-segmentation, 2021.  [Online; accessed 25-April-2022].
  * Kirillov et al. [2019] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9404–9413, 2019.
  * Kofler et al. [2021] Florian Kofler, Ivan Ezhov, Fabian Isensee, Christoph Berger, Maximilian Korner, Johannes Paetzold, Hongwei Li, Suprosanna Shit, Richard McKinley, Spyridon Bakas, et al.  Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient.  _arXiv preprint arXiv:2103.06205v1_ , 2021.
  * Konukoglu et al. [2012] Ender Konukoglu, Ben Glocker, Dong Hye Ye, Antonio Criminisi, and Kilian M Pohl.  Discriminative segmentation-based evaluation through shape dissimilarity.  _IEEE transactions on medical imaging_ , 31(12):2278–2289, 2012.
  * Krause et al. [2018] Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster.  Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy.  _Ophthalmology_ , 125(8):1264–1272, 2018.
  * Kuhn [1955] Harold W Kuhn.  The hungarian method for the assignment problem.  _Naval research logistics quarterly_ , 2(1-2):83–97, 1955.
  * Kulikov and Lempitsky [2020] Victor Kulikov and Victor Lempitsky.  Instance segmentation of biological images using harmonic embeddings.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3843–3851, 2020.
  * Kull et al. [2019] Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach.  Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration.  _Advances in neural information processing systems_ , 32, 2019.
  * Kumar et al. [2019] Ananya Kumar, Percy S Liang, and Tengyu Ma.  Verified uncertainty calibration.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lennerz et al. [2022] Jochen K Lennerz, Ursula Green, Drew FK Williamson, and Faisal Mahmood.  A unifying force for the realization of medical ai.  _npj Digital Medicine_ , 5(1):1–3, 2022.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * Maier-Hein et al. [2018] Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunovic, Andrew P Bradley, Aaron Carass, et al.  Why rankings of biomedical image analysis competitions should be interpreted with care.  _Nature communications_ , 9(1):1–13, 2018.  With this comprehensive analysis of biomedical image analysis competitions (challenges), the authors initiated a shift in how such challenges are designed, performed, and reported in the biomedical domain. Its concepts and guidelines have been adopted by reputed organizations such as MICCAI.
  * Maier-Hein et al. [2022] Lena Maier-Hein, Annika Reinke, Evangelia Christodoulou, Ben Glocker, Patrick Godau, Fabian Isensee, Jens Kleesiek, Michal Kozubek, Mauricio Reyes, Michael A Riegler, et al.  Metrics reloaded: Pitfalls and recommendations for image analysis validation.  _arXiv preprint arXiv:2206.01653_ , 2022.
  * Margolin et al. [2014] Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal.  How to evaluate foreground maps?  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 248–255, 2014.
  * Maška et al. [2014] Martin Maška, Vladimír Ulman, David Svoboda, Pavel Matula, Petr Matula, Cristina Ederra, Ainhoa Urbiola, Tomás España, Subramanian Venkatesan, Deepak MW Balak, et al.  A benchmark for comparison of cell tracking algorithms.  _Bioinformatics_ , 30(11):1609–1617, 2014.
  * Matthews [1975] Brian W Matthews.  Comparison of the predicted and observed secondary structure of t4 phage lysozyme.  _Biochimica et Biophysica Acta (BBA)-Protein Structure_ , 405(2):442–451, 1975.
  * Muschelli [2020] John Muschelli.  Roc and auc with a binary predictor: a potentially misleading metric.  _Journal of classification_ , 37(3):696–708, 2020.
  * Naeini et al. [2015] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _Twenty-Ninth AAAI Conference on Artificial Intelligence_ , 2015.
  * Nai et al. [2021] Ying-Hwey Nai, Bernice W Teo, Nadya L Tan, Sophie O’Doherty, Mary C Stephenson, Yee Liang Thian, Edmund Chiong, and Anthonin Reilhac.  Comparison of metrics for the evaluation of medical segmentations using prostate mri dataset.  _Computers in Biology and Medicine_ , 134:104497, 2021.
  * Nasa et al. [2021] Prashant Nasa, Ravi Jain, and Deven Juneja.  Delphi methodology in healthcare research: how to decide its appropriateness.  _World Journal of Methodology_ , 11(4):116, 2021.
  * Nikolov et al. [2021] Stanislav Nikolov, Sam Blackwell, Alexei Zverovitch, Ruheena Mendes, Michelle Livne, Jeffrey De Fauw, Yojan Patel, Clemens Meyer, Harry Askham, Bernadino Romera-Paredes, et al.  Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study.  _Journal of Medical Internet Research_ , 23(7):e26151, 2021.
  * Ounkomol et al. [2018] Chawin Ounkomol, Sharmishtaa Seshamani, Mary M Maleckar, Forrest Collman, and Gregory R Johnson.  Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy.  _Nature methods_ , 15(11):917–920, 2018.
  * Pham et al. [2020] Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan.  Problems and opportunities in training deep learning software systems: An analysis of variance.  In _Proceedings of the 35th IEEE/ACM international conference on automated software engineering_ , pages 771–783, 2020.
  * Popordanoska et al. [2022] Teodora Popordanoska, Raphael Sayer, and Matthew B Blaschko.  A consistent and differentiable lp canonical calibration error estimator.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Powers [2012] David Martin Ward Powers.  The problem with kappa.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 345–355, 2012.
  * Reinke et al. [2021a] Annika Reinke, Matthias Eisenmann, Minu D Tizabi, Carole H Sudre, Tim Rädsch, Michela Antonelli, Tal Arbel, Spyridon Bakas, M Jorge Cardoso, Veronika Cheplygina, Keyvan Farahani, Ben Glocker, Doreen Heckmann-Nötzel, Fabian Isensee, Pierre Jannin, Charles Kahn, Jens Kleesiek, Tahsin Kurc, Michal Kozubek, Bennett A Landman, Geert Litjens, Klaus Maier-Hein, Anne L Martel, Henning Müller, Jens Petersen, Mauricio Reyes, Nicola Rieke, Bram Stieltjes, Ronald M Summers, Sotirios A Tsaftaris, Bram van Ginneken, Annette Kopp-Schneider, Paul Jäger, and Lena Maier-Hein.  Common limitations of image processing metrics: A picture story.  _arXiv preprint arXiv:2104.05642_ , 2021a.
  * Reinke et al. [2021b] Annika Reinke, Matthias Eisenmann, Minu D Tizabi, Carole H Sudre, Tim Rädsch, Michela Antonelli, Tal Arbel, Spyridon Bakas, M Jorge Cardoso, Veronika Cheplygina, et al.  Common limitations of image processing metrics: A picture story.  _arXiv preprint arXiv:2104.05642_ , 2021b.
  * Roberts et al. [2017] Brock Roberts, Amanda Haupt, Andrew Tucker, Tanya Grancharova, Joy Arakaki, Margaret A Fuqua, Angelique Nelson, Caroline Hookway, Susan A Ludmann, Irina A Mueller, et al.  Systematic gene tagging using crispr/cas9 in human stem cells to illuminate cell organization.  _Molecular biology of the cell_ , 28(21):2854–2874, 2017.
  * Rosenfeld and Pfaltz [1966] Azriel Rosenfeld and John L Pfaltz.  Sequential operations in digital picture processing.  _Journal of the ACM (JACM)_ , 13(4):471–494, 1966.
  * Saha et al. [2021] Anindo Saha, Joeran Bosma, Jasper Linmans, Matin Hosseinzadeh, and Henkjan Huisman.  Anatomical and diagnostic bayesian segmentation in prostate mri $-$ should different clinical objectives mandate different loss functions?  _arXiv preprint arXiv:2110.12889_ , 2021.
  * Schmidt et al. [2018] Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers.  Cell detection with star-convex polygons.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 265–273. Springer, 2018.
  * Shit et al. [2021] Suprosanna Shit, Johannes C Paetzold, Anjany Sekuboyina, Ivan Ezhov, Alexander Unger, Andrey Zhylka, Josien PW Pluim, Ulrich Bauer, and Bjoern H Menze.  cldice-a novel topology-preserving loss function for tubular structure segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16560–16569, 2021.
  * Stringer et al. [2021] Carsen Stringer, Tim Wang, Michalis Michaelos, and Marius Pachitariu.  Cellpose: a generalist algorithm for cellular segmentation.  _Nature methods_ , 18(1):100–106, 2021.
  * Taha and Hanbury [2015] Abdel Aziz Taha and Allan Hanbury.  Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool.  _BMC medical imaging_ , 15(1):1–28, 2015.  The paper discusses the importance of effective metrics for evaluating the accuracy of 3D medical image segmentation algorithms. The authors analyze existing metrics, propose a selection methodology, and develop a tool to aid researchers in choosing appropriate evaluation metrics based on the specific characteristics of the segmentation task.
  * Taha et al. [2014] Abdel Aziz Taha, Allan Hanbury, and Oscar A Jimenez del Toro.  A formal method for selecting evaluation metrics for image segmentation.  In _2014 IEEE international conference on image processing (ICIP)_ , pages 932–936. IEEE, 2014.
  * Tharwat [2020] Alaa Tharwat.  Classification assessment methods.  _Applied Computing and Informatics_ , 2020.
  * Tran et al. [2022] Thuy Nuong Tran, Tim Adler, Amine Yamlahi, Evangelia Christodoulou, Patrick Godau, Annika Reinke, Minu Dietlinde Tizabi, Peter Sauer, Tillmann Persicke, Jörg Gerhard Albert, et al.  Sources of performance variability in deep learning-based polyp detection.  _arXiv preprint arXiv:2211.09708_ , 2022.
  * Vaassen et al. [2020] Femke Vaassen, Colien Hazelaar, Ana Vaniqui, Mark Gooding, Brent van der Heyden, Richard Canters, and Wouter van Elmpt.  Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy.  _Physics and Imaging in Radiation Oncology_ , 13:1–6, 2020.
  * Vaicenavicius et al. [2019] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön.  Evaluating model calibration in classification.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pages 3459–3467. PMLR, 2019.
  * Van Ginneken et al. [2010] Bram Van Ginneken, Samuel G Armato III, Bartjan de Hoop, Saskia van Amelsvoort-van de Vorst, Thomas Duindam, Meindert Niemeijer, Keelin Murphy, Arnold Schilham, Alessandra Retico, Maria Evelina Fantacci, et al.  Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: the anode09 study.  _Medical image analysis_ , 14(6):707–722, 2010.
  * Van Rijsbergen [1979] C Van Rijsbergen.  Information retrieval: theory and practice.  In _Proceedings of the Joint IBM/University of Newcastle upon Tyne Seminar on Data Base Systems_ , volume 79, 1979.
  * Viana et al. [2023] Matheus P Viana, Jianxu Chen, Theo A Knijnenburg, Ritvik Vasan, Calysta Yan, Joy E Arakaki, Matte Bailey, Ben Berry, Antoine Borensztejn, Eva M Brown, et al.  Integrated intracellular organization and its variations in human ips cells.  _Nature_ , pages 1–10, 2023.
  * Vickers and Elkin [2006] Andrew J Vickers and Elena B Elkin.  Decision curve analysis: a novel method for evaluating prediction models.  _Medical Decision Making_ , 26(6):565–574, 2006.
  * Vickers et al. [2016] Andrew J Vickers, Ben Van Calster, and Ewout W Steyerberg.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.  _bmj_ , 352, 2016.
  * Wack et al. [2012] David S Wack, Michael G Dwyer, Niels Bergsland, Carol Di Perri, Laura Ranza, Sara Hussein, Deepa Ramasamy, Guy Poloni, and Robert Zivadinov.  Improved assessment of multiple sclerosis lesion segmentation agreement via detection and outline error estimates.  _BMC medical imaging_ , 12(1):1–10, 2012.
  * Warrens [2012] Matthijs J Warrens.  Some paradoxical results for the quadratically weighted kappa.  _Psychometrika_ , 77(2):315–323, 2012.
  * Widmann et al. [2019] David Widmann, Fredrik Lindsten, and Dave Zachariah.  Calibration tests in multi-class classification: A unifying framework.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Wiesenfarth et al. [2021] Manuel Wiesenfarth, Annika Reinke, Bennett A Landman, Matthias Eisenmann, Laura Aguilera Saiz, M Jorge Cardoso, Lena Maier-Hein, and Annette Kopp-Schneider.  Methods and open-source toolkit for analyzing and visualizing challenge results.  _Scientific Reports_ , 11(1):1–15, 2021.
  * Yeghiazaryan and Voiculescu [2015] Varduhi Yeghiazaryan and Irina Voiculescu.  An overview of current evaluation methods used in medical image segmentation.  _Department of Computer Science, University of Oxford_ , 2015.
  * Yeghiazaryan and Voiculescu [2018] Varduhi Yeghiazaryan and Irina D Voiculescu.  Family of boundary overlap metrics for the evaluation of medical image segmentation.  _Journal of Medical Imaging_ , 5(1):015006, 2018.
  * Zhu [2020] Qiuming Zhu.  On the performance of matthews correlation coefficient (mcc) for imbalanced dataset.  _Pattern Recognition Letters_ , 136:71–80, 2020.

## Supplementary Notes

##  1\. Metric Fundamentals

The present work focuses on biomedical image analysis problems that can be
interpreted as classification tasks at the image, object, or pixel level. The
vast majority of metrics for these problem categories are directly or
indirectly based on epidemiological principles of True Positive (TP), False
Negative (FN), False Positive (FP), True Negative (TN), i.e., the
cardinalities of the so-called confusion matrix. The TP/FN/FP/TN are
henceforth referred to as cardinalities. In the case of more than two classes
$C$, we also refer to the entries of the $C\times C$ confusion matrix as
cardinalities. For simplicity and clarity in notation, we restrict ourselves
to the binary case in most examples. Cardinalities can be computed at the
image (segment), object, or pixel level. They are typically computed by
comparing the prediction of the algorithm to a reference annotation. Modern
neural network-based approaches commonly require a threshold to be set in
order to convert the algorithm output comprising predicted class scores (also
referred to as continuous class scores) to a confusion matrix. For the purpose
of metric recommendation, the available metrics can be broadly classified as
follows (see also [Cao et al., 2020]):

  * •

Counting metrics operate directly on the confusion matrix and express the
metric value as a function of the cardinalities. In the context of
segmentation, they are typically referred to as overlap-based metrics [Taha
and Hanbury, 2015]. We distinguish multi-class counting metrics, which are
defined for an arbitrary number of classes and invariant under class order,
from per-class counting metrics, which are computed by treating one class as
foreground/positive class and all other classes as background. Popular
examples for the former include MCC or Accuracy, while examples for the latter
are Sensitivity, Specificity and DSC.

  * •

Multi-threshold metrics operate on a dynamic confusion matrix, reflecting the
conflicting properties of interest, such as high Sensitivity and high
Specificity. Popular examples include the AUROC and AP.

  * •

Distance-based metrics have been designed for semantic and instance
segmentation tasks. They operate exclusively on the TPs and rely on the
explicit definition of object boundaries. Popular examples are the HD and the
NSD.

Depending on the context (e.g., image-level classification vs. semantic
segmentation task) and the community (e.g., medical imaging community vs.
computer vision community), identical metrics are referred to with different
terminology. For example, Sensitivity, True Positive Rate (TPR) and Recall
refer to the same concept. The same holds true for the DSC and the F1 Score.
The most relevant metrics for the problem categories in the scope of this
paper are introduced in the following.

Most metrics are recommended to be applied per class (except for the multi-
class counting metrics), meaning that a potential multi-class problem is
converted to multiple binary classification problems, such that each relevant
class serves as the positive class once. This results in different confusion
matrices depending on which class is used as the positive class.

###  1.1. Image-level Classification

Image-level classification refers to the process of assigning one or multiple
labels, or classes, to an image. Modern algorithms usually output predicted
class scores (or continuous class scores) between 0 and 1 for every image and
class, indicating the probability of the image belonging to a specific class.
By introducing a threshold (e.g., 0.5), predictions are considered as positive
(e.g., cancer = true) if they are above the threshold, or negative if they are
below the threshold. Subsequently, predictions are assigned to the
cardinalities (e.g., a cancer patient with prediction cancer = true is
considered as TP) [Davis and Goadrich, 2006]. The most popular classification
metrics are counting metrics, operating on a confusion matrix with fixed
threshold on the class probabilities, and multi-threshold metrics, as detailed
in the following.

##### Counting metrics

As stated previously, counting metrics rely on the confusion matrix. We
distinguish between per-class and multi-class counting metrics. Popular multi-
class counting metrics include:

Accuracy:

[Tharwat, 2020]: Fig. SN 3.40

Balanced Accuracy (BA):

[Tharwat, 2020]: Fig. SN 3.41

Expected Cost (EC):

(also referred to as Expected Prediction Error or Expected Loss) [Bishop and
Nasrabadi, 2006, Hastie et al., 2009, Ferrer, 2022]: Fig. SN 3.44

Matthews Correlation Coefficient (MCC):

(also referred to as Phi Coefficient) [Matthews, 1975]: Fig. SN 3.48

Weighted Cohen’s Kappa (WCK):

(also referred to as Weighted Cohen‘s Kappa Coefficient, Weighted Kappa
Statistic or Weighted Kappa Score) [Cohen, 1960]: Fig. SN 3.56

Popular per-class counting metrics include:

Fβ Score:

[Van Rijsbergen, 1979, Chinchor, 1992]: Fig. SN 3.45

Net Benefit (NB):

[Vickers and Elkin, 2006]: Fig. SN 3.49

Negative Predictive Value (NPV):

[Tharwat, 2020]: Fig. SN 3.50

Positive Predictive Value (PPV):

(also referred to as Precision) [Tharwat, 2020]: Fig. SN 3.53

Sensitivity:

(also referred to as Recall, TPR or Hit Rate) [Tharwat, 2020]: Fig. SN 3.54

Specificity:

(also referred to as Selectivity or True Negative Rate (TNR)) [Tharwat, 2020]:
Fig. SN 3.55

##### Multi-threshold metrics

The classical counting metrics presented above rely on fixed thresholds to be
set on the predicted class probabilities (if available), resulting in them
being based on the cardinalities of the confusion matrix. Multi-threshold
metrics overcome this limitation by calculating metric scores based on
multiple thresholds. Popular examples are:

Area under the Receiver Operating Characteristic Curve (AUROC):

(also referred to as Area under the Curve (AUC), AUC - ROC (Area under the
Curve - Receiver Operating Characteristics), C-Index, C-Statistics) [Hanley
and McNeil, 1982]: Fig. SN 3.57

Average Precision (AP):

[Lin et al., 2014, Everingham et al., 2015]: Fig. SN 3.58

##### Calibration metrics

While most research in biomedical image analysis focuses on the discrimination
capabilities of classifiers, a complementary property of relevance is the
calibration of predicted class scores (also known as confidence scores).
Intuitively speaking, a system is well-calibrated if the predicted class
scores (i.e., the output of the model) reflect the true probabilities of the
outcome. In practice, this means that calibrated scores match the empirical
success rate of associated predictions. For a binary classification task,
calibration implies that of all the data samples assigned a predicted score of
$0.8$ for the positive class, empirically, $80\%$ belong to this class.
Popular examples are:

Brier Score (BS):

[Gneiting and Raftery, 2007]: Fig. SN 3.66

Class-Wise Calibration Error (CWCE):

[Kumar et al., 2019, Kull et al., 2019]: Fig. SN 3.67

Expected Calibration Error (ECE):

[Naeini et al., 2015]: Fig. SN 3.68

Expected Calibration Error Kernel Density Estimate (ECEKDE):

[Popordanoska et al., 2022] : Fig. SN 3.69

Kernel Calibration Error (KCE):

[Gruber and Buettner, 2022, Widmann et al., 2019]: Fig. SN 3.70

Negative Log Likelihood (NLL):

[Cybenko et al., 1998]: Fig. SN 3.71

Root Brier Score (RBS):

[Gruber and Buettner, 2022]: Fig. SN 3.72

###  1.2. Semantic Segmentation

Semantic segmentation is commonly defined as the process of partitioning an
image into multiple segments/regions. To this end, one or multiple labels are
assigned to every pixel such that pixels with the same label share certain
characteristics. Semantic segmentation can therefore also be regarded as
pixel-level classification. As in image-classification problems, predicted
class probabilities are typically calculated for each pixel, deciding on the
class affiliation based on a threshold over the class scores [Asgari Taghanaki
et al., 2021]. In semantic segmentation problems, the pixel-level
classification is typically followed by a post-processing step, in which
connected components are defined as objects, and object boundaries are created
accordingly. Semantic segmentation metrics can roughly be classified into: (1)
counting metrics or overlap-based metrics, for measuring the overlap between
the reference annotation and the prediction of the algorithm, (2) distance-
based or boundary-based metrics, for measuring the distance between object
boundaries, and (3) problem-specific metrics, measuring, for example, object
volumes.

##### Counting metrics

The most frequently used segmentation metrics are counting metrics. In the
context of segmentation they are also referred to as overlap-based metrics, as
they essentially measure the overlap between a reference mask and the
algorithm prediction. Popular examples of overlap-based metrics include:

Dice Similarity Coefficient (DSC):

(also referred to as Sørensen–Dice Coefficient, F1 Score, Balanced F Score)
[Dice, 1945]: Fig. SN 3.43

Intersection over Union (IoU):

(also referred to as Jaccard Index, Tanimoto Coefficient) [Jaccard, 1912]:
Fig. SN 3.47

centerline Dice Similarity Coefficient (clDice):

[Shit et al., 2021]: Fig. SN 3.42

##### Distance-based metrics

Overlap-based metrics are often complemented by distance-based metrics that
operate exclusively on the TPs and compute one or several distances between
the reference and the prediction. Besides few exceptions, distance-based
metrics are often boundary-based metrics which focus on assessing the accuracy
of object boundaries. Popular examples include:

Average Symmetric Surface Distance (ASSD):

(also referred to as Weighted Bilateral Mean Contour Distance) [Yeghiazaryan
and Voiculescu, 2015]: Fig. SN 3.60

Boundary Intersection over Union (Boundary IoU):

[Cheng et al., 2021]: Fig. SN 3.61

Hausdorff Distance (HD):

(also referred to as Maximum Symmetric Surface Distance, Hausdorff Metric,
Pompeiu–Hausdorff Distance) [Huttenlocher et al., 1993]: Fig. SN 3.62

Hausdorff Distance 95th Percentile (HD95):

[Huttenlocher et al., 1993]: Fig. SN 3.65

Mean Average Surface Distance (MASD):

(also referred to as Mean Surface Distance) [Beneš and Zitová, 2015]: Fig. SN
3.63

Normalized Surface Distance (NSD):

(also referred to as Normalized Surface Dice, Surface Distance, Surface Dice,
Surface DSC) [Nikolov et al., 2021]: Fig. SN 3.64

##### Problem-specific segmentation metrics

While overlap- and distance-based metrics are the standard metrics used by the
general computer vision community, biomedical applications often have special
domain-specific requirements. In medical imaging, for example, the actual
volume of an object (e.g., a tumor) may be of particular interest. In this
case, volume metrics such as the Absolute or Relative Volume Error and the
Symmetric Relative Volume Difference can be computed [Nai et al., 2021].

###  1.3. Object Detection

Object detection refers to the detection of one or multiple objects (or:
instances) of a particular class (e.g., lesion) in an image [Lin et al.,
2014]. The following description assumes single-class problems, but
translation to multi-class problems is straightforward, as validation for
multiple classes on object level is performed individually per class. Notably,
as multiple predictions and reference instances may be present in one image,
the predictions need to include localization information, such that reference
and predicted objects can be matched. Important design choices with respect to
the validation of object detection methods include:

  1. (1)

How to represent an object? Representation is typically composed of location
information and a class affiliation. The former may for example take the form
of a bounding box (i.e., a list of coordinates), a pixel mask, or the object’s
center point. Additionally, modern algorithms typically assign a confidence
value to each object, representing the probability of a prediction
corresponding to an actual object of the respective class. Note that a
confusion matrix is later computed for a fixed threshold on the predicted
class probabilities.444Please note that we will use the term confidence scores
analogously to predicted class probabilities in the context of object
detection and instance segmentation.

  2. (2)

How to decide whether a reference instance was correctly detected? This step
is achieved by applying the localization criterion. A localization criterion
may, for example, be based on comparing the object centers of the reference
and prediction or computing their overlap.

  3. (3)

How to resolve assignment ambiguities? The above step might lead to ambiguous
matchings, such as two predictions being assigned to the same reference
object. Several strategies exist for resolving such cases.

The following sections provide details on (1) applying the localization
criterion, (2) applying the assignment strategy, and (3) computing the actual
performance metrics.

##### Localization criterion

As one image may contain multiple objects or no object at all, the
localization criterion or hit criterion measures the (spatial) similarity
between a prediction (represented by a bounding box, pixel mask, center point
or similar) and a reference object. It defines whether the prediction
hit/detected (TP) or missed (FP) the reference. Any reference object not
detected by the algorithm is defined as FN. Please note that TNs are not
defined for object detection tasks. Popular localization criteria include:

Box/Approx Intersection over Union (IoU):

[Jaccard, 1912]: Fig. SN 3.76

Mask IoU ¿ 0:

[Jaccard, 1912, Wack et al., 2012]: Fig. SN 3.77

Center Distance:

[Gurcan et al., 2010]: Fig. SN 3.74

Point inside Mask/ Box/ Approx:

     555https://cada.grand-challenge.org/Assessment/

: Fig. SN 3.78

##### Assignment strategy

The localization criterion alone is not sufficient to extract the final
confusion matrix based on a fixed threshold for the predicted class
probabilities (confidence scores), as ambiguities can occur. For example, two
predictions may have been assigned to the same reference object in the
localization step, or vice versa. These ambiguities need to be resolved in a
further assignment step. This assignment and thus the resolving of potential
assignment ambiguities can be done via different strategies:

Greedy (by Score) Matching:

[Everingham et al., 2015]: Fig. SN 3.79

Optimal (Hungarian) Matching:

[Kuhn, 1955]: Fig. SN 3.81

Matching via Overlap ¿ 0.5:

[Everingham et al., 2006]: Fig. SN 3.82

Greedy (by Localization Criterion) Matching:

[Maier-Hein et al., 2022]: Fig. SN 3.80

##### Metric computation

Similar to image-level classification and semantic segmentation algorithms,
object detection algorithms are commonly assessed with counting metrics,
assuming a fixed confusion matrix. Popular examples include:

Fβ Score:

[Van Rijsbergen, 1979, Chinchor, 1992]: Fig. SN 3.45

False Positives per Image (FPPI):

[Van Ginneken et al., 2010, Bandos et al., 2009]: Fig. SN 3.46

Positive Predictive Value (PPV):

(also referred to as Precision) [Tharwat, 2020]: Fig. SN 3.53

Sensitivity:

(also referred to as Recall, TPR or Hit Rate) [Tharwat, 2020]: Fig. SN 3.54

Similarly, multi-threshold metrics rely on a range of thresholds. Popular
examples are:

Average Precision (AP):

[Lin et al., 2014, Everingham et al., 2015]: Fig. SN 3.58

Free-Response Receiver Operating Characteristic (FROC) Score:

[Van Ginneken et al., 2010]: Fig. SN 3.59

###  1.4. Instance Segmentation