# Weakly Supervised Learning Significantly Reduces the
Number of Labels Required for Intracranial
Hemorrhage Detection on Head CT

Jacopo Teneggi111Department of Computer Science, Johns Hopkins University,
Baltimore, MD, 21218 222Mathematical Institute for Data Science (MINDS), Johns
Hopkins University, Baltimore, MD, 21218 Paul H. Yi333University of Maryland
Medical Intelligent Imaging (UM2ii) Center, Department of Diagnostic Radiology
and Nuclear Medicine, University of Maryland School of Medicine, Baltimore,
MD, 21201 Jeremias Sulam444Department of Biomedical Engineering, Johns Hopkins
University, Baltimore, MD, 21218 22footnotemark: 2

###### Abstract

Modern machine learning pipelines, in particular those based on deep learning
(DL) models, require large amounts of labeled data. For classification
problems, the most common learning paradigm consists of presenting labeled
examples during training, thus providing _strong supervision_ on what
constitutes positive and negative samples. As a result, the adequate training
of these models demands the curation of large datasets with high-quality
labels. This constitutes a major obstacle for the development of DL models in
radiology—in particular for cross-sectional imaging (e.g., computed tomography
[CT] scans)—where labels must come from manual annotations by expert
radiologists at the image or slice-level. These differ from examination-level
annotations, which are coarser but cheaper, and could be extracted from
radiology reports using natural language processing techniques. This work
studies the question of _what kind of labels_ should be collected for the
problem of intracranial hemorrhage detection in brain CT. We investigate
whether _image_ -level annotations should be preferred to _examination_ -level
ones. By framing this task as a multiple instance learning (MIL) problem, and
employing modern attention-based DL architectures, we analyze the degree to
which different levels of supervision improve detection performance. We find
that strong supervision (i.e., learning with local image-level annotations)
and weak supervision (i.e., learning with only global examination-level
labels) achieve comparable performance in examination-level hemorrhage
detection (the task of selecting the images in an examination that show signs
of hemorrhage) as well as in image-level hemorrhage detection (highlighting
those signs within the selected images). Furthermore, we study this behavior
as a function of the number of labels available during training. Our results
suggest that local labels may not be necessary at all for these tasks,
drastically reducing the time and cost involved in collecting and curating
datasets.

##  1 Introduction

Modern Deep Learning (DL) models continue to drive exciting advances across
several medical imaging tasks, from image reconstruction and enhancement [2,
17, 3, 87, 47], to automatic lesion detection and segmentation [16, 35, 52].
DL models for classification and detection are especially desirable for
Computer-Aided Diagnosis (CAD) systems in radiology, potentially supporting
clinicians in their decision-making by providing a second opinion on subtle
cases, or prioritizing the most severe ones [61, 73, 85]. Indeed, recent
results indicate that the performance of these machine learning models can be
comparable to that of expert physicians in many scenarios [64, 69], and they
hold significant promise for the automation of diagnosis, especially in
underserved areas where access to radiology expertise might be limited [48,
21, 51, 62, 90, 7].

In this work, we center our attention on the development of DL models for
Intracranial Hemorrhage (ICH) detection in head Computed Tomography (CT). In
this context, given a new CT scan, the task is to detect the presence of any
type of brain hemorrhage. ICH is a potentially life-threatening condition
consisting of bleeding inside of the brain which can have several different
causes, from trauma to drug abuse [33]. ICH accounts for approximately 10% to
20% of all strokes [5], and expert radiologists can diagnose ICH from
unenhanced head CT scans by analyzing the location, shape, and size of the
lesions [33]. The large number of head CT scans produced daily, and the
importance of a quick diagnosis for an effective treatment of severe cases,
make ICH detection one of the most popular applications of deep learning in
radiology thus far [12]. Many recent works have explored deep learning
solutions to different challenges in developing machine learning pipelines for
ICH detection, such as the volumetric nature of CT data, the windowing range,
and the lack of confidence in _black-box_ predictors [92, 46, 54, 53].

At the same time, the development of these high-performing models can be
notoriously time-consuming and expensive, largely due to the significant
amount of required training data. The most common approach to training DL
models for medical imaging classification and detection is _supervised
learning_ , wherein a collection of images with ground-truth labels are
presented to the model. These examples serve the purpose of describing what
constitutes a sample from a given class, or how a specific finding looks like
in a given image. Naturally, this requires having access to large amounts of
labeled data that must be collected by radiologists who manually annotate
hundreds or thousands of images—a laborsome and time-consuming process that
often results in very high costs [33].

Some recent research efforts have explored ways of alleviating these
limitations. _Semi supervised_ learning approaches, for example, extract low-
quality labels automatically from clinical notes stored in the Electronic
Health Record (EHR) system of a medical institution. The authors in [32] and
[84] show how weak labels extracted automatically from clinical reports enable
whole-body abnormality detection in PET/CT and body CT, respectively. Although
semi supervised learning alleviates the need for large amounts of data with
ground-truth annotations, collecting _some_ amount of annotated data remains
central to training and, importantly, testing these models, and the central
aforementioned limitations persist.

In detection problems in particular—where the label of a sample is determined
by the presence of a specific finding—it remains unclear _what kind_ of labels
should be sought after. In the hemorrhage detection problem described above,
should ground-truth binary labels be collected for every image in an
examination? This can be implemented by labeling an image as ‘1’ if it
contains signs of hemorrhage, or ‘0’ otherwise. Or would coarse, examination-
level annotations that only indicate the presence of hemorrhage somewhere in
the scan (but not in which image) suffice? On the one hand, it is clear that
the amount of information in each label decreases as we provide coarser
annotation. That is, there might be other findings in a scan (e.g., midline
shift effects, external hemotomas, signs of prior surgery, asymmetries) that
may be highly correlated with intracranial hemorrhage in the training data. A
coarse examination-level binary label may not provide enough information to
disambiguate them. At the same time, coarser annotations can lead to huge
improvements in data curation time and annotation, since radiologists need
only to provide a binary response for each examination.

In this work, we address these fundamental questions using a weakly supervised
approach different from semi supervised learning: _Multiple Instance Learning_
(MIL) [29, 59, 89]. In MIL problems, one regards every input as a _bag of
instances_ , and the label of the bag is determined by the labels of its
instances. This framework naturally fits the problem of hemorrhage detection
in head CT, since an examination is considered positive (i.e., its coarse,
global label is positive) as soon as it contains at least one image with
evidence of hemorrhage (i.e., it contains an image with a positive local
label). MIL is a particular case of weakly supervised learning, wherein labels
are only available for bags (i.e., examinations) instead of instances (i.e.,
images). By employing a state-of-the-art MIL model [43] that can be trained
with either global or local labels, we study whether strong supervision with
expensive local labels leads to significantly higher performance in hemorrhage
detection in head CT, or whether weak supervision—which is cheaper to
obtain—can provide comparable models.

### Summary of contributions

We show that weakly supervised learning can produce DL models for ICH
detection with performance matching that of DL models trained using strong
supervision—all while using $\approx 35$-times fewer labels. Furthermore,
these weakly supervised models had better generalization on at least one
external dataset. Finally, we show that weakly supervised DL models have
comparable localization ability of ICH on both the image- and pixel-levels,
which is a key feature towards explainability and building trust with
clinician end-users. These results inform how data should be collected for
this and other similar tasks in radiology, providing a solution to the primary
bottleneck in development of high-performing DL models in medical imaging.

##  2 Results

For a positive head CT scan, we will refer to _examination-level_ hemorrhage
detection as the task of retrieving the images that contain signs of ICH; and
_image-level_ hemorrhage detection as the task of highlighting these findings
within the retrieved images. We rephrase both examination- and image-level
hemorrhage detection as MIL binary classification problems [29, 59, 89] (see
Section 4.1 for details on supervised learning and MIL), and evaluate the
performance of models trained with local (image-level) annotations and global
(examination-level) labels. We refer to the former as a strong learner
(${\mathcal{SL}}$), as it is trained via strong supervision, and weak learner
(${\mathcal{WL}}$) to the latter, since it only uses weak supervision.

###  2.1 Datasets

We train a strong and a weak learner on the RSNA 2019 Brain CT Hemorrhage
Challenge dataset [33], which comprises 21,784 examinations (with a positive
rate of 41%) for a total of 752,803 images (with a positive rate of
14%).555For the sake of simplicity, we will refer to the RSNA 2019 Brain CT
Hemorrhage Challenge dataset as “RSNA dataset”, which is available at:
https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection. Every image
in the RSNA dataset was labeled by expert neuroradiologists with the type(s)
of hemorrhage present (i.e., epidural, intraparenchymal, intraventricular,
subarachnoid, or subdural). We use 80% of the data for training and 20% for
validation. Splits were created by random sampling of examinations, rather
than images, and the same splits were used for both models in order to
guarantee a fair comparison between them. Table 1 shows the distribution of
positive and negative labels—note that while the total _number of images_ is
the same for each model, the weak learner has access to $\approx 35$-times
fewer _total labels_ , which is the average number of images in a scan across
the dataset.

Table 1: Number of positive and negative labels in the RSNA dataset for strong and weak learners. | Training | Validation
---|---|---
Learner | Positive labels | Negative labels | Positive labels | Negative labels
full supervision | 86,295 ($\approx$ 14%) | 515,635 ($\approx$ 86%) | 21,489 ($\approx$ 14%) | 129,003 ($\approx$ 86%)
weak supervision | 7,100 ($\approx$ 40%) | 10,288 ($\approx$ 60%) | 1,776 ($\approx$ 40%) | 2,572 ($\approx$ 60%)
Table 2: Number of positive and negative examinations in the CQ500 and CT-ICH datasets, alongside the total number of images contained in the two datasets. Dataset | Positive examinations | Negative examinations | Total images
---|---|---|---
CQ500 [20] | 212 ($\approx$ 49%) | 224 ($\approx$ 51%) | 15,156
CT-ICH [41] | 36 ($\approx$ 48%) | 39 ($\approx$ 52%) | 2,539

In addition to the validation split of the RSNA dataset, we evaluate our
resulting models on two external test sets—the CQ500 dataset (436 examinations
with a positive rate of 49%) [20] and the CT-ICH dataset (75 examinations with
a positive rate of 48%) [41, 42, 36].666The CQ500 dataset is available at:
http://headctstudy.qure.ai/dataset; the CT-ICH dataset is available at:
https://physionet.org/content/ct-ich/1.3.1/. Table 2 shows the distribution of
positive and negative examinations in the two external test sets and their
total number of images. We note that the CQ500 dataset only provides
examination-level labels, while the CT-ICH dataset provides both image-level
labels and manual pixel-level segmentations of the bleeds performed by two
expert radiologists. Hence, we extend the CQ500 dataset with the ICH bounding
box annotations provided for this dataset by three radiologists with varying
degree of experience, available in the BHX dataset [70, 36].777The BHX dataset
is available at: https://physionet.org/content/bhx-brain-bounding-box/1.1/. We
include details on the preprocessing of the images for all three datasets in
Section 4.3.

###  2.2 Attention-based MIL enables training with local or global labels

We frame the ICH detection task as an MIL binary classification problem. We
include a detailed description of the model architectures and their training
procedures in Sections 4.2 and 4.4, respectively.888Code to reproduce the
experiments in this paper is available at: https://github.com/Sulam-
Group/MIL_ICH. Here, we briefly describe how state-of-the-art attention-based
MIL models [43] enable us to precisely investigate whether classical strong
supervision with expensive local labels provides an advantage over weak
supervision with cheap global labels.

(a) Pictorial representation of a strong learner $h$ (i.e., ${\mathcal{SL}}$).

(b) Pictorial representation of a weak learner $H$ (i.e., ${\mathcal{WL}}$).

Figure 1: Model architectures. Fig. 1(a): The strongly supervised model makes
a prediction on every input image, and it requires their labels for training.
Fig. 1(b): The weakly supervised model makes a prediction for the entire
examination, and thus only requires their labels for training. Note that both
learners use the same encoder (i.e., a ResNet18 [40]) and the same fully-
connected layer as the final classifier.

We regard an individual image in a CT scan as a vector $x\in{\mathbb{R}}^{d}$.
Each image is associated with a binary label $y$ that indicates the presence
$(y=1)$ or absence $(y=0)$ of signs of hemorrhage in the image. A single CT
scan of a patient can be naturally seen as the stacking of $r$ images along
the scanner’s axis, i.e.
$X=[x^{(1)},x^{(2)},\dots,x^{(r)}]\in{\mathbb{R}}^{dr}$. Analogously to
images, examinations are also associated with a binary label $Y$ indicating
whether any image in the examination presents signs of hemorrhage $(Y=1)$, or
every image in the examination is healthy $(Y=0)$. Note that the label of an
examination can be determined from the labels of the images in the examination
(if they are available), since the presence of hemorrhage in any image implies
the presence of hemorrhage in the examination. This observation can be
formalized by stating that the examination’s label is the logical OR function
of the labels of the images in the examination, i.e.
$Y={\texttt{OR}}(y^{(1)},y^{(2)},\dots,y^{(r)})$.

In the traditional strongly supervised setting, a predictor $h$ is trained on
a collection of $n_{s}$ training samples $\\{(x_{i},y_{i})\\}_{i=1}^{n_{s}}$,
with the goal of obtaining $h(x)$ as an accurate predictor of the label of a
new sample $x$, i.e. as an approximation of the conditional expectation of $y$
given $x$. In this work, $h$ is given by the composition of a feature
extractor $f$, implemented by a Convolutional Neural Network (CNN) that
encodes a $d$-dimensional input (here $d=512\times 512$ pixels) into a feature
vector of size 256, with a binary classifier $g$ that receives the feature
vector and returns a value in the unit interval $[0,1]$. To summarize,
$h:\leavevmode\nobreak\ {\mathbb{R}}^{d}\to[0,1]$ such that $h(x)=g(f(x))$. We
remark that training $h$ requires the collection, annotation, and curation of
$n_{s}$ pairs of input images with their respective labels, which is time-
consuming and costly.

To grant these functions the ability to learn with global labels only, we
propose an attention-based MIL DL architecture [43] that can predict the
presence of hemorrhage on entire examinations of arbitrary length $r$
directly, which we denote $H:\leavevmode\nobreak\ {\mathbb{R}}^{dr}\to[0,1]$.
This predictor accepts an entire stack of images as input and it predicts the
presence or absence of ICH in it. Unlike the previous case, training such a
predictor $H$ only requires collecting $n_{w}$ training samples of pairs
$\\{(X_{i},Y_{i})\\}_{i=1}^{n_{w}}$, where $Y_{i}$ are the global labels of
the examinations—and thus, the local labels of each image are not needed.
Since there are a large number of images per examination ($r$ is about 30 for
a typical scan), the number of examination labels is much lower than that of
image labels, i.e. $n_{w}\ll n_{s}$.

The predictor $H$ is a multiple instance learning (MIL) model, as it receives
as input a collection (i.e., a bag) of $r$ images (i.e., instances). MIL has a
long tradition in machine learning [29, 74, 76, 75] and in biomedical imaging
in particular [4, 68, 18, 88]. However, its applications to the task of ICH
detection remain limited [72, 91, 57]. Similarly to previous works [72, 91,
57] we use the attention-based MIL framework recently developed in [43], which
parametrizes such an MIL predictor $H$ by composing $r$ instance-wise encoders
with an attention mechanism [9, 86], $a$, and a final classifier $g$.
Succinctly, we can write
$H(X)=g(a([f(x^{(1)}),f(x^{(2)}),\dots,f(x^{(r)})]))$. This MIL predictor, as
well as the strongly supervised one, are depicted in Fig. 1.

Importantly, to make comparisons between the local (strong) and global (weak)
predictors, the feature extractor $f$ that encodes each image in an
examination, as well as the binary classifier $g$, in $H$ are the same as the
ones described in the context of traditional supervised learning and used in
the image-level predictor $h$. Moreover, if an examination $X$ contains a
single image (i.e., $r=1$) the attention mechanism $a$ reduces to the identity
map. It follows that for these cases $H(X)=g(f(X))=h(x)$, and thus $H$ is
equivalent to the fully supervised predictor $h$. For this reason, the MIL
model $H$ generalizes the image-wise predictor $h$ while maintaining the core
feature extractor and classifier ($f$ and $g$, respectively). In this work, we
compare the resulting image-wise classifier $h$, trained using the local
annotations from every image, and examination-wise classifier $H$, trained
using only global labels for every examination.

###  2.3 MIL provides comparable performance on examination-level binary
classification

We compare the strong and weak learners on the examination-level binary
classification problem, i.e. the task of predicting whether a new examination
$X$ (with $r$ images) contains any signs of hemorrhage. For the MIL learner
$H$, the examination-level prediction is simply ${\widehat{Y}}_{w}=H(X)$. On
the other hand, the strongly supervised predictor $h$ can predict on single
images only. Since the ground-truth examination-level label $Y$ can be
expressed as the logical OR of the labels of the images in the examination, it
is natural to define the examination-level prediction of $h$ as
${\widehat{Y}}_{s}=\max(h(x^{(1)}),h(x^{(2)}),\dots,h(x^{(r)}))$, which
extends the logical OR to real-valued functions on the unit interval $[0,1]$.

Fig. 2 shows the ROC curves with their AUC’s for the strong learner $h$ (i.e.,
${\mathcal{SL}}$: _strong learner_) and the MIL learner $H$ (i.e.,
${\mathcal{WL}}$: _weak learner_) on the validation split of the RSNA dataset
as well as the CQ500 and CT-ICH datasets. AUC’s are compared via a one-sided
DeLong’s test [27]. Figs. 2(a) and 2(b) demonstrate that there is virtually no
difference in performance between the strong and the weak learner on the
validation split of the RSNA dataset and the CQ500 dataset, respectively. The
strong learner achieves an AUC of $0.961$, whereas the weak learner obtains an
AUC of $0.960$ ($p=0.636$) on the validation split of the RSNA dataset, and of
$0.901$ and $0.921$ ($p=0.147$), respectively, on the CQ500 dataset. In fact,
Fig. 2(c) suggests that the weak learner has significantly better
generalization power on the CT-ICH dataset (AUC’s of $0.924$ and $0.954$,
respectively, $p=0.032$).

(a) RSNA dataset
$(p=0.636)$. (b) CQ500 dataset
$(p=0.147)$. (c) CT-ICH dataset
$(p=0.032)$.

Figure 2: Comparison of a strong learner (${\mathcal{SL}}$) and an MIL learner
(${\mathcal{WL}}$) on the examination-level binary classification problem.
AUC’s are compared via a one-sided DeLong’s test.

###  2.4 MIL provides comparable performance on examination-level hemorrhage
detection

Recall that we refer to _examination-level_ hemorrhage detection as the task
of retrieving the positive images within a positive examination. That is,
identifying those images in a scan that show signs of hemorrhage (if any are
present). For the strong learner $h$, this is no different than predicting the
presence of hemorrhage individually on each of the images in the examination,
and selecting the predicted positive images. For the MIL learner $H$, on the
other hand, there is no unique way to perform this image-wise selection. A
very popular approach relies on employing the attention mechanism $a$ as an
instance selector, since this function explicitly assigns weights (between 0
and 1) to each instance in the bag, thus reflecting some notion of importance
towards the overall label of the bag [72, 53, 77, 71]. Alternatively, one can
resort to other notions of importance, such as those based on game-theoretic
principles. Shapley coefficients provide a natural way to do this by assigning
scores to each image that reflect their contributions towards the overall
examination prediction [79, 58, 82]. We explore both of these approaches here.

Intuitively, we expect an accurate MIL predictor $H$ to assign large attention
weights to the positive images within a positive examination. Then, we select
those images whose attention weights are no smaller than a certain threshold,
$t$. In this work we use $t=1/r$, which corresponds to uniform attention
across all $r$ images in an examination, but this choice for $t$ is not
crucial and other options exist.999Indeed, since we employ sparse-max [60, 67,
23] for the attention mechanism, this behavior is relatively independent of
the chosen threshold. Although attention weights are extensively used in
recent literature [72, 53, 77, 71] to select important instances, their
theoretical underpinnings remain scarce [45, 31]. On the contrary, the Shapley
value [79] has gained substantial popularity in the machine learning
literature [24] because of its precise theoretical properties. Here, we
introduce the first Shapley-based explanation method specifically designed for
deep set predictors [94] (such as the MIL predictor $H$) by extending _h-Shap_
[82], a hierarchical extension of the Shapley value (see Section 4.5.1 for
details).101010h-Shap is available at https://github.com/Sulam-Group/h-shap.
Similarly to the attention-based selection method, we select the images which
have a Shapley value no smaller than $t=1/r$.

(a) RSNA dataset (b) CT-ICH dataset

Figure 3: Comparison of a strong learner (${\mathcal{SL}}$) and a weak learner
(${\mathcal{WL}}$) on examination-level hemorrhage detection. 3(a) Average
recall (TPR) as a function of hemorrhage sequence length on the validation
split of the RSNA dataset. 3(b) Average TPR on the CT-ICH dataset.

We compare the strong and the weak learners on examination-level hemorrhage
detection by means of their examination-level $f_{1}$ score over the true
hemorrhage sequences (i.e., series of consecutive positive images) in true
positive examinations (see Section 4.6 for details). Fig. 3(a) shows the
average recall on the RSNA validation split as a function of the number of
consecutive positive images in a hemorrhage sequence. We can appreciate how
there is no significant gap between the performance of the strong learner
${\mathcal{SL}}$ compared to the weak learner ${\mathcal{WL}}$, with either
detection strategy. Fig. 3(a) shows that, as expected, it is in general harder
for all learners to detect short hemorrhage sequences that may comprise only a
few consecutive positive images. Fig. 3(b) shows the average recall on the CT-
ICH dataset. In this case, we do not stratify the results as a function of
hemorrhage sequence length given the relatively small amount of examinations
in the dataset. We see here as well that there in no significant
generalization power difference across learners or detection strategies, with
a slight advantage for the weak learner with the Shapley-based selection
method.

(a) CQ500 dataset.

(b) CT-ICH dataset.

Figure 4: Example saliency maps on some predicted positive images that contain
hemorrhage. Saliency maps where obtained by applying GRAD-CAM to the last
convolutional layer of both the strong learner (${\mathcal{SL}}$) and the weak
learner (${\mathcal{WL}}$), and by means of h-Shap. All saliency maps are
thresholded using Otsu’s method to reduce noise.

###  2.5 MIL provides comparable performance on image-level hemorrhage
detection

_Image-level_ hemorrhage detection refers to the localization of the
hemorrhage, or its signs, within the selected images (e.g., hyperdense
regions, spots, asymmetries). Recall that in this work, we train learners on
binary classification tasks either locally (on images for the strong learner
$h$), or globally (on examinations for the MIL learner $H$) rather than on a
segmentation task. In particular, the RSNA dataset does not provide ground-
truth segmentations of the bleeds. In this context, one can attempt to
localize the signs of hemorrhage by explaining the models’ predictions and
find their most important features (i.e., pixels) in order to bridge
classification with detection. To this end, we use and compare two machine
learning explainability methods [14, 56]: $(i)$ Grad-CAM [78]111111Grad-CAM is
available at https://github.com/jacobgil/pytorch-grad-cam., which is a
saliency method based on sensitivity analysis and very popular in radiology
[63, 26, 19], and $(ii)$ h-Shap [82], an efficient hierarchical extension to
Shapley-based image explanations that provably retains several of the
theoretical benefits of game theoretic explanations (see Section 4.5.2 for
details). To our knowledge, this is the first time explanation methods have
been used to explain the predictions of a bag-level [4] MIL classifier at the
pixel-level [18]. Even though the saliency maps produced by explanation
methods provide a weaker sense of localization, they allow users to interpret
a model’s prediction and investigate their complex mechanisms.

(a) CQ500 dataset. (b) CTICH dataset.

Figure 5: Image-level hemorrhage detection performance for a strong learner
(${\mathcal{SL}}$) and weak learner (${\mathcal{WL}}$) on the CQ500 and CT-ICH
datasets with saliency maps obtained with GRAD-CAM and h-Shap. The $f_{1}$
scores are computed between the binarized saliency maps and the ground-truth
bounding box annotations. For a fair comparison, we show the $f_{1}$ score
distributions of true positive images explained by both models (i.e., 1,162
images for the CQ500 dataset and 130 images for the CT-ICH dataset).

Fig. 4 presents an example of saliency maps for both strong and weak learners,
using both Grad-CAM and h-Shap, for every type of hemorrhage in the CQ500 and
CT-ICH datasets: epidural (EDH), intraparenchymal (IPH), intraventricular
(IVH), subarachnoid (SAH), and subdural (SDH). We remark that the CT-ICH
dataset provides manual segmentations of the ground-truth lesions, while we
use the BHX extension of the CQ500 dataset to obtain ground-truth bounding-
boxes for this latter case. Fig. 4 demonstrates that saliency maps produced by
either predictor align well with the ground-truth annotations, with no clear
advantage of the strongly supervised model. Interestingly, we can also
appreciate how the saliency maps concentrate around the target lesions rather
than other findings that may correlate well with the presence of ICH in the
training set (such as external hematomas due to injury, midline shift effects,
or compression of the ventricles).

We further quantitatively evaluate the alignment of the binarized saliency
maps with the ground-truth annotations via pixel-level $f_{1}$ scores. Fig. 5
depicts the distribution of these $f_{1}$ scores stratified by hemorrhage type
for the CQ500 and CT-ICH datasets. For a fair comparison between the strong
learner (${\mathcal{SL}}$) and the MIL weak learner (${\mathcal{WL}}$) we show
the distribution of the $f_{1}$ scores for true positive images that were both
predicted by the strong learner to contain signs of hemorrhage and selected by
the weak learner via Shapley coefficients thresholding. In particular, we
compare 1,162 images from the CQ500 dataset and 130 images from the CT-ICH
dataset. Fig. 5 confirms that there is no clear advantage of strong
supervision for image-level hemorrhage detection. Specifically, there is no
combination of learner and explanation method that consistently outperforms
all others across all types of hemorrhage and datasets. Thus, these results
suggest that image-level hemorrhage detection can be performed with comparable
performance completely without image-level labels.

###  2.6 Attention-based MIL significantly reduces the number of labels
required

So far, we investigated the performance of a strong and a weak learner trained
on the entire training split of the RSNA dataset—as typically done in these
applications [80, 13, 38]. We now study this behavior as a function of the
number of labels available to each model during training, $m$. For the strong
learner, $m$ thus refers to the number of labeled images, whereas for the weak
learner, which only has access to examination-level information, $m$ refers to
the numbers of labeled examinations. Note that this quantification is useful
because, if the cost of obtaining a label is comparable in both image- and
examination-wise cases, this number $m$ reflects an overall cost associated
with the annotation of a dataset. Note that in practice, it is a much easier
and faster task for an expert radiologist to quickly scroll through the images
in an examination and determine whether the whole scan has signs of
hemorrhage, rather than having to label all the images in the scan
individually. In order to account for the increased variance of the training
process with smaller number of samples, we repeat the training process an
increasing number of times on random subsets of images or examinations as $m$
decreases (see Section 4.7 for details). The obtained models are evaluated on
a fixed subset of 1,000 examinations from the validation split of the RSNA
dataset.

Fig. 6 shows the mean AUC’s of strong and weak learners, with their 95%
confidence intervals, on the examination-level binary classification problem
as a function of the number of labels available to each model during training,
$m$. MIL learners show a slight advantage over strong learners on the CQ500
and CT-ICH datasets, while they overlap for the most part with strong learners
on the validation split of the RSNA dataset. Furthermore, the performance of
MIL learners show a larger variance compared to strong learners. These results
suggest that although MIL learners can provide comparable or better
performance than strong learners on the examination-level binary
classification problem, they might be harder to train. This agrees with
intuition that the MIL framework does provide a weaker sense of supervision,
and the learners might need to disambiguate the true concept (i.e., ICH) from
others that might correlate well with examination-level labels.

(a) RSNA dataset (b) CQ500 dataset (c) CT-ICH dataset

Figure 6: Mean performance and 95% confidence interval of strong
(${\mathcal{SL}}$) and weak (${\mathcal{WL}}$) learners on the examination-
level binary classification problem as a function of number of labels $m$. For
the RSNA dataset, we validate models on a fixed subset of 1,000 examinations.
Note that the points after $m=1\times 10^{4}$ have zero variance because we
repeat the training process only once.

Finally, Fig. 7 depicts the mean hemorrhage-level detection $f_{1}$ scores and
their 95% confidence intervals over the validation split of the RSNA dataset
as a function of the numbers of labels available. Confidence intervals are
computed across repetitions of the training process with the same number of
labels, thus capturing the variance of the training process. Since we train
only one model for $m>10^{4}$ labels, the variance vanishes. We see that for
$m\leq 10^{4}$, strong supervision does in fact provide a significant
advantage over weak supervision. However, MIL learners quickly outperform
strongly supervised ones for $m\gtrsim 10^{4}$. Importantly, these results
confirm that attention-based models trained on examination-level binary labels
can provide comparable performance to traditional classifiers trained on
image-level labels while requiring $\approx 35$-times fewer labels. Note that
the curves for the weak learners interrupt after $10^{4}$ labels because they
reach the limit of the training data size—the training split of the RSNA
dataset contains $\approx 17\times 10^{4}$ labeled examinations, whereas there
are about $35$ times more image labels.

Figure 7: Mean examination-level hemorrhage detection $f_{1}$ score as a
function of number of labels $m$ on a fixed subset of 1,000 examinations from
the validation split of the RSNA dataset.

##  3 Discussion

In this study, we compared the performance of predictive models for ICH
detection in Head CT scans trained with strong supervision (i.e., having one
label per image within an examination) or weak supervision (i.e., using a
single label for each entire examination). The methodology is based on recent
Multiple Instance Learning (MIL) approaches via attention mechanisms [86, 43],
which strictly generalizes predictors that are trained with strong
supervision. This framework enabled the use of models that have the same
architecture and main components in either setting, making these comparisons
precise and fair. We found that weakly supervised models had comparable
performance to strongly supervised models, despite using approximately 35
times fewer labels. On one external dataset, the weakly supervised models
actually had significantly higher performance, suggesting better
generalizability. Importantly, weakly supervised models also had comparable
ability to localize ICH on image- and pixel-levels. Altogether, these findings
indicate that image-level annotations are not necessary to train high-
performing and explainable DL models for diagnosis of ICH on head CT.

Our first result demonstrated that strong supervision is not at all necessary
for weak, or global, prediction tasks, as long as sufficient data is
available. More precisely, Fig. 2 demonstrates that whether a predictor is
trained on image-wise or examination-wise labels, they obtain virtually the
same AUC in the task of predicting the presence or absence of ICH. In one of
the studied datasets (CT-ICH [41]), there is in fact a slight advantage to the
latter, suggesting better generalizability to different clinical populations.
This is not surprising, as the weak learner is precisely trained on the task
of detecting hemorrhage at the examination level. Nevertheless, this
generalizability advantage is important, given the well-documented drops in
performance of DL models for medical imaging diagnosis on external test sets
[95], which threaten the safe deployment of DL models in real-world clinical
practice.

Interestingly, our results further demonstrate that these observations hold
even for the case of examination-level hemorrhage detection, i.e. the task of
finding the images within each examination where signs of hemorrhage are
present. More precisely, even though the MIL model was only trained on global
examination-wise labels, one can make predictions about each of the
constituent images by either studying their attention weights, or by employing
game-theoretic tools like the Shapley value [79, 58]. In either case, the
ability to detect the positive images within positive examination is
comparable to the performance of a model trained with strong supervision, i.e.
with a label per image (Fig. 3(b)). We emphasize that the ability to identify
examination-level hemorrhage is of critical practical importance for
radiologists’ workflows. In current triage use cases of DL models in
radiology, cases with potentially actionable findings, such as ICH, are
flagged by DL models for radiologist review, after which a radiologist must
confirm whether they agree or disagree with the diagnosis. Having image
“flags” beyond the mere presence or absence of hemorrhage that show which
specific images have a hemorrhage prediction are critical for: $(i)$ allowing
a radiologist to expeditiously confirm presence or absence of hemorrhage, and
$(ii)$ building trust with radiologists and other physician end-users, who
have been shown to be less trusting of diagnostic results generated by
automated systems in medical imaging compared to those provided by human
experts [34].

When further analyzing the resulting models with both a popular saliency
method (Grad-CAM [78]) as well as newer approaches to interpretability with
theoretical guarantees (h-Shap [82]), both models indicate having captured the
same semantic concepts that constitute ICH in head CT (Fig. 4). Indeed, our
qualitative and quantitative comparisons of these saliency maps indicate that
the ability to find the corresponding hemorrhages within each of the images is
virtually the same, and only mild differences exist once stratified per ICH
type (Fig. 5). To this point, we remark that to verify whether a model did
learn the desired concept (i.e., ICH) instead of other spurious correlations
in the training data is especially important in medical imaging. As modern
machine learning models continue becoming increasingly complex, gaining
insights on their decision making process is fundamental for a responsible use
in real-world scenarios. As discussed above, building trust with physician
end-users is paramount, and providing pixel-level explanations for specific
disease identification will be important towards this goal. Furthermore,
medical institutions may be required by certain laws to provide explanations
of what lead an automatic systems to recommend a certain treatment or to
provide a specific diagnosis [37].

Our work also has limitations. First, we evaluated only a single diagnostic
use case of ICH detection on CT scans of the head, albeit with multiple
datasets from different clinical populations. However, our approach is
applicable to any other medical imaging use case that utilizes cross-sectional
imaging, including diagnosis of disease on CT of other body parts, as well as
on other imaging modalities, such as MRI. Future studies will apply our
approach to other use cases to validate its generalizability in other
diagnostic scenarios and imaging modalities. Second, while this study
demonstrated that indeed the examination-level annotations suffice for ICH
detection in CT once enough training data is available, some image-level
annotations were needed to validate our methodology. In future extensions to
other diagnostic tasks or imaging modalities, this minimal amount of locally
annotated data will also be necessary for validation purposes. This number of
local annotations is very small, however: in this work we employed 1,000
examinations of the validation split of the RSNA dataset to this end,
requiring about $35\times 10^{3}$ image-level labels. This represents less
than 6% of the number of image-level labels needed to train an alternative
strongly supervised model. Third, the weakly supervised method currently only
evaluates medical imaging data; given the potential improvements in imaging
diagnoses using multimodal AI models [1] incorporating multiple types of
medical data (e.g., imaging, clinical symptoms, laboratory values), developing
weakly supervised DL models that can incorporate multiple data types is an
important topic for future study. Finally, although Convolutional Neural
Networks (CNNs)—such as the models used in this work—remain the most popular
deep learning architecture in medical imaging, it remains important to
investigate whether these results extend to other parametrization of the
predictors and architecture choices, for example to Vision Transformers (ViTs)
[30, 49]—which are rapidly gaining popularity in the field.

In summary, our results indicate that training DL models with weak or strong
supervision provides comparable performance for the tasks of ICH detection in
head CT across three different levels of granularity: $(i)$ global binary
prediction, $(ii)$ examination-level detection, and $(iii)$ image-level
detection. Our last results explore these points further by studying the
performance of strong and weak learners on the global binary classification
problem, as well as on examination-level hemorrhage detection as a function of
the number of labels available during training. These results indicate that,
indeed, weakly supervised learning enables a significant reduction in the need
for annotations: once the number of labels provided is large enough ($m\gtrsim
5\times 10^{3}$) weakly supervised models achieve comparable performance to
strongly supervised models at a fraction of the provided labels. However, for
the strongly supervised predictor, these labels represent the number of
labeled images, whereas the for the MIL predictors, $m$ represents only global
information of the entire examination—which can be easily collected, e.g. from
clinical reports. This approach could apply to other 3D cross-sectional
imaging tasks, such as MRI diagnosis, potentially saving thousands of hours of
annotation labor by radiologists [33], thereby alleviating the biggest
bottleneck in developing high-performing DL models for medical imaging
diagnosis.

##  4 Methods

###  4.1 Learning paradigms

####  4.1.1 Supervised learning

In supervised learning settings, given input and output domains
${\mathcal{X}}$ and ${\mathcal{Y}}$, one is interested in predicting a
response $y\in{\mathcal{Y}}$ on an input $x\in{\mathcal{X}}$ with a predictor
$h:\leavevmode\nobreak\ {\mathcal{X}}\to{\mathcal{Y}}$. Given a loss function
$\ell(y,h(x))$ that penalizes the dissimilarity between the true label $y$ and
the predicted label $h(x)$, we search for a predictor $h$ with low risk over a
suitable family of predictors (e.g., Convolutional Neural Networks). This
search is usually carried out by minimizing the empirical loss over a training
set $\\{(x_{i},y_{i})\\}_{i=1}^{n_{s}}$ such that

$h=\operatorname*{arg\,min}_{h^{\prime}}\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}\ell(y_{i},h^{\prime}(x_{i})).$
(1)

####  4.1.2 Multiple-Instance Learning (MIL)

Multiple Instance Learning (MIL) [29, 59, 89] generalizes the supervised
learning framework to _bags_ of inputs. Formally, recall that ${\mathcal{X}}$
and ${\mathcal{Y}}$ are input and output domains, and let
$X=(x^{(1)},x^{(2)},\dots,x^{(r)})\in{\mathcal{X}}^{r}$, $r\in{\mathbb{N}}$
indicate a bag with $r$ _instances_. Furthermore, the MIL paradigm assumes
that the bag-level response $Y\in{\mathcal{Y}}$ is a known function of the
instance-level responses $y^{(1)},y^{(2)},\dots,y^{(r)}$, which can encompass
a wide variety of choices [11, 8, 6, 74, 75, 88].

In this work, we focus on MIL binary classification, such that

$Y=\texttt{OR}(y^{(1)},y^{(2)},\dots,y^{(r)}),$ (2)

and we search for a _bag-level_ classifier $H:\leavevmode\nobreak\
{\mathcal{X}}^{r}\to{\mathcal{Y}}$ with low risk over a suitable class of
predictors. Similarly to the supervised learning paradigm, given a loss
function $\ell(Y,H(X))$ that penalizes wrong predictions, $H$ is found by
optimization of the empirical loss over a training set
$\\{(X_{i},Y_{i})\\}_{i=1}^{n_{w}}$ of labeled bags, such that

$H=\operatorname*{arg\,min}_{H^{\prime}}\frac{1}{n_{w}}\sum_{i=1}^{n_{w}}\ell(Y_{i},H^{\prime}(X_{i})).$
(3)

Importantly, we remark that an MIL learner does not have access to the
underlying instance-level labels. Finally, note that:

  * •

The examination-level binary classification problem satisfies the MIL
assumption in Eq. 2, as the global label of an examination is positive as soon
as it contains at least one image with signs of hemorrhage (i.e., a positive
image). Equivalently,

  * •

The local image-level labels can also be phrased as an instance of Eq. 2. In
particular, an image should be labeled positively as soon as it contains signs
of hemorrhage.

###  4.2 Model architecture details

####  4.2.1 Strong learner

The strong learner $h$ is the composition of a feature extractor $f$ with a
binary classifier $g$ implemented by a fully connected layer with sigmoid
activation. In this work, $f$ is a ResNet18 [40] pretrained on ImageNet [28]
that encodes an input image of size $512\times 512$ pixels into a vector of
size 256, as illustrated in Fig. 1(a).

####  4.2.2 Weak learner

In addition to the same feature extractor $f$ and final classifier $g$
employed for the strong learner, the weak learner $H$ comprises a two-layer
attention mechanism $a$ as proposed in [43] (see Fig. 1(b)). For an input
examination with $r$ images, the attention mechanism combines the $r$ image-
level feature vectors into a single examination-level feature vector which can
be expressed as a convex combination of the image-level feature vectors. In
this work—differently from the original work in [43]—we use the _sparsemax_
activation function [60, 67, 23] rather then the softmax function to favor
sparse attention weights.121212The entmax package is available at:
https://github.com/deep-spin/entmax.

###  4.3 Data preprocessing

The images in the three datasets used in this work were annotated by expert
neuroradiologists of varying degree of expertise with the type(s) of
hemorrhage present in the image. We group the original classes into ‘normal’
(i.e., label 0, no type of hemorrhage) and ‘with hemorrhage’ (i.e., label 1,
any type of hemorrhage). Images are provided in DICOM and NIfTI format, so we:

  1. 1.

Convert them to Hounsfield Units (HUs) [15], then

  2. 2.

Window them using the standard brain window setting, i.e. WL = 40 and WW = 80
[83], and finally

  3. 3.

Normalize them with min-max normalization.

This way, pixel intensities represent the same HU value (and hence, tissue)
across all datasets.

###  4.4 Training procedures

Experiments were performed on Nvidia Quadro RTX 5000 GPU’s on a private
cluster and on the Azure Machine Learning (ML) platform [10] via Microsoft
Research’s Project InnerEye open-source software tools
(https://aka.ms/InnerEyeOSS).131313Project InnerEye is available at:
https://github.com/microsoft/InnerEye-DeepLearning.

To account for the high label imbalance in the training split of RSNA dataset
and for the gap in difficulty between the prediction of the presence of
hemorrhage compared to predicting its absence, models were trained using
_focal loss_ [55]—a variation of binary cross-entropy loss. All models were
trained for 15 epochs with a learning rate decay of 0.3 every 3 epochs. We
chose the best performing model across epochs according to validation
accuracy.

####  4.4.1 Image-level augmentation

We use TorchIO’s [65] library of spatial and intensity
transformations.141414TorchIO is available at
https://github.com/fepegar/torchio. Specifically, every image is augmented
independently via random flips, affine transformations, deformations and
rotations, and one out of addition of random noise, random bias field, random
anisotropy, random gamma transformation, or random ghosting artifacts.

####  4.4.2 Examination-level augmentation

We randomly sample without replacement between 10 and $r$ images within the
same examination. This sub-sampling augmentation strategy does not rely on
image-level labels and it can be used in weakly supervised scenarios where
only examination-level labels are available. Intuitively, sampling at least 10
images controls the probability of flipping a positive examination to a
negative one. That is, sampling a subset of all negative images from a
positive examination would result in a wrong label (i.e., the subset would be
labeled positively even if it did not contain any positive images). Although
we cannot completely rule out this event without knowing local labels, we can
reduce its probability to a tolerable level for the weak learner. Formally,
given a positive examination $X$ of length $r$ with $K$ positive images,
sample a random subset $S$ of images without replacement. Denote $Y_{S}$ the
true global label of the subset, and note that
$p_{\text{flip}}={\mathbb{P}}[Y_{S}=0\mid Y=1]$ is a decreasing function of
the size of the subset $S$ and it follows a hypergeometric distribution. In
this work, we estimate $p_{\text{flip}}$ over the training split of the RSNA
dataset, and obtain $p_{\text{flip}}\leq 4\times 10^{-3}$. We remark that to
estimate $p_{\text{flip}}$ we used the image-level labels provided in the
training split of the RSNA dataset. In practical scenarios this information
can easily be replaced by prior knowledge of expert radiologists about the
problem at hand.

####  4.4.3 Training strong learners

Strong learners were trained using Adam optimizer [50] with learning rate of
$1\times 10^{-5}$, weight decay of $1\times 10^{-7}$, and batch size of 64.
During training, we add a dropout layer with $p=0.50$ between the encoder $f$
and the binary classifier $g$.

####  4.4.4 Training weak learners

Weak learners were trained using Stochastic Gradient Descent (SGD) with
momentum equal to $0.9$ [81], learning rate of $1\times 10^{-3}$, weight decay
of $1\times 10^{-4}$, and batch size of 1. We remark that the choice of batch
size equal to 1 comes both from memory limitations and gradient propagation
imbalances through the attention mechanism for volumes with different numbers
of images. During training, we add both a dropout layer with $p=0.50$ between
the encoder $f$ and the binary classifier $g$ and a dropout layer with
$p=0.25$ after the first layer of the attention mechanism.

###  4.5 Explaining model predictions with h-Shap

We use h-Shap [82], a Shapley-based explanation method with provable runtime
and accuracy for problems that satisfy the binary MIL assumption in Eq. 2 to
select the positive images in an examination, and to highlight signs of
hemorrhage within the selected images.

####  4.5.1 Examination-level hemorrhage detection

We extend the original implementation of h-Shap to explain the examination-
level prediction of a weak learner. Since the global binary label satisfies
Eq. 2, one can explore a binary tree of the input examination and
hierarchically compute the exact Shapley coefficient of every image in the
examination [see 82, Theorem 3.4]. The symmetry axiom of the Shapley value
[79] implies that the positive images in an examination should receive the
same coefficient. Thus, one can use an importance threshold $t=1/r$ and select
those images whose Shapley values are $\geq t$. We remark that—as recently
noted by others [25, 44]—explaining predictions on sets with the Shapley value
is particularly attractive because it does not require to sample an
uninformative baseline to mask features [58]. In fact, the weak learner can
predict on sequences of arbitrary length and it is permutation invariant [94],
hence one can simply remove images from an examination without having to
replace them.

####  4.5.2 Image-level hemorrhage detection

Bleeds can present complex and irregular shapes. However, h-Shap explores
fixed quad-trees of the input image. Thus, we extend the original
implementation with standard ideas of cycle spinning [22]. Denote $s$ the
minimal feature size in h-Shap (i.e., the size of the smallest leaf explored
by the algorithm), and let $\bm{\rho}=\\{\rho_{i}\\}_{i=1}^{n_{\rho}}$ be
$n_{\rho}$ equally spaced radii between 0 and $s$, and let
$\bm{\alpha}=\\{\alpha_{i}\\}_{i=1}^{n_{\alpha}}$ be $n_{\alpha}$ equally
spaced angles between 0 and $2\pi$. Then, we average the saliency maps
obtained by cycle spinning the original partition by the vector
$(\rho\cos(\alpha),\rho\sin(\alpha))$, $\rho\in\bm{\rho},\leavevmode\nobreak\
\alpha\in\bm{\alpha}$. Finally, we note that we use the unconditional
expectation over the training split of the RSNA dataset to mask features,
which is a valid choice in MIL binary classification problems [82]. In this
work, we use h-Shap with an absolute importance tolerance $\tau$ equal to 0
(i.e. h-Shap explores all partitions with a positive Shapley coefficient),
minimal feature size $s=64$, number of radii $n_{\rho}=3$, and number of
angles $n_{\alpha}=12$.

###  4.6 Comparing strong and weak learners on examination-level hemorrhage
detection

In this section we expand on the methodology and choice of parameters for
comparing strong and weak learners on examination-level hemorrhage detection.
All choices were made to provide a fair comparison between strongly supervised
and weakly supervised models.

####  4.6.1 Choosing the classification threshold

Examination-level hemorrhage detection is performed only for predicted
positive examinations. Recall that both strong and weak learners are real-
valued functions on the unit interval $[0,1]$. Thus, a threshold $t\in[0,1]$
(e.g., $0.5$) is required to binarize their predictions. The choice of $t$
induces a False Positive Rate (FPR) and a True Positive Rate (TPR) on images
(for a strong learner) or on examinations (for a weak learner). In this work,
we use Youden’s $J$ statistic [93, 66, 39] to find the threshold $t^{*}$ that
maximizes the difference of TPR and FPR, i.e. $J=\text{TPR}-\text{FPR}$. Then:

  * •

For a strong learner $h$, we choose the threshold $t^{*}_{s}$ that maximizes
$J$ on the image-level labels, and,

  * •

For a weak learner $H$, we choose the threshold $t^{*}_{w}$ that maximizes $J$
on the examination-level labels.

We remark that there exist other methods to choose the threshold $t^{*}$, and
the main results discussed in this work do not strongly depend on this choice.
For completeness, Figs. A.2 and A.3 show the equivalent of Figs. 3 and 7 where
instead of maximizing Youden’s $J$, $t^{*}$ is chosen to minimize the distance
to the $(0,1)$ point (perfect classification), which can be written as
$d=\sqrt{\text{FPR}^{2}+\left(1-\text{TPR}\right)^{2}}$ and is also common in
the literature [66, 39].

####  4.6.2 Choosing the best minimal sequence length

To reduce the false positive rate in the predicted hemorrhage sequences, we
fine-tune the minimal number of consecutive positive images that have to be
selected by each method in order for a series of consecutive selected images
to be considered a candidate hemorrhage sequence. We set this length to 4 for
strong learner, to 2 for weak learners when using attention weights to select
images, and to 3 for weak learners when using Shapley values, guaranteeing the
best performance for each method. Fig. A.1 depicts the examination-level
$f_{1}$ score as a function of this minimal sequence length on the validation
split of the RSNA dataset with both Youden’s $J$ and distance to the $(0,1)$
point, which motivate these choices.

####  4.6.3 Computing the examination-level $f_{1}$ score

Denote $T=\\{T_{1},T_{2},\dots,T_{n}\\}$ the set of true hemorrhage sequences
(i.e., non-overlapping series of consecutive positive images) in a positive
examination. That is, $T_{i}$ contains the indices of the images in the
$i^{\text{th}}$ hemorrhage sequence. Let $S=\\{s_{1},s_{2},\dots,s_{r}\\}$ be
the local estimator used to select positive images depending on the type of
learner: single-image logits for a strong learner, and either attention
weights or Shapley values for a weak learner. Denote
$P=\\{P_{1},P_{2},\dots,P_{n}\\}$ the predicted hemorrhage sequences by the
learner. We define the True Positive (TP), False Positive (FP), and Predicted
Positive (PP) sequences as

$\displaystyle{\text{TP}}\coloneqq\\#\\{i\in[n]:\leavevmode\nobreak\ \exists
P_{j}\in P:\leavevmode\nobreak\ (\operatorname*{arg\,max}_{k\in
P_{j}}\leavevmode\nobreak\ s_{k})\in T_{i}\\}$ (4)
$\displaystyle{\text{FP}}\coloneqq\\#\\{j\in[m]:\leavevmode\nobreak\ \nexists
T_{i}\in T:\leavevmode\nobreak\ (\operatorname*{arg\,max}_{k\in
P_{j}}\leavevmode\nobreak\ s_{k})\in T_{i}\\}$ (5)
$\displaystyle{\text{PP}}={\text{TP}}+{\text{PP}}.$ (6)

Put into words, for every true hemorrhage sequence $T_{i}\in T$, we count one
true positive prediction if there exists a predicted hemorrhage sequence in
$P_{j}\in P$ such that the image with the larges estimator value within
$P_{j}$ is contained in $T_{i}$. Note that this definition of TP does not
double count predicted sequences that may correspond to the same true one, and
using the $\operatorname*{arg\,max}$ avoids the trivial case where a model may
select all images, or a few very long sequences that could include multiple
true ones. Similarly, we count one false positive prediction for every
predicted sequence $P_{j}$ for which there does not exists a corresponding
true one. The $f_{1}$ score is then defined as the harmonic mean of precision
and recall, i.e.

$\displaystyle\text{precision}=\frac{{\text{TP}}}{{\text{PP}}},\quad\text{recall}=\frac{TP}{\lvert
T\rvert}$ (7) $\displaystyle
f_{1}=2\cdot\frac{\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}}.$
(8)

We note that this procedure reflects how a machine learning model could be
deployed in a clinical setting to detect hemorrhage sequences.

###  4.7 Training multiple times on the same number of labels

When training models on a fixed number of labels $m$, we randomly sample
without replacement a subset of the original training split of the RSNA
dataset (of images for the strong learners, and of examinations for the weak
learners) that maintains the label proportions of the original dataset. In
particular, we use 15 distinct values of $m$: $\bm{m}=[12,\ 24,\ 40,\ 52,\
64,\ 80,\ 100,\ 152,\ 200,\ 252,\allowbreak\ 520,\ 796,\ 10^{3},\ 10\times
10^{3},\ 17\times 10^{3}]$. For each choice of $m$, we train a decreasing
number of models to account for the variance in the training process. In
particular, we repeat the training process 20 times when $m\leq 252$; 15 times
for $m=520$; 10 times for $m=796,\ 10^{3}$; 6 times for $m=10\times 10^{3}$;
and 1 time for $m\geq 17\times 10^{3}$. Finally, instead of training models
for a fixed number of epochs, we set a _patience_ parameter such that training
is terminated if the validation accuracy of a model does not increase for more
than 3 consecutive epochs.

##  Appendix A Figures

(a) Youden’s $J$ (b) Distance to $(0,1)$

Figure A.1: Examination-level $f_{1}$ score as a function of minimal sequence
length for a strong learner (${\mathcal{SL}}$) and a weak learner
(${\mathcal{WL}}$) on the validation split of the RSNA dataset. 1(a) Results
with Youden’s $J$ statistic. 1(b) Results with distance to $(0,1)$ point. Note
that the best minimal sequence length does not depend on the threshold
$t^{*}$.

(a) RSNA dataset (b) CT-ICH dataset

Figure A.2: Comparison of a strong learner (${\mathcal{SL}}$) and a weak
learner (${\mathcal{WL}}$) on examination-level hemorrhage detection. 3(a)
Average recall (TPR) as a function of hemorrhage sequence length on the
validation split of the RSNA dataset. 3(b) Average TPR on the CT-ICH dataset.
These results are computed by choosing the threshold $t^{*}$ that minimizes
the distance to the $(0,1)$ point. Figure A.3: Mean examination-level
hemorrhage detection $f_{1}$ score as a function of number of labels $m$ on a
fixed subset of 1,000 examinations from the validation split of the RSNA
dataset. These results are computed by choosing the threshold $t^{*}$ that
minimizes the distance to the $(0,1)$ point.

## References

  * Acosta et al. [2022] Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol.  Multimodal biomedical ai.  _Nature Medicine_ , 28(9):1773–1784, September 2022.
  * Ahishakiye et al. [2021] Emmanuel Ahishakiye, Martin Bastiaan Van Gijzen, Julius Tumwiine, Ruth Wario, and Johnes Obungoloch.  A survey on deep learning in medical image reconstruction.  _Intelligent Medicine_ , 1(03):118–127, 2021\.
  * Alenezi and Santosh [2021] Fayadh Alenezi and KC Santosh.  Geometric regularized hopfield neural network for medical image enhancement.  _International Journal of Biomedical Imaging_ , 2021, 2021.
  * Amores [2013] Jaume Amores.  Multiple instance classification: Review, taxonomy and comparative study.  _Artificial intelligence_ , 201:81–105, 2013.
  * An et al. [2017] Sang Joon An, Tae Jung Kim, and Byung-Woo Yoon.  Epidemiology, risk factors, and clinical features of intracerebral hemorrhage: an update.  _Journal of stroke_ , 19(1):3, 2017.
  * Andrews et al. [2002] Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann.  Support vector machines for multiple-instance learning.  _Advances in neural information processing systems_ , 15, 2002.
  * Attia et al. [2021] Zachi I Attia, David M Harmon, Elijah R Behr, and Paul A Friedman.  Application of artificial intelligence to the electrocardiogram.  _European heart journal_ , 42(46):4717–4730, 2021\.
  * Auer et al. [1998] Peter Auer, Philip M Long, and Aravind Srinivasan.  Approximating hyper-rectangles: learning and pseudorandom sets.  _Journal of Computer and System Sciences_ , 57(3):376–388, 1998.
  * Bahdanau et al. [2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ , 2014.
  * Barnes [2015] Jeff Barnes.  Azure machine learning.  In _Microsoft Azure Essentials_. Microsoft, 2015.
  * Blum and Kalai [1998] Avrim Blum and Adam Kalai.  A note on learning from multiple-instance examples.  _Machine learning_ , 30(1):23–29, 1998.
  * Buchlak et al. [2022] Quinlan D Buchlak, Michael R Milne, Jarrel Seah, Andrew Johnson, Gihan Samarasinghe, Ben Hachey, Nazanin Esmaili, Aengus Tran, Jean-Christophe Leveque, Farrokh Farrokhi, et al.  Charting the potential of brain computed tomography deep learning systems.  _Journal of Clinical Neuroscience_ , 99:217–223, 2022.
  * Burduja et al. [2020] Mihail Burduja, Radu Tudor Ionescu, and Nicolae Verga.  Accurate and efficient intracranial hemorrhage detection and subtype classification in 3d ct scans with convolutional and long short-term memory neural networks.  _Sensors_ , 20(19):5611, 2020.
  * Burkart and Huber [2021] Nadia Burkart and Marco F Huber.  A survey on the explainability of supervised machine learning.  _Journal of Artificial Intelligence Research_ , 70:245–317, 2021.
  * Buzug [2011] Thorsten M Buzug.  Computed tomography.  In _Springer handbook of medical technology_ , pages 311–342. Springer, 2011.
  * Cai et al. [2020] Lei Cai, Jingyang Gao, and Di Zhao.  A review of the application of deep learning in medical image classification and segmentation.  _Annals of translational medicine_ , 8(11), 2020.
  * Chen et al. [2022] Yutong Chen, Carola-Bibiane Schönlieb, Pietro Liò, Tim Leiner, Pier Luigi Dragotti, Ge Wang, Daniel Rueckert, David Firmin, and Guang Yang.  Ai-based reconstruction for fast mri—a systematic review and meta-analysis.  _Proceedings of the IEEE_ , 110(2):224–245, 2022\.
  * Cheplygina et al. [2019] Veronika Cheplygina, Marleen de Bruijne, and Josien PW Pluim.  Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis.  _Medical image analysis_ , 54:280–296, 2019.
  * Chien et al. [2022] Jong-Chih Chien, Jiann-Der Lee, Ching-Shu Hu, and Chieh-Tsai Wu.  The usefulness of gradient-weighted cam in assisting medical diagnoses.  _Applied Sciences_ , 12(15):7748, 2022.
  * Chilamkurthy et al. [2018] Sasank Chilamkurthy, Rohit Ghosh, Swetha Tanamala, Mustafa Biviji, Norbert G Campeau, Vasantha Kumar Venugopal, Vidur Mahajan, Pooja Rao, and Prashant Warier.  Deep learning algorithms for detection of critical findings in head ct scans: a retrospective study.  _The Lancet_ , 392(10162):2388–2396, 2018.
  * Choy et al. [2018] Garry Choy, Omid Khalilzadeh, Mark Michalski, Synho Do, Anthony E Samir, Oleg S Pianykh, J Raymond Geis, Pari V Pandharipande, James A Brink, and Keith J Dreyer.  Current applications and future impact of machine learning in radiology.  _Radiology_ , 288(2):318, 2018.
  * Coifman and Donoho [1995] Ronald R Coifman and David L Donoho.  Translation-invariant de-noising.  In _Wavelets and statistics_ , pages 125–150. Springer, 1995.
  * Correia et al. [2019] Gonçalo M Correia, Vlad Niculae, and André FT Martins.  Adaptively sparse transformers.  In _Proc. EMNLP-IJCNLP (to appear)_ , 2019.
  * Covert et al. [2021] Ian Covert, Scott Lundberg, and Su-In Lee.  Explaining by removing: A unified framework for model explanation.  _Journal of Machine Learning Research_ , 22(209):1–90, 2021.
  * Covert et al. [2022] Ian Covert, Chanwoo Kim, and Su-In Lee.  Learning to estimate shapley values with vision transformers.  _arXiv preprint arXiv:2206.05282_ , 2022.
  * Deepika et al. [2022] Pon Deepika, Prasad Sistla, Ganesh Subramaniam, and Madhav Rao.  Deep learning based automated screening for intracranial hemorrhages and grad-cam visualizations on non-contrast head computed tomography volumes.  In _2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI)_ , pages 01–05. IEEE, 2022.
  * DeLong et al. [1988] Elizabeth R DeLong, David M DeLong, and Daniel L Clarke-Pearson.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.  _Biometrics_ , pages 837–845, 1988.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dietterich et al. [1997] Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez.  Solving the multiple instance problem with axis-parallel rectangles.  _Artificial intelligence_ , 89(1-2):31–71, 1997\.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Ethayarajh and Jurafsky [2021] Kawin Ethayarajh and Dan Jurafsky.  Attention flows are shapley value explanations.  _arXiv preprint arXiv:2105.14652_ , 2021.
  * Eyuboglu et al. [2021] Sabri Eyuboglu, Geoffrey Angus, Bhavik N Patel, Anuj Pareek, Guido Davidzon, Jin Long, Jared Dunnmon, and Matthew P Lungren.  Multi-task weak supervision enables anatomically-resolved abnormality detection in whole-body fdg-pet/ct.  _Nature communications_ , 12(1):1–15, 2021.
  * Flanders et al. [2020] Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree Kalpathy-Cramer, Robyn Ball, John T Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, et al.  Construction of a machine learning dataset through collaboration: the rsna 2019 brain ct hemorrhage challenge.  _Radiology: Artificial Intelligence_ , 2(3):e190211, 2020.
  * Gaube et al. [2021] Susanne Gaube, Harini Suresh, Martina Raue, Alexander Merritt, Seth J Berkowitz, Eva Lermer, Joseph F Coughlin, John V Guttag, Errol Colak, and Marzyeh Ghassemi.  Do as AI say: susceptibility in deployment of clinical decision-aids.  _npj Digital Medicine_ , 4(1):31, 2021.  ISSN 2398-6352.  doi: 10.1038/s41746-021-00385-9.  URL https://doi.org/10.1038/s41746-021-00385-9.
  * Giger [2018] Maryellen L Giger.  Machine learning in medical imaging.  _Journal of the American College of Radiology_ , 15(3):512–520, 2018.
  * Goldberger et al. [2000] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley.  Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals.  _circulation_ , 101(23):e215–e220, 2000.
  * Goodman and Flaxman [2017] Bryce Goodman and Seth Flaxman.  European union regulations on algorithmic decision-making and a “right to explanation”.  _AI magazine_ , 38(3):50–57, 2017.
  * Gudigar et al. [2021] Anjan Gudigar, U Raghavendra, Ajay Hegde, Girish R Menon, Filippo Molinari, Edward J Ciaccio, and U Rajendra Acharya.  Automated detection and screening of traumatic brain injury (tbi) using computed tomography images: a comprehensive review and future perspectives.  _International journal of environmental research and public health_ , 18(12):6499, 2021.
  * Habibzadeh et al. [2016] Farrokh Habibzadeh, Parham Habibzadeh, and Mahboobeh Yadollahie.  On determining the most appropriate test cut-off value: the case of tests with continuous results.  _Biochemia medica_ , 26(3):297–307, 2016.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hssayeni [2020] Murtadha Hssayeni.  Computed tomography images for intracranial hemorrhage detection and segmentation.  _Intracranial Hemorrhage Segmentation Using A Deep Convolutional Model. Data_ , 5(1), 2020.
  * Hssayeni et al. [2020] Murtadha D Hssayeni, Muayad S Croock, Aymen D Salman, Hassan Falah Al-khafaji, Zakaria A Yahya, and Behnaz Ghoraani.  Intracranial hemorrhage segmentation using a deep convolutional model.  _Data_ , 5(1):14, 2020.
  * Ilse et al. [2018] Maximilian Ilse, Jakub Tomczak, and Max Welling.  Attention-based deep multiple instance learning.  In _International conference on machine learning_ , pages 2127–2136. PMLR, 2018.
  * Jain et al. [2022] Saachi Jain, Hadi Salman, Eric Wong, Pengchuan Zhang, Vibhav Vineet, Sai Vemprala, and Aleksander Madry.  Missingness bias in model debugging.  _arXiv preprint arXiv:2204.08945_ , 2022.
  * Jain and Wallace [2019] Sarthak Jain and Byron C Wallace.  Attention is not explanation.  _arXiv preprint arXiv:1902.10186_ , 2019.
  * Kaka et al. [2021] Hussam Kaka, Euan Zhang, and Nazir Khan.  Artificial intelligence and deep learning in neuroradiology: exploring the new frontier.  _Canadian Association of Radiologists Journal_ , 72(1):35–44, 2021.
  * Kang et al. [2017] Eunhee Kang, Junhong Min, and Jong Chul Ye.  A deep convolutional neural network using directional wavelets for low-dose x-ray ct reconstruction.  _Medical physics_ , 44(10):e360–e375, 2017.
  * Kawooya [2012] Michael G Kawooya.  Training for rural radiology and imaging in Sub-Saharan africa: Addressing the mismatch between services and population.  _J. Clin. Imaging Sci._ , 2, 2012.
  * Khan et al. [2022] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  _ACM computing surveys (CSUR)_ , 54(10s):1–41, 2022.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Langlotz [2019] Curtis P Langlotz.  Will artificial intelligence replace radiologists?  _Radiology. Artificial intelligence_ , 1(3), 2019.
  * Latif et al. [2019] Jahanzaib Latif, Chuangbai Xiao, Azhar Imran, and Shanshan Tu.  Medical imaging using machine learning and deep learning algorithms: a review.  In _2019 2nd International conference on computing, mathematics and engineering technologies (iCoMET)_ , pages 1–5. IEEE, 2019.
  * Lee et al. [2019] Hyunkwang Lee, Sehyo Yune, Mohammad Mansouri, Myeongchan Kim, Shahein H Tajmir, Claude E Guerrier, Sarah A Ebert, Stuart R Pomerantz, Javier M Romero, Shahmir Kamalian, et al.  An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets.  _Nature biomedical engineering_ , 3(3):173–182, 2019.
  * Lee et al. [2020] Ji Young Lee, Jong Soo Kim, Tae Yoon Kim, and Young Soo Kim.  Detection and classification of intracranial haemorrhage on ct images using a novel deep-learning algorithm.  _Scientific Reports_ , 10(1):1–7, 2020.
  * Lin et al. [2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017.
  * Linardatos et al. [2020] Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis.  Explainable ai: A review of machine learning interpretability methods.  _Entropy_ , 23(1):18, 2020.
  * López-Pérez et al. [2022] Miguel López-Pérez, Arne Schmidt, Yunan Wu, Rafael Molina, and Aggelos K Katsaggelos.  Deep gaussian processes for multiple instance learning: Application to ct intracranial hemorrhage detection.  _Computer Methods and Programs in Biomedicine_ , 219:106783, 2022.
  * Lundberg and Lee [2017] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30, 2017.
  * Maron and Lozano-Pérez [1997] Oded Maron and Tomás Lozano-Pérez.  A framework for multiple-instance learning.  _Advances in neural information processing systems_ , 10, 1997.
  * Martins and Astudillo [2016] Andre Martins and Ramon Astudillo.  From softmax to sparsemax: A sparse model of attention and multi-label classification.  In _International conference on machine learning_ , pages 1614–1623. PMLR, 2016.
  * Montagnon et al. [2020] Emmanuel Montagnon, Milena Cerny, Alexandre Cadrin-Chênevert, Vincent Hamilton, Thomas Derennes, André Ilinca, Franck Vandenbroucke-Menu, Simon Turcotte, Samuel Kadoury, and An Tang.  Deep learning workflow in radiology: a primer.  _Insights into imaging_ , 11(1):1–15, 2020.
  * Panesar and Panesar [2020] Arjun Panesar and Harkrishan Panesar.  Artificial intelligence and machine learning in global healthcare.  _Handbook of Global Health_ , pages 1–39, 2020.
  * Panwar et al. [2020] Harsh Panwar, PK Gupta, Mohammad Khubeb Siddiqui, Ruben Morales-Menendez, Prakhar Bhardwaj, and Vaishnavi Singh.  A deep learning and grad-cam based color visualization approach for fast detection of covid-19 cases using chest x-ray and ct-scan images.  _Chaos, Solitons & Fractals_, 140:110190, 2020.
  * Patel et al. [2019] Bhavik N Patel, Louis Rosenberg, Gregg Willcox, David Baltaxe, Mimi Lyons, Jeremy Irvin, Pranav Rajpurkar, Timothy Amrhein, Rajan Gupta, Safwan Halabi, Curtis Langlotz, Edward Lo, Joseph Mammarappallil, A J Mariano, Geoffrey Riley, Jayne Seekins, Luyao Shen, Evan Zucker, and Matthew Lungren.  Human-machine partnership with artificial intelligence for chest radiograph diagnosis.  _NPJ Digit Med_ , 2:111, November 2019.
  * Pérez-García et al. [2021] Fernando Pérez-García, Rachel Sparks, and Sebastien Ourselin.  Torchio: a python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning.  _Computer Methods and Programs in Biomedicine_ , 208:106236, 2021.
  * Perkins and Schisterman [2006] Neil J Perkins and Enrique F Schisterman.  The inconsistency of “optimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve.  _American journal of epidemiology_ , 163(7):670–675, 2006.
  * Peters et al. [2019] Ben Peters, Vlad Niculae, and André F. T. Martins.  Sparse sequence-to-sequence models.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1504–1519, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1146.  URL https://aclanthology.org/P19-1146.
  * Quellec et al. [2017] Gwenolé Quellec, Guy Cazuguel, Béatrice Cochener, and Mathieu Lamard.  Multiple-instance learning for medical image and video analysis.  _IEEE reviews in biomedical engineering_ , 10:213–234, 2017\.
  * Rajpurkar et al. [2018] Pranav Rajpurkar, Jeremy Irvin, Robyn L Ball, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis P Langlotz, Bhavik N Patel, Kristen W Yeom, Katie Shpanskaya, Francis G Blankenberg, Jayne Seekins, Timothy J Amrhein, David A Mong, Safwan S Halabi, Evan J Zucker, Andrew Y Ng, and Matthew P Lungren.  Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists.  _PLoS Med._ , 15(11):e1002686, November 2018.
  * Reis et al. [2020] Eduardo Pontes Reis, Felipe Nascimento, Mateus Aranha, F Mainetti Secol, Birajara Machado, Marcelo Felix, Anouk Stein, and Edson Amaro.  Brain hemorrhage extended (bhx): Bounding box extrapolation from thick to thin slice ct images, 2020.
  * Roscher et al. [2020] Ribana Roscher, Bastian Bohn, Marco F Duarte, and Jochen Garcke.  Explainable machine learning for scientific insights and discoveries.  _Ieee Access_ , 8:42200–42216, 2020.
  * Saab et al. [2019] Khaled Saab, Jared Dunnmon, Roger Goldman, Alex Ratner, Hersh Sagreiya, Christopher Ré, and Daniel Rubin.  Doubly weak supervision of deep learning models for head ct.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 811–819. Springer, 2019.
  * Saba et al. [2019] Luca Saba, Mainak Biswas, Venkatanareshbabu Kuppili, Elisa Cuadrado Godia, Harman S Suri, Damodar Reddy Edla, Tomaž Omerzu, John R Laird, Narendra N Khanna, Sophie Mavrogeni, et al.  The present and future of deep learning in radiology.  _European journal of radiology_ , 114:14–24, 2019.
  * Sabato and Tishby [2009] Sivan Sabato and Naftali Tishby.  Homogeneous multi-instance learning with arbitrary dependence.  In _COLT_. Citeseer, 2009.
  * Sabato and Tishby [2012] Sivan Sabato and Naftali Tishby.  Multi-instance learning with any hypothesis class.  _The Journal of Machine Learning Research_ , 13(1):2999–3039, 2012.
  * Sabato et al. [2010] Sivan Sabato, Nathan Srebro, and Naftali Tishby.  Reducing label complexity by learning from bags.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , pages 685–692. JMLR Workshop and Conference Proceedings, 2010.
  * Schlemper et al. [2019] Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert.  Attention gated networks: Learning to leverage salient regions in medical images.  _Medical image analysis_ , 53:197–207, 2019.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * Shapley [1953] Lloyd S Shapley.  A value for n-person games.  _Contributions to the Theory of Games_ , 2(28):307–317, 1953.
  * Shehab et al. [2022] Mohammad Shehab, Laith Abualigah, Qusai Shambour, Muhannad A Abu-Hashem, Mohd Khaled Yousef Shambour, Ahmed Izzat Alsalibi, and Amir H Gandomi.  Machine learning in medical applications: A review of state-of-the-art methods.  _Computers in Biology and Medicine_ , 145:105458, 2022.
  * Sutskever et al. [2013] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton.  On the importance of initialization and momentum in deep learning.  In _International conference on machine learning_ , pages 1139–1147. PMLR, 2013.
  * Teneggi et al. [2022] Jacopo Teneggi, Alexandre Luster, and Jeremias Sulam.  Fast hierarchical games for image explanations.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Turner and Holdsworth [2011] PJ Turner and G Holdsworth.  Ct stroke window settings: an unfortunate misleading misnomer?  _The British journal of radiology_ , 84(1008):1061–1066, 2011.
  * Tushar et al. [2021] Fakrul Islam Tushar, Vincent M D’Anniballe, Rui Hou, Maciej A Mazurowski, Wanyi Fu, Ehsan Samei, Geoffrey D Rubin, and Joseph Y Lo.  Classification of multiple diseases on body ct scans using weakly supervised deep learning.  _Radiology: Artificial Intelligence_ , 4(1):e210026, 2021.
  * Ueda et al. [2019] Daiju Ueda, Akitoshi Shimazaki, and Yukio Miki.  Technical and clinical overview of deep learning in radiology.  _Japanese journal of radiology_ , 37(1):15–33, 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2021] Tonghe Wang, Yang Lei, Yabo Fu, Jacob F Wynne, Walter J Curran, Tian Liu, and Xiaofeng Yang.  A review on medical imaging synthesis using deep learning and its clinical applications.  _Journal of applied clinical medical physics_ , 22(1):11–36, 2021.
  * Wang et al. [2022] Zhenzhen Wang, Carla Saoud, Sintawat Wangsiricharoen, Aaron W. James, Aleksander S. Popel, and Jeremias Sulam.  Label cleaning multiple instance learning: Refining coarse annotations on single whole-slide images.  _IEEE Transactions on Medical Imaging_ , pages 1–1, 2022.  doi: 10.1109/TMI.2022.3202759.
  * Weidmann et al. [2003] Nils Weidmann, Eibe Frank, and Bernhard Pfahringer.  A two-level learning method for generalized multi-instance problems.  In _European Conference on Machine Learning_ , pages 468–479. Springer, 2003.
  * Weissglass [2021] Daniel E Weissglass.  Contextual bias, the democratization of healthcare, and medical artificial intelligence in low-and middle-income countries.  _Bioethics_ , 2021.
  * Wu et al. [2021] Yunan Wu, Arne Schmidt, Enrique Hernández-Sánchez, Rafael Molina, and Aggelos K Katsaggelos.  Combining attention-based multiple instance learning and gaussian processes for ct hemorrhage detection.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 582–591. Springer, 2021.
  * Yeo et al. [2021] Melissa Yeo, Bahman Tahayori, Hong Kuan Kok, Julian Maingard, Numan Kutaiba, Jeremy Russell, Vincent Thijs, Ashu Jhamb, Ronil V Chandra, Mark Brooks, et al.  Review of deep learning algorithms for the automatic detection of intracranial hemorrhages on computed tomography head imaging.  _Journal of neurointerventional surgery_ , 13(4):369–378, 2021.
  * Youden [1950] William J Youden.  Index for rating diagnostic tests.  _Cancer_ , 3(1):32–35, 1950.
  * Zaheer et al. [2017] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola.  Deep sets.  _Advances in neural information processing systems_ , 30, 2017.
  * Zech et al. [2018] John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, and Eric Karl Oermann.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.  _PLOS Medicine_ , 15(11):1–17, 11 2018.  doi: 10.1371/journal.pmed.1002683.  URL https://doi.org/10.1371/journal.pmed.1002683.