1]Big Data Lab, Baidu Inc, Haidian, 100085, Beijing, China 2]HDMI Lab, South
China University of Technology, Guangzhou, Guangdong, China 3]Department of
Data Science, New Jersey Institute of Technology, Dr Martin Luther King Jr
Blvd, Newark, 07102, NJ

# Explanations of Classifiers Enhance Medical Image Segmentation via End-to-
end Pre-training

Jiamin Chen<EMAIL_ADDRESS>Xuhong Li<EMAIL_ADDRESS>Yanwu
Xu<EMAIL_ADDRESS>Mengnan Du<EMAIL_ADDRESS>Haoyi Xiong
<EMAIL_ADDRESS>[  [  [

###### Abstract

Medical image segmentation aims to identify and locate abnormal structures in
medical images, such as chest radiographs, using deep neural networks. These
networks require a large number of annotated images with fine-grained masks
for the regions of interest, making pre-training strategies based on
classification datasets essential for sample efficiency. Based on a large-
scale medical image classification dataset, our work collects explanations
from well-trained classifiers to generate pseudo labels of segmentation tasks.
Specifically, we offer a case study on chest radiographs and train image
classifiers on the CheXpert dataset to identify 14 pathological observations
in radiology. We then use Integrated Gradients (IG) method to distill and
boost the explanations obtained from the classifiers, generating massive
diagnosis-oriented localization labels (DoLL). These DoLL-annotated images are
used for pre-training the model before fine-tuning it for downstream
segmentation tasks, including COVID-19 infectious areas, lungs, heart, and
clavicles. Our method outperforms other baselines, showcasing significant
advantages in model performance and training efficiency across various
segmentation settings.

###### keywords:

Segmentation, pre-training, Chest X-ray, Explanation

##  1 Introduction

Medical image segmentation, crucial for identifying and localizing abnormal
structures in medical images, such as diagnosing pneumonia, heart failure and
hiatal hernia from chest radiographs (a.k.a chest X-ray or CXR), has
significantly benefited from deep neural networks (DNNs) [1, 2]. Many studies
have been proposed towards the medical image segmentation for chest X-rays
through supervised deep learning [3, 4, 5, 1]. However, all these work rely on
a large number of annotated images with fine-grained masks for the regions of
interest, posing a challenge regarding sample efficiency. Consequently, pre-
training strategies leveraging image classification datasets have become
essential to address this issue [6, 7].

To effectively pre-train a segmentation model for medical image segmentation,
one can either utilize natural image classification datasets, such as ImageNet
[8] and Grayscale ImageNet [6], or leverage datasets specifically curated for
medical image classification [7, 9, 10, 11]. The backbone of classifiers
trained on such datasets can then be adapted to serve as pre-trained weights
for the segmentation model (backbone+segmentation module), facilitating
improved performance in medical image segmentation. However, there still
remain some challenges for the above studies. For the backbone pre-training,
as they only initialize the backbone weights and leave the segmentation module
randomly initialized, it still needs much annotated data for fine-tuning. Some
works [12, 13] propose the end-to-end pre-training strategies with large-scale
annotated datasets of natural images, such as Microsoft COCO [14]. However,
for CXR images, there does not exist so far any large annotated dataset.
Unlike natural images with more morphological details to annotate, the
semantic information inside chest X-rays is non-obvious and can be perceived
under different views [15].

(a) Generating Diagnosis-oriented Localization Labels (DoLL)

(b) End-to-end Pre-training and Fine-tuning processes

Figure 1: An Illustration of DoLL-based Pre-training Approach based on CXR
Classification Datasets, Classifiers, and Explainers

On the other hand, recent studies in explainable artificial intelligence (XAI)
have demonstrated the feasibility of using activation maps, saliency maps, or
even input gradients of a model to interpret the decisions made by DNNs [16].
Clinicians and researchers adopted these interpretations to explain the
diagnostic results to patients or even advance the precise diagnostics that
can be undetectable to the naked eye [17, 18, 19]. More specifically, our
previous studies found the explanation result of an image classifier would be
spatially closed to the location of visual objects for classification in the
image [20]. Through aggregating the explanation results from multiple well-
trained DNNs on the same image, it is possible to obtain a cross-model
consensus of explanations via pixel-wise majority voting [20]. Such cross-
model consensus of explanations could be further used as the _Pseudo Semantic
Segmentation Label_ of the image (which was originally annotated for
classification) to improve the performance of segmentation tasks through pre-
training [13, 21].

In this study, we introduce a novel strategy, the Diagnosis-oriented
Localization Labels (DoLL), to seamlessly pre-train deep neural networks
(DNNs) for medical image segmentation tasks using only classification
datasets. This approach is exemplified through a suite of chest X-ray (CXR)
segmentation tasks. As illustrated in our framework shown in Fig. 2, we refine
and enhance the explanatory power of image classifiers across 14 clinically
relevant chest radiographic observations, adhering to the Fleischner Society’s
glossary standards [22]. Distinct from the traditional segmentation pre-
training-fine-tuning workflow, which typically initializes only the model’s
backbone, our method endows the entire model, including the segmentation
module, with superior pre-trained weights. This setup significantly
streamlines the fine-tuning process by concentrating updates solely within the
segmentation module, referred to as downstream segmentation adapters. Our
contributions are manifold:

  1. 1.

We devise the DoLL method for automatically annotating chest X-rays, by
distilling and boosting the explanations of classifiers trained on the
frontal-view CXRs on CheXpert [23] for 14 pathological observations. With this
method, we elaborate a large-scale annotated dataset CheXpert-DoLL for pre-
training. It supports many downstream CXR segmentation tasks concerning bones,
heart, lung, pleura, and the abnormalities within these regions. The CheXpert-
DoLL dataset is publicly available at http://somewhere.

  2. 2.

We present an end-to-end pre-training method for chest X-ray segmentation
based on CheXpert-DoLL. By associating X-ray pixels with meaningful categories
according to clinical practice, this end-to-end pre-training method leaves the
whole segmentation model, including both backbone and segmentation module,
with a deep and complete understanding to chest X-ray images. In contrast to
conventional approaches that apply pre-training solely to the backbone
weights, our method allows for the use of pre-trained weights for both the
backbone and the segmentation module. This comprehensive pre-training process
significantly enhances the adaptation of the model to downstream segmentation
tasks during fine-tuning.

  3. 3.

We conduct extensive experiments with different network architectures on
various downstream settings including lung segmentation, COVID-19 infection
segmentation, and multi-organ segmentation of lungs, heart and clavicles. Both
the experimental results and training efficiency demonstrate the great
advantages of pre-training with DoLL against the other self-supervised pre-
training methods, such as MoCo-CXR [24] and MoCo-v2 [25], and pretrained
backbones on ImageNet, Grayscale ImageNet [6] and CheXpert [23]. The CheXpert-
DoLL dataset and the pretrained segmentation models will be released publicly
for future usage.

##  2 Related Works

Pre-training strategies and the subsequent fine-tuning have become a crucial
area of focus in recent research [26], with implications extending across
various domains including medical imaging [27, 28, 29]. While fully supervised
pre-training and self-supervised pre-training have seen significant
advancements [27, 28, 30], the territory of weakly-supervised pre-training
remains comparatively underexplored.

In the realm of visual recognition systems, Singh et al. revisit the potential
of weakly-supervised pre-training, utilizing models that are pre-trained with
hashtag-based supervision and demonstrating that such approaches can rival the
performance of self-supervised methods [26]. Furthermore, Ghadiyaram et al.
explored the application of large-scale weakly supervised pre-training for
video models within the context of action recognition, showcasing the
versatility of weak supervision beyond static imagery [31]. Within medical
imaging, despite the proposition of several weakly-supervised techniques in
recent years, challenges persist. Some methods prove to be incompatible with
segmentation tasks [32], while others may impose prohibitive time and resource
demands for effective large-scale pre-training [33]. Liao  _et al._ proposed a
weakly-supervised pre-training strategy that combines unsupervised contrastive
learning and supervised continual learning tasks into one pre-training
pipeline for advanced performance based on X-ray images [29].

Our work follows the concept of “Learning from explanations”, which offers a
novel perspective—incorporating insights gleaned from Explainable AI (XAI)
techniques into the learning process to enhance the model’s interpretability
and accuracy, particularly for segmentation tasks. Class Activation Maps (CAM)
paved the way by employing explanatory visual cues to locate discriminative
regions within classification models [34]. These techniques have since
evolved, with methods like ACoL that integrate CAM into an adversarial
learning framework for object detection [35], and Puzzle-CAM which refines the
quality of CAM-generated segmentation using reconstructive regularization loss
[36]. The notion of pre-training segmentation models with explanations was
further advanced by Pseudo Semantic Segmentation Labels (PSSL) which
repurposes the explanations of classifier to annotate images for semantic
segmentation pre-training [13].

As we strive to harness weakly-supervised pre-training for the specialized
needs of medical image analysis, particularly segmentation, these varied
approaches provide a rich tapestry of strategies to draw upon. They highlight
the untapped potential of weak supervision not only as a tool for reducing
reliance on extensive annotated datasets but also as a means to improve model
interpretability and performance in a cost-effective manner. Our work builds
on these insights, aiming to close the gap in the current landscape of weakly
supervised methods for medical image segmentation and propel them to the
forefront of pre-training methodologies.

##  3 Methods

In this section, we introduce how the diagnosis-oriented localization labels
are built to enable an end-to-end pre-training process for segmentation
models. For each CXR image, DoLL localizes 14 sets of regions of interest via
different pathological observations by distilling and boosting the
explanations of weak classifiers. Applying this method to the massive images
from CheXpert dataset, we pretrain the entire model and only finetune the
segmentation adapters for various downstream settings with a unified backbone.

###  3.1 DoLL: Diagnosis-orinted Localization Labels

As illustrated in Fig. 1a, we present how the explanations of weak learners
are used to construct the 14 sets of diagnosis-oriented labels. CheXpert [23]
is a large-scale multi-label classification dataset for chest X-rays. It
consists of 224,316 images of 65,240 patients labeled for the presence of 14
common chest radiographic observations such as “Enlarged Cardiomediastinum”,
“Lung Opacity”, “Pneumothorax”, “Fracture”, “Support Devices”, etc. Here, we
select in total 191,027 frontal-view CXRs and train several multi-label
classifiers for predicting the probability of each pathological observation.

Integrated gradient [37] is a gradient-based axiomatic attribution method for
deep neural network, which requires almost no modification to the original
network. By following a designed path from baseline to the input image, it
computes in each step the integral of gradients, which reduces the unexpected
noise on irrelevant pixels. Here we set the baseline as a black image with a
linear path and apply the Riemann approximation for calculating the integral.
$T$ denotes the total discretizing steps, $E_{i}^{c}$ the explanation result
of integrated gradient for model $i$ while predicting label $c$,
$\mathcal{L}^{c}_{i}$ the loss function of model $i$ for label $c$, and $X$
the input image. For each model $i$, we have:

$\displaystyle E^{c}_{i}$
$\displaystyle=\frac{1}{T}\sum_{t=1}^{T}\frac{\partial\mathcal{L}^{c}_{i}(\frac{t}{T}X)}{\partial
X}\overset{\rm
T\to\infty}{\to}\int_{\alpha=0}^{1}\frac{\partial\mathcal{L}^{c}_{i}(\alpha
X)}{\partial X}d\alpha\quad.$ (1)

####  3.1.1 Boosted Distillation of Explanations

The feature attribution map of a single model can be biased [38]. Here, the
case is even more complicated, since we not only have multiple models but each
of them predicts 14 observations with varying performances. In order to
maximize the plausibility and faithfulness of our generated DoLL masks, we
adopt a boosting strategy for distilling the helpful knowledge of each model
when predicting each observation.

Assuming $M$ weak classifiers with $C$ observations, we first compute the
weights $W_{m,c}\in\mathbb{R}^{M\times C}$ with $N$ samples according to their
predicting performance on CheXpert, as illustrated in Algorithm 1.

Algorithm 1 Boosting the explanations for weak learners

$y_{n}^{c}$ the CheXpert label of sample $n$ for observation $c$

for $c\in[1,C]$ do $\triangleright$ $C$ observations

Initialize weights $S:S_{1},S_{2},\dots,S_{N}=\frac{1}{N}$

for $m\in[1,M]$ do $\triangleright$ $M$ weak learners

Fit classifier $m$ to $N$ samples and obtain the predicted label
$\hat{y}_{n}^{c}$

$e_{i}=\frac{\sum_{n=1}^{N}S_{n}I(\hat{y}_{n}^{c}\neq
y_{n}^{c})}{\sum_{n=1}^{N}S_{n}}$ $\triangleright$ Calculating the error

$W_{m,c}=\log(\frac{1-e_{i}}{e_{i}})+\log(K-1)$ $\triangleright$ weight for
model $m$ on observation $c$

$S_{n}=S_{n}e^{W_{m,c}I(\hat{y}_{n}^{c}\neq y_{n}^{c})}$ for $S_{n}\in S$

$S_{n}=\frac{S_{n}}{\sum_{n}S_{n}}$ for $S_{n}\in S$ $\triangleright$ Re-
normalize $S$

end for

end for

return $W$

Moreover, we only consider the region of interest for the models with
relatively high probability scores (beyond the threshold $\tau$) and have:

$\displaystyle N^{c}$ $\displaystyle=\\{\ i\ |\
p_{i}^{c}>\tau,i\in\\{1,\dots,M\\}\ \\}\quad,$ (2)
$\displaystyle\tilde{E}^{c}$ $\displaystyle=\frac{1}{|N^{c}|}\sum_{m\in
N^{c}}W_{m,c}E^{c}_{n}\quad,$ (3)

where $p_{i}^{c}$ denotes the probability score of model $i$ for label $c$,
$N^{c}$ the set of models with the probability score beyond the threshold
$\tau$, and $|N^{c}|$ the cardinality of the set $N^{c}$.

At last, for each observation $c$, we binarize $\tilde{E}^{c}$ with a
threshold and convert the explanation result into Boolean values. In this way,
we obtain 14 segmentation labels for each chest radiograph which localize the
discriminative pixels from different pathological views.

Figure 2: Example of our DoLLs and its ground truth masks

Based on this method, we equip each frontal CXR in CheXpert with 14 sets of
binary masks, where each describes the region of interest under a specific
pathological perspective. Thereby, we obtain a new large-scale dataset
CheXpert-DoLL, which enables an end-to-end pre-training for segmentation
models. As presented in DoLLs under certain observations are coherent with
ground truth masks for different segmentation tasks, indicating the advantage
of pre-training with DoLLs in supporting a large variety of downstream
settings.

###  3.2 Downstream Segmentation Adapters

For the pre-training stage, we pretrain both backbone and segmentation modules
with DoLLs served as the segmentation labels. We minimize the binary cross
entropy between DoLL and the output mask, that is

$\sum_{c=1}^{14}\ -d_{c,x}\ {\rm log}(p_{c,x})-(1-d_{c,x})\ {\rm
log}(1-p_{c,x})$ (4)

where $p_{c,x}$ is the probability and $d_{c,x}$ the DoLL for label $c$ on
pixel $x$. The benefits of pre-training in an end-to-end manner is that the
backbone and the segmentation module can be better correlated in their
pretrained weights, thus decreasing the training difficulty in fine-tuning
process. Besides, the strong clinical relevance and the specificity for CXR
can help the model extract more general features and reach better performances
on various downstream tasks.

Given both backbone and segmentation module pretrained, we apply them for
various downstream tasks. As shown in Fig. 1b, we freeze the backbone and only
update the segmentation module during fine-tuning. We name the finetuned
modules as downstream segmentation adapters. The reasons for freezing the
backbone are two-fold: First, only updating the parameters in the segmentation
module demands less time and space for better training efficiency. Second, it
adds more flexibility to implementations and requires less storage space. The
model can be adapted to different segmentation tasks by only switching the
adapter layers.

##  4 Results

In this section, we present the results of our work. We first introduce the
experiment settings, and then present the overall comparison results between
DoLL and baselines. Ablation and case studies have been done to confirm the
advantages of DoLL.

###  4.1 Experiment Settings

Here, we introduce DoLL generation from classification datasets, datasets used
for pre-training and fine-tuning, baseline algorithms for comparisons.

###  4.2 CheXpert-DoLL Generation

For the generation of CheXpert-DoLL, we train the following 17 models for a
multi-label classification task on CheXpert: DenseNet121, MobileNetV2,
AlexNet, ResNet50, ResNet50-32$\times$4d, ResNet101-32$\times$8d, Wide-
ResNet50, Wide-ResNet101, Vgg16, ResNet152, ShuffleNetv2, DenseNet161,
ShuffleNetv2-x2, ResNet101, EfficientNetB4, MobileNetV3 and ViT-base-32, with
the mean AUC around 0.8 for each model. As shown in Fig. 1a, the overall idea
of DoLL generation is to first aggregate the explanation results (e.g.,
sailency maps obtained by Integrated Gradients) extracted from these well-
trained classifiers, and then to establish the “consensus” of visual
explanations among these classifiers as the pseudo labels for semantic
segmentation tasks [13, 20]. Please refer to Section 3.1 for details on the
algorithms for generating DoLL on images. In general, the greater number of
models and the better classification performance can definitely improve its
exactness for the DoLL generation, and lead to better pre-training results.
For a compromise between time consumption and exactness, we suggest the number
of models to be set around 15-20.

Then, we generate the CheXpert-DoLL over these observations with the total
number of models $M=17$, the threshold $\tau=0.1$, the $80$-th percentile for
the binarizing threshold, the steps in IG $T=5$, and $K=3$ for the boosting. A
larger value of binarizing threshold leads to a more concentrated region of
interest, and vice versa for a smaller value. Based on our empirical findings,
more concentrated DoLL can improve the segmentation performance for the lung-
related regions but degrade the model’s generalization ability, leading to
worse performance for the segmentation of heart, clavicles, supporting
devices, etc.

Table 1: Overview of datasets used in our experiments Datasets | Task | Train | Val | Test | Purpose
---|---|---|---|---|---
CheXpert (frontal) | Multi-label classification | 191,027 | - | - | pre-training
COVID-QU-EX | Lung segmentation | 7658 | 1903 | 2395 | fine-tuning
COVID-19 infection segmentation | 1864 | 466 | 583 | fine-tuning
JSRT | Multi-organ segmentation | 171 | 25 | 50 | fine-tuning

####  4.2.1 Segmentation Tasks

In the experiments, we evaluate the pre-training methods on three downstream
tasks: lung segmentation on COVID-QU-EX [2] (Lung), COVID-19 infectious region
segmentation on COVID-QU-EX (Infection) and multi-organ segmentation including
lungs, heart and clavicles on Chest X-ray Landmark Segmentation Dataset [39]
(JSRT). COVID-QU-EX is a dataset collected by the researchers of Qatar
University. It is composed of X-ray images of the human chest labelled as
either “Healthy”, “COVID-19”, or “Pneumonia”. It provides as well the
corresponding ground-truth infection segmentation masks and lung segmentation
masks. In our experiments, we use the lung masks for lung segmentation tasks,
and the infection masks for a direct segmentation from X-ray images without
mediating any lung mask. Chest x-ray Landmark Segmentation Dataset contains
911 landmark annotations for chest X-ray images from JSRT, Shenzhen,
Montgomery and Padchest datasets. Here, we use the subset with JSRT images for
the multi-organ segmentation of “Right Lung”, “Left Lung”, “Heart”, ”Left
Clavicle”, and “Right Clavicle”. More detailed descriptions of these datasets
can be found in Table 1. Also, please refer to Section 3.2 for details on the
algorithms for fine-tuning CheXpert-DoLL pre-trained models for segmentation
tasks.

####  4.2.2 Baselines and Setups

To validate our DoLL pre-training strategies, we propose the following pre-
training baselines. We consider both supervised pre-training with large-scale
classification datasets for natural and medical images, and self-supervised
pre-training strategies.

  * •

ImageNet: we finetune the entire 3-channel model on target datasets with
initialization from officially-released pretrained backbone weights;

  * •

CheXpert [23]: the 3-channel backbone is pretrained with a multi-label
classification task on CheXpert and we finetune the entire model for
downstream tasks;

  * •

MoCo-CXR [24]: it is an adaptation of Momentum Contrast (MoCo) to produce
better representation and initialization for the detection of pathologies in
chest X-rays. We initialize the backbone with the officially-released
pretrained weights in MoCo-CXR, and finetune the entire model for the
downstream tasks;

  * •

MoCo-v2 [40]: the method implements the SimCLR’s design in the MoCo framework,
by using an MLP head and better data augmentation strategies. We here
initialize the backbone with the officially-released pretrained weights on
ImageNet, and finetune the entire model for the downstream tasks;

  * •

GrayScale ImageNet [6]: we adopt both the official released single-channel
backbone structure and weights, and finetune the entire model;

  * •

Scratch: for segmentation models without pretrained backbones such as U-Net,
we train the model directly on the target dataset.

For a more fair comparison, we propose DoLL with 3-channel model input, and
DoLL-1channel for single channel. We select three representative segmentation
network architectures: the classic PSPNet [12]-ResNet50 [41], the Transformer-
based Segformer-Mix Transformer [42], and the U-Net [3] which is designed for
medical image segmentation. The segmentation performance is evaluated via the
metrics including the mean Intersection over Union (mIoU), mean Accuracy
(mAcc) and Dice Similarity Coefficient (Dice). For the pre-training, we train
30 epochs for each model with learning rate 0.01, batch size 128, input image
size 224 and data augmentation methods including random crops, flipping and
rotations. For the fine-tuning, we train each model or adapter for 20,000
iterations, and select the checkpoint with highest IoU on validation set.

###  4.3 Overall Results

We present in Table 2 the results of baselines and ours on all the downstream
tasks. By only updating the segmentation module, our method can reach better
performance than all the other entirely finetuned baselines which are entirely
finetuned, including the large-scale classification datasets and self-
supervised learning strategies. Moreover, our methods show great improvements
regardless of different model architectures. For COVID-19 infection
segmentation, the outperformance is more evident, which indicates that pre-
training with DoLL reaches a deeper extraction of visual features for CXRs,
not only just in the contour, but also the pathological interpretations.

We have also conducted experiments with fixed backbones for the baseline
methods. Most of the metrics get worse compared to the entire fine-tuning.
This demonstrates that our pre-trained weights are better correlated between
backbone and segmentation module. As for the visualization of segmentation, we
provide in Fig. 3 an example of multi-organ segmentation from JSRT dataset. We
observe that our method is closer to the ground-truth segmentation mask, and
shows significant improvement in segmenting clavicles.

Table 2: Overall results on testing set for COVID-19 infection segmentation (COVID-19), lung segmentation (Lung) and multi-organ segmentation (JSRT) Model | pre-training | COVID-19 | Lung | JSRT
---|---|---|---|---
IoU | Acc | IoU | Acc | mIoU | mAcc | mDice
PSPNET -ResNet50 | ImageNet | 66.74 | 80.05 | 89.37 | 94.39 | 86.54 | 90.84 | 92.65
CheXpert | 72.73 | 84.21 | 90.7 | 95.12 | 85.56 | 89.69 | 92.00
MoCo-CXR | 72 | 83.72 | 94.83 | 97.34 | 89.96 | 92.70 | 94.62
MoCo-v2 | 72.39 | 83.98 | 94.96 | 97.41 | 89.78 | 92.85 | 94.49
GrayScale ImageNet | 69.75 | 82.18 | 91.67 | 95.65 | 89.13 | 93.18 | 94.09
DoLL | 77.48 | 87.31 | 95.31 | 97.60 | 90.98 | 93.96 | 95.20
DoLL-1channel | 79.3 | 88.45 | 94.53 | 97.19 | 90.03 | 93.31 | 94.63
Segformer-MiT | ImageNet | 72.13 | 83.81 | 93.97 | 96.89 | 93.28 | 95.14 | 96.49
CheXpert | 83.67 | 91.11 | 94.46 | 97.15 | 93.09 | 95.32 | 96.38
DoLL | 92.12 | 95.9 | 95.91 | 97.91 | 93.66 | 95.72 | 96.69
U-Net∗ | Scratch | 75.42 | 85.99 | 94.15 | 96.99 | 70.64 | 80.44 | 82.32
DoLL | 81.05 | 89.54 | 96.54 | 97.75 | 74.24 | 82.84 | 84.87
Figure 3: Visualization of segmentation results on JSRT for ImageNet and DoLL
pretrained models

####  4.3.1 Adaptation for Downstream Tasks

In our experiments, we consider a great variety of downstream settings to
validate our pre-training method. Besides the common lung segmentation, we
include as well infectious region segmentation and few-shot multi-organ
segmentation. The results well support the conclusion that by generating
labels via the 14 observations, our pretrained model reaches a better
generalization ability and a deeper understanding to the CXR images, leading
to an all-round enhancement on various downstream segmentation tasks.

Figure 4: Record of mIoU on validation set during the PSPNet fine-tuning
process for JSRT, COVID-19 and Lung segmentation tasks

####  4.3.2 Efficiency

The freezing of backbone during the fine-tuning process, i.e. the downstream
segmentation adapters, not only facilitates the future implementations by
requiring less storage space and less time for loading and switching, but also
improves the training efficiency. As illustrated in Fig. 4, we record the mIoU
score on validation set for each epoch. For all the downstream settings, our
pretrained model presents a stable performance and fast convergence. By
contrast, ImageNet pre-training shows an easy tendency of overfitting under
few-shot setting (JSRT) and slow convergence with lung segmentation. Moreover,
we freeze the backbone for ImageNet pre-training as well, whereas the results
are not as satisfactory as ours. First, the less powerful backbone limits its
performance on downstream tasks. Besides, the random initialization of
segmentation module leads to a broken relation between backbone and
segmentation module. Failing to get correlated adds extra difficulty to the
fine-tuning process, since the model not only needs to adapt into downstream
tasks, but also to build correlations between backbone and segmentation
module.

###  4.4 Ablation Study

We conduct an ablation study to validate the effectiveness of the boosting
module in our proposed annotating method. Table 3 presents the IoU results on
our three downstream segmentation settings with PSPNET-ResNet50. We average
the results from different explanation methods instead of adopting the
boosting coefficients. The results demonstrate the advantages of boosting weak
learners with improvements in all metrics.

Table 3: IoU results of ablation study with PSPNET-R50 PSPNET-R50 |  COVID-19 |  Lung | JSRT
---|---|---|---
RL | LL | H | RCLA | LCLA
Averaged | 76.10 | 95.01 | 95.73 | 94.99 | 93.00 | 85.51 | 84.36
Boosted | 77.48 | 95.31 | 96.07 | 95.59 | 93.06 | 85.66 | 84.50

###  4.5 Case Studies

Here, we use case studies to analyze the time consumption of DoLL and the
potential impact of explanation accuracy in DoLL to the overall performance.

####  4.5.1 Time Cost

Questions may come up if our method costs too much time compared to the
conventional backbone pre-training pipeline, since it involves the generation
of DoLL, the end-to-end pre-training, and the downstream fine-tuning process.
First, the generation of DoLL does not cost much time and effort. By using 17
classification models with mean AUC around 0.8, we can already construct the
DoLL with pretty high quality. The training of these classification models can
be easily achieved without the necessity of bringing special tricks. Second,
we would like to address that both the DoLL and pretrained checkpoints will be
released publicly. We demonstrated in the previous section that pre-training
with the DoLL can largely improve the efficiency during fine-tuning, costing
less time and space. Besides, once the DoLLs are generated, it can be directly
applied for other pre-training tasks. Even if the desired network architecture
is not involved in the released checkpoints, the end-to-end pre-training will
not cost much more time than the conventional pre-training pipeline, and
largely improve the efficiency in the fine-tuning process.

####  4.5.2 What if the explanations are inaccurate ?

We have adopted several techniques to guarantee the faithfulness of generated
DoLLs. As mentioned, we distill and boost the explanations of weak learners to
avoid the biasedness from misclassification. Besides, increasing the number of
classifiers, improving their classifying performances, tuning the threshold
$\tau$ in Equation (2), and modifying the binarizing threshold can all be
helpful to further improve the exactness. In our experiment, we generate DoLLs
with 17 models and pretrain the entire segmentation model for 30 epochs
without any special designs. As a result, we have achieved very satisfactory
performance on downstream segmentation tasks, with evident improvements
compared to previous pretrainig pipelines.

##  5 Discussions

This work introduces DoLL, an innovative pre-training method for medical image
segmentation specifically tailored for chest X-rays. The extensive CheXpert-
DoLL dataset, generated using interpretations of classifiers trained for 14
pathological observations with CheXpert, significantly boost model performance
on tasks such as lung and COVID-19 infection segmentation, and few-shot multi-
organ segmentation. Compared to traditional pre-training that only applies to
backbone weights, DoLL enables end-to-end pre-training for both backbone and
segmentation modules, effectively reducing the need for large annotated
datasets. The resulting approach not only deepens feature extraction
capabilities but also generalizes better to new tasks, allowing for the
backbone to be fixed during fine-tuning processes. As part of contribution, we
have compiled and released the CheXpert-DoLL dataset, set to catalyze future
advancements in chest X-ray segmentation research and clinical applications.

Despite the notable successes, there remain open research issues that invite
further exploration.

  * •

Generalizability. One significant area is the generalizability of DoLL across
other imaging modalities beyond chest radiographs. As medical imaging
encompasses a wide array of modalities - each with its own unique set of
characteristics and challenges - future research could investigate the
applicability and adaptation of DoLLs to CT scans, MRI, and ultrasound images.
Some earlier work that leverages 2D images to pre-train 3D models actually
shed the light to this area [43].

  * •

Robustness. Another open issue pertains to the robustness of the generated
DoLL against variations in pathology presentations and imaging conditions.
While the paper details the success in uncovering 14 pathological
observations, the diversity of pathological conditions and their
manifestations in images suggest the need to validate and possibly enhance the
robustness of the extracted explanations against such variations. Furthermore,
there is the potential to expand the utility of DoLL to enhance not just
segmentation tasks but also broader aspects of computer-aided diagnosis such
as prognostic modeling and personalized medicine, which could significantly
impact patient care.

  * •

Interpretability. In terms of the technical aspects, there lies an avenue for
research in optimizing the extraction and utilization of classifier
explanations. For instance, DoLL are derived using Integrated Gradients, but
further research could explore alternative methods such as LIME, SHAP or their
variants [44], potentially providing an enriched set of localization labels
that could further refine the pre-training process.

  * •

Ethics. Lastly, ethical considerations and biases within AI applications in
medicine necessitate ongoing research. Ensuring that the algorithms are fair,
transparent, and equitable across different populations is a multifaceted
challenge that intersects with data diversity, algorithmic design, and
regulatory compliance.

The work undertaken has laid a robust foundation that can catalyze future
explorations within these open research areas. By addressing these challenges,
the scientific community can further the advancements in medical image
analysis, ultimately contributing to the enhancement of patient outcomes and
the optimization of healthcare delivery.

## References

  *   * [1] Rajaraman, S., Folio, L. R., Dimperio, J., Alderson, P. O. & Antani, S. K.  Improved semantic segmentation of tuberculosis—consistent findings in chest x-rays using augmented training of modality-specific u-net models with weak localizations.  _Diagnostics_ 11, 616 (2021).
  * [2] Tahir, A. M. _et al._ COVID-19 infection localization and severity grading from chest x-ray images.  _Comput. Biol. Medicine_ 139, 105002 (2021).
  * [3] Ronneberger, O., Fischer, P. & Brox, T.  _U-net: Convolutional networks for biomedical image segmentation_ , Vol. 9351 of _Lecture Notes in Computer Science_ , 234–241 (Springer, 2015).
  * [4] Badrinarayanan, V., Kendall, A. & Cipolla, R.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _IEEE Trans. Pattern Anal. Mach. Intell._ 39, 2481–2495 (2017).
  * [5] Liu, W. _et al._ Automatic lung segmentation in chest x-ray images using improved u-net.  _Scientific Reports_ 12, 8649 (2022).
  * [6] Xie, Y. & Richmond, D.  _Pre-training on grayscale imagenet improves medical image classification_ , Vol. 11134 of _Lecture Notes in Computer Science_ , 476–484 (Springer, 2018).
  * [7] Liao, W. _et al._ _MUSCLE: multi-task self-supervised continual learning to pre-train deep models for x-ray images of multiple body parts_ , Vol. 13438 of _Lecture Notes in Computer Science_ , 151–161 (Springer, 2022).
  * [8] Deng, J. _et al._ _Imagenet: A large-scale hierarchical image database_ , 248–255 (IEEE Computer Society, 2009).
  * [9] Kalapos, A. & Gyires-Tóth, B.  _Self-supervised pretraining for 2d medical image segmentation_ , Vol. 13807 of _Lecture Notes in Computer Science_ , 472–484 (Springer, 2022).
  * [10] Huang, X., Liu, M., Belongie, S. J. & Kautz, J.  _Multimodal unsupervised image-to-image translation_ , Vol. 11207 of _Lecture Notes in Computer Science_ , 179–196 (Springer, 2018).
  * [11] Liang, G. _et al._ Contrastive cross-modal pre-training: A general strategy for small sample medical imaging.  _IEEE J. Biomed. Health Informatics_ 26, 1640–1649 (2022).
  * [12] Zhao, H., Shi, J., Qi, X., Wang, X. & Jia, J.  _Pyramid scene parsing network_ , 6230–6239 (IEEE Computer Society, 2017).
  * [13] Li, X. _et al._ Distilling ensemble of explanations for weakly-supervised pre-training of image segmentation models.  _Machine Learning_ 112, 2193–2209 (2023).
  * [14] Lin, T. _et al._ _Microsoft COCO: common objects in context_ , Vol. 8693 of _Lecture Notes in Computer Science_ , 740–755 (Springer, 2014).
  * [15] Wen, Y., Chen, L., Deng, Y. & Zhou, C.  Rethinking pre-training on medical imaging.  _J. Vis. Commun. Image Represent._ 78, 103145 (2021).
  * [16] Li, X. _et al._ Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond.  _Knowledge and Information Systems_ 64, 3197–3234 (2022).
  * [17] Essemlali, A., St-Onge, E., Descoteaux, M. & Jodoin, P.-M.  _Understanding alzheimer disease’s structural connectivity through explainable ai_ , 217–229 (PMLR, 2020).
  * [18] El-Sappagh, S., Alonso, J. M., Islam, S. R., Sultan, A. M. & Kwak, K. S.  A multilayer multimodal detection and prediction model based on explainable artificial intelligence for alzheimer’s disease.  _Scientific reports_ 11, 2660 (2021).
  * [19] Borys, K. _et al._ Explainable ai in medical imaging: An overview for clinical practitioners–beyond saliency-based xai approaches.  _European journal of radiology_ 110786 (2023).
  * [20] Li, X., Xiong, H., Huang, S., Ji, S. & Dou, D.  Cross-model consensus of explanations and beyond for image classification models: An empirical study.  _Machine Learning_ 112, 1627–1662 (2023).
  * [21] Liu, Y., Zhang, S., Chen, J., Chen, K. & Lin, D.  Pixmim: Rethinking pixel reconstruction in masked image modeling.  _arXiv preprint arXiv:2303.02416_ (2023).
  * [22] Hansell, D. M. _et al._ Fleischner society: glossary of terms for thoracic imaging.  _Radiology_ 246, 697–722 (2008).
  * [23] Irvin, J. _et al._ _Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison_ , Vol. 33, 590–597 (2019).
  * [24] Sowrirajan, H., Yang, J., Ng, A. Y. & Rajpurkar, P.  _Moco pretraining improves representation and transferability of chest x-ray models_ , Vol. 143 of _Proceedings of Machine Learning Research_ , 728–744 (PMLR, 2021).
  * [25] Chen, L., Papandreou, G., Schroff, F. & Adam, H.  Rethinking atrous convolution for semantic image segmentation.  _CoRR_ abs/1706.05587 (2017).
  * [26] Singh, M. _et al._ _Revisiting weakly supervised pre-training of visual perception models_ , 794–804 (IEEE, 2022).
  * [27] Xie, Y. & Richmond, D.  _Pre-training on grayscale imagenet improves medical image classification_ , 0–0 (2018).
  * [28] Wen, Y., Chen, L., Deng, Y. & Zhou, C.  Rethinking pre-training on medical imaging.  _Journal of Visual Communication and Image Representation_ 78, 103145 (2021).
  * [29] Liao, W. _et al._ _Muscle: Multi-task self-supervised continual learning to pre-train deep models for x-ray images of multiple body parts_ , 151–161 (Springer, 2022).
  * [30] Wolf, D. _et al._ Self-supervised pre-training with contrastive and masked autoencoder methods for dealing with small datasets in deep learning for medical imaging.  _Scientific Reports_ 13, 20260 (2023).
  * [31] Ghadiyaram, D., Tran, D. & Mahajan, D.  _Large-scale weakly-supervised pre-training for video action recognition_ , 12046–12055 (Computer Vision Foundation / IEEE, 2019).
  * [32] Zhou, H.-Y. _et al._ _Comparing to learn: Surpassing imagenet pretraining on radiographs by comparing image representations_ , 398–407 (Springer, 2020).
  * [33] Viniavskyi, O., Dobko, M. & Dobosevych, O.  _Weakly-supervised segmentation for disease localization in chest x-ray images_ , 249–259 (Springer, 2020).
  * [34] Zhou, B., Khosla, A., Lapedriza, À., Oliva, A. & Torralba, A.  _Learning deep features for discriminative localization_ , 2921–2929 (IEEE Computer Society, 2016).
  * [35] Zhang, X., Wei, Y., Feng, J., Yang, Y. & Huang, T. S.  _Adversarial complementary learning for weakly supervised object localization_ , 1325–1334 (Computer Vision Foundation / IEEE Computer Society, 2018).
  * [36] Jo, S. & Yu, I.  _Puzzle-cam: Improved localization via matching partial and full features_ , 639–643 (IEEE, 2021).
  * [37] Sundararajan, M., Taly, A. & Yan, Q.  _Axiomatic attribution for deep networks_ , Vol. 70 of _Proceedings of Machine Learning Research_ , 3319–3328 (PMLR, 2017).
  * [38] Li, X., Xiong, H., Huang, S., Ji, S. & Dou, D.  Cross-model consensus of explanations and beyond for image classification models: An empirical study.  _CoRR_ abs/2109.00707 (2021).
  * [39] Gaggion, N., Mansilla, L., Mosquera, C., Milone, D. H. & Ferrante, E.  Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis.  _IEEE Transactions on Medical Imaging_ (2022).
  * [40] Chen, X., Fan, H., Girshick, R. B. & He, K.  Improved baselines with momentum contrastive learning.  _CoRR_ abs/2003.04297 (2020).
  * [41] He, K., Zhang, X., Ren, S. & Sun, J.  _Deep residual learning for image recognition_ , 770–778 (IEEE Computer Society, 2016).
  * [42] Xie, E. _et al._ Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P. & Vaughan, J. W. (eds) _Segformer: Simple and efficient design for semantic segmentation with transformers_.  (eds Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P. & Vaughan, J. W.) _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , 12077–12090 (2021).
  * [43] Zhang, Y. _et al._ _Video4mri: An empirical study on brain magnetic resonance image analytics with cnn-based video classification frameworks_ , 1–5 (IEEE, 2023).  URL https://doi.org/10.1109/ISBI53787.2023.10230371.
  * [44] Li, X. _et al._ G-lime: Statistical learning for local interpretations of deep neural networks using global priors.  _Artificial Intelligence_ 314, 103823 (2023).