:
2024

# Multi-modal Masked Siamese Network Improves
Chest X-Ray Representation Learning

Saeed Shurrab∗<EMAIL_ADDRESS>
Department of Computer Engineering
New York University Abu Dhabi
Abu Dhabi  United Arab Emirates  Alejandro Guerra-Manzanares∗
<EMAIL_ADDRESS>
Department of Computer Engineering
New York University Abu Dhabi
Abu Dhabi  United Arab Emirates  Farah E. Shamout<EMAIL_ADDRESS>
Department of Computer Engineering
New York University Abu Dhabi
Abu Dhabi  United Arab Emirates
∗These authors contributed equally to this work

###### Abstract

Self-supervised learning methods for medical images primarily rely on the
imaging modality during pretraining. While such approaches deliver promising
results, they do not leverage associated patient or scan information collected
within Electronic Health Records (EHR). Here, we propose to incorporate EHR
data during self-supervised pretraining with a Masked Siamese Network (MSN) to
enhance the quality of chest X-ray representations. We investigate three types
of EHR data, including demographic, scan metadata, and inpatient stay
information. We evaluate our approach on three publicly available chest X-ray
datasets, MIMIC-CXR, CheXpert, and NIH-14, using two vision transformer (ViT)
backbones, specifically ViT-Tiny and ViT-Small. In assessing the quality of
the representations via linear evaluation, our proposed method demonstrates
significant improvement compared to vanilla MSN and state-of-the-art self-
supervised learning baselines. Our work highlights the potential of EHR-
enhanced self-supervised pre-training for medical imaging. The code is
publicly available at: https://github.com/nyuad-cai/CXR-EHR-MSN

##  1 Introduction

Supervised training of deep neural networks requires large amounts of quality
annotated data (LeCun et al., 2015). This is not always straightforward in
applications involving clinical tasks, due to the time, cost, effort and
expertise required to collect labeled data (Taleb et al., 2020). Self-
supervised learning has recently demonstrated great success in leveraging
unlabeled data, such as in natural language processing (Lan et al., 2019) and
computer vision (Jing and Tian, 2020). Such frameworks aim to learn useful
underlying representations during pretraining, without any labels, which are
then used in downstream prediction tasks via supervised linear evaluation.

Considering the state-of-the-art performance of self-supervised pretraining
with large unlabeled data compared to end-to-end supervised learning, a
plethora of recent applications in healthcare sought to harness the power of
self-supervised learning by focusing on a specific type of data, usually a
single modality (Shurrab and Duwairi, 2022). For example, Xie et al. (2020)
applied spatial augmentations for 3D image segmentation, while Azizi et al.
(2021) applied transformations to Chest X-Ray (CXR) images and dermatology
images to predict radiology labels and skin conditions, respectively. Zhang et
al. (2022a) preserved time-frequency consistency of time-series data for
several tasks, such as detection of epilepsy, while Kiyasseh et al. (2021)
leveraged electrocardiogram signals to learn patient-specific representations
for classification of cardiac arrhythmia.

In clinical practice, health practitioners rely on several sources of
information to perform a diagnosis or interpret medical scans (Cui and Zhang,
2021). For instance, patient sex could be a decisive determinant of
diagnostics and therapeutic responses (Mauvais-Jarvis et al., 2020), while
patient age could influence clinical decision-making in managing certain
health issues (Adams et al., 2006). Health practitioners generally consider
additional inputs, such as vital-sign measurements and lab test results, to
enhance their understanding of various diseases. Hence, medical data is
inherently multi-modal, encompassing various types of modalities, such as
medical images, Electronic Health Records (EHR), clinical notes, and omics
(Kline et al., 2022). Consequently, we hypothesize that making use of
additional modalities during self-supervised pretraining can improve the
quality of the representations for downstream classification tasks (Krishnan
et al., 2022).

Hence, we propose to incorporate EHR data during self-supervised pretraining
with a Masked Siamese Network (Assran et al., 2022) for CXR representation
learning. In summary, our pretraining framework includes two visual encoders,
adopted from the vanilla MSN, one non-imaging encoder for the EHR, and a
projection module that fuses the modalities to encode each CXR-EHR pair. We
investigate the inclusion of three types of EHR data, including (a)
demographic variables, (b) scan metadata, and (c) inpatient stay information.
Our main contributions can be summarized as follows:

  * •

We design a multi-modal MSN to incorporate EHR data during self-supervised
pretraining for CXR, with the goal of improving the quality of learned
representations and enhancing downstream classification. This specifically
involves the introduction of an EHR encoder and projection head for the
multimodal representation of EHR and CXR. We compare our approach with vanilla
MSN, and compare MSN to other state-of-the-art methods for CXR.

  * •

We conduct a comprehensive evaluation of relevant EHR features, analyzing
their individual and combined impact on self-supervised pretraining for
downstream classification.

  * •

We extensively evaluate our proposed approach using three publicly available
datasets: MIMIC-CXR (Johnson et al., 2019) for pretraining and internal
validation, and ChexPert (Irvin et al., 2019) and NIH-14 (Wang et al., 2017)
for external validation.

We summarize the related literature in Section 2, our proposed methodology in
Section 3, the experimental settings in Section 4, the results in Section 5,
and finally the discussion and concluding remarks in Section 6.

##  2 Related Work

###  2.1 Overview of Self-Supervised Learning

Self-supervised learning methods learn task-agnostic feature representations
using hand-crafted pretext tasks or joint embedding architectures (Kinakh et
al., 2021; Bardes et al., 2021). Hand-crafted pretext tasks rely on the use of
pseudo-labels generated from unlabeled data. Examples of such tasks include
rotation prediction (Gidaris et al., 2018), jigsaw puzzle solving (Noroozi and
Favaro, 2016), colorization (Zhang et al., 2016), and in-painting (Pathak et
al., 2016). Joint embedding methods utilize siamese networks (Bromley et al.,
1993) to learn useful representations by discriminating between different
views of samples based on a specific objective function (Bardes et al., 2021),
without the need for human annotation or pseudo-labels.

Joint embedding methods can be further categorized into contrastive and non-
contrastive methods, where the latter encompasses clustering, distillation,
and information maximization methods (Bardes et al., 2021). Contrastive
methods learn representations by maximizing the agreement between positive
pairs and minimizing the agreement between negative pairs (Van den Oord et
al., 2018). Some prominent examples include SimCLR (Chen et al., 2020),
contrastive predictive coding (Van den Oord et al., 2018), and MoCo (He et
al., 2020). Non-contrastive methods focus on optimizing different forms of
similarity metrics across the learned embeddings. Examples include BYOL (Grill
et al., 2020), SimSiam (Chen and He, 2021), and VICReg (Bardes et al., 2021).
While most existing work considers convolutional networks as backbones for
input encoders, recent approaches explore the role of vision transformers
(ViT) (Dosovitskiy et al., 2020) for self-supervision, such as DINO (Caron et
al., 2021) and MSN (Assran et al., 2022). MSN is a state-of-the-art self-
supervised learning architecture that operates on the principle of mask-
denoising, without reconstruction, as well as transformation invariance with
transformers. MSN has limited applications in healthcare-related tasks, and is
promising considering its computational scalability.

Figure 1: Overview of the multi-modal MSN pre-training framework. We use
$x_{ehr}$ as an auxiliary modality during model pretraining. In downstream
classification, we freeze $f_{target}$ and use it as a feature extractor along
with a classification head for multi-label image classification. Components of
the vanilla MSN are outlined in black.

###  2.2 Self-Supervised Learning in Healthcare

Self-supervised learning methods have shown great promise in learning
representations of different types of medical images, such as computed
tomography and magnetic resonance imaging (Jamaludin et al., 2017; Zhuang et
al., 2019; Taleb et al., 2020), optical coherence tomography and fundus
photography (Holmberg et al., 2020; Hervella et al., 2020; Li et al., 2021),
and endoscopy images (Ross et al., 2018). Several studies investigated self-
supervised learning for applications involving CXR images. For example,
Sowrirajan et al. (2021), Chen et al. (2021), and Sriram et al. (2021)
utilized MoCo (He et al., 2020) as a pretraining strategy for chest disease
diagnosis and prognosis tasks. Azizi et al. (2021) showed that initializing
SimCLR (Chen et al., 2020) during pretraining with ImageNet weights improves
downstream performance in CXR classification. Van der Sluijs et al. (2024)
explored the impact of various image augmentation techniques on siamese
representation learning for CXR.

Some studies investigated the use of other sources of information in designing
the self-supervised learning framework for CXR, mostly focusing on vision-
language pretraining. Vu et al. (2021) proposed MedAug, which considers
patient metadata for generating positive pairs in MoCo framework (He et al.,
2020). Tiu et al. (2022) achieved expert-level performance by utilizing raw
radiology reports associated with CXR as supervisory signals during self-
supervised pretraining. Similarly, Zhang et al. (2022b) proposed ConVIRT, a
self-supervised learning framework that learns representations from paired CXR
and textual data. None of these methods explored the incorporation of static
EHR data during self-supervised pretraining, nor did they investigate the use
of MSN for CXR. To address these gaps, we specifically focus on MSN with the
goal of enhancing chest X-ray representation learning using transformer
pretraining with EHR data.

##  3 Methods

###  3.1 Problem Formulation

Consider $x_{cxr}\in\mathbb{R}^{h\times w}$, where $h$ and $w$ represent the
image height and width, to be a CXR image collected from patient during their
hospital stay, the goal is to predict a set of disease labels $y_{cxr}$. We
assume that each $x_{cxr}$ is associated with $x_{ehr}\in\mathbb{R}^{n}$, a
vector of static features derived from the patient’s EHR data, where $n$ is
the number of variables. We use both modalities to learn the CXR
representation within our pretraining framework. An overview of the multi-
modal MSN architecture for pretraining is shown in Figure 1. The network
consists of two components: (i) the visual encoders for CXR (ii) a multi-modal
branch that incorporates the EHR data, as described in the following sections.

Figure 2: Visualization of the masking strategies. (a) Original image. (b)
Random masking: drops patches at random locations within the image. (c) Focal
masking: keeps a set of neighboring patches and drops the remaining ones.

###  3.2 Masked Siamese Network

We adopt the MSN (Assran et al., 2022) as the base model for our proposed
framework due to its computational scalability and the need for pretraining
transformers efficiently. For a given unlabeled input image, the goal is to
align the anchor and target views, denoted by $x^{anchor}_{cxr}$ and
$x^{target}_{cxr}$, respectively. For each image, a random transformation
function $T(.)$ is used to generate a set of $M$ anchor views and a single
target view. The transformations include image resizing, random resized
cropping, and random horizontal flipping with a probability of $0.5$,
following Sowrirajan et al. (2021). However, we excluded color jittering and
Gaussian blurring as the former does not apply to grayscale images, while the
latter may distort disease-related information (Sowrirajan et al., 2021).
Since the views are patchified to serve as input to a ViT, the anchor views
are further masked via patch dropping (either random or focal masking as shown
in Figure 2), while leaving $x^{target}_{cxr}$ unmasked.

Two encoders, $f_{anchor}$ and $f_{target}$, parameterized with a ViT
(Dosovitskiy et al., 2020), are trained to generate the image embeddings:

$v_{cxr}=f_{anchor}(x^{anchor}_{cxr})\And
v_{cxr+}=f_{target}(x^{target}_{cxr}).$ (1)

The target embedding $v_{cxr+}$ is further processed by a projection head
$h^{+}$ to obtain $z^{+}$, while $v_{cxr}$ is used in the multi-modal mapping
as described in the next section. MSN does not compute the loss directly on
the generated embeddings based on a certain similarity metric. Instead, it
utilizes a set of prototypes and seeks to map the projected embeddings of a
given sample into the same learned prototype, where the mappings are used to
compute the loss. Further background information is provided in Appendix A.

###  3.3 Multi-modal Pretraining

Instead of solely relying on CXR, our proposed framework encourages MSN to
leverage additional information extracted from the patient’s EHR. In
particular, we introduce three additional modules to the standard MSN
architecture. First, we encode the static EHR data with $f_{ehr}$ to learn a
representation vector, such that:

$v_{ehr}=f_{ehr}(x_{ehr}).$ (2)

We then concatenate ($\oplus$) the EHR and CXR latent representations,
$v_{ehr}$ and $v_{cxr}$. At this stage, there exists dimensionality mismatch
between the concatenated representations and $v_{cxr+}$. To address this, we
project the concatenated representation into the same dimensional space of
$v_{cxr+}$ using the projection head $g$, resulting in $v_{fused}$:

$v_{fused}=g(v_{ehr}\oplus v_{cxr}).$ (3)

Lastly, the fused representation is processed by a projection head $h(.)$ to
compute $z$. Both $z$ and $z^{+}$ are used in the clustering of prototypes
during pretraining. Overall, this simple procedure learns a joint
representation of the EHR data and the anchor embeddings, with the goal of
improving the mapping ability of MSN during pretraining.

###  3.4 Downstream Classification

After pretraining the encoders $f_{target}$, $f_{anchor}$, and $f_{ehr}$, we
use $f_{target}$ as a feature extractor. We then include a classification
model $f_{c}$ to predict the image labels:

$\hat{y}_{cxr}=f_{c}(v_{cxr+})$ (4)

The main assumption behind our proposed method is that introducing additional
data related to the patient or the CXR scan during pretraining would provide
the model with valuable contextual information. This information would
contribute to enhancing the quality of the learned representations for
downstream classification.

Table 1: Summary of EHR features incorporated during pretraining. We pretrain
multi-modal MSN using each of the EHR features summarized below.

Group | Feature | Type | Values | Dimension
---|---|---|---|---
Demographic ($x_{D}$) | Age | Numeric | $\\{x\in\mathbb{N}|18\leq x\leq 100\\}$ | $x_{age}\in\mathbb{R}^{1}$
Sex | Binary | $\\{Male,Female\\}$ | $x_{sex}\in\mathbb{R}^{2}$
Scan medatadata ($x_{SM}$) | View | Multiclass |  $\\{AP,L,PA,LL\\}$ | $x_{view}\in\mathbb{R}^{4}$
Position | Binary | $\\{Erect,Recumbent\\}$ | $x_{pos}\in\mathbb{R}^{2}$
Inpatient stay information ($x_{SI}$) | ICU admission | Binary | $\\{Negative,Positive\\}$ | $x_{icu}\in\mathbb{R}^{2}$
In-hospital mortality | Binary | $\\{Negative,Positive\\}$ | $x_{mort}\in\mathbb{R}^{2}$

Table 2: Summary of datasets and data splits. We summarize the size of the
training, validation, and test sets in terms of the number of images used in
our experiments. We used MIMIC-CXR for self-supervised pretraining and
downstream classification and CheXPert to obtain an external test set only.
For NIH-14, we used the training set during downstream classification since it
has different labels from MIMIC-CXR.

Dataset | Purpose | Training | Validation | Test
---|---|---|---|---
MIMIC-CXR | Internal validation | 325,188 | 15,282 | 36,625
ChexPert | External validation | - | - | 688
NIH-14 | External validation | 32,457 | 3,567 | 15,735

##  4 Experiments

###  4.1 Dataset

We conducted our experiments using the MIMIC-CXR dataset, which is a publicly
available dataset of CXR images (Johnson et al., 2019). MIMIC-CXR includes
377,110 chest radiographs gathered from 227,835 studies between 2011 and 2016.
Each image is associated with a set of ground-truth labels
${y}_{cxr}\in\mathbb{R}^{14}$ that indicates the presence of any of 14 medical
conditions extracted from corresponding radiology reports. The medical
conditions are Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged
Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, Pleural Effusion,
Pneumonia, Pneumothorax, Pleural Other, Support Devices, and No Finding. After
pre-processing, the dataset consisted of 376,201 individual CXR images related
to 65,152 patients and 82,506 stays.

We used MIMIC-IV (Johnson et al., 2023), an EHR database associated with
MIMIC-CXR, to extract EHR variables related to the CXR images. Table 1
provides a summary of the extracted EHR features, including data types,
possible values, and dimensionality. After a thorough investigation of the
MIMIC-IV database, we identified three sets of relevant features: patient
demographics ($x_{D}$), scan metadata ($x_{SM}$), and inpatient stay
information ($x_{SI}$). For patient demographics, we retrieved patient age
($x_{age})$ for each specific admission and normalized it using _MinMax_
normalization, as well as patient sex ($x_{sex}$) encoded as a one-hot vector.
For scan metadata, we extracted the scan view ($x_{view}$) and patient
position ($x_{pos}$) and encoded them as one-hot vectors. The possible values
for view are antero-posterior (AP), lateral (L), postero-anterior (PA), or
left-lateral (LL), while for patient position the possible values are erect or
recumbent. For inpatient information, we defined $x_{icu}$ and $x_{mort}$ as
binary features indicating whether the patient required intensive care
treatment or experienced in-hospital mortality, respectively. We provide a
summary of the EHR features distributions in Appendix B.

We used MIMIC-CXR and MIMIC-EHR for pretraining and downstream classification.
We followed the work of Hayat et al. (2022) to obtain the training,
validation, and test splits. The data splits are provided in Table 2.

###  4.2 External Validation

We also performed external validation of our pretrained encoders in downstream
classification using the ChexPert (Irvin et al., 2019) and National Institutes
of Health Chest X-Ray Dataset (NIH-14) (Wang et al., 2017) datasets. ChexPert
is a publicly available medical imaging dataset composed of 224,316 chest
radiographs of 65,240 patients, collected between 2002-2017, including
inpatient and outpatient centers. It includes images and associated radiology
reports, which were used to generate labels indicating the presence of 14
diseases. We evaluated the pretrained encoder and classifier obtained with
MIMIC-CXR on the ChexPert test set since they have the same labels.

NIH-14 is a publicly available medical imaging dataset composed of 112,120
frontal-view CXR images of 30,805 patients, collected between 1992-2015. Each
image is assigned 14 common disease labels. We follow the same split reported
in Irvin et al. (2019) and perform linear evaluation using the NIH-14 training
set, since it has different labels from MIMIC-CXR. The data splits are shown
in Table 2.

###  4.3 Evaluation Protocol

To assess the quality of the learned representations using our proposed
pretraining framework, we conducted linear evaluation as the main evaluation
protocol following previous work in self-supervised learning (Bardes et al.,
2021; Garrido et al., 2023; Huang et al., 2022). We froze the encoder
$f_{target}$ after pretraining, and trained the classification head $f_{c}$,
which is randomly initialized, with embeddings of the whole training set.

We report performance using the Area Under the Receiver Operating
characteristic Curve (AUROC) and the Area Under the Precision-Recall Curve
(AUPRC) across all experiments. We report the 95% Confidence Intervals (CI)
using the bootstrapping method (Puth et al., 2015). We also performed
statistical significance testing by comparing our proposed approach with the
best performing baseline.

###  4.4 Uni-modal Self-supervised Pre-training Baselines

We compared the results of our proposed method with self-supervised and
supervised baselines that are transformer-based for a fair comparison:

  1. 1.

Vanilla MSN (Assran et al., 2022): MSN is a self-supervised learning model
that leverages the idea of mask-denoising while avoiding pixel and token-level
reconstruction. We trained the vanilla version following the architecture in
the original work.

  2. 2.

DINO (Caron et al., 2021): DINO is a transformer-based architecture that
learns representation via knowledge distillation from a teacher backbone to a
student backbone without labels.

  3. 3.

Masked AutoEncoder (MAE) (He et al., 2022): MAE is a transformer-based
encoder-decoder architecture that learns representation by reconstructing the
missing pixels within an input image.

  4. 4.

Supervised: We train ViT-S and ViT-T in an end-to-end fashion with random and
ImageNet (Deng et al., 2009) weight initializations.

In our experiments, we used ViT-Small (ViT-S) as the backbone for our proposed
method and the state-of-the-art self-supervised baselines. We also conducted
experiments using the ViT-Tiny (ViT-T) backbone. These are smaller versions
than the ones defined in Dosovitskiy et al. (2020) and we chose them due to
significantly faster training time with comparable performance compared to the
original implementation with ViT-Base (ViT-B). All implementation details and
model training settings are provided in Appendix A.

Table 3: Performance results for linear evaluation of our proposed method
using ViT-S backbone on the MIMIC-CXR dataset. We summarize the AUROC and
AUPRC performance on the test set for the self-supervised methods as well as
the 95% confidence intervals. The best results are shown in blue. We use
$\uparrow$ and $\downarrow$ to indicate whether the performance of a given
model is 0.5-1.5% better or worse than the reference model (i.e., vanilla
MSN). $\uparrow\uparrow$ and $\downarrow\downarrow$ indicate whether the
difference is 1.5% better or worse than the reference model, respectively. $-$
indicates that the performance difference is less than 0.5% compared to the
reference model.

Pretraining | AUROC (CI) | AUPRC (CI)
---|---|---
ImageNet | 0.703 (0.699, 0.706) $\downarrow\downarrow$ | 0.269 (0.266, 0.272) $\downarrow\downarrow$
DINO | 0.714 (0.711, 0.718) $\downarrow\downarrow$ | 0.278 (0.276, 0.281) $\downarrow$
MAE | 0.649 (0.645, 0.653) $\downarrow\downarrow$ | 0.223 (0.221, 0.225) $\downarrow\downarrow$
MSN | 0.731 (0.727, 0.734) | 0.291 (0.289, 0.294)
MSN$+x_{gen}$ | 0.751∗ (0.748, 0.754) $\uparrow\uparrow$ | 0.311∗ (0.309, 0.314) $\uparrow\uparrow$
MSN$+x_{age}$ | 0.746∗ (0.743, 0.749) $\uparrow\uparrow$ | 0.307∗ (0.305, 0.310) $\uparrow\uparrow$
MSN$+x_{view}$ | 0.747∗ (0.744, 0.750) $\uparrow\uparrow$ | 0.307∗ (0.305, 0.310) $\uparrow\uparrow$
MSN$+x_{pos}$ | 0.748∗ (0.745, 0.752) $\uparrow\uparrow$ | 0.306∗ (0.303, 0.309) $\uparrow\uparrow$
MSN$+x_{mort}$ | 0.748∗ (0.744, 0.751) $\uparrow\uparrow$ | 0.308∗ (0.306, 0.312) $\uparrow\uparrow$
MSN$+x_{icu}$ | 0.746∗ (0.742, 0.749) $\uparrow\uparrow$ | 0.305∗ (0.303, 0.308) $\uparrow$
MSN$+x_{SD}$ | 0.751∗ (0.748, 0.754) $\uparrow\uparrow$ | 0.310∗ (0.308, 0.313) $\uparrow\uparrow$
MSN$+x_{SM}$ | 0.744∗ (0.741, 0.747) $\uparrow$ | 0.306∗ (0.303, 0.309) $\uparrow\uparrow$
MSN$+x_{IS}$ | 0.749∗ (0.746, 0.752) $\uparrow\uparrow$ | 0.308∗ (0.305, 0.311) $\uparrow\uparrow$
MSN$+x_{SD+SM}$ | 0.742∗ (0.739, 0.746) $\uparrow$ | 0.302∗ (0.300, 0.305) $\uparrow$
MSN$+x_{SD+SI}$ | 0.744∗ (0.740, 0.747) $\uparrow$ | 0.306∗ (0.304, 0.309) $\uparrow\uparrow$
MSN$+x_{SM+SI}$ | 0.748∗ (0.744, 0.751) $\uparrow\uparrow$ | 0.307∗ (0.304, 0.310) $\uparrow\uparrow$
MSN$+x_{SD+SM+SI}$ | 0.739† (0.736, 0.743) $\uparrow$ | 0.301∗ (0.299, 0.305) $\uparrow$
∗ Statistical significance results with respect to vanilla MSN ($p<0.001$)
† Statistical significance results with respect to vanilla MSN ($p<0.01$)

##  5 Results

Here, we present the performance results of our proposed method and all self-
supervised baselines using the ViT-S. The performance results for ViT-T and
all other baselines, including self-supervised and supervised learning
methods, are reported in Appendix C.

###  5.1 Linear Evaluation Results

Table 3 summarizes the performance results of the linear evaluation
experiments using ViT-S. First, we note that all of the models that
incorporate EHR during pretraining achieve a better performance compared to
the best performing baseline, vanilla MSN. The improvement is at least $1\%$
in terms of AUROC and AUPRC, with a maximum improvement of $2\%$ when
incorporating demographic information, 0.751 AUROC with $x_{sex}$ or $x_{D}$,
compared to 0.731 AUROC with MSN. Additionally, in most experiments where we
pretrain with a single EHR feature achieve improvements greater than $1.5\%$,
while pretraining with groups of features yields slightly lower performance
improvements.

Moreover, our proposed strategies demonstrate a significant improvement
compared to state-of-the-art self-supervised learning baselines, specifically
MAE and DINO, as well as the supervised ImageNet-initialized baseline,
reaching an improvement of $10\%$ compared to the worst performing model. In
summary, the results of linear evaluation demonstrate that our approach, which
incorporates EHR data during pretraining, enhances the quality of the learned
representations and downstream performance in a multi-label disease
classification task, surpassing both the vanilla MSN and other baselines. In
Appendix C, we provide further results with ViT-T as the model backbone. In
general, we observe similar patterns, demonstrating consistent behaviour with
different backbone architectures.

For the sake of completeness we also report in Appendix C the results of a
secondary evaluation protocol by fine-tuning under low data regimes including
$1\%$, $5\%$, and $10\%$ of the full training set. In this setting, we trained
both the pretrained encoder $f_{target}$ and the classification head $f_{c}$
with the training set. We note that the results are on par with vanilla MSN,
which is expected since fine-tuning dilutes the effects of incorporating EHR
during pretraining.

Table 4: Performance results for external validation of our proposed methods
using ViT-S backbone under linear evaluation. We summarize AUROC and AUPRC
results on CheXpert and NIH-14 test sets including 95% confidence intervals.
The best results are shown in blue. We use $\uparrow$ and $\downarrow$ to
indicate whether the performance of a given model is 0.5-1.5% better or worse
than the reference model (i.e., vanilla MSN). $\uparrow\uparrow$ and
$\downarrow\downarrow$ indicate whether the difference is 1.5% better or worse
than the reference model, respectively. $-$ indicates that the performance
difference is less than 0.5% compared to the reference model.

Model | CheXpert | NIH-14
---|---|---
AUROC (CI) | AUPRC (CI) | AUROC (CI) | AUPRC (CI)
MSN | 0.770 (0.755, 0.785) | 0.420 (0.409, 0.453) | 0.699 (0.695, 0.702) | 0.220 (0.217, 0.226)
MSN$+x_{sex}$ | 0.768 (0.754, 0.788) $-$ | 0.417 (0.414, 0.457) $-$ | 0.734∗ (0.730, 0.738) $\uparrow\uparrow$ | 0.265∗ (0.261, 0.274)$\uparrow\uparrow$
MSN$+x_{age}$ | 0.772 (0.754, 0.797) $-$ | 0.424 (0.416, 0.446) $-$ | 0.736∗ (0.732, 0.739) $\uparrow\uparrow$ | 0.263∗ (0.257, 0.270) $\uparrow\uparrow$
MSN$+x_{view}$ | 0.801† (0.785, 0.825) $\uparrow\uparrow$ | 0.445† (0.420, 0.469) $\uparrow\uparrow$ | 0.732∗ (0.728, 0.736) $\uparrow\uparrow$ | 0.261∗ (0.258, 0.270) $\uparrow\uparrow$
MSN$+x_{pos}$ | 0.785 (0.776, 0.798) $\uparrow\uparrow$ | 0.426 (0.412, 0.465) $\uparrow$ | 0.734∗ (0.730, 0.737) $\uparrow\uparrow$ | 0.261∗ (0.257, 0.270) $\uparrow\uparrow$
MSN$+x_{mort}$ | 0.778 (0.743, 0.811) $\uparrow$ | 0.424 (0.412, 0.451) $-$ | 0.737∗ (0.734, 0.742) $\uparrow\uparrow$ | 0.265∗ (0.261, 0.275) $\uparrow\uparrow$
MSN$+x_{icu}$ | 0.773 (0.739, 0.789) $-$ | 0.422 (0.408, 0.445) $-$ | 0.727∗ (0.722, 0.731) $\uparrow\uparrow$ | 0.256∗ (0.251, 0.264) $\uparrow\uparrow$
MSN$+x_{D}$ | 0.776 (0.757, 0.802) $\uparrow$ | 0.419 (0.406, 0.439) $-$ | 0.736∗ (0.732, 0.740) $\uparrow\uparrow$ | 0.267∗ (0.262, 0.276) $\uparrow\uparrow$
MSN$+x_{SM}$ | 0.796† (0.775, 0.813) $\uparrow\uparrow$ | 0.427 (0.418, 0.473) $\uparrow$ | 0.732∗ (0.728, 0.735) $\uparrow\uparrow$ | 0.258∗ (0.254, 0.268) $\uparrow\uparrow$
MSN$+x_{SI}$ | 0.771 (0.753, 0.788) $-$ | 0.423 (0.402, 0.443) $-$ | 0.738∗ (0.734, 0.741) $\uparrow\uparrow$ | 0.270∗ (0.264, 0.278) $\uparrow\uparrow$
MSN$+x_{D+SM}$ | 0.776 (0.756, 0.791) $\uparrow$ | 0.412 (0.397, 0.447) $\downarrow$ | 0.728∗ (0.725, 0.732) $\uparrow\uparrow$ | 0.254∗ (0.250, 0.263) $\uparrow\uparrow$
MSN$+x_{D+SI}$ | 0.757 (0.724, 0.769) $\downarrow$ | 0.418 (0.404, 0.441) $-$ | 0.728∗ (0.724, 0.732) $\uparrow\uparrow$ | 0.269∗ (0.263, 0.279) $\uparrow\uparrow$
MSN$+x_{SM+D}$ | 0.782 (0.761, 0.803) $\uparrow$ | 0.431 (0.418, 0.452) $\uparrow$ | 0.734∗ (0.730, 0.738) $\uparrow\uparrow$ | 0.263∗ (0.258, 0.273) $\uparrow\uparrow$
MSN$+x_{D+SM+SI}$ | 0.775 (0.753, 0.798) $\uparrow$ | 0.427 (0.413, 0.452) $\uparrow$ | 0.725∗ (0.721, 0.728) $\uparrow\uparrow$ | 0.251∗ (0.248, 0.258) $\uparrow\uparrow$
∗ Statistical significance results with respect to vanilla MSN ($p<0.001$)
† Statistical significance results with respect to vanilla MSN ($p<0.01$)

###  5.2 External Validation Results

Table 4 presents the external validation results for linear evaluation of our
proposed methods using the ViT-S backbone on the ChexPert Irvin et al. (2019)
and NIH-14 Wang et al. (2017) datasets. The results indicate that our multi-
modal pretraining enhances the robustness of self-supervised models, resulting
in higher validation scores and improved performance compared to vanilla MSN
in most scenarios. In the best scenario, the performance gain reaches
$3$-$4\%$ in both evaluation datasets. However, we observe better performance
improvements in the NIH-14 dataset, since its training set was used to train
the linear evaluation classifier, compared to off-the-shelf evaluation with
CheXpert. The results for the VIT-T backbone and fine-tuning are reported in
Appendix C. We observe similar patterns that are in line with the findings of
internal validation with MIMIC-CXR.

Figure 3: Embedding visualization via t-SNE. We plot the resulting embeddings
obtained via t-SNE for (a): Vanilla MSN (best performing baseline) (b):
MSN$+x_{sex}$ (best performing proposed variant) (c): MSN$+x_{D}$ (best
performing proposed variant) (d): MSN$+x_{D+SM+SI}$ (worst performing proposed
variant). The first column depicts the clusters of the four most prevalent
diseases in the dataset, the second column corresponds to the second four most
prevalent diseases, and the last column represents the four least prevalent
diseases.

###  5.3 t-SNE Analysis Results

We applied t-SNE to a subset of samples with a single label in the training
set (n=68,000) of MIMIC-CXR for the ease of interpretability, considering the
challenge of visualizing multi-label samples in the latent space. Figure 3
visualizes the embeddings obtained with vanilla MSN (baseline), MSN$+x_{sex}$,
MSN$+x_{D}$, and MSN$+x_{D+SM+SI}$ for the ViT-S backbone. Based on our
qualitative assessment, we observe that the clustering quality is enhanced
using MSN$+x_{sex}$, MSN$+x_{D}$ and MSN$+x_{D+SM+SI}$ compared to vanilla
MSN. More specifically, the clusters of the best performing approaches
(MSN$+x_{sex}$, and MSN$+x_{D}$) show denser clusters, with lower intra-
cluster distance and larger inter-cluster distances, which is typically
related to a better clustering quality in the embedding space. We hypothesize
that this enhanced clustering quality yields an improved downstream
performance.

##  6 Discussion

In this paper, we propose the incorporation of EHR data during self-supervised
pretraining for CXR representation learning, with the aim of improving
performance in downstream classification. We present our proposed approach as
an enhanced version of an existing self-supervised learning method, MSN
(Assran et al., 2022), by including clinical imaging-related patient EHR data
during pretraining. We also comprehensively evaluated our proposed approach on
the MIMIC-CXR dataset (internal validation), and two datasets for external
validation, ChexPert and NIH-14. Our results demonstrate that the integration
of EHR metadata during pretraining with MSN significantly improves performance
in the downstream linear evaluation protocol. We also observe that, in
general, the inclusion of a single EHR feature yields better quality
embeddings compared to pretraining with combinations of EHR features, as shown
by the t-SNE embeddings in Figure 3. In addition, our proposed method shows
superior performance compared with other state-of-the-art self-supervised
baselines, including DINO (Caron et al., 2021) and MAE, (He et al., 2022) as
well as fully supervised baselines pretrained with ImageNet (Deng et al.,
2009). Our extensive evaluation highlights that our proposed approach
generalizes well to external validation datasets, significantly outperforming
vanilla MSN in various settings for CheXpert and in all cases for the NIH-14
dataset.

A relevant insight derived from our extensive experimentation using single
features and combinations of them is that it is _not guaranteed that the
addition of more EHR features leads to improved performance_. We hypothesize
that such behavior could be attributed to feature interactions, which is an
area of future work.

#### Limitations

Despite the performance improvements shown by our framework during linear
evaluation, our approach has some limitations. First, our approach did not
show significant improvements during end-to-end fine-tuning. That is, it
performed on-par with vanilla MSN, or slightly lower in the worst scenario as
shown in the additional results in Appendix C. We hypothesize that fine-tuning
the target encoder without the EHR data dilutes the impact of incorporating it
during pretraining. Specifically, the learned representations during the
pretraining stage (with EHR data) are updated during fine-tuning (without EHR
data), which impacts the quality of the representations. We aim to address
this limitation in our future work, where the EHR data is incorporated during
both linear evaluation and fine tuning, rather than solely during pre-training
for a fair comparison. Furthermore, we only tested our proposed methodology
with a transformer-based self-supervised learning method. In future work, we
will explore its applicability to other self-supervised learning architectures
and generalization as a framework for other medical imaging datasets and
clinical prediction tasks.

## References

  * Adams et al. (2006) Ann Adams, Christopher D Buckingham, Sara Arber, John B McKinlay, Lisa Marceau, and Carol Link.  The influence of patient’s age on clinical decision-making about coronary heart disease in the usa and the uk.  _Ageing & society_, 26(2):303–321, 2006.
  * Assran et al. (2022) Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas.  Masked siamese networks for label-efficient learning.  In _European Conference on Computer Vision_ , pages 456–473. Springer, 2022.
  * Azizi et al. (2021) Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al.  Big self-supervised models advance medical image classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3478–3488, 2021.
  * Bardes et al. (2021) Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  _arXiv preprint arXiv:2105.04906_ , 2021.
  * Bromley et al. (1993) Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.  Signature verification using a” siamese” time delay neural network.  _Advances in neural information processing systems_ , 6, 1993.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Chen et al. (2021) Xiaocong Chen, Lina Yao, Tao Zhou, Jinming Dong, and Yu Zhang.  Momentum contrastive learning for few-shot covid-19 diagnosis from chest ct images.  _Pattern recognition_ , 113:107826, 2021.
  * Chen and He (2021) Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 15750–15758, 2021.
  * Cui and Zhang (2021) Miao Cui and David Y Zhang.  Artificial intelligence and computational pathology.  _Laboratory Investigation_ , 101(4):412–422, 2021.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Garrido et al. (2023) Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun.  Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank.  In _International Conference on Machine Learning_ , pages 10929–10974. PMLR, 2023.
  * Gidaris et al. (2018) Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  _arXiv preprint arXiv:1803.07728_ , 2018.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  _arXiv preprint arXiv:2006.07733_ , 2020.
  * Hayat et al. (2022) Nasir Hayat, Krzysztof J Geras, and Farah E Shamout.  Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images.  _arXiv preprint arXiv:2207.07027_ , 2022.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Hervella et al. (2020) Álvaro S Hervella, Lucía Ramos, José Rouco, Jorge Novo, and Marcos Ortega.  Multi-modal self-supervised pre-training for joint optic disc and cup segmentation in eye fundus images.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 961–965. IEEE, 2020.
  * Holmberg et al. (2020) Olle G Holmberg, Niklas D Köhler, Thiago Martins, Jakob Siedlecki, Tina Herold, Leonie Keidel, Ben Asani, Johannes Schiefelbein, Siegfried Priglinger, Karsten U Kortuem, et al.  Self-supervised retinal thickness prediction enables deep learning from unlabelled data to boost classification of diabetic retinopathy.  _Nature Machine Intelligence_ , 2(11):719–726, 2020.
  * Huang et al. (2022) Lang Huang, Chao Zhang, and Hongyang Zhang.  Self-adaptive training: Bridging supervised and self-supervised learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 2022.
  * Irvin et al. (2019) Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 590–597, 2019.
  * Jamaludin et al. (2017) Amir Jamaludin, Timor Kadir, and Andrew Zisserman.  Self-supervised learning for spinal mris.  In _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_ , pages 294–302. Springer, 2017.
  * Jing and Tian (2020) Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(11):4037–4058, 2020.
  * Johnson et al. (2019) Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng.  Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.  _arXiv preprint arXiv:1901.07042_ , 2019.
  * Johnson et al. (2023) Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al.  Mimic-iv, a freely accessible electronic health record dataset.  _Scientific data_ , 10(1):1, 2023.
  * Kinakh et al. (2021) Vitaliy Kinakh, Olga Taran, and Svyatoslav Voloshynovskiy.  Scatsimclr: self-supervised contrastive learning with pretext task regularization for small-scale datasets.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1098–1106, 2021.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kiyasseh et al. (2021) Dani Kiyasseh, Tingting Zhu, and David A Clifton.  Clocs: Contrastive learning of cardiac signals across space, time, and patients.  In _International Conference on Machine Learning_ , pages 5606–5615. PMLR, 2021.
  * Kline et al. (2022) Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo.  Multimodal machine learning in precision health: A scoping review.  _npj Digital Medicine_ , 5(1):171, 2022.
  * Krishnan et al. (2022) Rayan Krishnan, Pranav Rajpurkar, and Eric J Topol.  Self-supervised learning in medicine and healthcare.  _Nature Biomedical Engineering_ , 6(12):1346–1352, 2022.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_ , 2019.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Li et al. (2021) Xiaomeng Li, Xiaowei Hu, Xiaojuan Qi, Lequan Yu, Wei Zhao, Pheng-Ann Heng, and Lei Xing.  Rotation-oriented collaborative self-supervised learning for retinal disease diagnosis.  _IEEE Transactions on Medical Imaging_ , 2021.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Mauvais-Jarvis et al. (2020) Franck Mauvais-Jarvis, Noel Bairey Merz, Peter J Barnes, Roberta D Brinton, Juan-Jesus Carrero, Dawn L DeMeo, Geert J De Vries, C Neill Epperson, Ramaswamy Govindan, Sabra L Klein, et al.  Sex and gender: modifiers of health, disease, and medicine.  _The Lancet_ , 396(10250):565–582, 2020.
  * Micikevicius et al. (2017) Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.  Mixed precision training.  _arXiv preprint arXiv:1710.03740_ , 2017.
  * Noroozi and Favaro (2016) Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In _European conference on computer vision_ , pages 69–84. Springer, 2016.
  * Pathak et al. (2016) Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2536–2544, 2016.
  * Puth et al. (2015) Marie-Therese Puth, Markus Neuhäuser, and Graeme D Ruxton.  On the variety of methods for calculating confidence intervals by bootstrapping.  _Journal of Animal Ecology_ , 84(4):892–897, 2015.
  * Ross et al. (2018) Tobias Ross, David Zimmerer, Anant Vemuri, Fabian Isensee, Manuel Wiesenfarth, Sebastian Bodenstedt, Fabian Both, Philip Kessler, Martin Wagner, Beat Müller, et al.  Exploiting the potential of unlabeled endoscopic video data with self-supervised learning.  _International journal of computer assisted radiology and surgery_ , 13(6):925–933, 2018.
  * Shurrab and Duwairi (2022) Saeed Shurrab and Rehab Duwairi.  Self-supervised learning methods and applications in medical imaging analysis: A survey.  _PeerJ Computer Science_ , 8:e1045, 2022.
  * Sowrirajan et al. (2021) Hari Sowrirajan, Jingbo Yang, Andrew Y. Ng, and Pranav Rajpurkar.  Moco pretraining improves representation and transferability of chest x-ray models.  In Mattias Heinrich, Qi Dou, Marleen de Bruijne, Jan Lellmann, Alexander Schläfer, and Floris Ernst, editors, _Proceedings of the Fourth Conference on Medical Imaging with Deep Learning_ , volume 143 of _Proceedings of Machine Learning Research_ , pages 728–744. PMLR, 07–09 Jul 2021.
  * Sriram et al. (2021) Anuroop Sriram, Matthew Muckley, Koustuv Sinha, Farah Shamout, Joelle Pineau, Krzysztof J Geras, Lea Azour, Yindalon Aphinyanaphongs, Nafissa Yakubova, and William Moore.  Covid-19 prognosis via self-supervised representation learning and multi-image prediction.  _arXiv preprint arXiv:2101.04909_ , 2021.
  * Taleb et al. (2020) Aiham Taleb, Winfried Loetzsch, Noel Danz, Julius Severin, Thomas Gaertner, Benjamin Bergner, and Christoph Lippert.  3d self-supervised methods for medical imaging.  _Advances in Neural Information Processing Systems_ , 33:18158–18172, 2020.
  * Tiu et al. (2022) Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar.  Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning.  _Nature Biomedical Engineering_ , 6(12):1399–1406, 2022.
  * Van den Oord et al. (2018) Aaron Van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv e-prints_ , pages arXiv–1807, 2018.
  * Van der Sluijs et al. (2024) Rogier Van der Sluijs, Nandita Bhaskhar, Daniel Rubin, Curtis Langlotz, and Akshay S Chaudhari.  Exploring image augmentations for siamese representation learning with chest x-rays.  In _Medical Imaging with Deep Learning_ , pages 444–467. PMLR, 2024.
  * Vu et al. (2021) Yen Nhi Truong Vu, Richard Wang, Niranjan Balachandar, Can Liu, Andrew Y Ng, and Pranav Rajpurkar.  Medaug: Contrastive learning leveraging patient metadata improves representations for chest x-ray interpretation.  _arXiv preprint arXiv:2102.10663_ , 2021.
  * Wang et al. (2017) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2097–2106, 2017.
  * Xie et al. (2020) Yutong Xie, Jianpeng Zhang, Zehui Liao, Yong Xia, and Chunhua Shen.  Pgl: prior-guided local self-supervised learning for 3d medical image segmentation.  _arXiv preprint arXiv:2011.12640_ , 2020.
  * Zhang et al. (2016) Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In _European conference on computer vision_ , pages 649–666. Springer, 2016.
  * Zhang et al. (2022a) Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik.  Self-supervised contrastive pre-training for time series via time-frequency consistency.  _arXiv preprint arXiv:2206.08496_ , 2022a.
  * Zhang et al. (2022b) Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz.  Contrastive learning of medical visual representations from paired images and text.  In _Machine Learning for Healthcare Conference_ , pages 2–25. PMLR, 2022b.
  * Zhuang et al. (2019) Xinrui Zhuang, Yuexiang Li, Yifan Hu, Kai Ma, Yujiu Yang, and Yefeng Zheng.  Self-supervised feature learning for 3d medical images by playing a rubik’s cube.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 420–428. Springer, 2019.

##  Appendix A Implementation details

In this section, we present additional information pertaining to the
implementation details.

###  A.1 Masked Siamese Network

The MSN network (Assran et al., 2022) is a self-supervised pretraining
architecture that leverages mask de-noising and transformation invariance. Let
$B\geq 1$ be a mini-batch of chest X-ray images sampled from an unlabeled
training set. For each image in the dataset, a set of random transformations
is applied to obtain $M$ anchor views $x_{i,1},x_{i,2}...x_{i,M}$ and a single
target view denoted as $x_{i}^{+}$. As the images serve as input to a Vision
Transformer (ViT), the anchor and target views are initially _patchified_ ,
i.e., converted into a set of patches with dimensions $p\times p$ pixels,
where $p$ represents the patch size. Before the patches are processed by the
transformer-based encoders, random and focal masking are applied to the anchor
views. Figure 2 illustrates the masking strategies: random masking drops
random patches at different locations across the input image, while focal
masking drops a block of adjacent patches.

The anchor patches are processed by the encoder $f_{anchor}$, while the target
patches are processed by the encoder $f_{target}$ (see Figure 1). The weights
of $f_{target}$ are updated through an exponential moving average of
$f_{anchor}$. Both encoders are parameterized as ViT encoders. The [CLS] token
of each encoder is processed by a three-layer projection head $h$ to obtain
the final image embeddings $z_{i,m}$ and $z_{i}^{+}\in\mathbb{R}^{n}$ for each
encoder.

There also exists a set of learnable prototypes
$\mathbf{q}\in\mathbb{R}^{n\times k}$, where $K$ is the number of prototypes.
The network learns to assign the learned embeddings to the prototypes based on
cosine similarity, such that
$p_{i,m}:=\text{softmax}(\frac{z_{i,m}\cdot\mathbf{q}}{\tau})$ and
$p_{i}^{+}:=\text{softmax}(\frac{z_{i}^{+}\cdot\mathbf{q}}{\tau^{+}})$, where
$\tau$ and $\tau^{+}$ are the temperature hyperparameters. As described in
Assran et al. (2022), the overall loss function is defined using a standard
cross-entropy loss $H(p_{i}^{+},p_{i,m})$ and a mean entropy maximization
regularizer $H(\bar{p})$ following Equation 5:

$\mathcal{L}=\frac{1}{MB}\sum^{B}_{i=1}\sum^{M}_{m=1}H(p^{+}_{i},p_{i,m})-\lambda
H(\bar{p}),$ (5)

where $B$ is the batch size, $M$ is the number of anchor views, and $\lambda$
is the weight of the mean entropy maximization regularizer. The value of
$\bar{p}$ in the regularization term is calculated as presented in Equation 6:

$\bar{p}=\frac{1}{MB}\sum^{B}_{i=1}\sum^{M}_{m=1}p_{i,m}$ (6)

Concerning masking, we use a masking ratio of $0.15$ for MSN and $0.6$ for
MAE. We use exponential moving average with a momentum value of $0.996$ to
update the target encoder in MSN and the teacher encoder in DINO, along with
their respective projection heads. For focal view generation in MSN, we crop
the image to $96\times 96$. We set $M=11$ anchor views of an image and
generate 1 random mask and 10 focal masks. We also use ImageNet Deng et al.
(2009) statistics to normalize the input CXR scans during pretraining.

###  A.2 Model Pretraining

We follow the same hyperparameter settings as in the original MSN (Assran et
al., 2022), DINO (Caron et al., 2021), and MAE (He et al., 2022). We use the
AdamW optimizer (Loshchilov and Hutter, 2017) for MSN and the other self-
supervised baselines with a batch size of $64$. To select the learning rate,
we conduct several experiments using different combinations of learning rate
and weight decay values. We empirically found that the best learning rate and
weight decay values for MSN are $1e-4$ and $1e-3$, respectively, while the
best values for DINO and MAE are $1e-5$ and $1e-4$, respectively. For all
pertaining experiments, we employ the cosine annealing method as a learning
rate scheduler. We pretrain each model for $100$ epochs with early stopping,
using a patience of $5$ epochs and a minimum delta of $1e-5$ for the training
loss.

To incorporate EHR data in our proposal, we implement both $f_{ehr}$ and $g$
as linear layers (see Figure 1). We fix the output dimension of $f_{ehr}$ as
$128$ and match the output dimension of $g$ to the hidden size $D$ according
to the backbone used (see Table A1). Table A1 provides an overview of the size
and architecture of the Vision Transformer architectures used.

Table A1: Summary of Vision Transformer (ViT) model variants. Descriptive
summary of the ViT variants employed in our setup and the original Vision
Transformer architecture, ViT-Base.

Model | Layers | Hidden size $D$ | MLP size | Heads | Parameters
---|---|---|---|---|---
ViT-B | 12 | 768 | 3072 | 6 | 86M
ViT-S | 12 | 384 | 1536 | 6 | 21.7M
ViT-T | 12 | 192 | 768 | 6 | 5.5M

###  A.3 Downstream Classification

We use the Adam optimizer (Kingma and Ba, 2014) with a batch size of $64$ for
all downstream evaluation experiments. For each method evaluation, we perform
three experiments with learning rates in [$1e-3$,$5e-4$,$1e-4$] following
(Azizi et al., 2021), each with cosine annealing learning rate scheduler. In
this setup, we apply early stopping with a patience of 5 epochs if the
validation AUROC does not improve by $1e-4$. Additionally, we conduct two more
experiments with learning rates in [$5e-4$, $1e-4$] and use the Reduce on
Plateau learning rate scheduler without early stopping. We perform this setup
once for linear evaluation and replicate it for fine-tuning experiments,
resulting in $10$ experiments for each model. We select the best-performing
model on the validation set for test set evaluation. The maximum training
epochs for all downstream evaluation tasks are set to 50 epochs. For
augmentations, we apply random horizontal flip, random affine, and center crop
during training, and only center crop during inference. We use ImageNet (Deng
et al., 2009) statistics to normalize the CXR images during both training and
inference, following the same settings as during pretraining.

For low-data regime fine-tuning, we conduct five experiments on five randomly
selected subsets of the training set, corresponding to specified percentages
($1\%$, $5\%$, and $10\%$), and evaluate the model on the full test set. To
run these experiments, we use the same settings as the best-performing fine-
tuning setup on the full training set. The best performing run is selected
from the five runs based on the AUROC metric.

All pretraining experiments are conducted using a single Nvidia A100 GPU, and
downstream evaluation is performed using a single Nvidia V100 GPU. Pretraining
takes approximately 20-30 hours for $100$ epochs, while downstream evaluation
requires 2-5 hours for $50$ epochs, depending on the backbone used. We also
enable automatic mixed-precision (Micikevicius et al., 2017) in all
experiments to speed up training and reduce memory requirements.

##  Appendix B EHR Features

Figure B1 provides the frequency distributions for the EHR features considered
in our work during the self-supervised pretraining stage.

Figure B1: Distribution of EHR features. Absolute frequency distributions of
the EHR features employed in our experimental setup. Note that the Y-axis in
all plots provides the absolute frequency value, but the scale differs.

##  Appendix C Additional Results

In this section, we present additional results from our experimental setting.
These results complement or extend the ones provided in the main text,
including additional baselines or architectures (i.e., ViT-T).

###  C.1 Linear Evaluation

Table C1 provides performance results for linear evaluation of self-supervised
methods using the ViT-T architecture as backbone model.

Table C1: Performance results for linear evaluation of self-supervised methods
using ViT-T as backbone model. We summarize the AUROC and AUPRC performance on
the test set for the self-supervised methods as well as the 95% confidence
intervals. The best results are shown in blue. We use $\uparrow$ and
$\downarrow$ to indicate whether the performance of a given model is 0.5-1.5%
better or worse than the reference model (i.e., vanilla MSN).
$\uparrow\uparrow$ and $\downarrow\downarrow$ indicate whether the difference
is 1.5% better or worse than the reference model, respectively. $-$ indicates
that the performance difference is less than 0.5% compared to the reference
model.

Pretraining | AUROC (CI) | AUPRC (CI)
---|---|---
ImageNet | 0.684 (0.680, 0.688) $\downarrow\downarrow$ | 0.251 (0.249, 0.254) $\downarrow\downarrow$
DINO | 0.694 (0.691, 0.698) $\downarrow$ | 0.262 (0.259, 0.264) $\downarrow$
MAE | 0.633 (0.629, 0.637) $\downarrow\downarrow$ | 0.211 (0.209, 0.213) $\downarrow\downarrow$
MSN | 0.708 (0.704, 0.711) | 0.272 (0.270, 0.275)
MSN$+x_{sex}$ | 0.728∗ (0.724, 0.731) $\uparrow\uparrow$ | 0.290∗ (0.288, 0.293) $\uparrow\uparrow$
MSN$+x_{age}$ | 0.728∗ (0.724, 0.732) $\uparrow\uparrow$ | 0.289∗ (0.287, 0.292) $\uparrow\uparrow$
MSN+$x_{view}$ | 0.726∗ (0.722, 0.730) $\uparrow\uparrow$ | 0.288∗ (0.286, 0.291) $\uparrow\uparrow$
MSN+$x_{pos}$ | 0.729∗ (0.726, 0.733) $\uparrow\uparrow$ | 0.292∗ (0.290, 0.295) $\uparrow\uparrow$
MSN+$x_{mort}$ | 0.727∗ (0.723, 0.730) $\uparrow\uparrow$ | 0.291∗ (0.288, 0.294) $\uparrow\uparrow$
MSN+$x_{icu}$ | 0.726∗ (0.722, 0.729) $\uparrow\uparrow$ | 0.289∗ (0.287, 0.292) $\uparrow\uparrow$
MSN+$x_{D}$ | 0.727∗ (0.723, 0.731) $\uparrow\uparrow$ | 0.290∗ (0.287, 0.293) $\uparrow\uparrow$
MSN+$x_{SM}$ | 0.727∗ (0.723, 0.730) $\uparrow\uparrow$ | 0.290∗ (0.288, 0.293) $\uparrow\uparrow$
MSN+$x_{SI}$ | 0.725∗ (0.721, 0.728) $\uparrow\uparrow$ | 0.289∗ (0.286, 0.292) $\uparrow\uparrow$
MSN+$x_{D+SM}$ | 0.724∗ (0.720, 0.727) $\uparrow\uparrow$ | 0.285∗ (0.283, 0.288) $\uparrow$
MSN+$x_{D+SI}$ | 0.723∗ (0.720, 0.727) $\uparrow\uparrow$ | 0.287∗ (0.285, 0.291) $\uparrow\uparrow$
MSN+$x_{SM+SI}$ | 0.726∗ (0.723, 0.729) $\uparrow\uparrow$ | 0.288∗ (0.286, 0.291) $\uparrow\uparrow$
MSN+$x_{D+SM+SI}$ | 0.724∗ (0.720, 0.727) $\uparrow\uparrow$ | 0.287∗ (0.284, 0.289) $\uparrow\uparrow$
∗ Statistical significance results with respect to vanilla MSN ($p<0.001$)

###  C.2 Supervised learning

Table C2 reports performance results of reference architectures trained using
supervised learning for comparison with self-supervised methods.

Table C2: Supervised learning models. Reference models trained using
supervised learning.

Backbone | Initialization | AUROC (CI) | AUPRC (CI)
---|---|---|---
ViT-T | Random | 0.762 (0.758, 0.765) | 0.329 (0.326, 0.332)
ViT-T | ImageNet | 0.783 (0.780, 0.787) | 0.356 (0.353, 0.360)
ViT-S | Random | 0.763 (0.759, 0.766) | 0.330 (0.328, 0.334)
ViT-S | ImageNet | 0.782 (0.778, 0.785) | 0.353 (0.350, 0.358)

###  C.3 Fine-tuning

Table C3 provides performance results for fine-tuning of self-supervised
methods using the ViT-S architecture as backbone model. Table C4 provides the
results obtained using ViT-T architecture as backbone.

Table C3: Performance results for the fine-tuning of our proposed methods
using ViT-S as backbone and under low-data regime. We summarize AUROC and
AUPRC results on the test set including 95% confidence intervals. The best
results are shown in blue.We use $\uparrow$ and $\downarrow$ to indicate
whether the performance of a given model is 0.5-1.5% better or worse than the
reference model (i.e., vanilla MSN). $\uparrow\uparrow$ and
$\downarrow\downarrow$ indicate whether the difference is 1.5% better or worse
than the reference model, respectively. $-$ indicates that the performance
difference is less than 0.5% compared to the reference model.

Pretraining | Training data proportion
---|---
1% | 5% | 10%
AUROC (CI) | AUPRC (CI) | AUROC (CI) | AUPRC (CI) | AUROC (CI) | AUPRC (CI)
DINO | 0.690 (0.687, 0.694) $\downarrow\downarrow$ | 0.269 (0.267, 0.271) $\downarrow\downarrow$ | 0.730 (0.727, 0.734)$\downarrow\downarrow$ | 0.302 (0.300, 0.305)$\downarrow$ | 0.737 (0.733, 0.740)$\downarrow\downarrow$ | 0.305 (0.302, 0.308)$\downarrow\downarrow$
MAE | 0.659 (0.655, 0.664) $\downarrow\downarrow$ | 0.241 (0.240, 0.244) $\downarrow\downarrow$ | 0.702 (0.699, 0.706)$\downarrow\downarrow$ | 0.274 (0.272, 0.277)$\downarrow\downarrow$ | 0.727 (0.723, 0.731)$\downarrow\downarrow$ | 0.297 (0.294, 0.300)$\downarrow\downarrow$
MSN | 0.707 (0.703, 0.711) | 0.286 (0.284, 0.289) | 0.746 (0.743, 0.75) | 0.316 (0.313, 0.319) | 0.760 (0.756, 0.763) | 0.329 (0.327, 0.333)
MSN$+x_{sex}$ | 0.713 (0.709, 0.717) $\uparrow$ | 0.289 (0.287, 0.292) $-$ | 0.751 (0.747, 0.754) $\uparrow$ | 0.315 (0.313, 0.319) $-$ | 0.761 (0.758, 0.765) $-$ | 0.329 (0.327, 0.333) $-$
MSN$+x_{age}$ | 0.714 (0.710, 0.718) $\uparrow$ | 0.290 (0.288, 0.293) $-$ | 0.751 (0.748, 0.755) $\uparrow$ | 0.319 (0.316, 0.322) $-$ | 0.760 (0.756, 0.763) $-$ | 0.326 (0.324, 0.330) $-$
MSN$+x_{view}$ | 0.708 (0.704, 0.711) $-$ | 0.282 (0.280, 0.285) $-$ | 0.748 (0.744, 0.751) $-$ | 0.317 (0.315, 0.320) $-$ | 0.761 (0.757, 0.764) $-$ | 0.329 (0.327, 0.333) $-$
MSN$+x_{pos}$ | 0.713 (0.710, 0.717) $\uparrow$ | 0.290 (0.288, 0.292) $-$ | 0.749 (0.745, 0.752) $-$ | 0.317 (0.315, 0.320) $-$ | 0.761 (0.758, 0.764) $-$ | 0.329 (0.326, 0.333) $-$
MSN$+x_{mort}$ | 0.711 (0.707, 0.715) $-$ | 0.282 (0.279, 0.284) $-$ | 0.748 (0.744, 0.751) $-$ | 0.317 (0.314, 0.320) $-$ | 0.761 (0.757, 0.764) $-$ | 0.326 (0.323, 0.329) $-$
MSN$+x_{icu}$ | 0.706 (0.702, 0.710) $-$ | 0.284 (0.282, 0.287) $-$ | 0.747 (0.744, 0.751) $-$ | 0.314 (0.311, 0.317) $-$ | 0.758 (0.755, 0.762) $-$ | 0.324 (0.321, 0.327) $-$
MSN$+x_{D}$ | 0.710 (0.706, 0.713) $-$ | 0.290 (0.288, 0.293) $-$ | 0.749 (0.746, 0.752) $-$ | 0.315 (0.313, 0.318) $-$ | 0.763 (0.760, 0.766) $-$ | 0.330 (0.327, 0.333) $-$
MSN$+x_{SM}$ | 0.710 (0.706, 0.714) $-$ | 0.287 (0.285, 0.290) $-$ | 0.748 (0.744, 0.751) $-$ | 0.315 (0.313, 0.319) $-$ | 0.758 (0.755, 0.762) $-$ | 0.327 (0.324, 0.330) $-$
MSN$+x_{SI}$ | 0.708 (0.704, 0.712) $-$ | 0.286 (0.284, 0.289) $-$ | 0.749 (0.746, 0.753) $-$ | 0.315 (0.313, 0.318) $-$ | 0.762 (0.759, 0.766) $-$ | 0.327 (0.325, 0.331) $-$
MSN$+x_{D+SM}$ | 0.704 (0.701, 0.708) $-$ | 0.287 (0.285, 0.290) $-$ | 0.746 (0.742, 0.749) $-$ | 0.314 (0.312, 0.317) $-$ | 0.758 (0.754, 0.761) $-$ | 0.325 (0.323, 0.329) $-$
MSN$+x_{D+SI}$ | 0.703 (0.699, 0.706) $-$ | 0.286 (0.284, 0.288) $-$ | 0.745 (0.742, 0.749) $-$ | 0.315 (0.313, 0.318) $-$ | 0.762 (0.759, 0.765) $-$ | 0.328 (0.325, 0.331) $-$
MSN$+x_{SM+SI}$ | 0.708 (0.704, 0.711) $-$ | 0.282 (0.280, 0.284) $-$ | 0.746 (0.743, 0.750) $-$ | 0.315 (0.313, 0.319) $-$ | 0.760 (0.757, 0.763) $-$ | 0.326 (0.323, 0.329) $-$
MSN$+x_{D+SM+SI}$ | 0.706 (0.702, 0.709) $-$ | 0.285 (0.283, 0.288) $-$ | 0.746 (0.743, 0.750) $-$ | 0.314 (0.312, 0.318) $-$ | 0.756 (0.753, 0.759) $-$ | 0.326 (0.323, 0.330) $-$

Table C4: Performance results for the fine-tuning of our proposed methods
using ViT-T as backbone and under low-data regime. We summarize AUROC and
AUPRC results on the test set including 95% confidence intervals. The best
results are shown in blue. We use $\uparrow$ and $\downarrow$ to indicate
whether the performance of a given model is 0.5-1.5% better or worse than the
reference model (i.e., vanilla MSN). $\uparrow\uparrow$ and
$\downarrow\downarrow$ indicate whether the difference is 1.5% better or worse
than the reference model, respectively. $-$indicates that the performance
difference is less than 0.5% compared to the reference model.

Pretraining | Training data proportion
---|---
1% | 5% | 10%
AUROC (CI) | AUPRC (CI) | AUROC (CI) | AUPRC (CI) | AUROC (CI) | AUPRC (CI)
DINO | 0.685 (0.682, 0.689) $\downarrow$ | 0.266 (0.264, 0.268) $\downarrow$ | 0.727 (0.723, 0.730) $\downarrow\downarrow$ | 0.297 (0.295, 0.300) $\downarrow$ | 0.741 (0.737, 0.745) $\downarrow$ | 0.310 (0.308, 0.313) $\downarrow$
MAE | 0.640 (0.636, 0.644) $\downarrow\downarrow$ | 0.228 (0.227, 0.231) $\downarrow\downarrow$ | 0.700 (0.697, 0.704) $\downarrow\downarrow$ | 0.271 (0.269, 0.274) $\downarrow\downarrow$ | 0.721 (0.717, 0.725) $\downarrow\downarrow$ | 0.289 (0.286, 0.292) $\downarrow\downarrow$
MSN | 0.694 (0.69, 0.698) | 0.276 (0.274, 0.279) | 0.742 (0.739, 0.746) | 0.311 (0.309, 0.314) | 0.754 (0.751, 0.758) | 0.322 (0.320, 0.326)
MSN$+x_{sex}$ | 0.700 (0.696, 0.704) $\uparrow$ | 0.280 (0.278, 0.283) $-$ | 0.739 (0.735, 0.742) $-$ | 0.310 (0.308, 0.313) $-$ | 0.754 (0.750, 0.758) $-$ | 0.320 (0.318, 0.324) $-$
MSN$+x_{age}$ | 0.697 (0.693, 0.701) $-$ | 0.278 (0.276, 0.281) $-$ | 0.740 (0.737, 0.744) $-$ | 0.312 (0.310, 0.315) $-$ | 0.753 (0.750, 0.756) $-$ | 0.321 (0.318, 0.324) $-$
MSN$+x_{view}$ | 0.698 (0.693, 0.701) $-$ | 0.280 (0.278, 0.283) $-$ | 0.742 (0.738, 0.746) $-$ | 0.310 (0.307, 0.313) $-$ | 0.752 (0.748, 0.755) $-$ | 0.320 (0.317, 0.323) $-$
MSN$+x_{pos}$ | 0.702 (0.698, 0.705) $\uparrow$ | 0.279 (0.277, 0.281) $-$ | 0.739 (0.734, 0.742) $-$ | 0.308 (0.306, 0.312) $-$ | 0.754 (0.750, 0.757) $-$ | 0.319 (0.316, 0.322) $-$
MSN$+x_{mort}$ | 0.698 (0.694, 0.702) $-$ | 0.283 (0.280, 0.285) $\uparrow$ | 0.738 (0.735, 0.741) $-$ | 0.307 (0.305, 0.311) $-$ | 0.752 (0.748, 0.755) $-$ | 0.318 (0.316, 0.321) $-$
MSN$+x_{icu}$ | 0.700 (0.696, 0.704) $\uparrow$ | 0.278 (0.276, 0.281) $-$ | 0.739 (0.736, 0.743) $-$ | 0.308 (0.305, 0.311) $-$ | 0.751 (0.748, 0.754) $-$ | 0.319 (0.316, 0.322) $-$
MSN$+x_{D}$ | 0.698 (0.694, 0.701) $-$ | 0.273 (0.271, 0.276) $-$ | 0.738 (0.734, 0.742) $-$ | 0.309 (0.306, 0.311) $-$ | 0.752 (0.748, 0.755) $-$ | 0.318 (0.316, 0.321) $-$
MSN$+x_{SM}$ | 0.699 (0.695, 0.703) $\uparrow$ | 0.278 (0.276, 0.281) $-$ | 0.740 (0.736, 0.743) $-$ | 0.308 (0.306, 0.311) $-$ | 0.754 (0.75, 0.757) $-$ | 0.322 (0.320, 0.326) $-$
MSN$+x_{SI}$ | 0.702 (0.698, 0.706) $\uparrow$ | 0.282 (0.280, 0.285) $\uparrow$ | 0.739 (0.736, 0.743) $-$ | 0.309 (0.306, 0.311) $-$ | 0.750 (0.746, 0.754) $-$ | 0.317 (0.315, 0.320) $\downarrow$
MSN$+x_{D+SM}$ | 0.697 (0.693, 0.701) $-$ | 0.277 (0.275, 0.280) $-$ | 0.735 (0.731, 0.738) $\downarrow$ | 0.307 (0.304, 0.310) $-$ | 0.749 (0.745, 0.752) $\downarrow$ | 0.317 (0.315, 0.320) $\downarrow$
MSN$+x_{D+SI}$ | 0.697 (0.694, 0.701) $-$ | 0.278 (0.276, 0.281) $-$ | 0.740 (0.737, 0.744) $-$ | 0.307 (0.304, 0.310) $-$ | 0.753 (0.750, 0.757) $-$ | 0.321 (0.318, 0.324) $-$
MSN$+x_{SM+SI}$ | 0.699 (0.694, 0.702) $\uparrow$ | 0.276 (0.273, 0.278) $-$ | 0.739 (0.735, 0.742) $-$ | 0.306 (0.304, 0.310) $\downarrow$ | 0.750 (0.746, 0.753) $-$ | 0.316 (0.313, 0.319) $\downarrow$
MSN$+x_{D+SM+SI}$ | 0.701 (0.698, 0.705) $\uparrow$ | 0.280 (0.278, 0.283) $-$ | 0.736 (0.733, 0.740) $\downarrow$ | 0.305 (0.303, 0.308) $\downarrow$ | 0.752 (0.749, 0.756) $-$ | 0.321 (0.319, 0.325) $-$

###  C.4 External Validation

Table C5 reports external validation results on downstream classification for
fine tuning of ViT-S self-supervised models on the ChexPert and NIH-14
datasets.

Table C5: Performance results for external validation of our proposed methods
using ViT-S backbone under fine-tuning self-supervised. We summarize AUROC and
AUPRC results on CheXpert and NIH-14 test sets including 95% confidence
intervals. The best results are shown in blue. We use $\uparrow$ and
$\downarrow$ to indicate whether the performance of a given model is 0.5-1.5%
better or worse than the reference model (i.e., vanilla MSN).
$\uparrow\uparrow$ and $\downarrow\downarrow$ indicate whether the difference
is 1.5% better or worse than the reference model, respectively. $-$ indicates
that the performance difference is less than 0.5% compared to the reference
model.

| CheXpert | NIH-14
---|---|---
| AUROC (CI) | AUPRC (CI) | AUROC (CI) | AUPRC (CI)
MSN | 0.844 (0.827, 0.862) | 0.505 (0.483, 0.536) | 0.783 (0.780, 0.787) | 0.346 (0.339, 0.359)
MSN$+x_{sex}$ | 0.850 (0.832, 0.867) $\uparrow$ | 0.505 (0.480, 0.541) $-$ | 0.782 (0.778, 0.787) $-$ | 0.350 (0.341, 0.361) $-$
MSN$+x_{age}$ | 0.847 (0.830, 0.863) $-$ | 0.491 (0.471, 0.522) $\downarrow$ | 0.783 (0.779, 0.787) $-$ | 0.350 (0.342, 0.360) $-$
MSN$+x_{view}$ | 0.837 (0.819, 0.855) $\downarrow$ | 0.487 (0.470, 0.514) $\downarrow\downarrow$ | 0.777 (0.772, 0.781) $\downarrow$ | 0.338 (0.330, 0.349) $\downarrow$
MSN$+x_{pos}$ | 0.836 (0.819, 0.852) $\downarrow$ | 0.498 (0.476, 0.529) $\downarrow$ | 0.782 (0.778, 0.785) $-$ | 0.343 (0.337, 0.354) $-$
MSN$+x_{mort}$ | 0.841 (0.823, 0.859) $-$ | 0.521 (0.494, 0.559) $\uparrow\uparrow$ | 0.782 (0.776, 0.786) $-$ | 0.343 (0.336, 0.356) $-$
MSN$+x_{icu}$ | 0.847 (0.829, 0.867) $-$ | 0.510 (0.487, 0.550) $\uparrow$ | 0.781 (0.778, 0.786) $-$ | 0.343 (0.337, 0.354) $-$
MSN$+x_{D}$ | 0.830 (0.815, 0.846) $\downarrow$ | 0.477 (0.458, 0.507) $\downarrow\downarrow$ | 0.783 (0.777, 0.787) $-$ | 0.347 (0.339, 0.358) $-$
MSN$+x_{SM}$ | 0.828 (0.807, 0.848) $\downarrow\downarrow$ | 0.483 (0.464, 0.516) $\downarrow\downarrow$ | 0.780 (0.776, 0.784) $-$ | 0.343 (0.335, 0.353) $-$
MSN$+x_{SI}$ | 0.849 (0.830, 0.866) $\uparrow$ | 0.492 (0.474, 0.520) $\downarrow$ | 0.780 (0.776, 0.784) $-$ | 0.347 (0.340, 0.357) $-$
MSN$+x_{D+SM}$ | 0.850 (0.830, 0.869) $\uparrow$ | 0.510 (0.487, 0.546) $\uparrow$ | 0.775 (0.771, 0.780) $\downarrow$ | 0.338 (0.331, 0.349) $\downarrow$
MSN$+x_{D+SI}$ | 0.827 (0.807, 0.845) $\downarrow\downarrow$ | 0.480 (0.462, 0.511) $\downarrow\downarrow$ | 0.776 (0.772, 0.781) $\downarrow$ | 0.342 (0.335, 0.354) $-$
MSN$+x_{SM+D}$ | 0.834 (0.816, 0.852) $\downarrow$ | 0.489 (0.470, 0.525) $\downarrow\downarrow$ | 0.782 (0.778, 0.786) $-$ | 0.339 (0.333, 0.347) $\downarrow$
MSN$+x_{D+SM+SI}$ | 0.854 (0.835, 0.870) $\uparrow$ | 0.503 (0.481, 0.534) $-$ | 0.775 (0.769, 0.779) $\downarrow$ | 0.338 (0.331, 0.351) $\downarrow$

Table C6 report external validation results on downstream classification for
linear evaluation of ViT-S self-supervised models on the ChexPert and NIH-14
datasets. Table C7 provides the results for the fine-tuning of self-supervised
models.

Table C6: Performance results of external validation for linear evaluation of
self-supervised methods using ViT-T as backbone model. We summarize AUROC and
AUPRC results on CheXpert and NIH-14 test sets including 95% confidence
intervals. The best results are shown in blue. We use $\uparrow$ and
$\downarrow$ to indicate whether the performance of a given model is 0.5-1.5%
better or worse than the reference model (i.e., vanilla MSN).
$\uparrow\uparrow$ and $\downarrow\downarrow$ indicate whether the difference
is 1.5% better or worse than the reference model, respectively. $-$ indicates
that the performance difference is less than 0.5% compared to the reference
model.

| CheXpert | NIH-14
---|---|---
| AUROC (CI) | AUPRC (CI) | AUROC (CI) | AUPRC (CI)
MSN | 0.740 (0.720, 0.763) | 0.396 (0.382, 0.421) | 0.676 (0.671, 0.68) | 0.199 (0.197, 0.203)
MSN$+x_{sex}$ | 0.742 (0.721, 0.764) $-$ | 0.403 (0.389, 0.425) $\uparrow$ | 0.711∗ (0.707, 0.714) $\uparrow\uparrow$ | 0.233∗ (0.230, 0.242) $\uparrow\uparrow$
MSN$+x_{age}$ | 0.767 (0.746, 0.789) $\uparrow\uparrow$ | 0.390 (0.378, 0.410) $\downarrow$ | 0.711∗ (0.707, 0.715) $\uparrow\uparrow$ | 0.232∗ (0.229, 0.238) $\uparrow\uparrow$
MSN$+x_{view}$ | 0.765 (0.742, 0.788) $\uparrow\uparrow$ | 0.420∗ (0.405, 0.453) $\uparrow\uparrow$ | 0.711∗ (0.708, 0.715) $\uparrow\uparrow$ | 0.236∗ (0.232, 0.245) $\uparrow\uparrow$
MSN$+x_{pos}$ | 0.749 (0.728, 0.770) $\uparrow$ | 0.392 (0.380, 0.411) $-$ | 0.710∗ (0.706, 0.713) $\uparrow\uparrow$ | 0.235∗ (0.232, 0.241) $\uparrow\uparrow$
MSN$+x_{mort}$ | 0.750 (0.728, 0.772) $\uparrow$ | 0.413 (0.400, 0.435) $\uparrow\uparrow$ | 0.709∗ (0.705, 0.712) $\uparrow\uparrow$ | 0.236∗ (0.232, 0.244) $\uparrow\uparrow$
MSN$+x_{icu}$ | 0.744 (0.723, 0.765) $-$ | 0.398 (0.386, 0.422) $-$ | 0.704∗ (0.700, 0.708) $\uparrow\uparrow$ | 0.227∗ (0.225, 0.235) $\uparrow\uparrow$
MSN$+x_{D}$ | 0.746 (0.726, 0.767) $\uparrow$ | 0.379 (0.368, 0.399) $\downarrow\downarrow$ | 0.713∗ (0.709, 0.717) $\uparrow\uparrow$ | 0.238∗ (0.235, 0.245) $\uparrow\uparrow$
MSN$+x_{SM}$ | 0.766 (0.745, 0.786) $\uparrow\uparrow$ | 0.402 (0.387, 0.427) $\uparrow$ | 0.716∗ (0.712, 0.719) $\uparrow\uparrow$ | 0.241∗ (0.236, 0.249) $\uparrow\uparrow$
MSN$+x_{SI}$ | 0.765 (0.745, 0.783) $\uparrow\uparrow$ | 0.387 (0.375, 0.407) $\downarrow$ | 0.712∗ (0.708, 0.715) $\uparrow\uparrow$ | 0.234∗ (0.231, 0.241) $\uparrow\uparrow$
MSN$+x_{D+SM}$ | 0.752 (0.727, 0.778) $\uparrow$ | 0.391 (0.379, 0.423) $\downarrow$ | 0.704∗ (0.700, 0.707) $\uparrow\uparrow$ | 0.227∗ (0.224, 0.234) $\uparrow\uparrow$
MSN$+x_{D+SI}$ | 0.755 (0.738, 0.777) $\uparrow\uparrow$ | 0.388 (0.376, 0.408) $\downarrow$ | 0.709∗ (0.705, 0.712) $\uparrow\uparrow$ | 0.233∗ (0.230, 0.241) $\uparrow\uparrow$
MSN$+x_{SM+D}$ | 0.769 (0.752, 0.794) $\uparrow\uparrow$ | 0.406 (0.403, 0.432) $\uparrow$ | 0.708∗ (0.704, 0.711) $\uparrow\uparrow$ | 0.236∗ (0.232, 0.243) $\uparrow\uparrow$
MSN$+x_{D+SM+SI}$ | 0.770† (0.746, 0.781) $\uparrow\uparrow$ | 0.409 (0.403, 0.454) $\uparrow$ | 0.711∗ (0.706, 0.716) $\uparrow\uparrow$ | 0.233∗ (0.230, 0.240) $\uparrow\uparrow$
∗ Statistical significance results with respect to vanilla MSN ($p<0.001$)
† Statistical significance results with respect to vanilla MSN ($p<0.01$)

Table C7: Performance results of external validation for fine tuning of self-
supervised methods using ViT-T as backbone model. We summarize AUROC and AUPRC
results on CheXpert and NIH-14 test sets including 95% confidence intervals.
The best results are shown in blue. We use $\uparrow$ and $\downarrow$ to
indicate whether the performance of a given model is 0.5-1.5% better or worse
than the reference model (i.e., vanilla MSN). $\uparrow\uparrow$ and
$\downarrow\downarrow$ indicate whether the difference is 1.5% better or worse
than the reference model, respectively. $-$ indicates that the performance
difference is less than 0.5% compared to the reference model.

| CheXpert | NIH-14
---|---|---
| AUROC (CI) | AUPRC (CI) | AUROC (CI) | AUPRC (CI)
MSN | 0.854 (0.834, 0.878) | 0.494 (0.491, 0.541) | 0.773 (0.769, 0.777) | 0.330 (0.323, 0.342)
MSN$+x_{sex}$ | 0.842 (0.819, 0.854) $\downarrow$ | 0.510 (0.499, 0.540) $\uparrow$ | 0.768 (0.763, 0.772) $\downarrow$ | 0.325 (0.318, 0.335) $\downarrow$
MSN$+x_{age}$ | 0.846 (0.820, 0.864) $\downarrow$ | 0.492 (0.467, 0.519) $-$ | 0.772 (0.768, 0.776) $-$ | 0.330 (0.324, 0.341) $-$
MSN$+x_{view}$ | 0.835 (0.821, 0.849) $\downarrow\downarrow$ | 0.488 (0.452, 0.514) $\downarrow$ | 0.776 (0.771, 0.780) $-$ | 0.335 (0.328, 0.344) $\uparrow$
MSN$+x_{pos}$ | 0.839 (0.819, 0.860) $\downarrow$ | 0.495 (0.480, 0.533) $-$ | 0.768 (0.764, 0.773) $\downarrow$ | 0.329 (0.322, 0.341) $-$
MSN$+x_{mort}$ | 0.822 (0.806, 0.845) $\downarrow\downarrow$ | 0.486 (0.484, 0.514) $\downarrow$ | 0.772 (0.767, 0.776) $-$ | 0.326 (0.319, 0.337) $-$
MSN$+x_{icu}$ | 0.832 (0.810, 0.853) $\downarrow\downarrow$ | 0.465 (0.453, 0.538) $\downarrow\downarrow$ | 0.769 (0.765, 0.773) $-$ | 0.332 (0.325, 0.343) $-$
MSN$+x_{D}$ | 0.822 (0.811, 0.842) $\downarrow\downarrow$ | 0.478 (0.466, 0.526) $\downarrow\downarrow$ | 0.768 (0.763, 0.773) $\downarrow$ | 0.330 (0.323, 0.341) $-$
MSN$+x_{SM}$ | 0.824 (0.805, 0.831) $\downarrow\downarrow$ | 0.474 (0.468, 0.501) $\downarrow\downarrow$ | 0.773 (0.768, 0.777) $-$ | 0.335 (0.328, 0.345) $-$
MSN$+x_{SI}$ | 0.829 (0.800, 0.858) $\downarrow\downarrow$ | 0.491 (0.453, 0.527)$-$ | 0.772 (0.768, 0.775) $-$ | 0.330 (0.324, 0.340) $-$
MSN$+x_{D+SM}$ | 0.832 (0.804, 0.858) $\downarrow\downarrow$ | 0.494 (0.475, 0.535) $-$ | 0.767 (0.763, 0.771) $\downarrow$ | 0.318 (0.313, 0.327) $\downarrow$
MSN$+x_{D+SI}$ | 0.838 (0.815, 0.848) $\downarrow\downarrow$ | 0.482 (0.460, 0.515) $\downarrow\downarrow$ | 0.775 (0.770, 0.779) $\uparrow$ | 0.337 (0.330, 0.349) $\uparrow$
MSN$+x_{SM+D}$ | 0.840 (0.819, 0.846) $\downarrow$ | 0.496 (0.484, 0.536) $-$ | 0.772 (0.768, 0.777) $-$ | 0.334 (0.328, 0.346) $-$
MSN$+x_{D+SM+SI}$ | 0.846 (0.836, 0.866) $\downarrow$ | 0.492 (0.469, 0.531) $-$ | 0.772 (0.767, 0.776) $-$ | 0.334 (0.328, 0.344) $-$