# Rethinking Intermediate Layers design in Knowledge Distillation for Kidney
and Liver Tumor Segmentation

Knowledge distillation (KD) has demonstrated remarkable success across various
domains, but its application to medical imaging tasks, such as kidney and
liver tumor segmentation, has encountered challenges. Many existing KD methods
are not specifically tailored for these tasks. Moreover, prevalent KD methods
often lack a careful consideration of ‘what’ and ‘from where’ to distill
knowledge from the teacher to the student. This oversight may lead to issues
like the accumulation of training bias within shallower student layers,
potentially compromising the effectiveness of KD. To address these challenges,
we propose Hierarchical Layer-selective Feedback Distillation (HLFD). HLFD
strategically distills knowledge from a combination of middle layers to
earlier layers and transfers final layer knowledge to intermediate layers at
both the feature and pixel levels. This design allows the model to learn
higher-quality representations from earlier layers, resulting in a robust and
compact student model. Extensive quantitative evaluations reveal that HLFD
outperforms existing methods by a significant margin. For example, in the
kidney segmentation task, HLFD surpasses the student model (without KD) by
over 10%, significantly improving its focus on tumor-specific features. From a
qualitative standpoint, the student model trained using HLFD excels at
suppressing irrelevant information and can focus sharply on tumor-specific
details, which opens a new pathway for more efficient and accurate diagnostic
tools.

##  1 Introduction

Tumor segmentation in medical imaging enables clinicians to accurately
identify, assess, and manage malignancies. Leveraging neural networks, we
achieve automated, high-fidelity delineation of tumor boundaries in various
imaging modalities, including CT scans and MRIs [1, 2]. This technological
breakthrough elevates diagnostic accuracy and efficiency and streamlines
treatment planning [3], ultimately leading to enhanced patient care and
outcomes.

Significant challenges persist despite the remarkable successes of deep-
learning models in tumor segmentation. These models demand extensive datasets
and substantial computational resources, making deployment on resource-limited
devices a hurdle. Furthermore, the diversity in tumor appearances, irregular
sizes, unpredictable locations, and variations amplifies segmentation
complexity. To address these challenges, researchers are exploring innovative
strategies. For instance, lightweight networks [4, 5] have been explored for
real-time semantic segmentation, and recent works have delved into real-time
medical image segmentation. However, model simplification may hurt predictive
performance. Knowledge distillation (KD) [6] has emerged as a valuable
approach, facilitating knowledge transfer from the larger ‘teacher’ models to
the leaner ‘student’ models.

Existing works [7, 8, 9, 10, 11, 12] aim to enhance the final representations
of the student model by minimizing the difference in softmax representations
between the teacher and student models. However, this supervisory signal
originates solely from the final student layer. Hence, it tends to attenuate
with each layer during backpropagation, accumulating training bias within the
shallower student layers. This impairs the efficacy of knowledge transfer.
Other works [13, 14, 15, 16] focus on improving the alignment of latent
feature maps by mimicking intermediate representations. These intermediate
representations serve as solid indicators that facilitate learning the final
representation. However, when we replicate intermediate representations, we
are limited to capturing the knowledge acquired by that specific layer,
potentially missing out on global information. Recognizing this limitation,
capturing features from terminal representation at an earlier stage emerges as
a valuable strategy [17, 18]. However, these methods give suboptimal results
where boundary, shape, texture information, and a combination of low-level
features are essential, not only high-level class information.

Fig. 1: The input X and augmentation $X^{\prime}$, undergo encoding by both a
pre-trained teacher encoder and a randomly initialized student encoder,
resulting in representations $z^{t}_{early}$, $z^{t}_{\text{mid}j}$,
$z^{t}_{ter}$, and $z^{s}_{early}$, $z^{s}_{\text{mid}j}$, $z^{s}_{ter}$,
respectively. These representations contribute to feature-level loss
functions, $\mathcal{L}_{UFD}$ and $\mathcal{L}_{IFD}$. Additionally, the
teacher decoder decodes $z^{t}_{ter}$, producing representations
$p^{t}{early}$, $p^{t}_{\text{mid}j}$, $p^{t}_{ter}$, which are utilized in
pixel-level loss functions, $\mathcal{L}_{UPD}$ and $\mathcal{L}_{IPD}$. The
training process is further enhanced with the inclusion of a supervised focal
dice loss ($\mathcal{L}_{seg}$).

We have introduced hierarchical layer-selective feedback distillation (HLFD)
to address these challenges. HLFD comprises Feature-level LFD (FLFD) and
Pixel-level LFD (PLFD). FLFD, in turn, includes Unified Feature-level
Distillation (UFD) for unified representations and Individual Feature-level
Distillation (IFD) for middle-to-early and later-to-middle layer distillation.
PLFD, on the other hand, involves Unified Pixel-level Distillation (UPD) and
Individual Pixel-level Distillation (IPD), which transfers pixel-level
knowledge from the teacher decoder to the student through interpolated
features. HLFD integrates both FLFD and PLFD in a multi-task fashion,
promoting simultaneous learning of feature-level and pixel-level
representations. Our contributions are as follows:

  * •

We rethink the design of layers in the context of distillation and introduce
the Hierarchical Layer-selective Feedback Distillation (HLFD) framework.

  * •

We demonstrate HLFD’s capability to capture tumor-specific details from early
layers while effectively suppressing irrelevant information flow.

  * •

Extensive experiments conducted on kidney and liver tumor segmentation tasks
establish that our proposed method attains state-of-the-art (SOTA) results

##  2 Proposed Method

###  2.1 Feature-level Layer-selective Feedback Distillation (FLFD)

Given an input $X$, transformations occur through both the pre-trained teacher
encoder $f^{t}_{i}$ and the random student encoder $f^{s}_{i}$, denoted by i
for the number of blocks. This yields early representations $z^{t}_{early}$
and $z^{s}_{early}$, intermediate representations $z^{t}_{\text{mid}j}$ and
$z^{s}_{\text{mid}j}$ (where j is the number of middle layers), and terminal
representations $z^{t}_{late}$ and $z^{s}_{late}$. These representations form
the foundation of our framework. We propose an FLFD loss, defined as,
$\mathcal{L}_{F}=\mathcal{L}_{UFD}+\mathcal{L}_{IFD}$. These components are
defined below.

Unified Feature-level Distillation (UFD). Within this framework, we introduce
the concept of distilling the attentive knowledge from the teacher’s unified
representation of middle layers, $z^{t}_{mid}$, to the student’s early
representation, $z^{s}_{early}$. To achieve this, we propose the following
loss function.

$\displaystyle\mathcal{L}_{UFD}$
$\displaystyle=\frac{\|\mathcal{A}(z^{s}_{early})\|}{\|\mathcal{A}(z^{s}_{early})\|_{2}}-\frac{\|\mathcal{A}(z^{t}_{mid})\|}{\|\mathcal{A}(z^{t}_{mid})\|_{2}}$
(1)

To achieve $z^{t}_{mid}$, we perform interpolation on the middle layers with
the larger feature maps to ensure their spatial dimensions match the smallest
among them. Next, we concatenate all these interpolated representations along
the channel dimension. Finally, operation A(.) is employed first to rescale
the student’s representation $z^{s}_{early}$ to match the spatial dimension of
the teacher’s $z^{t}_{mid}$. Additionally, channel normalization is applied to
the rescaled student representation, assuming that the absolute value of a
neuron activation signifies its importance.

Individual Feature-level Distillation (IFD). Within this framework, we
introduce the concept of distilling the attentive knowledge from the teacher’s
late representation, $z^{t}_{late}$, to each student’s middle layers or
intermediate representation, $z^{s}_{\text{mid}_{j}}$. To achieve this, we
propose the following loss function.

$\displaystyle\mathcal{L}_{IFD}=\sum^{N}_{j=1}\frac{\|\mathcal{A}(z^{s}_{\text{mid}_{j}})\|}{\|\mathcal{A}(z^{s}_{\text{mid}_{j}})\|_{2}}-\frac{\|\mathcal{A}(z^{t}_{late})\|}{\|\mathcal{A}(z^{t}_{late})\|_{2}}$
(2)

Here, the operation $\mathcal{A(.)}$ is same as in Eq.1.

###  2.2 Pixel-level Layer-selective Feedback Distillation (PLFD):

In contrast to feature-level distillation, pixel-level segmentation-map
distillation is geared toward conveying pixel-wise predictions. In practice,
we distill pixel-level maps generated by the teacher’s decoder to interpolated
student maps. First, the teacher encoder output $z^{t}_{late}$ is passed
through pre-trained teacher decoder $d^{t}_{i}$ resulting in early predictive
map $p^{t}_{early}$, intermediate predictive map $p^{t}_{\text{mid}_{j}}$ and
terminal predictive map $p^{t}_{late}$. For students, we used an interpolated
representation map. We propose
$\mathcal{L}_{P}=\mathcal{L}_{UPD}+\mathcal{L}_{IPD}$. The components of PLFD
are as follows.

Unified Pixel-level Distillation (UPD). We propose distilling the precise
predictive information from the teacher’s unified pixel-wise predictive maps
of middle layers, $p^{t}_{mid}$, to the student’s early interpolated
representation, $p^{s}_{early}$.

$\displaystyle\mathcal{L}_{UPD}=\text{KL}(\mathcal{A}(p^{s}_{early})||p^{t}_{mid})$
(3)

Individual Pixel-level Distillation. Here, the teacher’s terminal predictive
map, denoted as $p^{t}_{late}$, distills precise information to the
intermediate predicted maps of the students individually, represented as
$p^{s}_{\text{mid}_{j}}$. This allows the student to capture detailed
knowledge about the exact pixel locations and their corresponding class
assignments within the image from much earlier layers. To achieve this, a KL-
divergence loss is employed between these maps:

$\displaystyle\mathcal{L}_{IPD}$
$\displaystyle=\frac{1}{N}\sum^{N}_{j}\text{KL}(\mathcal{A}(p^{s}_{\text{mid}_{j}})||p^{t}_{late})$
(4)

Here, $N$ is the number of middle-layer blocks in the student.

###  2.3 Hierarchical Layer-selective Feedback Distillation (HLFD)

Finally, distilling both feature-level and pixel-level representations allows
the student to learn fine-to-coarse hierarchical details at both the feature
and pixel levels. The multi-task loss function can be defined as:

$\displaystyle\mathcal{L}_{H}=\mathcal{L}_{Seg}+\beta*\mathcal{L}_{F}+\lambda*\mathcal{L}_{P}$
(5)

Where $\mathcal{L}_{seg}$ is the focal dice loss used for training the student
network in a supervised fashion. In the inference phase, post-sufficient
training, both the teacher network components and distillation modules are
discarded.

##  3 Experimental Platform

Datasets: We evaluated our techniques on kidney tumor segmentation (KiTS) [2]
and liver tumor segmentation (LiTS) [19] datasets. KiTS comprises 210
abdominal CT scans, where a 168:42 split is used for testing and training.
Similarly, the LiTS dataset consists of 201 CT scans and uses the split of
131:70.

Baselines: We compare our method with the following SOTA methods: i)
Structured Knowledge Distillation (SKD) [11]: Involves pair-wise distillation
to capture similarity at feature and pixel level. ii) Intermediate Feature
Distiller (IFD) [17]: Distills the teacher’s terminal representation into
concatenated branches of the student model. iii) Deep Knowledge Distillation
(DKD) [16]: Similar to [16] but without the Relational Knowledge
distillation(RKD) module. iv) Hierarchical Individual Feedback Knowledge
Distillation (HIFD) [18]: It distills the teacher’s terminal representation to
individual layers of the student. We extended this method for segmentation by
incorporating pixel-level feedback distillation loss functions. We maintained
identical implementation settings across all techniques.

Implementation Details: We employed UNet++ [20] architecture (36.1M
parameters) as the teacher network and ResNet18 [21] (11.6M parameters) as the
student network. Our segmentation networks and distillation processes,
inspired by [16], were trained using Adam optimizer with beta1 (0.9) and beta2
(0.999). The learning rate began at 0.001, utilizing CosineAnnealing for rate
scheduling, reaching a minimum of 0.000001. Data augmentation techniques such
as random rotation and flipping were applied, while experiments revealed that
Gaussian noise augmentation is unsuitable for medical images. Most networks
processed authentic $512\times 512$ CT images, requiring windowing of HU
values with radiological standards (e.g., -40 to 160 for the liver and -200 to
300 for the kidney). We use the PyTorch framework. We train all the networks
till convergence with up to 120 epochs. We report the result as $mean\pm std$
after three runs. For the Dice score (DSC), higher is better. For Relative
Volume Difference (RVD), smaller absolute values are desired, indicating a
closer match between the predicted and ground truth volumes. When comparing
RVD values, a smaller absolute value (closer to zero) is better, regardless of
whether the RVD is positive or negative. These metrics provide complementary
insights about the performance.

##  4 Results

Quantitative Results: As shown in Table 1, our method, HLFD, consistently
outshines both the supervised student and the baseline models. Notably, on the
KiTS dataset, we observe a substantial enhancement in DSC compared to the
student (without KD). Furthermore, both IFD and HIFD exhibit competitive or
superior outcomes than other baseline methods. These results underscore the
necessity of integrating KD and also emphasize the critical importance of
architecting layers that adeptly distill the ‘what’ and ‘where’ dimensions of
knowledge from the teacher model. On the RVD metric also, HLFD outperforms
baselines, including the student (without KD), by a significant margin. This
insight into volume differences holds valuable implications, especially in
tasks like tumor segmentation where accuracy in volume is of paramount
importance.

Table 1: Quantitative Results ($\beta=0.9$ and $\lambda=0.1$)  Method | KiTS | LiTS
---|---|---
DSC | RVD | DSC | RVD
Teacher |  64.50 $\pm$ 1.45 | -0.203 |  57.84 $\pm$ 1.55 | 1.434
Student(w/o KD) |  41.30 $\pm$ 2.30 | -0.421 |  41.19 $\pm$ 1.65 | 0.701
SKD [11] |  38.71 $\pm$ 2.53 | -0.411 |  42.09 $\pm$ 2.01 | 0.018
IFD [17] |  46.79 $\pm$ 1.25 | -0.275 |  45.36 $\pm$ 1.20 | 0.122
DKD [16] |  40.21 $\pm$ 2.35 | -0.496 |  43.72 $\pm$ 0.87 | 0.234
HIFD [18] |  42.50 $\pm$ 1.25 | -0.434 |  44.20 $\pm$ 1.22 | 0.187
HLFD(ours) |  52.18 $\pm$ 2.55 | -0.176 |  48.75 $\pm$ 2.23 | 0.173

On the LiTS dataset, HLFD (our method) consistently outperforms the baselines
on the DSC metric, whereas the SKD method is the best on the RVD metric.

Qualitative Results: The visualizations presented in Fig. 2 showcase the
superior performance of our proposed HLFD method. HLFD accurately segments the
Region of Interest (ROI) while effectively suppressing irrelevant information,
even at intermediate layers. The previous techniques fail completely in
recognizing the segmentation ROI at intermediate layers, especially for liver
segmentation. This underscores the importance of meticulously designing layers
to enhance the segmentation task’s representation quality.

Fig. 2: The green color highlights the regions of interest (ROI) representing
tumors. The segmentation maps are presented for both KiTS (first two rows) and
LiTS (last two rows) datasets, with ’G.T.’ denoting the ground truth.

The GradCAM maps presented in Fig. 3 showcase distinct patterns among methods.
SKD, which does not leverage intermediate layers, exhibits a flow of
irrelevant information, hindering focus on tumor-specific details. While DKD
shows some restriction of information, IFD and HIFD manage to suppress
irrelevant details. However, they face challenges in focusing on tumor-
specific information. In contrast, our method distinctly focuses on tumor-
specific information without capturing irrelevant details.

Fig. 3: Gradient-activated class maps for KiTS19, featuring CAM results for both the final and first layers. Remarkably, HLFD clearly focus on the tumor region, maintaining effectiveness in suppressing irrelevant information as the process advances to the final layer. Table 2: Impact of $\beta$ and $\lambda$ on DSC ${\beta}$ | ${\lambda}$ | KiTS | LiTS
---|---|---|---
0.9 | 0.1 |  52.18 $\pm$ 2.55 |  48.75 $\pm$ 2.23
1.8 | 0.1 |  51.58 $\pm$ 1.25 |  48.18 $\pm$ 1.63
0.9 | 0.2 |  51.95 $\pm$ 2.35 |  48.48 $\pm$ 2.05

Sensitivity Analysis: From Table 2, doubling the $\beta$ value (from 0.9 to
1.8) while maintaining $\lambda$ constant led to a slight deterioration in
performance. Conversely, increasing the value of $\lambda$ (from 0.1 to 0.2)
while keeping $\beta$ constant showed a similar trend but with slightly
improved performance compared to the previous case. This suggests that
$\mathcal{L}_{F}$ learns a rich representation of data, while
$\mathcal{L}_{P}$ learns the essential structure for the segmentation task.
Therefore, maintaining $\beta$ greater than $\lambda$ is crucial for optimal
results. The best performance was achieved with $\beta=0.9$ and $\lambda=0.1$.

##  5 Conclusion

We introduce a novel Knowledge Distillation (KD) framework for enhancing liver
and kidney tumor segmentation, redefining knowledge selection to distill and
the distillation source, and transitioning from teacher encoder layers to the
student. Quantitatively, HLFD has demonstrated remarkable superiority over
existing KD techniques and baseline models. Our method substantially improves
DSC, particularly in kidney tumor segmentation, where HLFD surpasses the
student model (without KD) by over 10%. Qualitatively, HLFD exhibits
exceptional capabilities in suppressing irrelevant information while
maintaining a sharp focus on tumor-specific details. The ability of HLFD to
accurately segment the region of interest (ROI) at both intermediate and final
layers showcases its effectiveness in enhancing the quality of segmentation
representations for kidney and liver tumor segmentation.

##  6 Compliance with ethical standards

This is a numerical simulation study for which no ethical approval was
required.

## References

  * [1] Abubaker Abdelrahman and Serestina Viriri,  “Kidney tumor semantic segmentation using deep learning: A survey of state-of-the-art,”  Journal of Imaging, vol. 8, no. 3, pp. 55, 2022.
  * [2] Nicholas Heller, Niranjan Sathianathen, Arveen Kalapara, Edward Walczak, Keenan Moore, Heather Kaluzniak, Joel Rosenberg, Paul Blake, Zachary Rengel, Makinna Oestreich, et al.,  “The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes,”  arXiv preprint arXiv:1904.00445, 2019.
  * [3] Philippe Meyer, Vincent Noblet, Christophe Mazzara, and Alex Lallement,  “Survey on deep learning for radiotherapy,”  Computers in biology and medicine, vol. 98, pp. 126–146, 2018.
  * [4] Dao-Hui Ge, Hong-Sheng Li, Liang Zhang, R Liu, P Shen, and Qi-Guang Miao,  “Survey of lightweight neural network,”  J. Softw, vol. 31, pp. 2627–2653, 2020.
  * [5] Taha Emara, Hossam E Abd El Munim, and Hazem M Abbas,  “Liteseg: A novel lightweight convnet for semantic segmentation,”  in 2019 Digital Image Computing: Techniques and Applications (DICTA), 2019, pp. 1–7.
  * [6] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao,  “Knowledge distillation: A survey,”  International Journal of Computer Vision, vol. 129, pp. 1789–1819, 2021.
  * [7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean,  “Distilling the knowledge in a neural network,”  arXiv preprint arXiv:1503.02531, 2015.
  * [8] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho,  “Relational knowledge distillation,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3967–3976.
  * [9] Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos,  “Knowledge distillation via softmax regression representation learning,”  in International Conference on Learning Representations, 2020.
  * [10] Yonglong Tian, Dilip Krishnan, and Phillip Isola,  “Contrastive representation distillation,”  arXiv preprint arXiv:1910.10699, 2019.
  * [11] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang,  “Structured knowledge distillation for semantic segmentation,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2604–2613.
  * [12] Mingi Ji, Byeongho Heo, and Sungrae Park,  “Show, attend and distill: Knowledge distillation via attention-based feature matching,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, vol. 35, pp. 7945–7952.
  * [13] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio,  “Fitnets: Hints for thin deep nets,”  arXiv preprint arXiv:1412.6550, 2014.
  * [14] Sergey Zagoruyko and Nikos Komodakis,  “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,”  arXiv preprint arXiv:1612.03928, 2016.
  * [15] Nikolaos Passalis and Anastasios Tefas,  “Learning deep representations with probabilistic knowledge transfer,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 268–284.
  * [16] Dian Qin, Jia-Jun Bu, Zhe Liu, Xin Shen, Sheng Zhou, Jing-Jun Gu, Zhi-Hua Wang, Lei Wu, and Hui-Fen Dai,  “Efficient medical image segmentation based on knowledge distillation,”  IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3820–3831, 2021.
  * [17] Jeongho Kim, Hanbeen Lee, and Simon S. Woo,  “Imf: Integrating matched features using attentive logit in knowledge distillation,”  in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence IJCAI, 2023.
  * [18] Shiya Luo, Defang Chen, and Can Wang,  “Knowledge distillation with deep supervision,”  in Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), 2023, pp. 1–8.
  * [19] Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al.,  “The liver tumor segmentation benchmark (lits),”  Medical Image Analysis, vol. 84, pp. 102680, 2023.
  * [20] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang,  “Unet++: A nested u-net architecture for medical image segmentation,”  in Proceedings of the international conference on Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, 2018, pp. 3–11.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.


Chest radiography (X-ray) is a modality that is routinely used for diagnostic procedures around the world [1].
It became the most common medical imaging examination for pulmonary diseases and allows a clear investigation of the thorax [2]. Chest X-ray imaging is therefore well-suited for diagnosing several pathologies including pulmonary nodules, masses, pleural effusions, pneumonia, COPD, and cardiac abnormalities [3]. It is also used for COVID-19 [4] screening, as abnormalities typical of those infected with the coronavirus can be detected in radiographs [5]. While chest radiography plays a crucial role in clinical care, discovering certain diseases and abnormalities in chest radiographs can be a challenging task for radiologists, which potentially results in undesirable misdiagnoses [6]. Therefore, computer-aided detection (CAD) systems based on DL [7] techniques have been developed in recent years to facilitate radiology workflows. These systems, characterized by their enormous benefits, can be utilized for a wide range of applications, e. g., for the automatic recognition of abnormalities in chest radiographs [3, 8] and the detection of tumors in mammography [9]. Some techniques even show the potential to exceed human performance [10]. However, the CAD systems are only treated as an additional source to support the radiologists and to increase certainty in their reading decisions.
On the one hand, the large variety of medical applications allows DL to grow and tackle real-life problems that were previously not solvable or improving solutions offered by traditional machine learning methods [7]. On the other hand, DL is a data-driven approach and well-known for its need for big data to train the neural networks [11, 12]. For these reasons, a vast amount of medical datasets have been published in recent years that enable researchers to develop diagnostic algorithms in the medical field in a reproducible way [13]. These include several large-scale chest radiography datasets, e. g., the CheXpert [14], the PLCO [15] and the ChestX-ray14 [16] datasets. But especially during the COVID-19 pandemic [17, 18], the number of publicly available chest radiography datasets increased rapidly. A few selected examples are the COVID-19 Image Data Collection [19], the Figure 1 COVID-19 Chest X-ray Dataset Initiative [20], the ActualMed COVID-19 Chest X-ray Dataset Initiative [21], and the COVID-19 Radiography Database [22].
Chest radiography datasets typically consist of two parts: First, the image data itself, which provides clinical information about the anatomical structure of the thorax. Second, the associated metadata, which contains sensitive patient-related information that is either stored in a separate file or embedded directly in the images [23].
Proper data anonymization constitutes an important step when preparing medical data for public usage to ensure that a patient's identity cannot be revealed in publicly available datasets [23].
In practice, any personally identifiable information is attempted to be removed from the data before it is shared.
These objectives and requirements are specified, e. g., by the Health Insurance Portability and Accountability Act (HIPAA) [24] in the United States or the General Data Protection Regulation (GDPR) [25] in Europe.
In 2017, Google entered into a project with the NIH to publish a dataset containing 100,000 chest radiographs. However, the release was canceled two days before publication after Google was informed by the NIH that the radiographs still contained personal information which indicates that the data was incorrectly anonymized [26, 27]. This major incident highlights that many potential pitfalls can arise when clinical and technological institutions collect and share large medical datasets to revolutionize health-care.
In the past, various data de-identification techniques have been proposed, including commonly-used methods like pseudonymization [28] and $k$-anonymity [29]. Pseudonymization describes a technique that replaces a true identifier, e. g., the name or the patient identification number by a pseudonym that is unique to the patient but has no relation to the person [28]. However, pseudonymization is a rather weak anonymization technique as the patient's identity may still be revealed, e. g., by cross-referencing with other publicly available datasets. In contrast, $k$-anonymity modifies the data before sharing in such a way that every sample in the published dataset can be associated with at least $k$ different subjects. In this way, the probability of performing identity disclosure is limited to at most $\nicefrac{1}{k}$ [29, 30]. Nevertheless, when background knowledge is available, $k$-anonymity is susceptible to many attacks.

General problem scenario: Comparing a given chest radiograph to publicly available dataset images by means of DL techniques would either result in discrete labels indicating whether or not the dataset images belong to the same patient as the given radiograph (verification scenario) or yield a ranked list of the most similar radiographs related to the given scan (retrieval scenario). Images belonging to the same patient are highlighted with the same color. The given radiograph is marked with an asterisk. The shown cases would enable a potential attacker to link sensitive patient-related information contained in the dataset to the image of interest.

To date, little attention has been paid to the possibility of re-identifying patients in large medical datasets by means of DL techniques. However in theory, medical data disclosure, as illustrated in <ref>, could be facilitated for potential attackers by using suitable DL approaches. Consider a publicly available dataset that is supposedly anonymized but contains further sensitive patient-related information, e. g., diagnosis, treatment history, and clinical institution. If a radiograph of known identity is accessible to a potential attacker and a properly working verification or re-identification model exists, then the model could be used to compare the given radiograph to each image in the dataset which would essentially result in a set of images belonging to the same patient (patient verification) or yield a ranked list of the most similar images to the given radiograph (patient re-identification). In this way, the patient's identity may be linked to sensitive data contained in the dataset. As a result, more patient-related information may have been leaked, highlighting the enormous data security and data privacy issues involved.
In our work, we investigated whether conventional anonymization techniques are secure enough and whether it is possible to re-identify and de-anonymize individuals from their medical data using DL-based methods. Therefore, we considered the public ChestX-ray14 dataset [16], which is one of the most widely used research datasets in the field of radiographic problems. Our algorithms are trained to determine whether two arbitrary chest radiographs can be recognized to belong to the same patient or not. Moreover, we showed that our proposed methods are able to perform a successful linkage attack on publicly available chest radiography datasets. Furthermore, this work aims to draw attention to the massive problem of releasing medical data without considering that DL systems can easily be used to reveal a patient's identity. Therefore, we call for reconsidering conventional anonymization techniques and developing more secure methods that resist potential attacks by DL algorithms.

§ PATIENT VERIFICATION

Overview of the obtained verification results for our experiments using varying training set sizes $N_s$ at different learning rates $\eta$. Moreover, different data handling techniques were used (FTS: Fixed training set; RNP: Randomized negative pairs). For each experiment, the training sets were balanced with respect to the amount of positive and negative image pairs. In this table, we present the AUC (together with the lower and upper bounds of the 95 confidence intervals from 10,000 bootstrap runs), the accuracy, the specificity, the recall, the precision, and the F1-score. Bold text emphasizes the overall highest AUC value.

First, we trained a SNN architecture on the ChestX-ray14 dataset to determine whether two individual chest radiographs correspond to the same patient or not. Our model was designed to process the two input images in two identical network branches, which are then combined by a merging layer. The fused information is fed through further network layers resulting in a single output score indicating the identity similarity.
<Ref> summarizes the outcomes of our evaluation. We analyzed a multitude of different experimental setups with varying LRs $\eta$ and differing balanced training set sizes $N_s$. Moreover, we investigated the effect of using epoch-wise randomized negative pairs (RNP) versus fixed training sets (FTS) for the entire learning procedure. When using RNP as the data handling technique, the negative image pairs were randomly constructed in each epoch, meaning that much more negative pairs could be utilized in a complete training run compared to FTS where the generated image pairs remain the same for the entire learning procedure. For all experiments on the ChestX-ray14 dataset, we used the same balanced validation and testing set with 50,000 and 100,000 image pairs, respectively, without patient overlap between any split. To assess the performance of the trained models, we performed a ROC analysis by computing the AUC value together with the 95 confidence intervals from 10,000 bootstrap runs. Moreover, we calculated the accuracy, specificity, recall, precision, and F1-score.

figureROC curves for different training set sizes $N_s$ and a fixed LR of $\eta=10^{-4}$. During training, the fixed data handling technique was employed.

figureConfusion matrix corresponding to the best experiment shown in <ref> (last row) giving clear insights into the performance of our trained model.

The results indicate that the amount of training data plays a crucial role in the patient verification task. We observe a significant performance increase as the training set size grows. For instance, when using a subset of 100,000 image pairs for training, we obtain an AUC value of 0.8610. In contrast, by enlarging the training set size to 800,000 image pairs (i. e. the total number of 400,000 positive image pairs combined with 400,000 negative pairs), we receive an AUC score of 0.9896. These findings have been visualized in the ROC curves shown in <ref> which illustrates the effect of the training set size on the verification performance when using fixed training sets. Note that <ref> only shows the best experiments per training set size $N_s$. Additional experiments were conducted to investigate the effect of the LR. The corresponding results are provided in a separate table in the appendix (see Supplementary Table 1).

We also observed that randomly constructing the negative image pairs in each epoch led to further improvements in the final model performance. By using this data handling technique, we achieved our overall best results. The respective outcomes are reported in <ref>. When training our network architecture with a total of 800,000 training samples with epoch-wise randomly constructed negative pairs, the AUC score improved from 0.9896 to 0.9940. Besides, the other reported evaluation metrics apart from the recall also increased compared to the results achieved by the model trained with the fixed set. <Ref> depicts the confusion matrix resulting from our best-trained model listed in <ref> (last row), thus giving clear insights into the patient verification performance.

TPR for image pairs with fig:age age differences, fig:abnorm with changes in the disease pattern, and fig:view with changes in the projection view. The absolute numbers of true positives and overall positives are given for each bin. Note that the number of image pairs with age differences of more than 12 years is comparatively small, which is why the corresponding TPRs are neglected in this figure.

We also analyzed how the model with the best recall (fourth row in <ref>) behaves when comparing images of the same patient where the acquisition dates are several years apart. The results are illustrated in <ref>. We received a TPR of 0.97 for image pairs that had small age differences of less than one year. As the age variation between the follow-up images and the initial scan increases, we observe a slight decrease in the TPR values. Nevertheless, our model still shows competitive results even if the patient's age in two images differs by several years.
Even for an age difference of twelve years, we can verify that two images belong to the same patient by 86. We only report the TPR for image pairs with follow-up intervals of up to 12 years in <ref> as the number of pairs with larger intervals is relatively small.

Exemplary image pairs are classified by our best performing verification model. Each column represents one image pair. The first four columns fig:FP1–fig:TP3 show true positive classifications. The last two columns fig:TP4 and fig:TP2 depict a false positive and a false negative classification, respectively.

Additionally, we investigated the model's verification capability in the case of new abnormality patterns appearing in follow-up scans that did not occur in previously acquired chest radiographs.
<Ref> shows that regardless of the abnormality, we nearly observe no decline in the TPR values, emphasizing the robustness of our trained SNN architecture. Note that the disease labels in the ChestX-ray14 dataset were extracted using natural language processing techniques. This could potentially have caused label noise which could have affected the results shown in <Ref>. Furthermore, <Ref> illustrates that changes in the projection view (e. g., one image taken in the anterior-posterior position and the other image acquired using the posterior-anterior view) hardly lead to any deterioration in the performance.

Comparison of the verification performance on two different subsets of the ChestX-ray14 dataset that either contain foreign material or not (first two rows). Furthermore, we show the verification results for the CheXpert dataset and the COVID-19 Image Data Collection (last two rows). We present the AUC (together with the lower and upper bounds of the 95 confidence intervals from 10,000 bootstrap runs), the accuracy, the specificity, the recall, the precision, and the F1-score.

Moreover, we perform a qualitative evaluation where we visually inspect some exemplary image pairs evaluated using our best-performing verification model. In <ref>, we show four TP classifications fig:FP1–fig:TP3, one pair that has been classified as a FP fig:TP4, and one example for a FN image pair fig:TP2. The shown images clearly illustrate the high technical variance present in the ChestX-ray14 dataset. The first image pair fig:FP1 shows two images belonging to the same patient with a difference of seven years. Clear differences in pixel intensities and lung shape are observed.
However, both images belong to the same person, cf. the small vascular clips in the area of the upper right lung.
Also, image pairs with large difference in scaling fig:FN1 or rotation fig:TP1 are verified correctly.
Our model is also robust to the patients' pathology: While the upper image of fig:FN1 shows characteristics of pneumothorax, the patient suffered from cardiomegaly, effusion, and masses in the lower image, according to the provided annotations. Similarly in fig:TP1, where the upper image indicates the presence of infiltration and pneumothorax, whereas the lower scan shows signs of infiltration and nodules.
<Ref> shows an exemplary image pair that has falsely been classified as positive. Conversely, fig:TP2 depicts a positive image pair that has been incorrectly classified as negative. To visually demonstrate which parts of the images are responsible for the verification task, we applied a siamese attention mechanism [31] to our network architecture which utilizes the Grad-CAM algorithm [32]. The obtained attention maps can be seen in Figures <ref> and <ref>. They clearly indicate that the human anatomy, especially the shape of the lungs and ribs, is the driving factor for the network decisions.

Grad-CAM visualizations for the first convolutional layer of the ResNet-50 incorporated in our SNN. Each column represents one image pair. The first four columns fig:FP1_map1–fig:TP3_map1 show true positive classifications. The last two columns fig:TP4_map1 and fig:TP2_map1 depict a false positive and a false negative classification, respectively. The shown images illustrate that the anatomical structure of, e. g. the breast (cf. fig:FP1_map1, fig:FN1_map1, fig:TP4_map1), the lungs (cf. fig:FP1_map1, fig:FN1_map1, fig:TP1_map1, fig:TP4_map1), and the heart (cf. fig:FP1_map1, fig:FN1_map1) have a high impact on the final model prediction. Furthermore, it can be seen that our network focuses on the collarbones (cf. fig:FP1_map1, fig:TP1_map1, fig:TP3_map1, fig:TP2_map1) and the ribs (cf. fig:FN1_map1, fig:TP1_map1, fig:TP4_map1). The upper images of fig:FP1_map1 and fig:TP2_map1 also highlight that our network pays attention to the contour of the diaphragm.

To investigate how foreign material (see <ref>) affects the verification performance, we evaluated our trained network on two small manually created subsets of around 200 images. The first one consisted only of images in which foreign material is visible, whereas the second one solely contained images without foreign material. When constructing the subsets, we selected the patients at random and then assigned the corresponding patient images to the respective subset after visual assessment. Furthermore, we ensured that no more than 5 images were used per patient. <Ref> summarizes the results indicating that the patient verification works with high performance regardless of the occurrence of foreign material. We even observe a slight improvement in performance for the subset where no foreign material is visible in the images.

Grad-CAM visualizations for an intermediate convolutional layer of the ResNet-50 incorporated in our SNN. Each column represents one image pair. The first four columns fig:FP1_map3–fig:TP3_map3 show true positive classifications. The last two columns fig:TP4_map3 and fig:TP2_map3 depict a false positive and a false negative classification, respectively. The obtained attention maps clearly illustrate that the selected network layer focuses on the ribs and the outline of the thorax.

Finally, to analyze whether our trained model is able to generalize to other datasets which have not been used during training, we evaluated our network on external datasets such as CheXpert and the COVID-19 Image Data Collection. For this, we utilized 16,486 image pairs from the CheXpert dataset and 1,548 image pairs from the COVID-19 Image Data Collection. The results are summarized in the last two rows of <ref>. It can be seen that our network still yields high AUC values of 0.9870 (CheXpert) and 0.9763 (COVID-19) for the verification task although the model has not been fine-tuned on the respective datasets. Also the other presented evaluation metrics show competitive values without deteriorating too much.

§ PATIENT RE-IDENTIFICATION

For our patient re-identification experiments, we trained another SNN architecture on the ChestX-ray14 dataset. In contrast to the verification model, we omitted all the layers from the merging layer onwards.
The main objective was to learn appropriate feature representations instead of directly determining whether the inputs belong to the same patient or not.
After training the network, we used the ResNet-50 backbone as a feature extractor for the actual image retrieval task. By computing the Euclidean distance between the embeddings of the query image and each other image, we obtained for each query image a ranked list of its most similar images in terms of identity. The used training, validation, and testing set consisted of 61,755, 10,815, and 25,596 images, respectively.

Overview of the obtained results for our image retrieval experiments. In this table, we report the $mAP@R$, the $R$-Precision, and the Precision@1. The first 4 rows show the results on the ChestX-ray14 dataset for different image resolutions used for evaluation. The fifth row shows the outcomes on the CheXpert dataset.
The last row indicates the results on the COVID-19 Image Data Collection. Bold text represents the overall highest performance metrics.

Comparison of the re-identification performance on two different subsets (ChestX-ray14) that either contain foreign material or not. We report the $mAP@R$, the $R$-Precision, and the Precision@1.

The results of the corresponding image retrieval experiments are summarized in <ref>. When using the original image size of 1024$\times$1024 pixels for evaluation, we obtain a precision@1 of more than 99 showing that the closest match nearly always is the same patient. The high mean average precision at $R$ (mAP@$R$) of about 97 further depicts that most of the most similar images are correctly identified. We observe a slight decrease in performance as the image size was reduced. Nevertheless, when the images were downsampled to a resolution of 512$\times$512 pixels, we still obtain high performance values. When the image size was reduced too aggressively, e. g., to 224$\times$224 pixels, the $mAP@R$ and the $R$-Precision rates drop. Yet, we still observed a high Precision@1 of more than 97.
Similar to the experiments in the patient verification section, we evaluated our best-trained re-identification model on two small subsets, one of which only contained images with visible foreign material and the other consisted exclusively of images without the presence of foreign material. The obtained results are presented in <ref>. As can be seen, we achieve high performance values for both subsets. Thus, we hypothesize that our outcomes are independent of foreign material which may occur only for specific patients.
Lastly, we analyzed the re-identification performance on external datasets such as CheXpert and the COVID-19 Image Data Collection. For this, we utilized 6,454 images from the CheXpert dataset and 781 images from the COVID-19 dataset. As can be seen in the last two rows of <ref>, we also  obtain high retrieval values although we haven't performed any fine-tuning on both datasets, which demonstrates the feasibility of the trained re-identification network on previously unseen datasets.

§ DISCUSSION AND CONCLUSION

In this paper, we investigated the patient verification and re-identification capabilities of DL techniques on chest radiographs. We have shown that well-trained SNN architectures are able to compare two individual frontal chest radiographs and reliably predict whether these images belong to the same patient or not. Moreover, we have shown that DL models have the potential to accurately retrieve relevant images in a ranked list. Our models have been evaluated on the publicly available ChestX-ray14 dataset and showed competitive results with an AUC of up to 0.9940 and classification accuracy of more than 95 % in the verification scenario and an $mAP@R$ of 97 and a precision@1 of about 99 in the image retrieval scenario. Especially the fact that basic SNN have the capability to re-identify patients despite potential age differences, disease changes or differing projection views demonstrated the effectiveness of DL techniques for this task. However, note that the shown results were obtained empirically, i. e. they do not necessarily reflect true measures of certainty.

As shown in <ref>, the used dataset suffers from a high technical variance which may occur due to various windowing techniques applied to the images. In a real-life scenario, the resulting variations in image contrast and brightness could be significantly mitigated by using dynamic normalization approaches [33]. Furthermore, we believe that variations in rotation and scaling can be counteracted by appropriate alignment algorithms. Nevertheless, even without such pre-processing steps, we were able to show that patient matching for chest radiographs is possible with a high performance by using DL techniques.

Moreover, we hypothesize that special noise patterns characteristic for unique patients appear in the images which might unintentionally improve the re-identification performance. For example, the initial anonymization strategy may be biased towards the clinical institution and, therefore, also towards follow-up images. To get a better impression of the re-identification capability of our SNN architecture, we also intend to investigate other datasets which show less or ideally no correlation between potential noise patterns and the patient identity. Therefore, further research on multiple datasets should ideally be considered. For our experiments, we already evaluated our models on two completely different datasets, the CheXpert dataset and the COVID-19 Image Data Collection. While the evaluation metrics are lower (possibly due to a domain shift or the severity of diseases), we still obtain AUC scores of over 97 (COVID-19) and 98 (CheXpert) and precision@1 values of more than 88 (COVID-19) and 94 (CheXpert) without fine-tuning on these datasets. This indicates that patient verification and re-identification is also applicable for data that was acquired in various hospitals around the world where other pre-processing steps may be taken before data publication compared to the ChestX-ray14 dataset.

The COVID-19 Image Data Collection is very heterogeneous, containing, e. g., images of different sizes, both gray-scale and color images, and images with visible markers, arrows or date displays. For our experiments, only those images in the COVID-19 Image Data Collection were used that were acquired using the anterior-posterior or the posterior-anterior view, while images taken in the lateral position and CT scans were discarded. Apart from this, no further steps were taken to ensure the quality of the dataset. Although some of the factors mentioned above (e. g., brandings such as markers, arrows or dates) may facilitate the patient re-identification, we hypothesize that the COVID-19 Image Data Collection poses a realistic example of a public medical dataset and we therefore consider the conducted experiment as an authentic real-life application scenario.

Furthermore, we want to accentuate that our trained network architectures are able to handle non-rigid transformations that may appear between two images of the same person in the ChestX-ray14 dataset. Such deformations can occur due to various breath states in follow-up scans or due to different positioning during X-ray acquisition. Hence, the shape of the heart and lungs, or the contours of the ribs may appear deformed compared to an initial scan. The obtained results lead to the assumption that our trained SNN architectures can withstand such deformations and can therefore be used for reliable patient re-identification on chest radiographs.

We conclude that DL techniques render medical chest radiographs biometric for everyone and allow a re-identification similar to a fingerprint. Therefore, publicly available medical chest X-ray data is not entirely anonymous. Using a DL-based re-identification network enables an attacker to compare a given radiograph with public datasets and to associate accessible metadata with the image of interest. The strength of our proposed method is that patients can be re-identified in a fully automated way without the need of expert knowledge. Thus, sensitive patient data is exposed to a high risk of falling into the unauthorized hands of an attacker who may disseminate the gained information against the will of the concerned patient. At this point, we want to emphasize that data leakage of this kind requires that the attacker has previously gained access to an image of a known person. This could happen, for example, through a stolen CD containing raw medical data of a specific patient, or through accidental data release by a radiological facility. Furthermore, data breaches due to inadequate data security measures at, e. g., healthcare institutions or health insurance companies, represent a possibility for attackers to obtain images of known patients, which could subsequently be utilized for a linkage attack as presented in our work. However, even if the attacker owns an image of an unknown identity, a re-identification model can be used to find the same patient across various datasets. Assuming multiple datasets contain the same patient but different metadata, an attacker would be able to obtain a more complete picture of the respective patient. We hypothesize that collecting patient information by this means could significantly help an attacker infer the true identity of the patient. We therefore urge that conventional anonymization techniques be reconsidered and that more secure methods be developed to resist the potential attacks by DL-based algorithms.

At this point, we would like to draw attention to the analogy of our work to the field of ASV. Speech signals contain a large amount of private data, e. g., age, gender, health and emotional state, ethnic origin, and more [34]. As such information is embedded in speech data itself, it can be exploited to reveal the speaker's identity by applying attack models in the form of ASV systems [34]. Therefore, in the speech community, raw speech signals are not considered anonymous. Instead, privacy challenges, such as the VoicePrivacy Challenge [35], were formed to develop solutions for the preservation of privacy in this field. With our work, we were able to demonstrate that the privacy issue with speech data is 1-to-1 transferable to the privacy issue with chest radiographs.

While the proposed algorithms may be used maliciously to produce harm in terms of patient privacy and data anonymity, we also want to draw attention to a potentially positive application area. Often, different datasets are used for training and evaluation of an algorithm. However, since in most cases these datasets have been anonymized using conventional techniques, it is not clear whether certain patients appear in more than one dataset. Therefore, in this context, our trained networks could be applied to check for mutual exclusiveness with respect to included patients between multiple datasets.

The publication of medical image datasets is an area of conflict. While, on the one hand, many patients may benefit from recent advances (e. g., the development of diagnostic algorithms), there are, on the other hand, patients who may be seriously harmed by the fact that their data is publicly available. With our work, we focus on providing empirical evidence for this issue and draw attention to the risks. The legal situation for the publication of medical data is currently regulated by the HIPAA (in the United States) and the GDPR (in Europe). We therefore contend that the corresponding ethics committees are responsible for weighing the benefits and the risks as well as for assessing the appropriateness of current regulations.

Potential solutions to the problems addressed in our work may be found in privacy-preserving approaches such as DP [36, 37] which guarantees that the global statistical distribution of a dataset is retained while individually recognizable information is reduced [38]. This means that an outside observer is unable to draw any conclusions about the presence or absence of a particular individual. Consequently, algorithms trained with DP are able to withstand linkage attacks attempting to reveal the identities of patients in the dataset used to train the algorithm. One commonly-used technique to achieve DP is to modify the input by adding noise to the dataset (local DP) [38, 39]. Furthermore, DP can be applied to the computation results (global DP) or to algorithm updates [38]. However, training models with DP degrades the quality of the model (privacy-utility trade-off) which is problematic in medicine where high diagnostic utility is required. Therefore, further exploration on these topics is necessary before general conclusions can be made.

Aside from perturbation-based privacy approaches, we want to mention that the use of collaborative decentralized learning protocols such as FL [40] can significantly contribute to a safer use of medical data. By training a machine learning model collaboratively without centralizing the data, the need of raw data sharing or dataset release is eliminated [41]. Thus, the medical data is able to reside with its owner, e. g., the healthcare institution where the data was acquired, which resolves data governance and ownership issues [38, 42]. However, FL itself does not provide full data security and privacy, meaning that some risks remain unless combined with other privacy-preserving methods.

§ METHODS

The research was carried out in accordance with the relevant guidelines and regulations of the institution conducting the experiments.

§.§ Siamese neural networks

To re-identify patients from their chest radiographs, we employ SNN architectures for both the classification and the retrieval tasks. A SNN receives two input images which are processed by two identical feature extraction blocks sharing the same set of network parameters. The resulting feature representations can then be used to compare  the inputs. The concept of a SNN was initially introduced by Bromley et al. [43] for handwritten signature verification. Taigman et al. [44] applied this idea in the field of face verification and proposed the DeepFace system. Moreover, Koch et al. [45] presented an approach for one-shot learning on the Omniglot [46] and MNIST [47] datasets.

§.§ NIH ChestX-ray14 dataset

With a total of 112,120 frontal-view chest radiographs from 30,805 unique patients, the NIH ChestX-ray14 [16] dataset counts to one of the largest publicly available chest radiography datasets in the scientific community. Due to follow-up scans, the image collection provides an average of 3-4 images per patient. The originally acquired radiographs were published as 8-bit gray-scale PNG images with a size of 1024$\times$1024 pixels. Associated metadata is available for all images in the dataset. The additional data comprises information about the underlying disease patterns (either no finding or a combination of up to 14 common thoracic pathologies), the number of follow-up images taken, the patients' age and gender, and the projection view (anterior-posterior or posterior-anterior) used for radiography acquisition. According to the publisher, the dataset was carefully screened to remove all personally identifiable information before release [48]. Therefore, the patient names were replaced by integer IDs. Moreover, personal data in the image domain itself has been made unrecognizable by placing black boxes over the corresponding image areas.

§.§ CheXpert dataset

The CheXpert [14] dataset contains 224,316 frontal and lateral chest radiographs of 65,240 patients, who underwent a radiographic examination from Standford University Medical Center between October 2002 and July 2017. The originally acquired radiographs were published as 8-bit gray-scale JPG images with varying image resolutions. Note that only frontal chest radiographs were used in our work, whereas lateral images were excluded.

§.§ COVID-19 Image Data Collection

The COVID-19 Image Data Collection [19] is a dataset that was created and published as an initiative to provide COVID-19 related chest radiographs and CT scans for machine learning tasks. It comprises data of 448 unique patients and a total of around 950 images with different image resolutions. In this work, only the available frontal radiographs were utilized, whereas the lateral images and CT scans were discarded.

§.§ Data preparation

Since SNN architectures require pairs of images for training and evaluation, we construct both positive and negative image pairs from the images contained in the ChestX-ray14 dataset. In this context, a positive pair consists of two images belonging to the same patient, whereas a negative pair comprises two images that belong to different patients. Mathematically, the constructed dataset can be described according to

\begin{equation}
\mathcal{S} = \{(\bm{x}_{11}, \bm{x}_{12}, y_1), ..., (\bm{x}_{m1}, \bm{x}_{m2}, y_m), ..., (\bm{x}_{M1}, \bm{x}_{M2}, y_M) \}\enspace,\enspace \text{with } y_m \in \{0,1\}\enspace,
\end{equation}

where the triplet $(\bm{x}_{m1}, \bm{x}_{m2}, y_m)$ represents one sample consisting of two images $\bm{x}_{m1}$ and $\bm{x}_{m2}$, and the corresponding label $y_m$. $M$ refers to the total number of samples and $m$ denotes an iterator variable in the range of $m \in [1,M]$. The class label $y_m$ symbolizes a binary variable that takes the value 0 for negative image pairs and 1 for positive image pairs.

To ensure that images from one patient only appear either in the training, validation, or testing set, we use the patient-wise splitting strategy. According to the official split provided with the ChestX-ray14 dataset, the data is randomly divided into 70 % training, 10 % validation, and 20 % testing. Based on this split, we construct the actual image pairs for each subset.

§.§.§ Offline mining

For patient verification, we follow an offline mining approach, meaning that the positive and negative image pairs are generated once before conducting the experiments. First, the positive pairs are generated by only considering the patients for whom multiple images exist in the respective subset. For each patient with follow-up images, we produce all possible tuple combinations assuming the images to be unique. By following this approach, we are able to construct a total of around 400,000 positive image pairs for our training set. The negative pairs in each subset are randomly generated and concatenated with the respective positive pairs afterwards.

§.§.§ Online mining

For the patient re-identification experiments, we choose an online mining approach, meaning that image pairs are formed in each batch during the training procedure. This means that the embeddings of all batch images are first computed and then subsequently used in all possible combinations as input for the loss function. Moreover, all patients with only one available image were discarded from the training set.

§.§ Patient verification

§.§.§ Deep learning architecture

For patient verification, the used SNN architecture (see <ref>) receives two images $\bm{x}_1$ and $\bm{x}_2$ of size 3$\times$256$\times$256. Both inputs are processed by a pre-trained ResNet-50 incorporated in each network branch. In its original version, the ResNet-50 was designed to classify images into 1,000 object categories trained on the ImageNet [49] dataset. To adapt the ResNet-50 to our specific needs, we replace its classification layer with a layer consisting of 128 output neurons producing the feature representations $\bm{z}_1$ and $\bm{z}_2$, respectively. To merge both network branches, the absolute difference of the sigmoid activations of the two feature vectors is computed. We add a FC layer to reduce the dimensionality to one neuron, followed by another sigmoid activation function $\sigma$ which yields the final output score $\hat{y} \in [0,1]$.

SNN architecture used for patient verification on the ChestX-ray14 [16] dataset. The feature extraction blocks (light blue) share the same set of network parameters and produce the feature representations $\bm{z}_1$ and $\bm{z}_2$ (yellow). After merging (orange) and an additional FC and sigmoid layer $\sigma$ (light blue), the network yields the final output score $\hat{y}$ (green). For our patient re-identification experiments, we used the same architecture but ejected all the layers from the merging layer onwards.

§.§.§ Training strategy

The verification model is trained using the BCE loss. The network parameters are optimized by combining mini-batch SGD [50, 51] with the Adam [52] method. The batch size $N_b$ is set to 32 in all our experiments. We use different LR to investigate their effect on the model's performance. Furthermore, we include an early stopping criterion with a patience $p=5$, which means that the training procedure stops as soon as the network does not improve for 5 epochs. We train the architecture using input dimensions of 3$\times$256$\times$256.

§.§.§ Evaluation techniques

We utilize ROC curves to visualize the trained verification models based on their performance. A ROC curve represents a two-dimensional graph in which the TPR is plotted against the FPR at various threshold settings [53], thus indicating how many true positive classifications can be gained as an increasing number of false positive classifications is allowed. Additionally, we calculate the AUC which reflects a proportion of the area of the unit square and will always range from 0 to 1 [53]. The higher the AUC score, the better the model's average performance. Nevertheless, it has to be mentioned that a classifier with a high AUC might perform worse in a specific region of ROC space than a classifier with a low AUC value. Moreover, we evaluate the performance by computing the accuracy, specificity, recall, precision and F1-score. Therefore, the threshold at the output neuron is set to $t=0.5$.

§.§ Patient re-identification

§.§.§ Deep learning architecture

For patient re-identification, we train a SNN architecture which receives two images $\bm{x}_1$ and $\bm{x}_2$ of size 3$\times$1024$\times$1024. Both inputs are processed by a pre-trained ResNet-50 incorporated in each network branch. However, the network head of the used ResNet-50 is slightly modified. The average pooling layer is replaced by an adaptive average pooling layer producing feature maps of size 5$\times$5. In addition to the adaptive average pooling layer, an adaptive max-pooling layer is applied which also yields feature maps of size 5$\times$5. The outputs of the pooling layers are concatenated and processed by a 1$\times$1 convolutional layer reducing the number of feature maps from 2048 to 100. The feature maps are then flattened, followed by two successive FC layers resulting in 128-dimensional feature representations $\bm{z}_1$ and $\bm{z}_2$ for the first and the second network branch.

§.§.§ Training strategy

The re-identification model is trained using the contrastive loss function [54] which is typically utilized to achieve a meaningful mapping $F$ from high to low dimensional space. By using the contrastive loss, the network learns to map similar inputs to nearby points on the output manifold while dissimilar inputs are mapped to distant points. Negative pairs contribute to the loss only if their distance is smaller than a certain margin $m$. In this work, the margin is set to $m=1$.
For our image retrieval experiments, the SNN architecture is optimized using the SGD algorithm in combination with the 1cycle learning policy [55, 56]. When using the 1cycle LR schedule, the LR $\eta$ steadily increases until it reaches a chosen maximum value and gradually decreases again thereafter. This schedule changes the LR after every single batch and is pursued a pre-defined number of epochs.
The upper bound is chosen at 0.1584 with the help of a LR finder. The lower bound is set to 0.0063. The L2 regularization technique is used with a decay factor of $10^{-5}$. Moreover, the batch size is adjusted to 32. We optimize the SNN architecture by first training the adapted network head of the incorporated ResNet-50 for 30 epochs with all other parameters being frozen. Then, the complete architecture is trained for another cycle, this time consisting of 50 epochs.
Since the batch size limits the task of constructing informative positive and negative pairs in the online mining strategy, the concept of cross-batch memory [57] is utilized to generate sufficient pairs across multiple mini-batches. This concept is based upon the observation that the embedding features generally tend to change slowly over time. This “slow drift” phenomenon allows the use of embeddings of previous iterations that would normally be considered out-dated and discarded. For our experiments, a memory size of 128 is chosen, meaning that the last 4 batches are considered for mining.

§.§.§ Evaluation techniques

To evaluate the re-identification performance of our trained model, several metrics are computed. $R$-Precision represents the precision at $R$, where $R$ denotes the number of relevant images for a given query image. In other words, if the top-$R$ retrieved images show $r$ relevant images, then $R$-Precision can be calculated from <ref>. Note that this value is then averaged over all query samples. Precision@1 constitutes a special case and evaluates how many times the top-1 images in the retrieved lists are relevant.

\begin{equation}
R\text{-}Precision = \frac{r}{R}\label{eq:r_prec}
\end{equation}

To further consider the order of the relevant images within the retrieved list, the mean average precision at $R$ ($mAP@R$) is computed according to <ref>. The $mAP@R$ denotes the mean of the average precision scores at $R$ ($AP@R$) over all $Q$ query images. The $AP@R$ (see <ref>) is the average of the precision values over all $R$ relevant samples, where $P@i$ refers to the precision at rank $i$ and $rel@i$ is an indicator function which equals 1 if the sample is relevant at rank $i$ and 0 if it is not relevant.

\begin{equation}
mAP@R = \frac{1}{Q} \sum_{i=1}^{Q} AP_{i}@R\label{eq:mAP}
\end{equation}

\begin{equation}
AP@R = \frac{1}{R} \sum_{i=1}^{R} P@i \times rel@i \label{eq:AP}
\end{equation}

§.§ Data availability

The NIH ChestX-ray14 dataset used throughout the current study is available via Box at <https://nihcc.app.box.com/v/ChestXray-NIHCC>. The COVID-19 Image Data Collection is available on GitHub at <https://github.com/ieee8023/covid-chestxray-dataset>. The CheXpert dataset can be requested at <https://stanfordmlgroup.github.io/competitions/chexpert>.

§.§ Code availability

The code used to train and evaluate both the patient verification and the patient re-identification models is available at <https://github.com/kaipackhaeuser/CXR-Patient-ReID>. Correspondence and requests for materials should be addressed to K.P.

§.§ Acknowledgements

The research leading to these results has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (ERC grant no. 810316).

§.§ Author contributions

A.M., V.C., S.G., C.S., and K.P. conceived the main idea. K.P. and N.M. performed the experiments and the evaluation. K.P. wrote the main part of the manuscript. S.G. offered continuous support during the experiments and the writing process. S.G., C.S., V.C., and A.M. provided expertise through intense discussions. All authors reviewed the manuscript.

§.§ Competing interests

The authors declare no competing interests.

url<#>1urlprefixURL doiprefixDOI:

[1]
authorMaier, A., authorSteidl, S.,
authorChristlein, V. & authorHornegger, J.
titleMedical Imaging Systems: An Introductory
Guide, vol. volume11111 (publisherSpringer,

[2]
authorRaoof, S. et al.
journaltitleInterpretation of Plain Chest
Chest volume141,
pages545–558 (year2012).

[3]
authorGündel, S. et al.
titleLearning to recognize Abnormalities in Chest
X-Rays with Location-Aware Dense Networks.
In booktitleIberoamerican Congress on Pattern
Recognition, pages757–765 (organizationSpringer,

[4]
authorWorld Health Organization (WHO).
note[Online; accessed: 21.12.2020].

[5]
authorWang, L., authorLin, Z. Q. &
authorWong, A.
journaltitleCOVID-Net: a tailored deep
convolutional neural network design for detection of COVID-19 cases from
chest X-ray images.
Scientific Reports
volume10, pages1–12 (year2020).

[6]
authorLee, C. S., authorNagy, P. G.,
authorWeaver, S. J. & authorNewman-Toker, D. E.
journaltitleCognitive and System Factors
Contributing to Diagnostic Errors in Radiology.
American Journal of Roentgenology
volume201, pages611–617

[7]
authorLeCun, Y., authorBengio, Y. &
authorHinton, G.
journaltitleDeep learning.
Nature volume521,
pages436–444 (year2015).

[8]
authorGündel, S. et al.
journaltitleMulti-task Learning for Chest
X-ray Abnormality Classification on Noisy Labels.
arXiv: 1905.06362  (year2019).

[9]
authorAkselrod-Ballin, A. et al.
titleA region based convolutional network for tumor
detection and classification in breast mammography.
In booktitleDeep learning and data labeling for
medical applications, pages197–205
(publisherSpringer, year2016).

[10]
authorRajpurkar, P. et al.
journaltitleCheXNet: Radiologist-Level
Pneumonia Detection on Chest X-Rays with Deep Learning.
arXiv: 1711.05225  (year2017).

[11]
authorRoh, Y., authorHeo, G. & authorWhang,
S. E.
journaltitleA Survey on Data Collection
for Machine Learning: A Big Data - AI Integration
IEEE Transactions on Knowledge and Data
Engineering  (year2019).

[12]
authorMaier, A., authorSyben, C.,
authorLasser, T. & authorRiess, C.
journaltitleA gentle introduction to deep
learning in medical image processing.
Zeitschrift für Medizinische Physik
volume29, pages86–101

[13]
authorOakden-Rayner, L.
journaltitleExploring Large-scale Public
Medical Image Datasets.
Academic Radiology
volume27, pages106–112

[14]
authorIrvin, J. et al.
titleCheXpert: A Large Chest Radiograph
Dataset with Uncertainty Labels and Expert Comparison.
In booktitleProceedings of the AAAI Conference on
Artificial Intelligence, vol. volume33,
pages590–597 (year2019).

[15]
authorGohagan, J. K., authorProrok, P. C.,
authorHayes, R. B. & authorKramer, B.-S.
titleThe Prostate, Lung, Colorectal and Ovarian
(PLCO) Cancer Screening Trial of the National Cancer Institute:
History, organization, and status.
In booktitleControlled Clinical Trials,
vol. volume21, pages251S–272S

[16]
authorWang, X. et al.
titleChestX-ray8: Hospital-scale Chest X-ray
Database and Benchmarks on Weakly-Supervised Classification and
Localization of Common Thorax Diseases.
In booktitleProceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages2097–2106

[17]
authorBandyopadhyay, D. et al.
journaltitleCovid-19 pandemic: cardiovascular
complications and future implications.
American Journal of Cardiovascular Drugs
pages1–14 (year2020).

[18]
authorSpinelli, A. & authorPellino, G.
journaltitleCovid-19 pandemic: perspectives on
an unfolding crisis.
The British journal of surgery

[19]
authorCohen, J. P. et al.
journaltitleCOVID-19 Image Data
Collection: Prospective Predictions are the Future.
arXiv: 2006.11988  (year2020).

[20]
authorChung, A.
titleFigure 1 COVID-19 Chest X-ray Dataset

[21]
authorChung, A.
titleActualMed COVID-19 Chest X-ray Dataset

[22]
authorRahman, T., authorChowdhury, M. &
authorKhandakar, A.
titleCOVID-19 Radiography Database.

[23]
authorWillemink, M. J. et al.
journaltitlePreparing Medical Imaging
Data for Machine Learning.
Radiology volume295,
pages4–15 (year2020).

[24]
authorCenters for Disease Control and Prevention.
titleHealth Insurance Portability and
Accountability Act of 1996 (HIPAA).
note[Online; accessed: 23.12.2020].

[25]
authorEuropean Union.
titleComplete guide to GDPR compliance.
note[Online; accessed: 23.12.2020].

[26]
authorO'Connor, M.
titleGoogle axed release of vast x-ray dataset following
NIH privacy concerns.
note[Online; accessed: 17.12.2020].

[27]
authorVincent, J.
titleGoogle scrapped the publication of 100,000 chest
x-rays due to last-minute privacy problems.
note[Online; accessed: 17.12.2020].

[28]
authorNoumeir, R., authorLemay, A. &
authorLina, J.-M.
journaltitlePseudonymization of Radiology
Data for Research Purposes.
Journal of Digital Imaging
volume20, pages284–295

[29]
authorSweeney, L.
journaltitlek-anonymity: A model for protecting
International Journal of Uncertainty, Fuzziness
and Knowledge-Based Systems volume10,
pages557–570 (year2002).

[30]
authorGkoulalas-Divanis, A. & authorLoukides, G.
titleMedical Data Privacy Handbook
(publisherSpringer, year2015).

[31]
authorZheng, M., authorKaranam, S.,
authorWu, Z. & authorRadke, R. J.
titleRe-Identification with Consistent Attentive
Siamese Networks.
In booktitleProceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages5735–5744

[32]
authorSelvaraju, R. R. et al.
titleGrad-CAM: Visual Explanations from Deep
Networks via Gradient-based Localization.
In booktitleProceedings of the IEEE International
Conference on Computer Vision, pages618–626

[33]
authorGündel, S. et al.
journaltitleRobust Classification from
Noisy Labels: Integrating Additional Knowledge for Chest
Radiography Abnormality Assessment.
Medical Image Analysis
pages102087 (year2021).

[34]
authorNautsch, A. et al.
journaltitlePreserving privacy in speaker and
speech characterisation.
Computer Speech & Language
volume58, pages441–480

[35]
authorTomashenko, N. et al.
titleThe VoicePrivacy 2022 Challenge Evaluation

[36]
authorDwork, C.
journaltitleA Firm Foundation for Private
Data Analysis.
Communications of the ACM
volume54, pages86–95

[37]
authorDwork, C., authorRoth, A. et al.
journaltitleThe Algorithmic Foundations of
Differential Privacy.
Foundations and Trends in Theoretical Computer
Science volume9, pages211–407

[38]
authorKaissis, G. A., authorMakowski, M. R.,
authorRückert, D. & authorBraren, R. F.
journaltitleSecure, privacy-preserving and
federated machine learning in medical imaging.
Nature Machine Intelligence
volume2, pages305–311

[39]
authorSarwate, A. D. & authorChaudhuri, K.
journaltitleSignal Processing and Machine
Learning with Differential Privacy: Algorithms and challenges for
continuous data.
IEEE signal processing magazine
volume30, pages86–94

[40]
authorKonečnỳ, J., authorMcMahan, H. B.,
authorRamage, D. & authorRichtárik, P.
journaltitleFederated Optimization:
Distributed Machine Learning for On-Device Intelligence.
arXiv preprint arXiv:1610.02527

[41]
authorRieke, N. et al.
journaltitleThe future of digital health with
federated learning.
NPJ digital medicine
volume3, pages1–7 (year2020).

[42]
authorKaissis, G. et al.
journaltitleEnd-to-end privacy preserving deep
learning on multi-institutional medical imaging.
Nature Machine Intelligence
volume3, pages473–484

[43]
authorBromley, J. et al.
journaltitleSignature Verification using a
“Siamese” Time Delay Neural Network.
International Journal of Pattern Recognition and
Artificial Intelligence volume7,
pages669–688 (year1993).

[44]
authorTaigman, Y., authorYang, M.,
authorRanzato, M. & authorWolf, L.
titleDeepFace: Closing the Gap to Human-Level
Performance in Face Verification.
In booktitleProceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages1701–1708

[45]
authorKoch, G., authorZemel, R. &
authorSalakhutdinov, R.
titleSiamese Neural Networks for One-shot Image
In booktitleICML Deep Learning Workshop

[46]
authorLake, B. M., authorSalakhutdinov, R. &
authorTenenbaum, J. B.
journaltitleHuman-level concept learning
through probabilistic program induction.
Science volume350,
pages1332–1338 (year2015).

[47]
authorLeCun, Y., authorBottou, L.,
authorBengio, Y. & authorHaffner, P.
journaltitleGradient-Based Learning
Applied to Document Recognition.
Proceedings of the IEEE
volume86, pages2278–2324

[48]
authorNational Institutes of Health (NIH).
titleNIH Clinical Center provides one of the largest
publicly available chest x-ray datasets to scientific community.
note[Online; accessed: 05.01.2021].

[49]
authorDeng, J. et al.
titleImageNet: A Large-Scale Hierarchical
Image Database.
In booktitle2009 IEEE Conference on Computer Vision
and Pattern Recognition, pages248–255 (year2009).

[50]
authorLeCun, Y. A., authorBottou, L.,
authorOrr, G. B. & authorMüller, K.-R.
titleEfficient BackProp.
In booktitleNeural Networks: Tricks of the Trade,
pages9–48 (publisherSpringer, year2012).

[51]
authorGoodfellow, I., authorBengio, Y. &
authorCourville, A.
titleDeep Learning (publisherMIT
Press, year2016).
noteURL: <http://www.deeplearningbook.org>.

[52]
authorKingma, D. P. & authorBa, J.
journaltitleAdam: A Method for
Stochastic Optimization.
arXiv preprint arXiv:1412.6980

[53]
authorFawcett, T.
journaltitleAn introduction to ROC
Pattern Recognition Letters
volume27, pages861–874

[54]
authorHadsell, R., authorChopra, S. &
authorLeCun, Y.
titleDimensionality Reduction by Learning an
Invariant Mapping.
In booktitle2006 IEEE Computer Society Conference on
Computer Vision and Pattern Recognition (CVPR'06),
vol. volume2, pages1735–1742
(organizationIEEE, year2006).

[55]
authorSmith, L. N. & authorTopin, N.
titleSuper-Convergence: Very Fast Training of
Neural Networks Using Large Learning Rates.
In booktitleArtificial Intelligence and Machine
Learning for Multi-Domain Operations Applications, vol.
volume11006, pages1100612
(organizationInternational Society for Optics and Photonics,

[56]
authorSmith, L. N.
titleCyclical Learning Rates for Training Neural
In booktitle2017 IEEE Winter Conference on
Applications of Computer Vision (WACV), pages464–472
(organizationIEEE, year2017).

[57]
authorWang, X., authorZhang, H.,
authorHuang, W. & authorScott, M. R.
titleCross-Batch Memory for Embedding Learning.
In booktitleProceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages6388–6397

§ SUPPLEMENTARY MATERIAL

[table]name=Supplementary Table

Additional patient verification results: Comparison of different data handling techniques (FTS and RNP), training set sizes $N_s$ and learning rates $\eta$. We present the AUC (together with the lower and upper bounds of the 95 % confidence intervals from 10,000 bootstrap runs), the accuracy, the specificity, the recall, the precision, and the F1-score.