11institutetext:  Division of Medical Image Computing, German Cancer Research
Center (DKFZ), Heidelberg, Germany 22institutetext: Medical Faculty
Heidelberg, Heidelberg University, Heidelberg, Germany 33institutetext:
Faculty of Mathematics and Computer Science, Heidelberg University,
Heidelberg, Germany 44institutetext: Department of Radiology, Charité -
Universitätsmedizin Berlin, Berlin, Germany 55institutetext: Interactive
Machine Learning Group (IML), German Cancer Research Center (DKFZ),
Heidelberg, Germany 66institutetext: Helmholtz Imaging, German Cancer Research
Center (DKFZ), Heidelberg, Germany. Pattern Analysis and Learning Group,
Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg,
Germany 77institutetext: National Center for Tumor Diseases (NCT) Heidelberg,
Germany 88institutetext: 88email<EMAIL_ADDRESS>
# Leveraging Foundation Models for Content-Based Medical Image Retrieval in
Radiology

Stefan Denner 1122 David Zimmerer 11 Dimitrios Bounias 1122 Markus Bujotzek
1122 Shuhan Xiao 1133 Lisa Kausch 11 Philipp Schader 1133 Tobias Penzkofer 55
Paul F. Jäger 5566 Klaus Maier-Hein 117788

###### Abstract

Content-based image retrieval (CBIR) has the potential to significantly
improve diagnostic aid and medical research in radiology. Current CBIR systems
face limitations due to their specialization to certain pathologies, limiting
their utility. In response, we propose using vision foundation models as
powerful and versatile off-the-shelf feature extractors for content-based
medical image retrieval. By benchmarking these models on a comprehensive
dataset of 1.6 million 2D radiological images spanning four modalities and 161
pathologies, we identify weakly-supervised models as superior, achieving a
$P@1$ of up to 0.594. This performance not only competes with a specialized
model but does so without the need for fine-tuning. Our analysis further
explores the challenges in retrieving pathological versus anatomical
structures, indicating that accurate retrieval of pathological features
presents greater difficulty. Despite these challenges, our research
underscores the vast potential of foundation models for CBIR in radiology,
proposing a shift towards versatile, general-purpose medical image retrieval
systems that do not require specific tuning.

###### Keywords:

Medical Image Retrieval Content-based image retrieval Foundation Models
Representation learning

##  1 Introduction

Content-based image retrieval (CBIR) systems offer significant potential for
enhancing teaching, diagnostics and research by the identification of similar
case histories and aiding the understanding of pathologies [4, 24, 26, 8].
These systems enable the indexing of large-scale medical-image datasets and
the retrieval of images that are visually similar to a query image, thus
supporting efficient browsing, searching, and retrieval operations [3]. CBIR
systems typically store low-dimensional representations of images in a
database and retrieve similar images based on representation similarity [24,
16]. In recent years, a key advancement in these systems has been the adoption
of deep learning models to encode the visual features of images [9]. However,
current CBIR approaches in the medical domain are typically limited to the few
pathologies that they were explicitly trained on [4, 12, 20, 18], therefore
lacking generalizability to unseen conditions. Furthermore, existing
approaches often rely on additional inputs, such as labels [4, 12], text [1],
or crops [20]. These limitations are in stark contrast to the evolving needs
of radiology, which demands a more general and versatile medical image
retrieval system capable of handling a broader scope of pathologies without
additional input [19, 24, 8].

Figure 1:  Schema of content-based image retrieval system workflow using
foundation models. During indexing, we store the embeddings of the medical
images created the foundation model. During retrieval, we compare the
embeddings of the query image to those in the index and retrieve the most
similar ones.

However, developing a feature extractor capable of accurately retrieving a
wide variety of pathologies presents several challenges. These include the
extensive data requirements for training as well as the complexity of
preprocessing and the training process itself [31]. A significant obstacle is
the nuanced similarity among medical images, where minor yet crucial
pathological features may be overshadowed by the anatomical similarities,
leading to potential underrepresentation in the embedding space and failure in
pathology differentiation [24, 2, 9].

To address these challenges, we investigate using vision foundation models as
powerful and versatile feature extractors for CBIR. Foundation models, having
been exposed to a diverse range of images, are hypothesized to possess a
nuanced understanding of relevant features, including those of modality,
anatomy, and pathology [32, 21, 30, 1, 16]. Using a comprehensive dataset of
1.6 million radiological 2D images across four modalities (CT, MR, X-ray, and
US), and 185 classes we answer the following questions:

  * •

What is the performance of vision foundation models as feature extractors for
medical image retrieval?

  * •

How does the performance of vision foundation models compare to a specialized,
pathology-specific CBIR system?

  * •

What effect does the index size have on the retrieval performance of medical
CBIR systems?

  * •

How well do embedding spaces of vision foundation models capture pathological
structures?

##  2 Materials and Methods

###  2.1 Datasets

To holistically evaluate foundation models’ performance in medical image
retrieval, we combine four public datasets – RadImageNet [23], NIH14 [29],
MIMIC [15] and CheXpert [14] – resulting in a comprehensive collection of over
1.6 million 2D images spanning four modalities, 12 anatomical regions, and 185
classes (161 pathological, 24 anatomical). Our dataset has a significant class
imbalance with the smallest and largest classes containing 13 and over 226,000
samples, respectively (supplementary Figure 4). We remove duplicates in
RadImageNet [23] and harmonize overlapping labels across datasets to ensure
consistent labeling. Furthermore, we exclude multi-label samples in all X-Ray
datasets to maintain a single-label focus. The dataset preparation also
includes a patient-wise split for CheXpert to create a validation set (ratio
90:10 of the training set). For other datasets, we use the provided dataset
splits. Details in supplementary Table 5.

###  2.2 Foundation models

Table 1:  Foundation models utilized for the benchmark.  Method | Architecture | Domain | #Params. | #Images | Datasets
---|---|---|---|---|---
Supervised
ResNet [11] | ResNet50 | Natural | 25 M | 1.3 M | IN1K [6]
ViT [7] | ViT-B/16 | Natural | 91 M | 14 M | IN21K [6]
SAM [17] | ViT-B/16 | Natural | 91 M | 300 M | SA-1B [17]
MedSAM [21] | ViT-B/16 | Medical | 91 M | 1.6 M | custom [21]
Weakly-Supervised
CLIP [27] | ViT-L/14 | Natural | 300 M | 400 M | WIT-400M [27]
MedCLIP [30] | Swin-T | Medical | 29 M | 600 K |  MIMIC [15] & CheXpert [14]
BiomedCLIP [32] | ViT-B/16 | Biomedical | 91 M | 15 M | PMC-15M [32]
Self-Supervised
MAE [10] | ViT-B/16 | Natural | 91 M | 14 M | IN21K [6]
DINOv2 [25] | ViT-B/14 | Natural | 86 M | 142 M | LVD-142M [25]

We choose a range of foundation models spanning self-supervised, weakly-
supervised, and fully-supervised learning schemes, trained on both medical or
natural images, ensuring a broad evaluation spectrum (Table 1). Our baselines
are ResNet [11] and Vision Transformer (ViT) [7], which are supervised models
trained on ImageNet [6]. We use their probability class predictions as feature
encodings. We also include the Segment Anything Model (SAM) [17] and its
medical adaptation (MedSAM) [21]. Both are supervised prompt-based
segmentation models, which showed comparable or even superior performance to
fully supervised segmentation models [17]. We use the averaged patch tokens of
the encoder as embeddings. CLIP [27] is a weakly-supervised learning approach,
trained to align the representations of images and texts in a shared embedding
space, allowing it to capture a wide range of semantic similarities. MedCLIP
[30] and BiomedCLIP [32] are adaptations of CLIP to the medical and biomedical
contexts, respectively. For all CLIP-based models, we use the vision encoder
representation as embeddings. For the self-supervised model category, we
choose ViT-based Masked Autoencoder (MAE) [10] and DINOv2 [25], as they have
shown state-of-the-art performance on classification tasks with no or minimal
fine-tuning. For both models, we utilize the class token of the ViT as feature
encodings.

###  2.3 Leveraging foundation models for image retrieval

Since our goal is to evaluate the effectiveness and versatility of foundation
models as feature encoders for medical image retrieval, we utilize them in an
off-the-shelf manner, i.e. without any fine-tuning. This approach allows us to
rigorously determine the models’ innate capacity to understand and represent
medical image pathologies. To this end, we (1) resize the images to the
required input size for each model, (2) extract the features as described
above, (3) perform $L2$ normalization on the models’ embeddings, and (4) store
these normalized embeddings in a vector database. During retrieval (5), we
measure the cosine similarity between the query image encoding and all other
image encodings in the index. We retrieve the top N images with the highest
similarity, sorted by descending order.

###  2.4 Evaluation

We measure the retrieval performance between the query image and the top N
retrieved images, quantified by Precision at N ($P@N$) [22], defined as:

$P@N=\frac{1}{Q}\sum_{q=1}^{Q}\frac{1}{N}\sum_{j=1}^{N}rel_{q}(j),$ (1)

where $rel_{q}(j)\in\\{0,1\\}$ is the relevance of the $j$-th image for the
$q$-th query, $Q$ is the number of queries and $N\in\\{1,3,5,10\\}$. In our
setting, an image is considered relevant if it belongs to the same class as
the query image. To gain a more comprehensive understanding of our model’s
performance, especially for classes with fewer examples, we calculate both
micro- and macro-averaged scores. Micro-averaging treats all samples equally,
regardless of their class. It measures the average score across all samples.
Macro-averaging focuses on each class individually. It first calculates the
average score per class, and then averages those class scores. This approach
is crucial for imbalanced datasets because it prevents the dominant class from
skewing the overall performance and provides a fair evaluation of how well the
model performs on less frequent classes.

Figure 2:  Qualitative results of image retrieval with BiomedCLIP. Rows
display the query image highlighted in orange on the left, followed by the six
most similar images, sorted by descending similarity. Green frames indicate a
match with the query image’s class, red indicates a mismatch.

##  3 Results and Discussion

Table 2: Retrieval performance of the evaluated models on the combined dataset. | Sample-wise (micro) | Class-wise (macro)
---|---|---
| P@1 | P@3 | P@5 | P@10 | P@1 | P@3 | P@5 | P@10
ResNet | 0.544 | 0.538 | 0.535 | 0.529 | 0.204 | 0.193 | 0.189 | 0.182
ViT | 0.560 | 0.554 | 0.550 | 0.543 | 0.217 | 0.204 | 0.199 | 0.190
SAM | 0.520 | 0.517 | 0.515 | 0.511 | 0.197 | 0.187 | 0.182 | 0.175
MedSAM | 0.489 | 0.483 | 0.478 | 0.470 | 0.154 | 0.146 | 0.142 | 0.134
CLIP | 0.571 | 0.567 | 0.565 | 0.559 | 0.222 | 0.213 | 0.209 | 0.202
MedCLIP | 0.566 | 0.561 | 0.559 | 0.554 | 0.223 | 0.211 | 0.205 | 0.198
BiomedCLIP | 0.594 | 0.590 | 0.588 | 0.583 | 0.240 | 0.230 | 0.224 | 0.217
MAE | 0.513 | 0.507 | 0.503 | 0.497 | 0.183 | 0.177 | 0.172 | 0.166
DINOv2 | 0.553 | 0.548 | 0.545 | 0.540 | 0.219 | 0.204 | 0.199 | 0.192
X-MIR | 0.478 | 0.471 | 0.466 | 0.457 | 0.149 | 0.142 | 0.138 | 0.131

####  3.0.1 Retrieval performance of foundation models.

Evaluating the retrieval performance of the foundation models on the combined
dataset, we observe that BiomedCLIP achieves the highest score across all
metrics (Table 2; qualitative results in Figure 2; t-SNE visualization in
supplementary Figure 5). In general, weakly-supervised approaches outperform
other training schemes. Segmentation models (SAM and MedSAM) show lower
performance, suggesting their embeddings are not well-suited for medical image
retrieval. Furthermore, medical adaptations (MedSAM and MedCLIP) fail to
surpass their natural image counterparts (SAM and CLIP), indicating that their
embeddings don’t generalize. The superior performance of CLIP-based approaches
can probably be attributed to its training objective of semantically
meaningful aligning the representations of images and texts. BiomedCLIP in
particular performs well, probably due to its exposure to a wide variety of
biomedical image-text pairs, leading to a nuanced understanding of
pathologies, which is reflected in the embedding space. However, sample-wise
scores are significantly higher than class-wise scores across all models,
0.594 and 0.240 for BiomedCLIP $P@1$, respectively.

####  3.0.2 Comparison to a specialized medical image retrieval system.

Table 3: Retrieval performance on the COVID-19 chest X-ray dataset. | Micro | Macro
---|---|---
| P@1 | P@3 | P@5 | P@10 | P@1 | P@3 | P@5 | P@10
DenseNet | 0.811 | 0.828 | 0.831 | 0.820 | 0.811 | 0.828 | 0.830 | 0.820
BiomedCLIP | 0.913 | 0.923 | 0.910 | 0.921 | 0.914 | 0.924 | 0.910 | 0.921
X-MIR | 0.954 | 0.934 | 0.934 | 0.932 | 0.954 | 0.933 | 0.934 | 0.931

To contextualize the efficacy of employing vision foundation models as feature
extractors, it is critical to benchmark their performance against image
retrieval systems designed for a narrower spectrum of pathologies. We
identified only one publicly available radiological image retrieval system
that operates without requiring additional user input: X-MIR by Hu et al.
[12], which utilizes a DenseNet121 [13] trained on a COVID-19 chest X-ray
dataset [28] for the retrieval of pneumonia, Covid-19, and normal cases.

Our evaluation unfolds in two phases. Firstly, we evaluate X-MIR on our
combined dataset to highlight its limited generalizability to unseen classes
(Table 2). Secondly, we evaluate the best performing foundation model on our
combined dataset, BiomedCLIP, on the COVID-19 chest X-ray dataset [28], which
X-MIR was specifically trained on.111Given updates to the dataset since its
initial publication, we evaluate the X-MIR model on a subset of the original
dataset, noting discrepancies from the originally reported results. Despite
BiomedCLIP not being fine-tuned on this dataset, it closely rivals X-MIR’s
specialized performance, underscoring the versatility and robustness of
foundation models in broader medical image retrieval applications.

Figure 3: Effect of the number of samples per class in the index on the
retrieval performance. (A) Class-wise performance ($P@1$) of BiomedCLIP. Color
encoded are anatomical (green) and pathological (red) classes. (B) Sample-wise
performance ($P@1$ micro) of the foundation models with varying numbers of
samples per class in the index.

####  3.0.3 Effect of index size on retrieval performance.

As noted in Section 3.0.1, there is a discrepancy between class-wise and
sample-wise scores indicating higher performance for more frequent classes on
our combined dataset. In Figure 3 A we show, exemplary for BiomedCLIP, that
classes with more samples in the index yield higher P@1. Additionally, our
findings show superior performance for anatomical over pathological classes,
highlighting the model’s challenges in accurately retrieving pathological
structures.

To investigate whether retrieval performance reaches saturation with a
specific number of samples per class in the index, we conduct an ablation
study focusing on classes with over 3000 samples, comprising 64 classes (42
pathological and 22 anatomical). We standardize the query dataset size to the
smallest class, containing 38 samples per class. Then, we randomly select $N$
cases per class, where $N$ ranges from 5 to 3000, to populate the index and
conduct retrievals using the fixed query set.

For the best models, P@1 starts saturating at around 1000 samples per class in
the index (Figure 3 B). This finding suggests, that up to 1000 samples per
class, providing more samples improves the retrieval performance. However,
beyond 1000 samples, additional samples do not significantly improve retrieval
outcomes, pointing to a need for advancements in the models’ clustering
capabilities for performance gains.

####  3.0.4 Analysis of the embedding space

Table 4: Performance of the foundation models on the combined dataset using kNN and linear classification. | kNN | Linear
---|---|---
| Micro | Macro | Micro | Macro
| AURPC | F1 | AURPC | F1 | AURPC | F1 | AURPC | F1
ResNet | 0.630 | 0.623 | 0.211 | 0.203 | 0.706 | 0.648 | 0.278 | 0.262
ViT | 0.644 | 0.633 | 0.220 | 0.210 | 0.734 | 0.668 | 0.300 | 0.275
SAM | 0.588 | 0.599 | 0.200 | 0.186 | 0.571 | 0.556 | 0.163 | 0.113
MedSAM | 0.561 | 0.567 | 0.151 | 0.139 | 0.497 | 0.492 | 0.100 | 0.060
CLIP | 0.673 | 0.648 | 0.243 | 0.221 | 0.743 | 0.674 | 0.308 | 0.278
MedCLIP | 0.668 | 0.643 | 0.241 | 0.212 | 0.720 | 0.660 | 0.286 | 0.246
BiomedCLIP | 0.711 | 0.667 | 0.269 | 0.231 | 0.713 | 0.651 | 0.273 | 0.232
MAE | 0.627 | 0.602 | 0.203 | 0.172 | 0.692 | 0.636 | 0.251 | 0.215
DINOv2 | 0.654 | 0.630 | 0.229 | 0.201 | 0.722 | 0.659 | 0.285 | 0.268

While image retrieval directly aims to find similar images, it inherently
relies on the quality of the underlying embedding space. By analyzing the
embedding space through classification tasks, we gain a deeper understanding
of the models’ ability to capture relevant features for the retrieval task
[22].

We employ k-nearest neighbors (kNN) classification [5] to evaluate clustering
within the embedding space, determining the clustering efficacy of similar
images [22]. Additionally, a single-layer linear classifier trained on these
representations helps assess whether the embedding space holds classification-
relevant features, serving as a benchmark for feature quality. Classification
performance is measured by F1-Score and AUPRC to adjust for dataset class
imbalance.

Overall, coherent with section 3.0.1, BiomedCLIP achieves the best kNN
classification performance, suggesting the best clustering of pathological
structures in the embedding space (Table 4). However, CLIP surpasses all
models in linear classification, indicating that even without finetuning it
encodes relevant features for pathological structures more effectively,
potentially leading to superior retrieval performance. Conversely, some
(self-)supervised models like DINOv2 and ViT exhibit strong performance in
linear classification despite not performing exceptionally well in the
retrieval task itself, demonstrating their potential capability as feature
extractors for image retrieval after fine-tuning with medical imaging data.

##  4 Conclusion

In this work, we have demonstrated the effectiveness of utilizing vision
foundation models as powerful and versatile feature extractors for content-
based medical image retrieval, benchmarking their performance across an
extensive dataset of over 1.6 million radiological 2D images, spanning four
modalities and 185 classes. Our analysis highlights the promising off-the-
shelf capabilities of weakly-supervised models, especially BiomedCLIP, in
providing superior feature extraction performance, rivaling that of
specialized models. Through comprehensive ablation studies, we have gained
valuable insights into the clustering and feature extraction capabilities of
these models. The encouraging outcomes of our research strongly support the
pursuit of further exploration into even more advanced vision foundation
models, that can accurately understand and represent pathological
similarities. We hope that our publicly shared resources – code, datasets, and
embeddings – facilitate future benchmarking of foundation models for medical
image retrieval.

## Acknowledgments

Funded by "NUM 2.0" (FKZ: 01KX2121)

## References

  * [1] Abacha, A.B., Santamaria-Pang, A., Lee, H.H., Merkow, J., Cai, Q., Devarakonda, S.T., Islam, A., Gong, J., Lungren, M.P., Lin, T., et al.: 3d-mir: A benchmark and empirical study on 3d medical image retrieval in radiology. arXiv preprint arXiv:2311.13752 (2023)
  * [2] Chen, C., Lu, M.Y., Williamson, D.F., Chen, T.Y., Schaumberg, A.J., Mahmood, F.: Fast and scalable search of whole-slide images via self-supervised deep learning. Nature Biomedical Engineering 6(12), 1420–1434 (2022)
  * [3] Chen, W., Liu, Y., Wang, W., Bakker, E.M., Georgiou, T., Fieguth, P., Liu, L., Lew, M.S.: Deep learning for instance retrieval: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [4] Choe, J., Hwang, H.J., Seo, J.B., Lee, S.M., Yun, J., Kim, M.J., Jeong, J., Lee, Y., Jin, K., Park, R., et al.: Content-based image retrieval by using deep learning for interstitial lung disease diagnosis with chest ct. Radiology 302(1), 187–197 (2022)
  * [5] Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE transactions on information theory 13(1), 21–27 (1967)
  * [6] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [8] Dubey, S.R.: A decade survey of content based image retrieval using deep learning. IEEE Transactions on Circuits and Systems for Video Technology 32(5), 2687–2704 (2021)
  * [9] Gordo, A., Almazán, J., Revaud, J., Larlus, D.: Deep image retrieval: Learning global representations for image search. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14. pp. 241–257. Springer (2016)
  * [10] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16000–16009 (2022)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [12] Hu, B., Vasu, B., Hoogs, A.: X-mir: Explainable medical image retrieval. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 440–450 (2022)
  * [13] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
  * [14] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 590–597 (2019)
  * [15] Johnson, A.E., Pollard, T.J., Shen, L., Lehman, L.w.H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., Mark, R.G.: Mimic-iii, a freely accessible critical care database. Scientific data 3(1), 1–9 (2016)
  * [16] Jush, F.K., Truong, T., Vogler, S., Lenga, M.: Medical image retrieval using pretrained embeddings. arXiv preprint arXiv:2311.13547 (2023)
  * [17] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [18] Kobayashi, K., Gu, L., Hataya, R., Mizuno, T., Miyake, M., Watanabe, H., Takahashi, M., Takamizawa, Y., Yoshida, Y., Nakamura, S., et al.: Sketch-based semantic retrieval of medical images. Medical Image Analysis 92, 103060 (2024)
  * [19] Langlotz, C.P.: The future of ai and informatics in radiology: 10 predictions (2023)
  * [20] Lee, H.H., Santamaria-Pang, A., Merkow, J., Oktay, O., Pérez-García, F., Alvarez-Valle, J., Tarapov, I.: Region-based contrastive pretraining for medical image retrieval with anatomic query. arXiv preprint arXiv:2305.05598 (2023)
  * [21] Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 15(1), 654 (2024)
  * [22] Manning, C.D.: An introduction to information retrieval. Cambridge university press (2009)
  * [23] Mei, X., Liu, Z., Robson, P.M., Marinelli, B., Huang, M., Doshi, A., Jacobi, A., Cao, C., Link, K.E., Yang, T., et al.: Radimagenet: an open radiologic deep learning research dataset for effective transfer learning. Radiology: Artificial Intelligence 4(5), e210315 (2022)
  * [24] Müller, H., Michoux, N., Bandon, D., Geissbuhler, A.: A review of content-based image retrieval systems in medical applications—clinical benefits and future directions. International journal of medical informatics 73(1), 1–23 (2004)
  * [25] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
  * [26] Qayyum, A., Anwar, S.M., Awais, M., Majid, M.: Medical image retrieval using deep convolutional neural network. Neurocomputing 266, 8–20 (2017)
  * [27] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [28] Wang, L., Lin, Z.Q., Wong, A.: Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific reports 10(1), 19549 (2020)
  * [29] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2097–2106 (2017)
  * [30] Wang, Z., Wu, Z., Agarwal, D., Sun, J.: Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163 (2022)
  * [31] Willemink, M.J., Koszek, W.A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L.R., Summers, R.M., Rubin, D.L., Lungren, M.P.: Preparing medical imaging data for machine learning. Radiology 295(1), 4–15 (2020)
  * [32] Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., et al.: Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 (2023)

Table 5:  Datasets utilized for the benchmark. For image retrieval, train dataset was used for indexing and test dataset for querying/retrieval. For kNN classification, validation dataset was for determining the best $K$. For linear classification, validation dataset was used for early stopping.  Dataset | Modality | Train | Val | Test |  Pathology/ Anatomy classes
---|---|---|---|---|---
NIH14 [29] | XR | 66,192 | 7,279 | 17,853 | 14/1
MIMIC [15] | XR | 229,597 | 1,775 | 2,473 | 12/1
CheXpert [14] | XR | 50,194 | 5,587 | 54 | 12/1
RadImageNet [23] | MR | 460,479 | 48,820 | 57,103 | 109/7
| CT | 202,521 | 24,660 | 63,198 | 32/2
| US | 313,569 | 27,417 | 31,072 | 14/1
Total |  | 1,322,552 | 115,538 | 171,753 | 161/24
Figure 4: Class imbalance of the combined dataset. Figure 5:  t-SNE projections of the BiomedCLIP embeddings of the combined dataset. Left: Color encoded are modalities. Center: Color encoded are anatomical structures. Right: Filtered for CT images only. Color encoded are pathologies.  Table 6: Training details for the linear classifier. Epochs | 100
---|---
Early stopping | 20
Learning rate | 1.00E-03
Optimizer | AdamW
Figure 6:  Qualitative results of image retrieval with BiomedCLIP. Rows
display the query image highlighted in orange on the left, followed by the six
most similar images, sorted by descending similarity. Green frames indicate a
match with the query image’s class, red indicates a mismatch.