# Foundation Models for Biomedical Image Segmentation: A Survey

Ho Hin Lee*, Yu Gu*, Theodore Zhao, Yanbo Xu, Jianwei Yang, Naoto Usuyama,
Cliff Wong, Mu Wei,
Bennett A. Landman, Yuankai Huo, Alberto Santamaria-Pang, Hoifung Poon
Microsoft Research, Vanderbilt University

###### Abstract

Recent advancements in biomedical image analysis have been significantly
driven by the Segment Anything Model (SAM). This transformative technology,
originally developed for general-purpose computer vision, has found rapid
application in medical image processing. Within the last year, marked by over
100 publications, SAM has demonstrated its prowess in zero-shot learning
adaptations for medical imaging. The fundamental premise of SAM lies in its
capability to segment or identify objects in images without prior knowledge of
the object type or imaging modality. This approach aligns well with tasks
achievable by the human visual system, though its application in non-
biological vision contexts remains more theoretically challenging. A notable
feature of SAM is its ability to adjust segmentation according to a specified
resolution scale or area of interest, akin to semantic priming. This
adaptability has spurred a wave of creativity and innovation in applying SAM
to medical imaging. Our review focuses on the period from April 1, 2023, to
September 30, 2023, a critical first six months post-initial publication. We
examine the adaptations and integrations of SAM necessary to address
longstanding clinical challenges, particularly in the context of 33 open
datasets covered in our analysis. While SAM approaches or achieves state-of-
the-art performance in numerous applications, it falls short in certain areas,
such as segmentation of the carotid artery, adrenal glands, optic nerve, and
mandible bone. Our survey delves into the innovative techniques where SAM’s
foundational approach excels and explores the core concepts in translating and
applying these models effectively in diverse medical imaging scenarios.

##  1 Introduction

In the rapidly evolving landscape of biomedical image analysis, the Segment
Anything Model (SAM) has emerged as a paradigm-shifting technology. Initially
conceived for general-purpose computer vision, SAM has quickly become
instrumental in the field of medical image processing. This survey provides an
in-depth examination of SAM’s applications and adaptations in biomedical
imaging, focusing on its remarkable proliferation in scholarly research over
the past year.
One of the most compelling attributes of SAM is its ability to segment objects
in medical images without requiring prior knowledge about the object types or
imaging modalities. This capability is of paramount importance in biomedical
imaging, where precise segmentation of diverse anatomical structures and
pathologies is critical. SAM’s methodology echoes the adaptability of the
human visual system in object recognition and segmentation, marking a
significant advancement in computational image analysis.
Our review concentrates on the period from April 1, 2023, to September 30,
2023. These six months post-SAM’s initial publication represent a crucial
phase in its development, characterized by extensive adaptations and
integrations of SAM to meet longstanding clinical challenges, especially in
the context of open datasets. Our analysis encompasses 33 such datasets,
offering a comprehensive assessment of the cutting-edge applications of SAM in
this dynamic area.
Moreover, we provide a critical evaluation of the domains where SAM’s
performance is currently lacking, including the segmentation of complex
anatomical regions such as the carotid artery, adrenal glands, optic nerve,
and mandible bone. This investigation aims to highlight the current
limitations and identify potential areas for future enhancements in SAM
technology.
This survey also explores the innovative techniques where SAM’s fundamental
approach demonstrates exceptional performance. We delve into the underlying
concepts essential for the effective translation and application of these
models across various medical imaging scenarios. Our objective is to delineate
the current state of SAM applications in biomedical imaging and to offer
strategic directions for future research in this vibrant and rapidly advancing
field.
In summary, as SAM propels forward in the realm of biomedical imaging, its
multifaceted impact becomes increasingly evident. Beyond the technical prowess
in image segmentation, SAM represents a beacon of interdisciplinary
collaboration, merging the frontiers of computer science, medicine, and data
analytics. This survey highlights not just SAM’s technical achievements but
also its role as a catalyst in redefining medical research paradigms. By
examining the integration of SAM across various medical domains, we offer
insights into how this technology is reshaping diagnostic methodologies,
enhancing patient care, and opening new avenues in personalized medicine. Our
exploration into SAM’s application in diverse medical contexts is not just an
academic exercise but a testament to the transformative potential of AI in
healthcare, providing a roadmap for future innovations that could further
revolutionize the field.

##  2 Background

###  2.1 Segment Anything Model (SAM)

Building upon foundational model concepts, the Segment Anything Model (SAM)
[80] serves as a pioneering framework that facilitates the expansion of
segmentation capabilities to previously unseen objects, leveraging user-
provided prompt guidance. Prompt inputs, in their diverse forms–ranging from
singular or multiple points to bounding boxes, or text descriptions
[170]–empower SAM to craft accurate instance segmentation masks for new image
samples without necessitating further training (a process known as zero-shot
segmentation). It should be highlighted, however, that the segmented objects
aren’t imbued with semantic meaning, a characteristic contingent upon the
reliability of the prompts utilized.

Structurally, SAM’s architecture can be divided into three primary components:
Image Encoder. This component incorporates a Masked AutoEncoder (MAE) [55]
pre-trained Vision Transformer (ViT) [42] to adapt to high-resolution images
across various scales. In order to explore scalability, the image encoder is
modulated from ViT-B to ViT-H or other model architectures, allowing for an
assessment of potential trade-offs in model efficiency. All images are resized
to dimensions of $1024\times 1024$ to conform to the standardized image size
established during pre-training.
Prompt Encoder. Two primary categories of prompts are delineated for
implementation: sparse and dense prompts. Sparse prompts encompass elements
like points and boxes, which are processed using ViT’s positional encoding
concept, integrated with their respective learned representations.
Additionally, text forms a part of the sparse prompt category, with features
extracted through a CLIP-pretrained text encoder. Meanwhile, dense prompts,
typified by masks, undergo convolution operations to establish element-wise
correlations with image features discerned by the image encoder.

Light-weighted Mask Decoder This segment aims to unify features discerned by
various encoders, fostering detailed predictions. It incorporates a
transformer decoder block followed by a dynamic prediction head, adept at
merging bidirectional self-attention and cross-attention mechanisms, thereby
facilitating concurrent updates of prompt-to-image and image-to-prompt
embeddings. To maintain computational efficiency, only two decoder blocks are
deployed to upsample the resolution of the acquired embeddings. A Multi-Layer
Perceptron (MLP) is utilized as the final layer that maps output tokens to a
dynamic linear classifier and ascertains foreground probabilities essential
for segmentation.
The entire SAM is trained on the large-scale dataset SA-1B, which consists of
11M high-resolution images with 1.1B high-quality segmentation masks, 400×
more masks than any existing segmentation dataset [80]. Such large-scale pre-
training enhances both the feasibility and stability of adapting unseen domain
samples for segmentation. While its success in the general domain is
noteworthy, medical imaging presents its own set of intricacies.

Figure 1: Evolution of SAM’s adaptation in medical research from April to
September 2023. The graph showcases the cumulative studies emphasizing four
phases: (i) Zero-shot Evaluation, (ii) Multi-dimensional Extension, (iii)
Domain-specific Tuning, and (iv) Knowledge Distillation, highlighting a
growing research interest in optimizing SAM for medical image segmentation.

###  2.2 Medical Image Segmentation

Medical image segmentation stands as an indispensable tool in the clinical
realm, enabling the precise extraction of anatomical structures and
pathological features from a plethora of imaging modalities. It’s instrumental
in facilitating accurate diagnoses, formulating treatment plans, guiding
surgical interventions, and propelling medical research.

####  2.2.1 Data

As medical images reconstructed with multi-dimensional property (i.e., 2D, 3D)
due to the significant variability in imaging protocols, manually generating
segmentation mask for organ-specific regions are time-consuming with
significant efforts from clinicians. Initial efforts have been put to generate
datasets with single organ labeled [34, 122, 119, 121, 120]. The single organ
labeled datasets provide an initial foundation basis to train deep learning
models for automatic segmentation, thus enhancing the extensive capabilities
to label multiple organ regions. The capability of deep learning models for
directly segmenting multiple regions also starts to explore with multi-
organ/multi-tissue labeled datasets [82, 18, 17, 10]. However, as shown in
Table 5, only limited number of semantic classes (i.e. 15 organs) have
demonstrated across different datasets [72], unlike SA-1B with large semantic
classes knowledge. TotalSegmentator dataset is a significant breakthrough with
117 organs labeled by adapting hierarchical segmentation network nnUNet and
refining the pseudo label manually. Such dataset demonstrates similarity with
SA-1B with approximately 100 semantic classes for each 2D slice-wise image.
However, with the significant manual efforts contributed in labeling, only
1228 imaging samples are labeled and there is still a significant difference
in sampling sizes comparing to natural image dataset.

####  2.2.2 Methods

Traditional Techniques. Grounded in algorithmic methods, early approaches like
thresholding [45], region-growing [110], and contour tracing provided
rudimentary segmentation [157, 130]. These were often manual or semi-
automatic, rendering them labor-intensive and sensitive to variations.

Convolutional Neural Networks. The rise of machine learning witnessed the
advent of CNNs, with architectures like U-Net revolutionizing medical image
segmentation[118, 33, 67, 85, 87]. By hierarchically processing images, CNNs
captured both local features and broader structures, albeit sometimes
grappling with global context due to fixed receptive fields.

Vision Transformers. Emerging as a recent trend, ViTs offer the capability to
understand global image patterns. Unlike CNNs, they aren’t bound by fixed
receptive fields, enabling them to capture both local details and broader
contexts. Hierarchical transformers, which blend CNN and transformer
strengths, further refine segmentation capabilities[27, 156, 150, 168, 53,
52].

###  2.3 SAM’s Role in Medical Image Segmentation

The application of SAM, initially optimized for general images, to the
specialized realm of medical imaging introduces both opportunities and
barriers. The following are pivotal considerations:

Domain Specificity. SAM’s general-domain proficiency is laudable, but medical
images introduce nuances not often seen in everyday images. Variations due to
patient demographics, imaging equipment, and protocols add layers of
complexity [105, 154, 101]. Such diversities question whether SAM, in its
original state, can deliver consistent performance across the vast spectrum of
medical images. Domain-specific fine-tuning and pretraining, incorporating
expert annotations or domain-adaptive techniques, may be needed for optimal
performance[49].

Dimenstional Mismatch. While 2D images are commonplace, medical imaging often
delves into the third dimension. Modalities like MRI and CT offer volumetric
insights, crucial for understanding anatomical structures and pathologies[85,
52], while SAM’s 2D-centric design is not natively cater to such data.
Adapting it would likely necessitate architectural tweaks and potential
integration with 3D convolutional layers or volumetric transformers.

Data Scarcity and Annotation Quality. Accurate segmentation hinges on high-
quality annotations, which, in the medical domain, are time-consuming,
expertise-dependent, and fraught with privacy concerns[167]. SAM’s broad
training on general images offers a potential springboard, but its true
utility in addressing the challenges of limited medical data and annotation
variance remains an open question.

##  3 Challenges in Contemporary Medical Image Segmentation Tasks

To understand the status of contemporary medical image segmentation, we
curated a collection of 33 medical image segmentation datasets in Table 1. We
order the datasets by the their most recent release year, ranging from 1998 to
2023, with the majority of them in the past decade. All the datasets in Table
1 are publicly available, thus have direct impact on the development of open
source models. We cover a variety of 7 modalities, 6 segmentation target
types, 17 anatomies in this collection. Although it is impossible to exhaust
every medical datasets that are publicly available, we view this collection as
a representative sub-sample to reflect the broad medical image segmentation
problems from a high level.

In the following parts of this section, we analyze the challenges of medical
image segmentation from two aspects: 1. input (imaging modality) and 2. output
(segmentation target), showing their versatility and how they are different
from the problems in general domain.

Table 1: Chronological timeline of medical image segmentation datasets. “Public” includes a link to each dataset (if available) or paper (if not). “Annotations” denotes the number of classes with ground-truth quality labels in each dataset. Year | Dataset | Public | Details
---|---|---|---
Modality | Anatomy | Data Size | Label Quality | # Targets | Seg. Target Type
1998 | JSRT [134] |  | X-Ray | Chest | 307 | Manual | 2 | Multi-Organ
2012 | VESSEL12 [124] |  | CT | Lung | 20 | Manual | 1 | Organ Parts
2012 | PROMISE12 [96] |  | MRI | Prostate | 100 | Manual | 1 | Single Organ
2013 | NCI-ISBI [21] |  | MRI | Prostate | 80 | Manual | 2 | Organ Parts
2015 | BTCV [82] |  | CT | Abdomen | 50 | Manual | 13 | Multi-Organ
2015 | CT-Lymph Nodes [119, 122, 34] |  | CT | Mediastinum | 176 | Manual | 1 | Single Organ
2015 | GlaS [135, 136] |  | Pathology | Colon | 165 | Manual | 1 | Cells
2016 | Pancreas-CT [120, 121, 34] |  | CT | Pancreas | 80 | Manual | 1 | Single Organ
2017 | LiTS [18] |  | CT | Liver | 131 | Manual | 2 | Tumor
2017 | ACDC [17] |  | MRI | Heart | 150 | Manual | 3 | Organ Parts
2018 | FUMPE [103] |  | CT | Lung | 35 | Exp.+Mdl. | 1 | Lesion
2018 | MSD [10] |  | CT, MRI | Multiple | 1411 CT, 1222 MRI | Manual | 18 | Multi-Task
2018 | DRIVE [138] |  | Fundus | Retina | 40 | Manual | 1 | Organ Parts
2018 | REFUGE [111] |  | Fundus | Retina | 1200 | Manual | 2 | Organ Parts
2019 | CHAOS [76, 74, 75] |  | CT, MRI | Abdomen | 40 CT, 40 MRI | Manual | 4 | Multi-Organ
2019 | SIIM-ACR Pneumothorax [160] |  | X-Ray | Chest | 12047 | Manual | 1 | Lesion
2019 | AbdomenUS [146] |  | Ultrasound | Abdomen | 61 Real, 926 Synth. | Real+Synth. | 8 | Multi-Organ
2019 | Breast Ultrasound Images [4] |  | Ultrasound | Breast | 780 | Manual | 3 | Tumor
2019 | CAMUS [83] |  | Ultrasound | Heart | 500 | Manual | 3 | Organ Parts
2020 | M&Ms [24] |  | MRI | Heart | 375 | Manual | 3 | Organ Parts
2020 | MosMed COVID-19 [109] |  | CT | Lung | 50 | Manual | 1 | Infection
2020 | COVID-19 Radiography [32, 116] |  | X-Ray | Chest | 21165 | Manual | 1 | Single Organ
2021 | COVID-QU-Ex [141, 142, 37, 32, 116] |  | X-Ray | Chest | 33920 | Manual | 2 | Infection
2021 | QaTa-COV19 [38] |  | X-Ray | Chest | 9258 | Manual | 1 | Infection
2021 | CT2US [137] |  | Ultrasound | Abdomen | 4586 | Synth. | 1 | Single Organ
2021 | PolypGen [7, 6, 5] |  | Endoscope | Colon | 8037 | Manual | 1 | Polyp
2022 | AbdomenCT-1K [102] |  | CT | Abdomen | 1112 | Exp.+Mdl. | 4 | Multi-Organ
2022 | AMOS [72] |  | CT, MRI | Abdomen | 500 CT, 100 MRI | Exp.+Mdl. | 15 | Multi-Organ
2023 | KiTS [57] |  | CT | Kidney | 599 | Exp.+Mdl. | 3 | Organ, Tumor
2023 | TotalSegmentator [153] |  | CT | Full Body | 1228 | Manual | 117 | Multi-Organ
2023 | BraTS [73, 13, 106, 14, 16, 15, 81, 107, 77, 2] |  | MRI | Brain | 4500 | Manual | 3 | Tumor
2023 | HaN-Seg [113] |  | CT, MRI | Head & Neck | 56 CT, 56 MRI | Manual | 30 | Multi-Organ
2023 | FH-PS-AOP [100] |  | Ultrasound | Transperineal | 6224 | Exp.+Mdl. | 2 | Multi-Organ

###  3.1 Versatile Medical Image Modalities

Imaging modality is one of the most critical property of a medical image,
which has a great impact on developing multi-task models [101]. While in
general domain “image” itself is a modality in comparison with text or audio,
the “modality” of medical images is defined by the mechanisms they were
produced, e.g. CT, MRI, X-Ray, etc.. Because the images of each modality are
constructed by sensing different signals (such as X-Ray, ultrasound, magnetic
resonance) to reflecting the distribution of tissues in human body, the
modalities are significantly different in terms of dimensionality, resolution,
noise level, number of channels, etc.. In general, medical imaging modalities
can be divided into the following categories.

  * •

3D Images: CT, MRI

  * •

2D Images:

    * –

Clinical: X-Ray, ultrasound, camera images (endoscope, fundus, dermoscope,
etc.)

    * –

Pathology

In Figure 2(a) we show the distribution of the medical imaging modalities in
the collection of public datasets. Among all the modalities, X-Rays composes
the largest portion in terms of number of scans. It is worth-noting that over
80% of the X-Ray data were curated during the COVID-19 pandemic, thanks to the
effort by numerous researchers in response to the public health crisis.
Ultrasound and camera imaging take similar portion of the 2D images. In
contrast, pathology images only take a tiny portion of 0.16%. Compared to
other types of images obtained in clinical practices, pathology images are
less in availability and harder for manual annotation, making the development
of high accuracy pathological models much more difficult.

The two 3D imaging modalities, CT and MRI, are nearly equal in their
percentage. Although the total number of 3D images only takes about one ninth
in terms of number of scans, each of them contains a number of 2D slices at
the scale of 10-100. When the 3D data are used for models like SAM which take
in 2D images, the most common practice is to use the 2D slices for training
and evaluation, giving the 3D images much higher weights in developing multi-
task models. On the other hand, the 2D slices from the same scan share spatial
connection, posing extra challenge to adapt SAM to a higher dimension. The
resolution and the spatial distance between the slices also vary across images
and datasets (see Table 5) in 3D adaptation.

(a) Data distribution by imaging modality

(b) Data distribution by segmentation target

Figure 2: The distribution of medical image segmentation data by number of
scans/images in the collection of public datasets in Table 1. Inclusion
criteria of the pie charts are: 1. the images are real world data, 2. the
annotation process involved domain experts.

###  3.2 Fine-grained Medical Segmentation Tasks

While general domain segmentation tasks are usually divided into semantic,
instance, or panoptic segmentation, medical image segmentation is more task-
tailored [11]. Specifically:

  * •

Segmentation is usually performed on a fixed vision field (certain part of the
human body to perform the scan).

  * •

The aim is to extract certain target(s) rather than segmenting the whole
vision field.

In general, the segmentation targets can be divided as 111Pathology images in
principle are different from other types of clinical images with segmentation
target at sell level. Their population is also tiny in the datasets, thus
requires separate discussion.:

  * •

Organs: the normal tissues parts of the body that are persistent for a certain
scan field. The target(s) can be: a single organ, multiple organs in the scan
field, and specific parts inside an organ.

  * •

Lesion: the abnormality in the body, e.g. tumor, infection, and various types
of other lesions.

The distribution of segmentation targets in the datasets is shown in Figure
2(b), with detailed statistics calculated for 2D and 3D images separately.
Lesion data outnumber organ data for both 2D and 3D images, but the fine-
grained distribution of the two types of modalities differs a lot. 2D images
typically exhibit coarser presentation of the tissues in body, thus are good
to segment single large organs like lung, but not less used to show multiple
organs or finer structures inside an organ. On the lesion side, the majority
of the data are for infection segmentation due to the large number of lung
infection data for chest X-Ray.

On the other hand, 3D images are able to provide a full reconstruction of the
spatial structure in the human body, thus enables fine-grained segmentation
like multi-organ and organ parts segmentation. On the lesion side, most of the
segmentation targets are various types of tumor, as the 3D imaging
technologies like CT and MRI are widely used to identify and measure the
tumor. In application, having a good estimate of the size of the tumor is
critical for cancer diagnosis and staging, demanding precise segmentation
tools.

##  4 Adaptation of SAM to the Medical Domain

Following the introduction of SAM on April 5, 2023, as illustrated in Figure
1, there has been a marked uptrend in its adoption in the medical domain.
Through a systematic review and analysis of the existing works, we identify
and categorize the adaptation of SAM into four primary methodologies.

Figure 3: Application of SAM Across Medical Imaging Modalities. The figure showcases Radiology, Pathology, and Camera Imaging examples. Central components of SAM, including the Image Encoder, Mask Decoder, and Prompt Encoder, are delineated. Methods ranging from Zero-shot Evaluation to Knowledge Distillation are accentuated within tan boxes. Figure 4: Decomposition of SAM Adaptation Methods in Medical Imaging. An illustrative overview of various adaptation strategies of SAM for the medical domain. The figure showcases five key methodologies: Zero-shot Evaluation, which assesses SAM’s inherent ability for medical image segmentation; Adapter, Projection, and Full Tuning, which represent different degrees of model fine-tuning; 3D Extension, highlighting SAM’s adaptation for volumetric data; and Knowledge Distillation, where SAM’s expertise is transferred to a student model. Each method’s flow from input to output, accompanied by specific components and modules, is visualized for clarity. Table 2: A comprehensive list of SAM-Related approaches. Proposed Perspectives | Years: Methods
---|---
Zero-Shot | Robustness Evaluation |  2023 April: [40], [123], [169], [61], [56], [105], [131], [71], [147], [30], [70], 2023 May: [104], [62], [69] 2023 June: [3], [64], [22] 2023 July: [26] 2023 August: [147], 2023 September: [117]
Image / Prompt Augmentation |  2023 April: [166], [158] 2023 June: [129], [89] 2023 July: [39], [36] 2023 August: [20], [155], [159]
Clinical Applications |  2023 April: [99], [148], [114], [108] 2023 May: [84], [140] 2023 June: [163], [127] 2023 July: [29], [60], [90]
Tuning | Adapter |  2023 April: [28], [162], 2023 June: [46], [149] 2023 August: [133]
Projection |  2023 April: [162] 2023 June: [64], [59] 2023 July: [35], [161], [132] 2023 August [44], [164]
Full |  2023 April: [63] 2023 May: [94] 2023 June: [59] 2023 July: [132] 2023 August: [112]
Multi-Dimensional Adaptation | 3D Extension |  2023 June: [48] 2023 September: [23], [25], [79]
Knowledge Distillation |  Weakly/Semi-Supervised Learning with Pseudo Prior |  2023 May: [54], [93] 2023 June: [78], [91] 2023 July: [35], [161] 2023 August: [66], [167]
Table 3: Overview of various existing works that build upon SAM.

Year-Month |  Method Tasks | 2D | 3D | A.P.P | P.A | E.Frozen | E.Finetune | R.N.M | T.P.H | T.P.E | T.A | Downstream Tasks
---|---|---|---|---|---|---|---|---|---|---|---|---
2023-April | SAM-Adaptor [28] |  2714 | - | - | - |  2714 | - | - | - | - |  2714 | Polyp
2023-April | SAMAug [166] |  2714 | - | - |  2714 |  2714 | - |  2714 | - | - | - | H&E, Polyp
2023-April | MedSAM Adaptor [154] |  2714 |  2714 | - | - | - | - |  2714 | - | - |  2714 | Abd, Opt, B.T, T.N
2023-April | LOSAM [166] |  2714 | - | - | - |  2714 | - | - | - |  2714 | - | Vessel & Lesion
2023-April | SAMed [162] |  2714 | - | - | - |  2714 | - | - |  2714 |  2714 |  2714 | Abd
2023-April | GazeSAM [148] |  2714 | - |  2714 | - |  2714 | - | - | - | - | - | Abd
2023-April | SkinSAM [63] |  2714 | - | - |  2714 | - |  2714 | - | - | - | - | S.L
2023-April | PiClick [158] |  2714 | - | - | - |  2714 | - | - | - | - | - | Neural Tissue
2023-May | Polyp-SAM [94] |  2714 | - | - | - | - |  2714 | - | - |  2714 | - | Polyp
2023-May | SAM-Track [31] |  2714 | - | - |  2714 |  2714 | - | - | - | - | - | Brain
2023-May | WS-SAM [54] |  2714 | - |  2714 | - |  2714 | - |  2714 | - | - | - | Polyp
2023-May | BreastSAM [62] |  2714 | - | - |  2714 |  2714 | - | - | - | - | - | Breast C.
2023-May | LuSAM [69] |  2714 | - | - | - |  2714 | - | - | - | - | - | Lung
2023-May | IAMSAM [84] |  2714 | - | - | - |  2714 | - | - | - | - | - | H & E
2023-June | DeSAM [46] |  2714 | - | - | - |  2714 | - | - | - |  2714 |  2714 | Prostate
2023-June | AutoSAM(1) [128] |  2714 | - | - | - |  2714 | - | - | - |  2714 | - | H & E, Polyp
2023-June | TEPO [129] |  2714 | - |  2714 | - |  2714 | - |  2714 | - | - | - | Brain
2023-June | RASAM [163] |  2714 | - | - | - |  2714 | - | - | - | - | - | Organ-at-risk
2023-June | 3DSAM-adaptor [48] | - |  2714 | - |  2714 |  2714 | - | - | - |  2714 |  2714 | Parts Tumor
2023-June | AutoSAM(2) [64] |  2714 | - | - | - |  2714 | - | - |  2714 | - | - | Cardiac Structure
2023-June | MedLSAM [89] |  2714 | - |  2714 | - |  2714 | - |  2714 | - | - | - | H & N, Abd, Lung
2023-June | CellViT [59] |  2714 | - | - | - | - |  2714 | - |  2714 | - | - | H & E
2023-July | SAM-U [39] |  2714 | - | - |  2714 |  2714 | - | - | - | - | - | Opt
2023-July | SAM${}^{\text{Med}}$ [149] |  2714 | - |  2714 |  2714 |  2714 | - | - | - | - |  2714 | Abd, Prostate
2023-July | SAMAug [36] |  2714 | - | - |  2714 |  2714 | - | - | - | - | - | Polyp, Lung
2023-July | All-in-SAM [35] |  2714 | - |  2714 |  2714 |  2714 | - | - |  2714 |  2714 | - | H & E
2023-July | SAM-Path [161] |  2714 | - | - |  2714 |  2714 | - | - |  2714 |  2714 | - | H & E
2023-July | CmAA [132] |  2714 | - | - | - |  2714 | - | - |  2714 | - | - | Glioma
2023-July | MedSAM [101] |  2714 | - | - | - | - | - |  2714 | - | - | - | 15 I.M, >30 C.T
2023-August | SAM-MLC [66] |  2714 | - |  2714 | - |  2714 | - |  2714 | - | - | - | Lung
2023-August | AdaptiveSAM [112] |  2714 | - | - |  2714 | - |  2714 | - |  2714 |  2714 | - | S.S
2023-August | Poly-SAM++ [20] |  2714 | - | - |  2714 |  2714 | - | - | - | - | - | Polyp
2023-August | SPSAM [155] |  2714 | - |  2714 |  2714 |  2714 | - | - | - |  2714 | - | Polyp, S.L
2023-August | SamDSK [167] |  2714 | - | - | - |  2714 | - |  2714 | - | - | - | Polyp, S.L, Breast C.
2023-August | AutoSAM Adaptor [90] | - |  2714 | - |  2714 |  2714 | - | - | - |  2714 |  2714 | Abd
2023-August | SAM-Med2D [30] |  2714 | - | - | - |  2714 | - | - | - |  2714 |  2714 | 9 MICCAI2023
2023-August | SAMedOCT [43] |  2714 | - | - | - |  2714 | - |  2714 | - | - | - | OCT
2023-September | SAM3D [23] |  2714 | - | - | - |  2714 | - | - |  2714 | - | - | Brain Lung,, Abd
2023-September | SAMUS [95] |  2714 | - | - | - |  2714 | - |  2714 | - | - | - | Ultrasound
2023-September | MA-SAM [25] | - |  2714 | - | - |  2714 | - | - | - | - |  2714 | Abd, Prostate, S.S
2023-September | MedVISTA-SAM [25] |  2714 |  2714 | - | - |  2714 | - | - | - | - |  2714 | Echocardiography

<Acronym: Meaning> A.P.P:Adapt Psuedo Prior; PA:Prompt Augmentation;
E.:Encoder; R.N.M: Retrain New Model; T.P.H:Train Projection Head; T.P.E:Train
Prompt Encoder; T.A:Train Adaptor; Abd:Abdomen; Opt:Optic; B.T:Brain Tumor;
T.N:Thyroid Nodule; I.M: Imaging Modalities; C.T: Cancer Types; Per.:
Peripheral; C.: Cancer

###  4.1 Zero-shot Segmentation Capabilities Evaluation

The zero-shot evaluation illuminates SAM’s ability to adapt to medical imaging
tasks without any prior domain-specific training [80]. Such zero-shot
capabilities primarily assesses whether SAM, known for its proficiency in
general domain image processing, can effectively generalize its learned
patterns to the nuanced and variable-rich world of medical imaging.

Medical imaging presents unique challenges, distinguished by factors like
varied imaging protocols and a wide range of patient demographics [40, 123,
169, 70, 62]. These complexities are not as predominant in standard domain
images, making SAM’s adaptability in this context particularly intriguing.
Comprehensive quantitative analyses have been conducted to compare SAM’s
zeroshot results with current SOTA techniques in various target areas and
regions. For detailed performance metrics and observations, readers are
referred to Section 5.1.

###  4.2 Domain-Specific Tuning

The versatility of SAM in instance segmentation, while noteworthy, has
demonstrated varying results across different contrast appearances and organ
morphologies inherent to medical images. To address this, researchers have
explored several domain-specific tuning strategies. This section provides an
objective overview of three prominent strategies:

####  4.2.1 Projection Tuning

One approach seeks to capitalize on the rich knowledge obtained from natural
image domains. By replacing the pretrained decoder with a new, task-specific
projection head, the model aims to harness generalized features (e.g., edges,
boundaries) that are prevalent in datasets like ImageNet. As shown in Figure
4, recent works propose to replace the projection head into various
architectures and train it from scratch, such as multi-layer perceptrons
(MLPs), convolutions, or vision transformers [162, 64, 44]. While this
strategy aims to maintain knowledge from the natural image domain, its
effectiveness can be contingent upon the quality of the prompt, especially
when dealing with significant domain shifts inherent to multi-modal images
[40, 35].

####  4.2.2 Adapter Tuning

To bolster the adaptability of SAM across various imaging modalities, recent
research has focused on incorporating adapter modules within the SAM encoder
blocks, as illustrated in Figure 4. These adapters, integrated without
modifying the core architecture, are designed to fine-tune the model’s
response to the specific challenges presented by medical imaging [28, 46, 149,
133]. This arrangement facilitates a blend of pre-existing knowledge from
natural images with nuanced, task-specific insights from the medical domain.
Yet, as indicated in Table 2, there have been relatively few studies that have
developed these adapter modules from the ground up specifically for domain
adaptation [132]. Consequently, the optimal configuration, placement, and
overall impact of these adapters within the SAM framework continue to be
subjects of active exploration in the field.

####  4.2.3 Full Tuning

Refining the entire SAM architecture to more closely resonate with the
subtleties of medical imaging represents a comprehensive strategy. This
approach entails not just minor adjustments but a substantial reconfiguration,
fine-tuning both the encoder and decoder of SAM to transition its generalized
knowledge base, originally derived from natural images, to a context more
specific to the medical domain. While this method promises a deeper
integration with the unique characteristics of medical data, it comes with the
caveat of potentially requiring more intensive training efforts and greater
resource allocation. Moreover, recent studies have highlighted the
difficulties in fine-tuning SAM’s architecture, especially when dealing with a
limited dataset . Thus, identifying an optimal approach to adapt and leverage
SAM’s foundational structure for medical applications, in a manner that
efficiently incorporates medical knowledge, remains a key challenge.

###  4.3 3D Imaging Modalities Extension

In the field of medical imaging, most protocols generate imaging modalities in
three-dimensional (3D) format. To align with SAM’s two-dimensional (2D)
framework, axial slices are typically processed to fit a 2D context, with
predictions made on a slice-by-slice basis for 3D images. This approach,
however, does not fully account for the spatial relationships between slices,
often resulting in imprecise organ boundary alignments. To mitigate this,
pioneering efforts have adopted a 2.5D approach, incorporating additional
convolutional modules that compute through-plane features to enhance the
spatial correlation between slices within the SAM encoder blocks [154, 48].
Yet, this method still focuses on axial and through-plane spatial correlations
on a slice-by-slice basis rather than establishing a comprehensive volumetric
correspondence. An alternative strategy involves the amalgamation of all slice
representations into a single volumetric map, followed by the adaptation of
spatial correlations at the feature level using a 3D decoder network [23].
Despite this advancement, the encoder network continues to depend on the
static weights from SAM’s original encoder, which limits robustness when
compared to various state-of-the-art transformer networks.

###  4.4 Knowledge Distillation

Beyond the intricacies of model architecture, leveraging SAM’s predictive
output as a form of prior knowledge has become an emerging trend in medical
imaging. Recent studies typically begin by generating a rough segmentation
mask using SAM, guided by carefully crafted prompts. However, using these
coarse masks as pseudo labels presents a challenge when training a more
accurate ’student’ model. To address this, an auxiliary label refinement
network is often employed to improve the quality of these initial segmentation
labels [66, 35]. The enhanced labels can then be used to train a task-specific
model from the ground up. Moreover, SAM’s zero-shot learning capabilities
markedly increase the efficiency of generating pseudo labels from vast
quantities of unlabeled data. Semi-supervised methodologies incorporating SAM
are also being explored to bolster the robustness of task-specific models
[167]. As such, in the realm of weakly and semi-supervised learning, the
distillation of pixel-level insights has become a crucial focus, enabling the
efficient utilization of large-scale datasets in model training.

##  5 SAM’s Performance on Medical Imaging Segmentation

In parallel, it is also equally crucial to explore its adaptive performance
across different anatomical regions and scales. Both the research community
and practicing clinicians are particularly keen on ensuring consistent and
reliable qualitative performance, especially when evaluating different
anatomical regions or adapting different imaging modalities. The distinctive
benefit of the concept "zero-shot segmentation" has been pivotal to propel
more intensive research, aiming to juxtapose the qualitative performance of
current SOTA approaches against the pre-trained SAM. Table 4, 6 and 7 present
a bird’s-eye view of these qualitative assessments across a spectrum of
imaging modalities. To dissect and understand these patterns more
comprehensively, we have organized the evaluation into two distinct macro-to-
micro aspects: 1) imaging modalities & downstream tasks and 2) methodologies
with SAM’s foundation.

###  5.1 Macro Aspect: Imaging Modalities & Downstream Tasks

Radiology Imaging: We begin our investigation with 2D radiology images,
categorizing them into two main modalities: OCT and X-ray. Figure
LABEL:fig:data-piechart reveals that X-ray images constitute a significant
portion of the data distribution assessed by SAM. However, this modality
primarily targets the chest area, delving deeper into specific lung conditions
such as lung opacity, viral pneumonia, and tuberculosis, as detailed in Table
4. Conversely, there are limited studies showcasing SAM’s effectiveness with
OCT imaging. When comparing this to 3D radiology imaging, we find it
intriguing that both CT and MR modalities encompass nearly the entirety of
anatomical regions, contrasting the specific focuses of X-ray and OCT.
Moreover, specific anatomical regions, including the brain, head & neck, and
abdomen, are frequently targeted. Given the diverse imaging protocols,
distinct boundaries are evident between different tissues and organs. As a
result, SAM demonstrates commendable performance across various organs of
interest, adapting well to a range of prompt guidance, such as 1 box, 1 p.f,
and 2 p.b.

Pathology Imaging: From Table 6, we observe that initial studies for pathology
imaging have been focused on H & E modality across variable anatomical
regions, such as skin, kidney and colon. Comparing to radiology imaging, fine-
grained semantics in multi-scale resolution such as cells, nuclei and
membranes, are targeted in H & E imaging. However, with the substantial
sparsity between neighboring cells, prompt such as a large bounding boxes are
difficult to provide fine-grained guidance and localize the target regions for
segmentation. Therefore, we can observe that numerous points as the prompt,
demonstrate to have the best performance and generate a fair qualitative
performance in different scenarios. However, still a substantial gap in
qualitative performances between SAM and current SOTAs are demonstrated in
most of the target cases.

Camera Imaging: As camera images are reconstructed in 2D, similar to the
format in natural images, SAM demonstrates a good transferrable capability to
segment targeted regions with clear boundaries. Most of the semantic targets
in Table 7 are easy to localize within the images and demonstrate a fair
quantitative performance such as surgical instruments (Dice: 0.893) and polyp
(Dice: 0.872). However, we observe that SAM demonstrates limited robustness to
segment fundus vasculature in retinal fundus images. The point prompt is
limited to provide an extensive guidance to localize the continuity of the
semantic structure, similar to the optic nerve segmentation in head & neck MR
images.

###  5.2 Micro Aspect: Methodologies with SAM’s Foundation

With Table 4, 6 & 7, we already have a general view on the zero-shot
performance of SAM with different imaging modalities and fine-grained tasks.
However, we can observe that the quantitative robustness is still limited with
the general knowledge prior from natural image domain and the specific prompts
guidance. Therefore, researchers begin to provide an alternative that adapts
each independent module (e.g., encoder network, mask encoder, prompt encoder)
as the foundation architecture and fine-tune for better performance. Table 3
demonstrates an overview of variable model architectures with SAM’s module as
the foundation basis from April 2023 to August 2023. We further summarize our
observation in Table 4 as four folds:

Image Dimensionality: The majority of previous studies have processed 16-bit
volumetric images (such as CT and MRI scans) into 8-bit 2D slices for
downstream segmentation tasks. However, a minority have begun to adapt SAM to
handle 3D volumetric data. Proposals include improving spatial correspondence
between slices by adding modules to the encoder and decoder networks that can
extract features from through-plane axes (such as the x-z and y-z planes, see
[154, 48]). These extracted features are then either combined with axial
features or used to create a 3-dimensional volume through depthwise
convolution for feature integration. In addition to module adaptation, another
emerging approach is to concatenate slice-wise embeddings into a volumetric
representation and integrate this with the decoder network to maintain spatial
coherence in segmentation tasks. Given the inherently three-dimensional nature
of radiological images, there is a clear demand for direct segmentation of 3D
structures. Extending SAM to accommodate 3D inputs shows promising potential
for real-world clinical application.

Data/Prompt Augmentation: Another perspective to enhance SAM’s performance is
the adaptation of prompts and the generated segmentation prior from SAM.
Instead of following the traditional human-in-the-loop prompts, multiple works
are proposed to generate multiple pseudo prompts (e.g., point prediction [36],
bounding box prediction [89]) with deep neural networks and apply it to SAM as
an end-to-end framework. Particularly for pathology images, numerous cells
exists within a gigapixel images and it is difficult to localize multiple
cellular regions with a single point / bounding box prompt only. Therefore,
multi-instance learning is initially performed to provide a coarse
localization on different cells. Such instance segmentation label are then
converted into circle-like bounding boxes for prompting and demonstrate a
significant enhancement in cell segmentation performance [35]. Another
augmentation technique is to additionally learn the outputs from pretrained
SAM. While the segmentation quality is fair, multiple works are proposed to
train a label refinement model with the coarse SAM output and leverage the
refined output for additional training [35, 66]. With the introduction of
label refinement idea, weakly-supervised learning may be another potential
alternative to train a model from scratch with good quality pseudo labels.

Encoder w/o Finetuning: Beyond the scope of data preparation, considerable
efforts have been devoted to developing innovative training strategies using
SAM as the foundational model for downstream segmentation tasks. While most
recent research has focused on utilizing the pretrained, frozen encoder
weights of SAM to accommodate medical images, only a handful of studies have
explored the full potential of fine-tuning the encoder network. Given that the
encoder has been extensively trained on natural images, acquiring a vast
repository of prior knowledge, the prevailing wisdom is to retain this
knowledge and enrich it with domain-specific insights from medical images to
boost performance. Nevertheless, the peculiarities of different imaging
domains present a unique set of challenges, making the establishment of a
universally optimal fine-tuning approach elusive. This dilemma extends to the
decision of whether to fine-tune the entirety of the encoder network or to
selectively fine-tune specific intermodular components within the encoder
block.

Additional Training: A considerable volume of recent research has been
directed at integrating SAM with a frozen encoder network, where additional
modules are introduced to either substitute specific components (such as the
decoder network or prompt encoder) or to implement an adaptor for translating
representations between encoder layers. Illustrated in Figure 4, there is a
discernible uptick in the number of studies employing freshly trained
projection head modules between June and August 2023. In contrast, there is a
noticeable decline in strategies that focus on training adaptor modules on a
per-layer basis. Notably, in the realm of pathology imaging, there is a
predilection towards retraining both the prompt encoder and projection
network. Meanwhile, for radiology imaging, the preference shifts to training
adaptor modules that operate on distinct blocks of layers. Although
traditional prompts have proven to be efficient for organ segmentation tasks,
they falter when tasked with adapting to multi-scale semantic segmentation
(such as cells and tissues). Consequently, the initial trends differ markedly
between imaging modalities.

Table 4: Quantitative Comparisons between the Zero-Shot Performance of SAM and Current State-Of-The-Art (SOTA) Approaches on Different Radiology Datasets. (p.f: foreground point, p.b: background point) Dim. | Modality | Region | Targets | Performance | Prompt Mode
---|---|---|---|---|---
SOTAs | MedSAM | SAM
3D |  CT | Brain | Intracranial Hemorrhage | 0.795 [92] | 0.940 | 0.867 [101] | 1 p.f, 2 p.b
Glioblastoma | 0.913 [52] | 0.943 | 0.744 [101] | 1 p.f, 2 p.b
Head & Neck | Head-Neck Cancer | 0.788[9] | 0.794 | 0.614 [101] | 1 p.f, 2 p.b
Lymph Nodes | 0.742 [50] | 0.821 | 0.771 [101] | 1 p.f, 2 p.b
Throat Cancer | 0.667 [12] | 0.803 | 0.281 [101] | 1 p.f, 2 p.b
Abdomen | Pancreas & Tumor | 0.828, 0.623 [97] | 0.872, 0.791 | 0.731, 0.741 [101] | 1 p.f, 2 p.b
Liver & Tumor | 0.950, 0.790 [97] | 0.980, 0.887 | 0.916, 0.766 [101] | 1 p.f, 2 p.b
Spleen | 0.974 [67] | 0.976 | 0.938 [65] | 1 box
Kidney & Tumor | 0.948, 0.763 [88] | 0.971, 0.902 | 0.947, 0.867 [101] | 1 p.f, 2 p.b
Aorta | 0.956 [87] | 0.956 | 0.912 [101] | 1 p.f, 2 p.b
Esophagus | 0.861 [87] | 0.737 | 0.845 [56] | 1 box
Stomach | 0.921 [87] | 0.962 | 0.855 [101] | 1 p.f, 2 p.b
Gallblader | 0.921 [87] | 0.918 | 0.872 [56] | 1 box
IVC | 0.924 [87] | 0.918 | 0.897 [56] | 1 box
|  | Adrenal Gland | 0.798 [87] | 0.661 | 0.742 [56] | 1 box
2-7 |  MR |  Brain | Brainstem | 0.860 [113] | 0.971 | 0.692 [101] | 1 p.f, 2 p.b
| Cerebellum | 0.915 [139] | 0.968 | 0.765 [101] | 1 p.f, 2 p.b
| Deep Grey Matter | 0.974 [67] | 0.956 | 0.496 [65] | 1 p.f, 2 p.b
| ventricles | 0.872 [86] | 0.900 | 0.639 [101] | 1 p.f, 2 p.b
| Glioma | 0.878, 0.928 [118] | 0.944, 0.962 | 0.763 (T1), 0.834 (FLAIR) [101] | 1 p.f, 2 p.b
| Glioma Enhancing Tumor | 0.956 [87] | 0.952 | 0.788 [101] | 1 p.f, 2 p.b
| Glioma Tumor Core | 0.956 [87] | 0.959 | 0.710 [101] | 1 p.f, 2 p.b
| Ischemic Stroke | 0.964 [58] | 0.923 | 0.613 [101] | 1 p.f, 2 p.b
| Meningioma | 0.946, 0.892 [118] | 0.979, 0.970 | 0.921 (T1-CE), 0.792 (T2-FLAIR) [101] | 1 p.f, 2 p.b
|  | Vestibular Schwannoma | 0.925 [41] | 0.952 | 0.853 [101] | 1 p.f, 2 p.b
|  Head & Neck | Eye PL | 0.930 [67] | 0.941 | 0.815 [101] | 1 p.f, 2 p.b
| Eye PR | 0.923 [67] | 0.940 | 0.819 [101] | 1 p.f, 2 p.b
| Optic Nerve | 0.699, 0.746 [67] | 0.613, 0.703 | 0.395 (L), 0.433 (R) [101] | 1 p.f, 2 p.b
| Bone Mandible | 0.944 [67] | 0.697 | 0.543 [101] | 1 p.f, 2 p.b
| Cricopharyngeus | 0.632 [67] | 0.902 | 0.614 [101] | 1 p.f, 2 p.b
| Glnd Lacrimal | 0.631, 0.621 [67] | 0.640, 0.687 | 0.613 (L), 0.599 (R) [101] | 1 p.f, 2 p.b
| Glnd Submand | 0.848, 0.840 [67] | 0.913, 0.909 | 0.779 (L), 0.797 (R) [101] | 1 p.f, 2 p.b
| Parotid | 0.871, 0.856 [67] | 0.917, 0.916 | 0.727 (L), 0.714 (R) [101] | 1 p.f, 2 p.b
| Glottis | 0.752 [67] | 0.850 | 0.301 [101] | 1 p.f, 2 p.b
|  | Larynx SG | 0.814 [67] | 0.882 | 0.540 [101] | 1 p.f, 2 p.b
|  | Lips | 0.722 [67] | 0.869 | 0.584 [101] | 1 p.f, 2 p.b
|  Abdomen | Left Kidney | 0.921 [68] | 0.948 | 0.912 [101] | 1 p.f, 2 p.b
| Right Kidney | 0.927 [68] | 0.948 | 0.921 [101] | 1 p.f, 2 p.b
| Liver | 0.920 [68] | 0.957 | 0.902 [101] | 1 p.f, 2 p.b
| Spleen | 0.894 [68] | 0.948 | 0.910 [101] | 1 p.f, 2 p.b
|  Heart | Left Atrium | 0.933 [97] | 0.973 | 0.836 [101] | 1 p.f, 2 p.b
| Left Ventricle | 0.959 [143] | 0.985 | 0.775 [101] | 1 p.f, 2 p.b
| Right Ventricle | 0.926 [143] | 0.972 | 0.903 [101] | 1 p.f, 2 p.b
| Artery Carotid | 0.874, 0.833 [152] | 0.620, 0.627 | 0.578 (L), 0.610 (R) [101] | 1 p.f, 2 p.b
| Whole Heart | 0.867 [19] | 0.963 | 0.521 [101] | 1 p.f, 2 p.b
|  Prostate | Prostate | 0.831 [67] | 0.985 | 0.872 [101] | 1 p.f, 2 p.b
| Prostate Cancer | 0.800 [126] | 0.969 | 0.693 [101] | 1 p.f, 2 p.b
|  |  Spine | Spine | 0.952 [125] | 0.918 | 0.808 [101] | 1 p.f, 2 p.b
|  | SpinalCord | 0.891 [165] | 0.774 | 0.559 [101] | 1 p.f, 2 p.b
2D | OCT | Eye | Diabetic Macular Edema | 0.983 [144] | 0.950 | 0.884 [101] | 1 p.f, 2 p.b
X-Ray | Chest | Heart | 0.950 [145] | 0.968 | 0.901 [101] | 1 p.f, 2 p.b
Lung | 0.979 [145] | 0.991 | 0.933 [101] | 1 p.f, 2 p.b
Viral Pneumonia | 0.992 [51] | 0.984 | 0.892 [101] | 1 p.f, 2 p.b
Pneumothorax | 0.891 [1] | 0.815 | 0.502 [101] | 1 p.f, 2 p.b
Tuberculosis | 0.978 [98] | 0.969 | 0.939 [101] | 1 p.f, 2 p.b
COVID-19 | 0.971 [47] | 0.989 | 0.782 [101] | 1 p.f, 2 p.b
Breast | Breast Cancer | 0.963 [8] | 0.833 | 0.665 [101] | 1 p.f, 2 p.b

##  6 Discussion

The continuous progression in SAM’s application to medical image segmentation
elucidates its potentiality in bridging the technical gap between general and
domain-specific applications. As the medical imaging community gravitates
towards the capabilities of SAM, it becomes paramount to reflect upon the
findings, draw analytical conclusions, and understand the multifaceted
challenges and requirements particular to this domain. In this section, we
delve deep into the limitations of current SAM adaptation methodologies and
highlight the distinctive requirements of medical image segmentation, offering
a comprehensive perspective for future endeavors.

###  6.1 Limitations in Current SAM Adaptation Approaches

As we scrutinize the implementation of SAM in the realm of medical image
segmentation, certain limitations inherent to the current adaptation
methodologies surface:

Generalization Discrepancy. While SAM is adept at adjusting to a variety of
tasks involving large-scale natural images, the medical imaging domain
presents unique challenges, such as patient population variability and extreme
pathological conditions. In addition, the trend in recent medical AI research
leans towards the development of models specialized for particular
applications—for instance, brain tumor segmentation or lung lesion
identification—due to the highly specific nature of the knowledge required for
each task. This specialization trend underscores the distinct knowledge
domains in medical tasks as opposed to the more generalizable learning
applicable to natural images. Thus, bridging the gap between SAM’s general
effectiveness with natural imagery and its potential in addressing the
specialized demands of medical imaging is an unresolved issue that continues
to be a focal point of research.

Fine-tuning Dilemma. From Table 2, we can observe that limited recent works
have been proposed to finetune the encoder network in SAM. As SAM consists of
rich semantic knowledge from large-scale natural images, it is difficult to
add on the medical domain knowledge with such foundation basis with limited
ground-truth labeled samples. Furthermore, with the task-specific discrepancy
in model training, the risk of overfitting or knowledge dilution persists and
establishing an optimal strategy that balances the strengths of pre-trained
weights with domain-specific adaptation remains nebulous.

Modality Inconsistencies. With the wide range of contrast intensity across
volumetric inputs (e.g, whole images, organ-specific patches), 16 bit
volumetric context are abstracted into 8-bit 2D slices (i.e. 0 to 255) as the
model input and demonstrate the unify capability of performing downstream
segmentation across imaging modalities and anatomical regions. However, from
Table 4, we observe that the segmentation performance varies substantially and
the risk of overfitting or knowledge dilution towards specific modalities may
persist.

###  6.2 Specific Requirement of Medical Image segmentation

####  6.2.1 Incorporation of Metadata

Medical imaging is unique in that the raw visual data is often supplemented by
a plethora of metadata, spanning patient demographics, clinical history, and
technical imaging specifications. This auxiliary information, while pivotal
for clinical decisions, can also enhance the richness of SAM’s segmentation
outcomes.

The heterogeneous nature of medical metadata presents a formidable challenge.
Differences in metadata format, magnitude, and relevance across different
databases and healthcare institutions necessitate the development of
sophisticated normalization and preprocessing pipelines. Successfully
integrating this data demands a deep understanding of the interplay between
visual features and metadata, ensuring enhancements without unintentional
biases. While preliminary attempts have been made in this direction, the
establishment of universally-accepted, metadata-assimilation strategies
harmonious with SAM’s segmentation algorithms is an ongoing pursuit.

####  6.2.2 Population Analysis

The realm of medical imaging mandates a departure from an image-centric
approach, propelling a shift towards population-based analyses. Here, images
are interconnected fragments of larger datasets, which, when viewed in
ensemble, unravel the intricate web of demographics, health trajectories, and
treatment pathways.

This transition, while invaluable, is riddled with complexities. Some
pathologies may predominantly affect certain demographic segments, while
others may manifest diversely across age groups. Time-lapsed images from a
patient can chronicle the evolution of ailments, offering insights into
disease progression or therapeutic effectiveness. Ensuring SAM captures such
population-driven subtleties, without being mired in the cacophony of
individual variations, is a challenge warranting targeted research.

###  6.3 Prospective Horizons

As we reflect upon the current landscape of SAM’s adaptation for medical image
segmentation, it is evident that the journey, while marked by promising
strides, is still in its nascent stages. From addressing the generalization
discrepancy and fine-tuning dilemmas to navigating the intricacies of metadata
incorporation and population analysis, the path forward is laden with
opportunities for innovation. With such horizon, we foresee concerted efforts
towards making SAM’s application in medical segmentation more robust,
interpretable, and clinically aligned. Here, we can summarize the potential
opportunity with SAM as follows:

####  6.3.1 Segmenting Unseen Classes

The current landscape of publicly available medical labels is heavily skewed
towards specific organs or tissues, which does not adequately reflect the
nuanced complexities of actual clinical environments. In response, researchers
are navigating two primary paths: first, by creating datasets that represent a
broader array of semantic categories, and second, by implementing incremental
learning methods. Yet, both strategies rely heavily on the procurement of new
annotations from a limited patient cohort for supplementary training,
perpetuating a reliance on data. The advent of zero-shot learning
capabilities, as exemplified by SAM, introduces a paradigm shift where human-
in-the-loop prompts can potentially delineate any structure with distinct
margins. Nevertheless, SAM’s ability to predict based on instance-level
semantics remains somewhat constrained. The emergence of CLIP has shown
promising strides in aligning textual semantics with visual features,
enhancing the ability to incorporate additional class information into
downstream image classification tasks. Leveraging SAM’s proficiency in
segmenting defined regions through varied prompts, there is potential to
synergize pixel-wise predictions with textual semantics. This approach could
enable the classification of novel categories without the necessity of extra
labeled data for training, thereby diminishing the model’s dependency on
extensive annotated datasets.

####  6.3.2 Enhancing Explainable Interpretability

Beyond considerations of data curation and training methodologies, SAM
operates as "black boxes," its internal workings shrouded by non-linear
complexities. This lack of transparency has catalyzed efforts to demystify
these mechanisms, notably through the implementation of visualization
techniques like feature map analysis and gradient mapping tools, such as
GradCAM, during the training process. These techniques create a tangible link
between the model’s predictions and the features it has learned, providing a
graphical exposition of how the model arrives at its conclusions. In tasks
related to classification, this transparency is further enriched by examining
the latent space—the realm within a network where the most complex features
reside. Here, features are organized into clusters that correspond to known
ground truth labels, which brings additional clarity to the model’s decision-
making process. The advent of prompt engineering introduces variable human-in-
the-loop prompts that promise to improve the intelligibility of the
relationship between feature representations and precise instance predictions.
Nevertheless, parsing out singular features that correspond to multiple
semantic categories, such as in multi-organ segmentation, remains a formidable
challenge. Refining the relationship between feature representations and
clinical outcomes, with SAM’s foundational model as a starting point,
represents a promising avenue toward enhancing the model’s applicability in
clinical settings. Such advances would contribute to the model’s ability to
provide more grounded, explainable results that align with clinical
interpretations.

##  7 Appendix

Table 5: Resolution of 3-D medical image segmentation datasets. x & y refers to the in-plane resolution of the pixels. z refers to the spacing between the slices. We show the range of the resolution for each dataset as it can vary for each scan in the dataset. Dataset | Modality | Anatomy | Resolution (mm)
---|---|---|---
x & y | z
BTCV [82] | CT | Abdomen | 0.594-0.977 | 2.5-5.0
Pancreas-CT [120, 121, 34] | CT | Pancreas | 0.664-0.977 | 1.5-2.5
LiTS [18] | CT | Liver & Tumor | 0.557-1.0 | 0.5-1.0
TotalSegmentator [153] | CT | Abdomen | 1.5 isotropic
FUMPE [103] | CT | Lung | - |  $\leq$ 1.5
VESSEL12 [124] | CT | Lung | 0.59-0.89 | 0.5-1.0
AbdomenCT-1K [102] | CT | Abdomen | - | 0.5-8.0
AMOS (CT) [72] | CT | Abdomen | - | 1.25-5.0
CHAOS (CT) [76, 74, 75] | CT | Abdomen | 0.7-0.8 | 3.0-3.2
HaN-Seg (CT) [113] | CT | Head & Neck | 0.51-1.56 | 2.0-3.0
MSD (Liver) [10] | CT | Liver | 0.5-1.0 | 0.45-6.0
MSD (Pancreas) [10] | CT | Abdomen | - | 2.5
ACDC [17] | MRI | Heart | 1.170-1.296 | 5.0-10.0
M&Ms [24] | MRI | Heart | 0.684-1.823 | 5.0-10.0
BraTS [73, 13, 106, 14, 16, 15, 81, 107, 77, 2] | MRI | Brain | 1.0 isotropic
NCI-ISBI [21] | MRI | Prostate | - | 3.0-4.0
PROMISE12 [96] | MRI | Prostate | 0.273-0.750, | 2.2-4.0
AMOS (MRI) [72] | MRI | Abdomen | - | 0.82-6.0
CHAOS (MRI) [76, 74, 75] | MRI | Abdomen | 1.36-1.89 | 5.5-9.0
HaN-Seg (MRI) [113] | MRI | Head and Neck | 0.47-0.82 | 3.0-5.0
MSD (Brain) [10] | MRI | Brain | 1.0 isotropic
MSD (Hippocampus) [10] | MRI | Brain | 1.0 isotropic
MSD (Heart) [10] | MRI | Heart | 1.25 | 2.7
MSD (Prostate) [10] | MRI | Prostate | 0.6 | 4.0
Table 6: Quantitative Comparisons between the Zero-Shot Performance of SAM and Current State-Of-The-Art (SOTA) Approaches on Different Pathology Datasets. Modality | Region | Tasks | Resolution | Performance | Prompt Mode
---|---|---|---|---|---
SOTAs | SAM
H & E | Skin | Tumor | 0.5 $\times$ | 0.720 [40] | 0.750 | 20 points
Kidney | Glomerular (CAP) | 5 $\times$ | 0.965 [40] | 0.801 | 20 points
Glomerular Tuft (TUFT) | 0.966 [40] | 0.799 | 20 points
Distal Tubular (DT) | 10 $\times$ | 0.810 [40] | 0.604 | 20 points
Proximal Tubular (PT) | 0.898 [40] | 0.666 | 20 points
Arteries (VES) |  | 0.851 [40] | 0.685 | 20 points
Peritubular Capillaries (PTC) | 40 $\times$ | 0.772 [40] | 0.646 | 20 points
Different Tumors | Nuclei | 0.818 [40] | 0.417 | 20 points
Colon | Adenocarcinoma, Benign Glands | 20 $\times$ | 0.797 [151] | 0.525 | classes prompt
Table 7: Quantitative Comparisons between the Zero-Shot Performance of SAM and Current State-Of-The-Art (SOTA) Approaches on Different Camera Datasets. Modality | Region | Tasks | Performance | Prompt Mode
---|---|---|---|---
SOTAs | SAM
Endoscopy | Abdomen | Cholecystectomy | 0.955 [101] | 0.718 | 1 p.f, 2 p.b
Instruments | 0.940 [56] | 0.893 | 1 box
Gastrectomy | 0.974 [101] | 0.815 | 1 p.f, 2 p.b
Colon | Polyp | 0.925 [56] | 0.872 | 1 box
Dermoscopy | Skin | Cancer | 0.906 [56] | 0.865 | 1 box
Fundus | Retina | Optic Disc | 0.983 [154] | 0.924 | 1 box
Fundus Vasculature | 0.809 [115] | 0.217 | 1 point

## References

  * [1] Ayat Abedalla, Malak Abdullah, Mahmoud Al-Ayyoub, and Elhadj Benkhelifa.  Chest x-ray pneumothorax segmentation using u-net with efficientnet and resnet architectures.  PeerJ Computer Science, 7:e607, 2021.
  * [2] Maruf Adewole, Jeffrey D Rudie, Anu Gbadamosi, Oluyemisi Toyobo, Confidence Raymond, Dong Zhang, Olubukola Omidiji, Rachel Akinola, Mohammad Abba Suwaid, Adaobi Emegoakor, et al.  The brain tumor segmentation (brats) challenge 2023: Glioma segmentation in sub-saharan africa patient population (brats-africa).  arXiv preprint arXiv:2305.19369, 2023.
  * [3] Mohsen Ahmadi, Masoumeh Farhadi Nia, Sara Asgarian, Kasra Danesh, Elyas Irankhah, Ahmad Gholizadeh Lonbar, and Abbas Sharifi.  Comparative analysis of segment anything model and u-net for breast tumor detection in ultrasound and mammography images.  arXiv preprint arXiv:2306.12510, 2023.
  * [4] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy.  Dataset of breast ultrasound images.  Data in brief, 28:104863, 2020.
  * [5] Sharib Ali, Mariia Dmitrieva, Noha Ghatwary, Sophia Bano, Gorkem Polat, Alptekin Temizel, Adrian Krenzer, Amar Hekalo, Yun Bo Guo, Bogdan Matuszewski, et al.  Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy.  Medical image analysis, 70:102002, 2021.
  * [6] Sharib Ali, Noha Ghatwary, Debesh Jha, Ece Isik-Polat, Gorkem Polat, Chen Yang, Wuyang Li, Adrian Galdran, Miguel-Angel González Ballester, Vajira Thambawita, et al.  Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge.  arXiv preprint arXiv:2202.12031, 2022.
  * [7] Sharib Ali, Debesh Jha, Noha Ghatwary, Stefano Realdon, Renato Cannizzaro, Osama E Salem, Dominique Lamarque, Christian Daul, Michael A Riegler, Kim V Anonsen, et al.  Polypgen: A multi-center polyp detection and segmentation dataset for generalisability assessment.  arXiv preprint arXiv:2106.04463, 2021.
  * [8] Mohammad Alkhaleefah, Tan-Hsu Tan, Chuan-Hsun Chang, Tzu-Chuan Wang, Shang-Chih Ma, Lena Chang, and Yang-Lang Chang.  Connected-segnets: A deep learning model for breast tumor segmentation from x-ray images.  Cancers, 14(16):4030, 2022.
  * [9] Vincent Andrearczyk, Valentin Oreiller, Sarah Boughdad, Catherine Cheze Le Rest, Hesham Elhalawani, Mario Jreige, John O Prior, Martin Vallières, Dimitris Visvikis, Mathieu Hatt, et al.  Overview of the hecktor challenge at miccai 2021: automatic head and neck tumor segmentation and outcome prediction in pet/ct images.  In 3D head and neck tumor segmentation in PET/CT challenge, pages 1–37. Springer, 2021.
  * [10] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.  The medical segmentation decathlon.  Nature communications, 13(1):4128, 2022.
  * [11] Reza Azad, Ehsan Khodapanah Aghdam, Amelie Rauland, Yiwei Jia, Atlas Haddadi Avval, Afshin Bozorgpour, Sanaz Karimijafarbigloo, Joseph Paul Cohen, Ehsan Adeli, and Dorit Merhof.  Medical image segmentation review: The success of u-net.  arXiv preprint arXiv:2211.14830, 2022.
  * [12] Xiaoyu Bai, Yan Hu, Guanzhong Gong, Yong Yin, and Yong Xia.  A deep learning approach to segmentation of nasopharyngeal carcinoma using computed tomography.  Biomedical Signal Processing and Control, 64:102246, 2021.
  * [13] Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al.  The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification.  arXiv preprint arXiv:2107.02314, 2021.
  * [14] S Bakas, H Akbari, A Sotiras, M Bilello, M Rozycki, J Kirby, J Freymann, K Farahani, and C Davatzikos.  Segmentation labels and radiomic features for the pre-operative scans of the tcga-gbm collection (2017).  DOI: https://doi. org/10.7937 K, 9.
  * [15] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin Kirby, John Freymann, Keyvan Farahani, and Christos Davatzikos.  Segmentation labels and radiomic features for the pre-operative scans of the tcga-lgg collection.  The cancer imaging archive, 286, 2017.
  * [16] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos.  Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features.  Scientific data, 4(1):1–13, 2017.
  * [17] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, Gerard Sanroma, Sandy Napel, Steffen Petersen, Georgios Tziritas, Elias Grinias, Mahendra Khened, Varghese Alex Kollerathu, Ganapathy Krishnamurthi, Marc-Michel Rohé, Xavier Pennec, Maxime Sermesant, Fabian Isensee, Paul Jäger, Klaus H. Maier-Hein, Peter M. Full, Ivo Wolf, Sandy Engelhardt, Christian F. Baumgartner, Lisa M. Koch, Jelmer M. Wolterink, Ivana Išgum, Yeonggul Jang, Yoonmi Hong, Jay Patravali, Shubham Jain, Olivier Humbert, and Pierre-Marc Jodoin.  Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved?  IEEE Transactions on Medical Imaging, 37(11):2514–2525, 2018.
  * [18] Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al.  The liver tumor segmentation benchmark (lits).  Medical Image Analysis, 84:102680, 2023.
  * [19] Benjamin Billot, Douglas N Greve, Oula Puonti, Axel Thielscher, Koen Van Leemput, Bruce Fischl, Adrian V Dalca, Juan Eugenio Iglesias, et al.  Synthseg: Segmentation of brain mri scans of any contrast and resolution without retraining.  Medical image analysis, 86:102789, 2023.
  * [20] Risab Biswas.  Polyp-sam++: Can a text guided sam perform better for polyp segmentation?  arXiv preprint arXiv:2308.06623, 2023.
  * [21] N Bloch, A Madabhushi, H Huisman, J Freymann, J Kirby, M Grauer, A Enquobahrie, C Jaffe, L Clarke, and K Farahani.  Nci-isbi 2013 challenge: Automated segmentation of prostate structures.  The Cancer Imaging Archive, 2015.
  * [22] Benedikt Blumenstiel, Johannes Jakubik, Hilde Kühne, and Michael Vössing.  What a mess: Multi-domain evaluation of zero-shot semantic segmentation.  arXiv preprint arXiv:2306.15521, 2023.
  * [23] Nhat-Tan Bui, Dinh-Hieu Hoang, Minh-Triet Tran, and Ngan Le.  Sam3d: Segment anything model in volumetric medical images.  arXiv preprint arXiv:2309.03493, 2023.
  * [24] Victor M Campello, Polyxeni Gkontra, Cristian Izquierdo, Carlos Martin-Isla, Alireza Sojoudi, Peter M Full, Klaus Maier-Hein, Yao Zhang, Zhiqiang He, Jun Ma, et al.  Multi-centre, multi-vendor and multi-disease cardiac segmentation: the m&ms challenge.  IEEE Transactions on Medical Imaging, 40(12):3543–3554, 2021.
  * [25] Cheng Chen, Juzheng Miao, Dufan Wu, Zhiling Yan, Sekeun Kim, Jiang Hu, Aoxiao Zhong, Zhengliang Liu, Lichao Sun, Xiang Li, et al.  Ma-sam: Modality-agnostic sam adaptation for 3d medical image segmentation.  arXiv preprint arXiv:2309.08842, 2023.
  * [26] Fang Chen, Lingyu Chen, Haojie Han, Sainan Zhang, Daoqiang Zhang, and Hongen Liao.  The ability of segmenting anything model (sam) to segment ultrasound images.  BioScience Trends, 2023.
  * [27] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  arXiv preprint arXiv:2102.04306, 2021.
  * [28] Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Shangzhan Zhang, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang.  Sam fails to segment anything?–sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more.  arXiv preprint arXiv:2304.09148, 2023.
  * [29] Ao Cheng, Guoqiang Zhao, Lirong Wang, and Ruobing Zhang.  Axoncallosumem dataset: Axon semantic segmentation of whole corpus callosum cross section from em images.  arXiv preprint arXiv:2307.02464, 2023.
  * [30] Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li.  Sam on medical images: A comprehensive study on three prompt modes.  arXiv preprint arXiv:2305.00035, 2023.
  * [31] Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang.  Segment and track anything.  arXiv preprint arXiv:2305.06558, 2023.
  * [32] Muhammad EH Chowdhury, Tawsifur Rahman, Amith Khandakar, Rashid Mazhar, Muhammad Abdul Kadir, Zaid Bin Mahbub, Khandakar Reajul Islam, Muhammad Salman Khan, Atif Iqbal, Nasser Al Emadi, et al.  Can ai help in screening viral and covid-19 pneumonia?  Ieee Access, 8:132665–132676, 2020.
  * [33] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: learning dense volumetric segmentation from sparse annotation.  In International conference on medical image computing and computer-assisted intervention, pages 424–432. Springer, 2016.
  * [34] Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al.  The cancer imaging archive (tcia): maintaining and operating a public information repository.  Journal of digital imaging, 26:1045–1057, 2013.
  * [35] Can Cui, Ruining Deng, Quan Liu, Tianyuan Yao, Shunxing Bao, Lucas W Remedios, Yucheng Tang, and Yuankai Huo.  All-in-sam: from weak annotation to pixel-wise nuclei segmentation with prompt-based finetuning.  arXiv preprint arXiv:2307.00290, 2023.
  * [36] Haixing Dai, Chong Ma, Zhengliang Liu, Yiwei Li, Peng Shu, Xiaozheng Wei, Lin Zhao, Zihao Wu, Dajiang Zhu, Wei Liu, et al.  Samaug: Point prompt augmentation for segment anything model.  arXiv preprint arXiv:2307.01187, 2023.
  * [37] Aysen Degerli, Mete Ahishali, Mehmet Yamac, Serkan Kiranyaz, Muhammad EH Chowdhury, Khalid Hameed, Tahir Hamid, Rashid Mazhar, and Moncef Gabbouj.  Covid-19 infection map generation and detection from chest x-ray images.  Health information science and systems, 9(1):15, 2021.
  * [38] Aysen Degerli, Serkan Kiranyaz, Muhammad E. H. Chowdhury, and Moncef Gabbouj.  Osegnet: Operational segmentation network for covid-19 detection using chest x-ray images.  In 2022 IEEE International Conference on Image Processing (ICIP), pages 2306–2310, 2022.
  * [39] Guoyao Deng, Ke Zou, Kai Ren, Meng Wang, Xuedong Yuan, Sancong Ying, and Huazhu Fu.  Sam-u: Multi-box prompts triggered uncertainty estimation for reliable sam in medical image.  arXiv preprint arXiv:2307.04973, 2023.
  * [40] Ruining Deng, Can Cui, Quan Liu, Tianyuan Yao, Lucas W Remedios, Shunxing Bao, Bennett A Landman, Lee E Wheless, Lori A Coburn, Keith T Wilson, et al.  Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging.  arXiv preprint arXiv:2304.04155, 2023.
  * [41] Reuben Dorent, Aaron Kujawa, Marina Ivory, Spyridon Bakas, Nicola Rieke, Samuel Joutard, Ben Glocker, Jorge Cardoso, Marc Modat, Kayhan Batmanghelich, et al.  Crossmoda 2021 challenge: Benchmark of cross-modality domain adaptation techniques for vestibular schwannoma and cochlea segmentation.  Medical Image Analysis, 83:102628, 2023.
  * [42] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [43] Botond Fazekas, José Morano, Dmitrii Lachinov, Guilherme Aresta, and Hrvoje Bogunović.  Samedoct: Adapting segment anything model (sam) for retinal oct.  arXiv preprint arXiv:2308.09331, 2023.
  * [44] Weijia Feng, Lingting Zhu, and Lequan Yu.  Cheap lunch for medical image segmentation by fine-tuning sam on few exemplars.  arXiv preprint arXiv:2308.14133, 2023.
  * [45] Yuncong Feng, Haiying Zhao, Xiongfei Li, Xiaoli Zhang, and Hongpeng Li.  A multi-scale 3d otsu thresholding algorithm for medical image segmentation.  Digital Signal Processing, 60:186–199, 2017.
  * [46] Yifan Gao, Wei Xia, Dingdu Hu, and Xin Gao.  Desam: Decoupling segment anything model for generalizable medical image segmentation.  arXiv preprint arXiv:2306.00499, 2023.
  * [47] Faeze Gholamiankhah, Samaneh Mostafapour, Nouraddin Abdi Goushbolagh, Seyedjafar Shojaerazavi, Parvaneh Layegh, Seyyed Mohammad Tabatabaei, and Hossein Arabi.  Automated lung segmentation from ct images of normal and covid-19 pneumonia patients.  arXiv preprint arXiv:2104.02042, 2021.
  * [48] Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, and Qi Dou.  3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation.  arXiv preprint arXiv:2306.13465, 2023.
  * [49] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.  Domain-specific language model pretraining for biomedical natural language processing.  ACM Transactions on Computing for Healthcare, 3(1):1–23, oct 2021\.
  * [50] Dazhou Guo, Jia Ge, Ke Yan, Puyang Wang, Zhuotun Zhu, Dandan Zheng, Xian-Sheng Hua, Le Lu, Tsung-Ying Ho, Xianghua Ye, et al.  Thoracic lymph node segmentation in ct imaging via lymph node station stratification and size encoding.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 55–65. Springer, 2022.
  * [51] Md Jahid Hasan, Md Shahin Alom, and Md Shikhar Ali.  Deep learning based detection and segmentation of covid-19 & pneumonia on chest x-ray image.  In 2021 International Conference on Information and Communication Technology for Sustainable Development (ICICT4SD), pages 210–214. IEEE, 2021.
  * [52] Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu.  Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images.  In International MICCAI Brainlesion Workshop, pages 272–284. Springer, 2022.
  * [53] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu.  Unetr: Transformers for 3d medical image segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 574–584, 2022.
  * [54] Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li.  Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping.  arXiv preprint arXiv:2305.11003, 2023.
  * [55] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [56] Sheng He, Rina Bao, Jingpeng Li, P Ellen Grant, and Yangming Ou.  Accuracy of segment-anything model (sam) in medical image segmentation tasks.  arXiv preprint arXiv:2304.09324, 2023.
  * [57] Nicholas Heller, Fabian Isensee, Dasha Trofimova, Resha Tejpaul, Zhongchen Zhao, Huai Chen, Lisheng Wang, Alex Golts, Daniel Khapun, Daniel Shats, et al.  The kits21 challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase ct.  arXiv preprint arXiv:2307.01984, 2023.
  * [58] Moritz R Hernandez Petzsche, Ezequiel de la Rosa, Uta Hanning, Roland Wiest, Waldo Valenzuela, Mauricio Reyes, Maria Meyer, Sook-Lei Liew, Florian Kofler, Ivan Ezhov, et al.  Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset.  Scientific data, 9(1):762, 2022.
  * [59] Fabian Hörst, Moritz Rempe, Lukas Heine, Constantin Seibold, Julius Keyl, Giulia Baldini, Selma Ugurel, Jens Siveke, Barbara Grünwald, Jan Egger, et al.  Cellvit: Vision transformers for precise cell segmentation and classification.  arXiv preprint arXiv:2306.15350, 2023.
  * [60] Md Shakhawat Hossain, MM Mahbubul Syeed, Mohammad Faisal Uddin, Mahady Hasan, Shivam Shingla, and Suresh Advani.  Robust her2 grading of breast cancer patients using zero-shot segment anything model (sam).  2023\.
  * [61] Chuanfei Hu and Xinde Li.  When sam meets medical images: An investigation of segment anything model (sam) on multi-phase liver tumor segmentation.  arXiv preprint arXiv:2304.08506, 2023.
  * [62] Mingzhe Hu, Yuheng Li, and Xiaofeng Yang.  Breastsam: A study of segment anything model for breast tumor detection in ultrasound images.  arXiv preprint arXiv:2305.12447, 2023.
  * [63] Mingzhe Hu, Yuheng Li, and Xiaofeng Yang.  Skinsam: Empowering skin cancer segmentation with segment anything model.  arXiv preprint arXiv:2304.13973, 2023.
  * [64] Xinrong Hu, Xiaowei Xu, and Yiyu Shi.  How to efficiently adapt large segmentation model (sam) to medical images.  arXiv preprint arXiv:2306.13731, 2023.
  * [65] Yuhao Huang, Xin Yang, Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen, Chaoyu Chen, et al.  Segment anything model for medical images?  arXiv preprint arXiv:2304.14660, 2023.
  * [66] Ziyi Huang, Hongshan Liu, Haofeng Zhang, Fuyong Xing, Andrew Laine, Elsa Angelini, Christine Hendon, and Yu Gan.  Push the boundary of sam: A pseudo-label correction framework for medical segmentation.  arXiv preprint arXiv:2308.00883, 2023.
  * [67] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature methods, 18(2):203–211, 2021.
  * [68] Fabian Isensee, Paul F Jäger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  Automated design of deep learning methods for biomedical image segmentation.  arXiv preprint arXiv:1904.08128, 2019.
  * [69] Rishika Iytha Sridhar and Rishikesan Kamaleswaran.  : A prompt-integrated framework for automated lung segmentation on icu chest x-ray images.  2023\.
  * [70] Ge-Peng Ji, Deng-Ping Fan, Peng Xu, Ming-Ming Cheng, Bowen Zhou, and Luc Van Gool.  Sam struggles in concealed scenes–empirical study on" segment anything".  arXiv preprint arXiv:2304.06022, 2023.
  * [71] Wei Ji, Jingjing Li, Qi Bi, Wenbo Li, and Li Cheng.  Segment anything is not always perfect: An investigation of sam on different real-world applications.  arXiv preprint arXiv:2304.05750, 2023.
  * [72] Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, et al.  Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.  arXiv preprint arXiv:2206.08023, 2022.
  * [73] Alexandros Karargyris, Renato Umeton, Micah J Sheller, Alejandro Aristizabal, Johnu George, Anna Wuest, Sarthak Pati, Hasan Kassem, Maximilian Zenk, Ujjwal Baid, et al.  Federated benchmarking of medical artificial intelligence with medperf.  Nature Machine Intelligence, pages 1–12, 2023.
  * [74] A. Emre Kavur, N. Sinem Gezer, Mustafa Barış, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Duc Duy Pham, Soumick Chatterjee, Philipp Ernst, Savaş Özkan, Bora Baydar, Dmitry Lachinov, Shuo Han, Josef Pauli, Fabian Isensee, Matthias Perkonigg, Rachana Sathish, Ronnie Rajan, Debdoot Sheet, Gurbandurdy Dovletov, Oliver Speck, Andreas Nürnberger, Klaus H. Maier-Hein, Gözde Bozdağı Akar, Gözde Ünal, Oğuz Dicle, and M. Alper Selver.  CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation.  Medical Image Analysis, 69:101950, Apr. 2021.
  * [75] A. Emre Kavur, Naciye Sinem Gezer, Mustafa Barış, Yusuf Şahin, Savaş Özkan, Bora Baydar, Ulaş Yüksel, Çağlar Kılıkçıer, Şahin Olut, Gözde Bozdağı Akar, Gözde Ünal, Oğuz Dicle, and M. Alper Selver.  Comparison of semi-automatic and deep learning based automatic methods for liver segmentation in living liver transplant donors.  Diagnostic and Interventional Radiology, 26:11–21, Jan. 2020.
  * [76] Ali Emre Kavur, M. Alper Selver, Oğuz Dicle, Mustafa Barış, and N. Sinem Gezer.  CHAOS - Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data, Apr. 2019.
  * [77] Anahita Fathi Kazerooni, Nastaran Khalili, Xinyang Liu, Debanjan Haldar, Zhifan Jiang, Syed Muhammed Anwar, Jake Albrecht, Maruf Adewole, Udunna Anazodo, Hannah Anderson, et al.  The brain tumor segmentation (brats) challenge 2023: Focus on pediatrics (cbtn-connect-dipgr-asnr-miccai brats-peds).  arXiv preprint arXiv:2305.17033, 2023.
  * [78] Evan Kellener, Ihina Nath, An Ngo, Thomas Nguyen, Joshua Schuman, Coen Adler, and Arnav Kartikeya.  Utilizing segment anything model for assessing localization of grad-cam in medical imaging.  arXiv preprint arXiv:2306.15692, 2023.
  * [79] Sekeun Kim, Kyungsang Kim, Jiang Hu, Cheng Chen, Zhiliang Lyu, Ren Hui, Sunghwan Kim, Zhengliang Liu, Aoxiao Zhong, Xiang Li, et al.  Medivista-sam: Zero-shot medical video analysis with spatio-temporal sam adaptation.  arXiv preprint arXiv:2309.13539, 2023.
  * [80] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything, 2023.
  * [81] Dominic LaBella, Maruf Adewole, Michelle Alonso-Basanta, Talissa Altes, Syed Muhammad Anwar, Ujjwal Baid, Timothy Bergquist, Radhika Bhalerao, Sully Chen, Verena Chung, et al.  The asnr-miccai brain tumor segmentation (brats) challenge 2023: Intracranial meningioma.  arXiv preprint arXiv:2305.07642, 2023.
  * [82] B Landman, Z Xu, J Igelsias, M Styner, T Langerak, and A Klein.  Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge.  In Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge, 2015.
  * [83] Sarah Leclerc, Erik Smistad, Joao Pedrosa, Andreas Østvik, Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Marc Jodoin, Thomas Grenier, et al.  Deep learning for segmentation using an open large-scale dataset in 2d echocardiography.  IEEE transactions on medical imaging, 38(9):2198–2210, 2019.
  * [84] Dongjoo Lee, Jeongbin Park, Seungho Cook, seong-jin Yoo, Daeseung Lee, and Hongyoon Choi.  Iamsam: Image-based analysis of molecular signatures using the segment-anything model.  bioRxiv, pages 2023–05, 2023.
  * [85] Ho Hin Lee, Shunxing Bao, Yuankai Huo, and Bennett A Landman.  3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation.  arXiv preprint arXiv:2209.15076, 2022.
  * [86] Ho Hin Lee, Shunxing Bao, Yuankai Huo, and Bennett A. Landman.  3d UX-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation.  In The Eleventh International Conference on Learning Representations, 2023.
  * [87] Ho Hin Lee, Quan Liu, Shunxing Bao, Qi Yang, Xin Yu, Leon Y Cai, Thomas Li, Yuankai Huo, Xenofon Koutsoukos, and Bennett A Landman.  Scaling up 3d kernels with bayesian frequency re-parameterization for medical image segmentation.  arXiv preprint arXiv:2303.05785, 2023.
  * [88] Ho Hin Lee, Quan Liu, Qi Yang, Xin Yu, Shunxing Bao, Yuankai Huo, and Bennett A Landman.  Deformux-net: Exploring a 3d foundation backbone for medical image segmentation with depthwise deformable convolution.  arXiv preprint arXiv:2310.00199, 2023.
  * [89] Wenhui Lei, Xu Wei, Xiaofan Zhang, Kang Li, and Shaoting Zhang.  Medlsam: Localize and segment anything model for 3d medical images.  arXiv preprint arXiv:2306.14752, 2023.
  * [90] Chengyin Li, Prashant Khanduri, Yao Qiang, Rafi Ibn Sultan, Indrin Chetty, and Dongxiao Zhu.  Auto-prompting sam for mobile friendly 3d medical image segmentation.  arXiv preprint arXiv:2308.14936, 2023.
  * [91] Ning Li, Lianjin Xiong, Wei Qiu, Yudong Pan, Yiqian Luo, and Yangsong Zhang.  Segment anything model for semi-supervised medical image segmentation via selecting reliable pseudo-labels.  Available at SSRN 4477443, 2023.
  * [92] Xiangyu Li, Gongning Luo, Kuanquan Wang, Hongyu Wang, Shuo Li, Jun Liu, Xinjie Liang, Jie Jiang, Zhenghao Song, Chunyue Zheng, et al.  The state-of-the-art 3d anisotropic intracranial hemorrhage segmentation on non-contrast head ct: The instance challenge.  arXiv preprint arXiv:2301.03281, 2023.
  * [93] Yunxiang Li, Meixu Chen, Wenxuan Yang, Kai Wang, Jun Ma, Alan C Bovik, and You Zhang.  Samscore: A semantic structural similarity metric for image translation evaluation.  arXiv preprint arXiv:2305.15367, 2023.
  * [94] Yuheng Li, Mingzhe Hu, and Xiaofeng Yang.  Polyp-sam: Transfer sam for polyp segmentation.  arXiv preprint arXiv:2305.00293, 2023.
  * [95] Xian Lin, Yangyang Xiang, Li Zhang, Xin Yang, Zengqiang Yan, and Li Yu.  Samus: Adapting segment anything model for clinically-friendly and generalizable ultrasound image segmentation.  arXiv preprint arXiv:2309.06824, 2023.
  * [96] Geert Litjens, Robert Toth, Wendy van de Ven, Caroline Hoeks, Sjoerd Kerkstra, Bram van Ginneken, Graham Vincent, Gwenael Guillard, Neil Birbeck, Jindang Zhang, Robin Strand, Filip Malmberg, Yangming Ou, Christos Davatzikos, Matthias Kirschner, Florian Jung, Jing Yuan, Wu Qiu, Qinquan Gao, Philip “Eddie” Edwards, Bianca Maan, Ferdinand van der Heijden, Soumya Ghose, Jhimli Mitra, Jason Dowling, Dean Barratt, Henkjan Huisman, and Anant Madabhushi.  Evaluation of prostate segmentation algorithms for mri: The promise12 challenge.  Medical Image Analysis, 18(2):359–373, 2014.
  * [97] Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou.  Clip-driven universal model for organ segmentation and tumor detection.  arXiv preprint arXiv:2301.00785, 2023.
  * [98] Wufeng Liu, Jiaxin Luo, Yan Yang, Wenlian Wang, Junkui Deng, and Liang Yu.  Automatic lung segmentation in chest x-ray images using improved u-net.  Scientific Reports, 12(1):8649, 2022.
  * [99] Yihao Liu, Jiaming Zhang, Zhangcong She, Amir Kheradmand, and Mehran Armand.  Samm (segment any medical model): A 3d slicer integration to sam.  arXiv preprint arXiv:2304.05622, 2023.
  * [100] Yaosheng Lu, Mengqiang Zhou, Dengjiang Zhi, Minghong Zhou, Xiaosong Jiang, Ruiyu Qiu, Zhanhong Ou, Huijin Wang, Di Qiu, Mei Zhong, Xiaoxing Lu, Gaowen Chen, and Jieyun Bai.  The jnu-ifm dataset for segmenting pubic symphysis-fetal head.  Data in Brief, 41:107904, 2022.
  * [101] Jun Ma and Bo Wang.  Segment anything in medical images.  arXiv preprint arXiv:2304.12306, 2023.
  * [102] Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, Shucheng Cao, Qi Zhang, Shangqing Liu, Yunpeng Wang, Yuhui Li, Jian He, and Xiaoping Yang.  Abdomenct-1k: Is abdominal organ segmentation a solved problem?  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6695–6714, 2022.
  * [103] Mojtaba Masoudi, Hamid-Reza Pourreza, Mahdi Saadatmand-Tarzjan, Noushin Eftekhari, Fateme Shafiee Zargar, and Masoud Pezeshki Rad.  A new dataset of computed-tomography angiography images for computer-aided detection of pulmonary embolism.  Scientific data, 5(1):1–9, 2018.
  * [104] Christian Mattjie, Luis Vinicius de Moura, Rafaela Cappelari Ravazio, Lucas Silveira Kupssinskü, Otávio Parraga, Marcelo Mussi Delucis, and Rodrigo Coelho Barros.  Exploring the zero-shot capabilities of the segment anything model (sam) in 2d medical imaging: A comprehensive evaluation and practical guideline.  arXiv preprint arXiv:2305.00109, 2023.
  * [105] Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang.  Segment anything model for medical image analysis: an experimental study.  Medical Image Analysis, page 102918, 2023.
  * [106] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  IEEE transactions on medical imaging, 34(10):1993–2024, 2014.
  * [107] Ahmed W Moawad, Anastasia Janas, Ujjwal Baid, Divya Ramakrishnan, Leon Jekel, Kiril Krantchev, Harrison Moy, Rachit Saluja, Klara Osenberg, Klara Wilms, et al.  The brain tumor segmentation (brats-mets) challenge 2023: Brain metastasis segmentation on pre-treatment mri.  arXiv preprint arXiv:2306.00838, 2023.
  * [108] Sovesh Mohapatra, Advait Gosai, and Gottfried Schlaug.  Brain extraction comparing segment anything model (sam) and fsl brain extraction tool.  arXiv preprint arXiv:2304.04738, 2023.
  * [109] Sergey P Morozov, AE Andreychenko, NA Pavlov, AV Vladzymyrskyy, NV Ledikhova, VA Gombolevskiy, Ivan A Blokhin, PB Gelezhe, AV Gonchar, and V Yu Chernina.  Mosmeddata: Chest ct scans with covid-19 related findings dataset.  arXiv preprint arXiv:2005.06465, 2020.
  * [110] D Muhammad Noorul Mubarak, M Mohamed Sathik, S Zulaikha Beevi, and K Revathy.  A hybrid region growing algorithm for medical image segmentation.  International Journal of Computer Science & Information Technology, 4(3):61, 2012.
  * [111] José Ignacio Orlando, Huazhu Fu, João Barbosa Breda, Karel Van Keer, Deepti R Bathula, Andrés Diaz-Pinto, Ruogu Fang, Pheng-Ann Heng, Jeyoung Kim, JoonHo Lee, et al.  Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs.  Medical image analysis, 59:101570, 2020.
  * [112] Jay N Paranjape, Nithin Gopalakrishnan Nair, Shameema Sikder, S Swaroop Vedula, and Vishal M Patel.  Adaptivesam: Towards efficient tuning of sam for surgical scene segmentation.  arXiv preprint arXiv:2308.03726, 2023.
  * [113] Gašper Podobnik, Primož Strojan, Primož Peterlin, Bulat Ibragimov, and Tomaž Vrtovec.  Han-seg: The head and neck organ-at-risk ct and mr segmentation dataset.  Medical physics, 50(3):1917–1927, 2023.
  * [114] Florian Putz, Johanna Grigo, Thomas Weissmann, Philipp Schubert, Daniel Hoefler, Ahmed Gomaa, Hassen Ben Tkhayat, Amr Hagag, Sebastian Lettmaier, Benjamin Frey, et al.  The segment anything foundation model achieves favorable brain tumor autosegmentation accuracy on mri to support radiotherapy treatment planning.  arXiv preprint arXiv:2304.07875, 2023.
  * [115] Zhongxi Qiu, Yan Hu, Heng Li, and Jiang Liu.  Learnable ophthalmology sam.  arXiv preprint arXiv:2304.13425, 2023.
  * [116] Tawsifur Rahman, Amith Khandakar, Yazan Qiblawey, Anas Tahir, Serkan Kiranyaz, Saad Bin Abul Kashem, Mohammad Tariqul Islam, Somaya Al Maadeed, Susu M Zughaier, Muhammad Salman Khan, et al.  Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images.  Computers in biology and medicine, 132:104319, 2021.
  * [117] Amin Ranem, Niklas Babendererde, Moritz Fuchs, and Anirban Mukhopadhyay.  Exploring sam ablations for enhancing medical segmentation in radiology and pathology.  arXiv preprint arXiv:2310.00504, 2023.
  * [118] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [119] H. Roth, L. Lu, A. Seff, K. M. Cherry, J. Hoffman, S. Wang, J. Liu, E. Turkbey, and R. M. Summers.  A new 2.5 d representation for lymph node detection in ct.  The Cancer Imaging Archive, 2015.  Data set.
  * [120] Holger R Roth, Amal Farag, E Turkbey, Le Lu, Jiamin Liu, and Ronald M Summers.  Data from pancreas-ct. the cancer imaging archive.  IEEE Transactions on Image Processing, 2016.
  * [121] Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers.  Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part I 18, pages 556–564. Springer, 2015.
  * [122] Holger R Roth, Le Lu, Ari Seff, Kevin M Cherry, Joanne Hoffman, Shijun Wang, Jiamin Liu, Evrim Turkbey, and Ronald M Summers.  A new 2.5 d representation for lymph node detection using random sets of deep convolutional neural network observations.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2014: 17th International Conference, Boston, MA, USA, September 14-18, 2014, Proceedings, Part I 17, pages 520–527. Springer, 2014\.
  * [123] Saikat Roy, Tassilo Wald, Gregor Koehler, Maximilian R Rokuss, Nico Disch, Julius Holzschuh, David Zimmerer, and Klaus H Maier-Hein.  Sam. md: Zero-shot medical image segmentation capabilities of the segment anything model.  arXiv preprint arXiv:2304.05396, 2023.
  * [124] Rina D Rudyanto, Sjoerd Kerkstra, Eva M Van Rikxoort, Catalin Fetita, Pierre-Yves Brillet, Christophe Lefevre, Wenzhe Xue, Xiangjun Zhu, Jianming Liang, Ilkay Öksüz, et al.  Comparing algorithms for automated vessel segmentation in computed tomography scans of the lung: the vessel12 study.  Medical image analysis, 18(7):1217–1232, 2014.
  * [125] Muhammad Usman Saeed, Wang Bin, Jinfang Sheng, Ghulam Ali, and Aqsa Dastgir.  3d mru-net: A novel mobile residual u-net deep learning model for spine segmentation using computed tomography images.  Biomedical Signal Processing and Control, 86:105153, 2023.
  * [126] Anindo Saha, Joeran Bosma, Jasper Twilt, Bram van Ginneken, Derya Yakar, Mattijs Elschot, Jeroen Veltman, Jurgen Fütterer, Maarten de Rooij, et al.  Artificial intelligence and radiologists at prostate cancer detection in mri—the pi-cai challenge.  In Medical Imaging with Deep Learning, short paper track, 2023.
  * [127] Federico Semeraro, Alexandre Quintart, Sergio Fraile Izquierdo, and Joseph C Ferguson.  Tomosam: a 3d slicer extension using sam for tomography segmentation.  arXiv preprint arXiv:2306.08609, 2023.
  * [128] Tal Shaharabany, Aviad Dahan, Raja Giryes, and Lior Wolf.  Autosam: Adapting sam to medical images by overloading the prompt encoder.  arXiv preprint arXiv:2306.06370, 2023.
  * [129] Chuyun Shen, Wenhao Li, Ya Zhang, and Xiangfeng Wang.  Temporally-extended prompts optimization for sam in interactive medical image segmentation.  arXiv preprint arXiv:2306.08958, 2023.
  * [130] Tian Shen, Hongsheng Li, and Xiaolei Huang.  Active volume models for medical image segmentation.  IEEE transactions on medical imaging, 30(3):774–791, 2010.
  * [131] Peilun Shi, Jianing Qiu, Sai Mu Dalike Abaxi, Hao Wei, Frank P-W Lo, and Wu Yuan.  Generalist vision foundation models for medical imaging: A case study of segment anything model on zero-shot medical segmentation.  Diagnostics, 13(11):1947, 2023.
  * [132] Xiaoyu Shi, Shurong Chai, Yinhao Li, Jingliang Cheng, Jie Bai, Guohua Zhao, and Yen-Wei Chen.  Cross-modality attention adapter: A glioma segmentation fine-tuning method for sam using multimodal brain mr images.  arXiv preprint arXiv:2307.01124, 2023.
  * [133] Dongik Shin, Beomsuk Kim, and Seungjun Baek.  Cemb-sam: Segment anything model with condition embedding for joint learning from heterogeneous datasets.  arXiv preprint arXiv:2308.06957, 2023.
  * [134] Junji Shiraishi, Shigehiko Katsuragawa, Junpei Ikezoe, Tsuneo Matsumoto, Takeshi Kobayashi, Ken-ichi Komatsu, Mitate Matsui, Hiroshi Fujita, Yoshie Kodera, and Kunio Doi.  Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules.  American Journal of Roentgenology, 174(1):71–74, 2000.
  * [135] Korsuk Sirinukunwattana, Josien P. W. Pluim, Hao Chen, Xiaojuan Qi, Pheng-Ann Heng, Yun Bo Guo, Li Yang Wang, Bogdan J. Matuszewski, Elia Bruni, Urko Sanchez, Anton Böhm, Olaf Ronneberger, Bassem Ben Cheikh, Daniel Racoceanu, Philipp Kainz, Michael Pfeiffer, Martin Urschler, David R. J. Snead, and Nasir M. Rajpoot.  Gland segmentation in colon histology images: The glas challenge contest, 2016.
  * [136] Korsuk Sirinukunwattana, David R. J. Snead, and Nasir M. Rajpoot.  A stochastic polygons model for glandular structures in colon histology images.  IEEE Transactions on Medical Imaging, 34(11):2366–2378, 2015.
  * [137] Yuxin Song, Jing Zheng, Long Lei, Zhipeng Ni, Baoliang Zhao, and Ying Hu.  Ct2us: Cross-modal transfer learning for kidney segmentation in ultrasound images with synthesized data.  Ultrasonics, 122:106706, 2022.
  * [138] Joes Staal, Michael D Abràmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Ginneken.  Ridge-based vessel segmentation in color images of the retina.  IEEE transactions on medical imaging, 23(4):501–509, 2004.
  * [139] Yue Sun, Limei Wang, Kun Gao, Shihui Ying, Weili Lin, Kathryn L Humphreys, Gang Li, Sijie Niu, Mingxia Liu, and Li Wang.  Self-supervised learning with application for infant cerebellum segmentation and analysis.  Nature Communications, 14(1):4717, 2023.
  * [140] Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Zhongyi Shui, Xiaoxuan Yu, Yizhi Zhao, Honglin Li, Yunlong Zhang, Ruojia Zhao, et al.  Pathasst: Redefining pathology through generative foundation ai assistant for pathology.  arXiv preprint arXiv:2305.15072, 2023.
  * [141] Anas M Tahir, Muhammad EH Chowdhury, Amith Khandakar, Tawsifur Rahman, Yazan Qiblawey, Uzair Khurshid, Serkan Kiranyaz, Nabil Ibtehaz, M Sohel Rahman, Somaya Al-Maadeed, et al.  Covid-19 infection localization and severity grading from chest x-ray images.  Computers in biology and medicine, 139:105002, 2021.
  * [142] Anas M. Tahir, Muhammad E. H. Chowdhury, Yazan Qiblawey, Amith Khandakar, Tawsifur Rahman, Serkan Kiranyaz, Uzair Khurshid, Nabil Ibtehaz, Sakib Mahmud, and Maymouna Ezeddin.  Covid-qu-ex dataset, 2022.
  * [143] Athanasios Tragakis, Chaitanya Kaul, Roderick Murray-Smith, and Dirk Husmeier.  The fully convolutional transformer for medical image segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3660–3669, 2023.
  * [144] Pamula Udayaraju, K Sreerama Murthy, P Jeyanthi, Bh VS Raju, T Rajasri, and N Ramadevi.  A combined u-net and multi-class support vector machine learning models for diabetic retinopathy macula edema segmentation and classification dme.  Soft Computing, pages 1–13, 2023.
  * [145] Ihsan Ullah, Farman Ali, Babar Shah, Shaker El-Sappagh, Tamer Abuhmed, and Sang Hyun Park.  A deep learning based dual encoder–decoder framework for anatomical structure segmentation in chest x-ray images.  Scientific Reports, 13(1):791, 2023.
  * [146] Santiago Vitale, José Ignacio Orlando, Emmanuel Iarussi, and Ignacio Larrabide.  Improving realism in patient-specific abdominal ultrasound simulation using cyclegans.  International journal of computer assisted radiology and surgery, 15(2):183–192, 2020.
  * [147] An Wang, Mobarakol Islam, Mengya Xu, Yang Zhang, and Hongliang Ren.  Sam meets robotic surgery: An empirical study on generalization, robustness and adaptation.  arXiv preprint arXiv:2308.07156, 2023.
  * [148] Bin Wang, Armstrong Aboah, Zheyuan Zhang, and Ulas Bagci.  Gazesam: What you see is what you segment.  arXiv preprint arXiv:2304.13844, 2023.
  * [149] Chenglong Wang, Dexuan Li, Sucheng Wang, Chengxiu Zhang, Yida Wang, Yun Liu, and Guang Yang.  Sammed: A medical image annotation framework based on large vision model.  arXiv preprint arXiv:2307.05617, 2023.
  * [150] Wenxuan Wang, Chen Chen, Meng Ding, Hong Yu, Sen Zha, and Jiangyun Li.  Transbts: Multimodal brain tumor segmentation using transformer.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 109–119. Springer, 2021.
  * [151] Xiyue Wang, Yuqi Fang, Sen Yang, Delong Zhu, Minghui Wang, Jing Zhang, Kai-yu Tong, and Xiao Han.  A hybrid network for automatic hepatocellular carcinoma segmentation in h&e-stained whole slide images.  Medical Image Analysis, 68:101914, 2021.
  * [152] Yu Wang and Yudong Yao.  Application of artificial intelligence methods in carotid artery segmentation: a review.  IEEE Access, 2023.
  * [153] Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth.  TotalSegmentator: Robust segmentation of 104 anatomic structures in CT images.  Radiology: Artificial Intelligence, 5(5), sep 2023.
  * [154] Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel.  Medical sam adapter: Adapting segment anything model for medical image segmentation.  arXiv preprint arXiv:2304.12620, 2023.
  * [155] Qi Wu, Yuyao Zhang, and Marawan Elbatel.  Self-prompting large vision models for few-shot medical image segmentation, 2023.
  * [156] Yutong Xie, Jianpeng Zhang, Chunhua Shen, and Yong Xia.  Cotr: Efficiently bridging cnn and transformer for 3d medical image segmentation.  In International conference on medical image computing and computer-assisted intervention, pages 171–180. Springer, 2021.
  * [157] Chenyang Xu, Dzung L Pham, and Jerry L Prince.  Image segmentation using deformable models.  Handbook of medical imaging, 2(20):0, 2000.
  * [158] Cilin Yan, Haochen Wang, Jie Liu, Xiaolong Jiang, Yao Hu, Xu Tang, Guoliang Kang, and Efstratios Gavves.  Piclick: Picking the desired mask in click-based interactive segmentation.  arXiv preprint arXiv:2304.11609, 2023.
  * [159] Xing Yao, Han Liu, Dewei Hu, Daiwei Lu, Ange Lou, Hao Li, Ruining Deng, Gabriel Arenas, Baris Oguz, Nadav Schwartz, et al.  False negative/positive control for sam on noisy medical images.  arXiv preprint arXiv:2308.10382, 2023.
  * [160] Anna Zawacki, Carol Wu, George Shih, Julia Elliott, Mikhail Fomitchev, Mohannad Hussain, Paras Lakhani, Phil Culliton, and Shunxing Bao.  Siim-acr pneumothorax segmentation, 2019.
  * [161] Jingwei Zhang, Ke Ma, Saarthak Kapse, Joel Saltz, Maria Vakalopoulou, Prateek Prasanna, and Dimitris Samaras.  Sam-path: A segment anything model for semantic segmentation in digital pathology.  arXiv preprint arXiv:2307.09570, 2023.
  * [162] Kaidong Zhang and Dong Liu.  Customized segment anything model for medical image segmentation.  arXiv preprint arXiv:2304.13785, 2023.
  * [163] Yichi Zhang and Rushi Jiao.  How segment anything model (sam) boost medical image segmentation?  arXiv preprint arXiv:2305.03678, 2023.
  * [164] Yiming Zhang, Tianang Leng, Kun Han, and Xiaohui Xie.  Self-sampling meta sam: Enhancing few-shot medical image segmentation with meta-learning.  arXiv preprint arXiv:2308.16466, 2023.
  * [165] Yichi Zhang, Lin Yuan, Yujia Wang, and Jicong Zhang.  Sau-net: Efficient 3d spine mri segmentation using inter-slice attention.  In Medical Imaging with Deep Learning, pages 903–913. PMLR, 2020\.
  * [166] Yizhe Zhang, Tao Zhou, Peixian Liang, and Danny Z Chen.  Input augmentation with sam: Boosting medical image segmentation with segmentation foundation model.  arXiv preprint arXiv:2304.11332, 2023.
  * [167] Yizhe Zhang, Tao Zhou, Shuo Wang, Ye Wu, Pengfei Gu, and Danny Z Chen.  Samdsk: Combining segment anything model with domain-specific knowledge for semi-supervised learning in medical image segmentation.  arXiv preprint arXiv:2308.13759, 2023.
  * [168] Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, and Yizhou Yu.  nnformer: Interleaved transformer for volumetric segmentation.  arXiv preprint arXiv:2109.03201, 2021.
  * [169] Tao Zhou, Yizhe Zhang, Yi Zhou, Ye Wu, and Chen Gong.  Can sam segment polyps?  arXiv preprint arXiv:2304.07583, 2023.
  * [170] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee.  Segment everything everywhere all at once.  arXiv preprint arXiv:2304.06718, 2023.