# MedLSAM: Localize and Segment Anything Model for 3D CT Images

Wenhui Lei Wei Xu Kang Li Xiaofan Zhang Shaoting Zhang School of Electronic
Information and Electrical Engineering, Shanghai Jiao Tong University,
Shanghai, China  Shanghai AI Lab, Shanghai, China  School of Biomedical
Engineering, Division of Life Sciences and Medicine, University of Science and
Technology of China, Hefei, China  West China Biomedical Big Data Center, West
China Hospital, Sichuan University, Chengdu, China

###### Abstract

The Segment Anything Model (SAM) has recently emerged as a groundbreaking
model in the field of image segmentation. Nevertheless, both the original SAM
and its medical adaptations necessitate slice-by-slice annotations, which
directly increase the annotation workload with the size of the dataset. We
propose MedLSAM to address this issue, ensuring a constant annotation workload
irrespective of dataset size and thereby simplifying the annotation process.
Our model introduces a 3D localization foundation model capable of localizing
any target anatomical part within the body. To achieve this, we develop a
Localize Anything Model for 3D Medical Images (MedLAM), utilizing two self-
supervision tasks: unified anatomical mapping (UAM) and multi-scale similarity
(MSS) across a comprehensive dataset of 14,012 CT scans. We then establish a
methodology for accurate segmentation by integrating MedLAM with SAM. By
annotating several extreme points across three directions on a few templates,
our model can autonomously identify the target anatomical region on all data
scheduled for annotation. This allows our framework to generate a 2D bbox for
every slice of the image, which is then leveraged by SAM to carry out
segmentation. We carried out comprehensive experiments on two 3D datasets
encompassing 38 distinct organs. Our findings are twofold: 1) MedLAM is
capable of directly localizing any anatomical structure using just a few
template scans, yet its performance surpasses that of fully supervised models;
2) MedLSAM not only aligns closely with the performance of SAM and its
specialized medical adaptations with manual prompts but achieves this with
minimal reliance on extreme point annotations across the entire dataset.
Furthermore, MedLAM has the potential to be seamlessly integrated with future
3D SAM models, paving the way for enhanced performance. Our code is public at
https://github.com/openmedlab/MedLSAM.

###### keywords:

SAM, Medical Image Segmentation, Contrastive Learning

††journal: Medical Image Analysis

##  1 Introduction

Fig. 1: The overall segmentation pipeline of MedLSAM operates as follows.
Given a dataset of any size, MedLSAM first applies a localization process
(MedLAM) to identify the six extreme points (in the z, x, and y directions) of
any anatomical structure of interest. This process results in the generation
of a 3D bbox encompassing the targeted organ or structure. Subsequently, for
each slice within this 3D bbox, a 2D bbox is derived, representing the
projection of the 3D bbox onto that specific slice. These 2D bboxes are then
utilized by the Segment Anything Model (SAM) to carry out precise segmentation
of the target anatomy, thereby automating the entire segmentation process.

Recently, there has been an increasing interest in the field of computer
vision to develop large-scale foundational models that can concurrently
address multiple visual tasks, such as image classification, object detection,
and image segmentation. For instance, CLIP [45], aligning a vast number of
images and text pairs collected from the web, could recognize new visual
categories using text prompts. Similarly, GLIP [32, 60], which integrates
object detection with phrase grounding in pre-training using web-sourced
image-text pairs, boasts strong zero-shot and few-shot transferability in
object-level detection tasks. On the segmentation front, the Segment Anything
Model (SAM) [27] has recently demonstrated remarkable capabilities in a broad
range of segmentation tasks with appropriate prompts, e.g. points, bounding
box (bbox) and text.

As foundational models increasingly demonstrate their prowess in general
computer vision tasks, the medical imaging domain, characterized by limited
image availability and high annotation costs, is taking keen notice of their
potential. Such challenges in medical imaging underscore the pressing need for
foundational models, drawing greater attention from researchers [62]. Some
studies have endeavored to design self-supervised learning tasks tailored to
dataset characteristics, pre-training on a vast amount of unlabeled data, and
then fine-tuning on specific downstream tasks to achieve commendable results
[53, 64, 51]. In contrast, other works have focused on pre-training with large
annotated datasets, either to fine-tune on new tasks [22] or to enable direct
segmentation without the necessity of further training or fine-tuning [7].
Overall, these models offer myriad advantages: they simplify the development
process, reduce dependence on voluminously labeled datasets, and bolster
patient data privacy.

In the field of segmentation, models such as SAM and its medical adaptations
[55, 37, 11, 19, 21, 18, 61, 39, 63] have demonstrated significant potential.
Both [21] and [11] have collected extensive medical image datasets and
conducted comprehensive testing on SAM. They discovered that, while SAM
exhibits remarkable performance on certain objects and modalities, it is less
effective or even fails in other scenarios. This inconsistency arises from
SAM’s primary training on natural images. To enhance SAM’s generalization in
medical imaging, studies like [55, 37] have fine-tuned the original SAM using
a large annotated medical image dataset, thereby validating its improved
performance over the base SAM model in test datasets. Despite these
advancements, the models still require manual prompts, such as labeled points
or bounding boxes, which remains a substantial limitation due to the
associated time and financial implications.

In scenarios requiring manual annotation prompts, annotating 3D images proves
especially time-consuming and labor-intensive. A single 3D scan often contains
dozens or even hundreds of slices. When each slice comprises multiple target
structures, the annotation requirements for the entire 3D scan grow
exponentially[54, 30]. To eliminate this vast annotation demand and facilitate
the use of SAM and its medical variants on 3D data, an effective approach
would be to automatically generate suitable prompts for the data awaiting
annotation. While this could be achieved by training a detector specifically
for the categories to be annotated, it introduces the additional burden of
annotating data for detector training, and the detectable categories remain
fixed[4]. A more optimal solution would be to develop a universal localization
model for 3D medical images. This model would only require users to provide
minimal annotations to specify the category of interest, allowing for direct
localization of the desired target in any dataset without further training.

However, to the best of our knowledge, foundational models specifically
tailored for 3D medical image localization are still limited. In the domain of
2D medical image localization, MIU-VL [43] stands out as a pioneering effort.
This approach ingeniously marries pre-trained vision-language models with
medical text prompts to facilitate object detection in medical imaging.
However, the application of these pre-trained models to specialized medical
imaging domains, particularly 3D modalities like CT scans, remains
challenging. This limitation is attributable to a significant domain gap, as
these models were originally trained on 2D natural images.

In response to the lack of localization foundation models for 3D medical
images and the persistent need for extensive interaction in SAM, we introduce
MedLSAM in this paper. MedLSAM is an automated medical image segmentation
solution that combines the Localize Anything Model for 3D Medical Image
(MedLAM), a dedicated 3D localization foundation module designed for
localizing any anatomical structure using a few prompts, with existing
segmentation foundation models. As illustrated in Fig. 1, MedLSAM employs a
two-stage methodology. The first stage involves MedLAM to automatically
identify the positions of target structures within volumetric medical images.
In the subsequent stage, the SAM model uses the bboxes from the first stage to
achieve precise image segmentation. The result is a fully autonomous pipeline
that eliminates the need for manual intervention.

The localization foundation model, MedLAM, is an extension of our conference
version [31] and is premised on the observation that the spatial distribution
of organs maintains strong similarities across different individuals. And we
assume that there exists a standard anatomical coordinate system, in which the
same anatomy part in different people shares similar coordinates. Therefore,
we can localize the targeted anatomy structure with a similar coordinate in
the unannotated scan. In our previous version, the model was trained
separately for different anatomical structures, each using only a few dozen
scan images. In contrast, this current study significantly expands the dataset
to include 14,012 CT scans from 16 different datasets. This allows us to train
a unified, comprehensive model capable of localizing any structure across the
entire body. The training process involves a projection network that predicts
the 3D physical offsets between any two patches within the same image, thereby
mapping every part of the scans onto a shared 3D latent coordinate system.

For the segmentation stage, we employ the original SAM and the well-
established MedSAM [37] as the foundation for our segmentation process.
MedSAM, previously fine-tuned on a comprehensive collection of medical image
datasets, has exhibited considerable performance in 2D and 3D medical image
segmentation tasks. The use of such a robust model bolsters the reliability
and effectiveness of our proposed pipeline.

Our contributions can be summarized as follows:

\- We introduce MedLSAM, the first completely automated medical adaptation of
the SAM model. It not only eliminates the need for manual intervention but
also propels the development of fully automated foundational segmentation
models in medical image analysis. To the best of our knowledge, this is the
first work to achieve complete automation in medical image segmentation by
integrating the SAM model;

\- Addressing the gap in localization foundation models for 3D medical
imaging, we present MedLAM. This model, an extension of our prior conference
work [31], incorporates pixel-level self-supervised tasks and is trained on a
comprehensive dataset of 14,012 CT scans covering the entire body. As a
result, it’s capable of directly localizing any anatomical structure within
the human body and surpassing the fully-supervised model by a significant
margin;

\- We validate the effectiveness of MedLSAM on two 3D datasets that span 38
organs. As showcased in Table 2(b), our localization foundation model, MedLAM,
exhibits localization accuracy that not only significantly outperforms
existing medical localization foundational models but also rivals the
performance of fully-supervised models. Concurrently, as demonstrated in Table
4(b), MedLSAM aligns with the performance of SAM and its medical adaptations,
substantially reducing the reliance on manual annotations. To further the
research in foundational models for medical imaging, we have made all our
models and codes publicly available.

##  2 Related Work

###  2.1 Medical Visual Foundation Models

Over recent years, large-scale pre-trained deep learning vision models have
marked remarkable milestones with models like ResNet[17], ConvNeXt[34], Vision
Transformers (ViT)[13], and DINO[8]. Owing to their profound representational
power and exceptional generalization capabilities, these models have
consistently excelled in various downstream applications.

In the realm of medical image analysis, the challenges are compounded by
concerns of data privacy and the often tedious nature of annotation tasks,
making the procurement of large-scale annotated data both challenging and
costly. This elevates the significance of visual foundational models in the
field. For instance, REMEDIS[3] harnesses the potential of self-supervised
learning on a diverse set of medical images spanning multiple modalities.
Impressively, it rivals the performance of heavily supervised baselines in
downstream disease classification tasks, requiring only between 1% to 33% of
the typical retraining data. Meanwhile, SAIS[3], capitalizing on the
foundational model DINO[8] pre-trained on standard images, showcases
impressive generalization and prowess in specialized tasks, such as surgical
gesture recognition and skill evaluation.

Turning the attention to segmentation, a key aspect within medical imaging,
there has been a surge of interest in the application of SAM for medical image
segmentation [11, 19, 55, 37, 21, 18, 61, 39, 63]. Notably, with additional
medical data collected for fine-tuning, the medical adaptations tailored for
SAM, specifically [55, 37], have exhibited commendable results in this domain.

However, a significant hurdle for SAM’s utility in medical image segmentation
is the necessity for manual annotations. This involves annotating specific
points or bboxes that demarcate the segmentation region, a process that is
both time-consuming and expensive. Intriguingly, comprehensive studies by [21]
and [11] have delved into the optimal prompts for SAM’s application in medical
imaging. They primarily investigated prompt types including bboxes, a
combination of bboxes with foreground and background points, and the exclusive
use of foreground and background points. Notably, in the majority of tasks,
bboxes alone showcased superior performance.

In light of these findings, one of the central focuses of this paper is the
automation of generating bbox prompts for SAM. Furthermore, we aim to advance
the full automation of SAM’s application in the medical domain, reducing the
reliance on manual annotations and propelling its utility.

Fig. 2: The learning process of MedLAM.

###  2.2 Medical Localization Models

Traditionally, localization tasks, such as object detection, have relied on
extensive annotations for a limited set of categories to enable models to
produce corresponding bboxes [46, 16, 12, 47]. This challenge has also been
extensively addressed in the realm of medical imaging, with a focus on areas
such as organ localization [56, 41, 24, 40], keypoint detection[10, 52], and
lesion detection[57, 59]. Both organ localization and keypoint detection often
serve as preliminary steps for many medical image analysis tasks, including
image registration[49], organ segmentation[33], and disease diagnosis[5].
While previous fully-supervised models have achieved commendable results in
predefined tasks, their primary limitation is their difficulty in generalizing
to new categories. The high cost and challenges associated with data
collection and annotation further exacerbate this issue, especially pronounced
in the field of medical imaging.

To reduce annotation costs, our previous work, the Relative Position
Regression-Net (RPR-Net) [31], along with another study, Cascade Comparing to
Detect (CC2D) [58], pioneered the localization of regions of interest or
landmarks using annotations from just a single instance. Specifically, CC2D
learns to project pixels associated with the same anatomical structures into
embeddings with high cosine similarities by addressing a self-supervised
”patch matching” proxy task in a coarse-to-fine manner. However, both RPR-Net
and CC2D were trained on limited datasets as independent models, without
evolving into foundational models for localization.

##  3 Method

As illustrated in Fig. 1, MedLSAM comprises two core components: the automated
localization algorithm MedLAM and the automated segmentation algorithm SAM. In
the subsequent sections, we first delve into the architecture and training of
MedLAM. Following that, we detail how MedLAM synergizes with SAM to realize
the complete autonomous segmentation framework.

###  3.1 Training of MedLAM

Our MedLAM model, depicted in Fig. 2, consists of three primary modules: a
feature encoder, a multilayer perceptron (MLP), and a feature decoder. Both
the feature encoder and decoder are composed of four convolutional blocks. The
encoder blocks each have two convolutional layers followed by a downsampling
layer, while the decoder blocks contain two convolutional layers succeeded by
an upsampling layer. The MLP is structured with three fully connected layers.

We start by selecting a volumetric image $\bm{v}$ from the unannotated
training set. We then extract two large image patches from $\bm{v}$, which
serve as the source for further extractions. These large patches undergo a
variety of transformations to produce two pairs of patches, namely, the
original patch pair $\bm{x}_{q},\bm{x}_{s}\in R^{D\times W\times H}$ and the
augmented patch pair $\bm{x_{q}^{\prime}},\bm{x_{s}^{\prime}}\in R^{D\times
W\times H}$. The augmentation techniques employed include rotation, elastic
transformations, and the addition of random noise. Our training objectives are
twofold: 1) Unified Anatomical Mapping (UAM): To project scan images from
different individuals onto a shared anatomical coordinate space by predicting
the relative distance between the original image patches $\bm{x}_{q}$ and
$\bm{x}_{s}$. 2) Multi-Scale Similarity (MSS): Ensure that the similarity
between features corresponding to the same region is maximized in both the
original image patches $\bm{x}_{q},\bm{x}_{s}$ and their augmented
counterparts $\bm{x_{q}^{\prime}},\bm{x_{s}^{\prime}}$.

####  3.1.1 Unified Anatomical Mapping (UAM)

In this step, we introduce the Unified Anatomical Mapping (UAM) framework,
building upon the foundation of our earlier methodology [31]. The central
principle remains consistent: mapping 3D scan images from diverse individuals
onto a unified implicit 3D anatomical coordinate system. By doing so,
identical anatomical structures from various individuals can be associated
with the same coordinate, facilitating an initial, coarse localization of the
point within the unannotated scan corresponding to our point of interest.

In contrast to our prior endeavors, UAM embodies a broader vision of creating
a model that captures the underlying anatomical structures across a wide range
of human body scans. While our previous methods primarily focused on training
specialized models for individual localized scan datasets, the UAM framework
takes advantage of an extensive collection of CT scans, covering the entire
human spectrum. This approach aims to provide a more holistic model capable of
operating seamlessly across different organs and body regions, increasing the
versatility and applicability of our method but also ensuring higher accuracy
and reliability in diverse medical contexts.

The RDR model aims to predict the 3D offset between the query patch
$\bm{x}_{q}$ and the support patch $\bm{x}_{s}$. Considering $\bm{e}\in R^{3}$
as the pixel spacing of $\bm{v}$, and $\bm{c}_{q},\bm{c}_{s}\in R^{3}$ as the
centroid coordinates of $\bm{x}_{q}$ and $\bm{x}_{s}$ in $\bm{v}$
respectively, the ground truth offset $\bm{d}^{\prime}_{qs}$ from $\bm{x}_{q}$
to $\bm{x}_{s}$ in the physical space can be calculated as:

$\bm{d}^{\prime}_{qs}=(\bm{c}_{s}-\bm{c}_{q})\cdot\bm{e}.$ (1)

Both $\bm{x}_{s}$ and $\bm{x}_{q}$ undergo processing via the feature encoder
to distill high-level features. Subsequently, MLP maps these features to their
corresponding 3D latent vectors, $\bm{p}_{s}$ and $\bm{p}_{q}\in R^{3}$. This
process leads to the predicted offset $\bm{d}_{qs}\in R^{3}$ from the query
patch $\bm{x}_{q}$ to the support patch $\bm{x}_{s}$ being computed as:

$\bm{d}_{qs}=r\cdot tanh(\bm{p}_{s}-\bm{p}_{q}).$ (2)

The utilization of the hyperbolic tangent function $tanh$ in conjunction with
the hyper-parameter $r$ is intended to dictate the upper and lower bound of
$\bm{d}_{qs}$, thereby covering the largest feasible offset. Lastly, to
measure the difference between $\bm{d}_{qs}$ and $\bm{d}^{\prime}_{qs}$, we
employ the Mean Square Error (MSE) loss function:

$L_{MSE}=||\bm{d}_{qs}-\bm{d}^{\prime}_{qs}||^{2}.$ (3)

Owing to the constraints set by the hyper-parameter $r$ and the inherent
anatomical similarities across different individuals, the model effectively
projects the same anatomical regions from various individuals onto similar
implicit coordinates after training. This projection mechanism aids in
achieving a consistent and unified representation across diverse datasets,
thereby enhancing the adaptability and precision of our MedLAM model.

Fig. 3: Deatils of the Multi Scale Similarity (MSS).

####  3.1.2 Multi Scale Similarity (MSS)

However, given the inherent variations in anatomical positioning across
different individuals, regions sharing the same latent coordinates in various
images may still correspond to different anatomical structures. Therefore, we
need to further refine the precision of our localization by extracting local
pixel-level features from our points of interest. This allows us to pinpoint
the most similar feature within the vicinity of the initially localized point,
thereby enhancing the overall localization accuracy. This is inspired by the
work in Cascade Comparing to Detect (CC2D) [58], which ensures that augmented
instances of the same image yield highly similar features for the same point,
while different points exhibit substantially divergent features.

More specifically, as illustrated in Fig. 3, the inputs to our MSS process
encompass normalized multi-scale feature maps, denoted as
$\bm{F}=\\{\bm{F}^{i},i=0,1,2\\}$ and
$\bm{F}^{\prime}=\\{\bm{F}^{{}^{\prime}i},i=0,1,2\\}$, extracted from $\bm{x}$
and $\bm{x}^{\prime}$, respectively. These multi-scale feature maps consist of
three different scales, where the size of each scale is $1/{2^{i}}$ times the
size of the original image, with the largest scale being the same size as the
original image.

Given a chosen point $\bm{c}_{j}$ from $\bm{x}$, which has a corresponding
point in $\bm{x}^{\prime}$ is denoted as $\bm{c}^{\prime}_{j}$. For every
scale $i$ of the multi-scale feature maps, we extract the feature vectors
$\bm{f}_{\bm{c}_{j}}^{i}$ and $\bm{f}_{\bm{c}^{\prime}_{j}}^{{}^{\prime}i}$
corresponding to points $\bm{c}_{j}$ and $\bm{c}^{\prime}_{j}$ respectively.
Formally, $\bm{f}_{\bm{c}_{j}}^{i}=\bm{F}^{i}(\bm{c}_{j})$ and
$\bm{f}_{\bm{c}^{\prime}_{j}}^{{}^{\prime}i}=\bm{F}^{{}^{\prime}i}(\bm{c}^{\prime}_{j})$.

To compute the similarity between these feature vectors and the corresponding
scale feature maps in $\bm{x^{\prime}}$, we use the cosine similarity metric.
The cosine similarity $s$ between two vectors $\bm{a}$ and $\bm{b}$ is given
by:

$s(\bm{a},\bm{b})=\frac{\bm{a}\cdot\bm{b}}{||\bm{a}||_{2}\cdot||\bm{b}||_{2}}.$
(4)

Since the $\bm{F}^{i}$ and $\bm{F}^{\prime i}$ are both normalized, for each
scale $i$, the cosine similarity map is computed by using the feature vector
$\bm{f}_{\bm{c}_{j}}^{i}$ as a $1\times 1\times 1$ convolutional kernel that
operates on the corresponding feature map from the augmented image. Our
optimization objective is to increase the similarity between
$\bm{f}_{\bm{c}_{j}}^{i}$ and $\bm{f}_{\bm{c}^{\prime}_{j}}^{{}^{\prime}i}$
relative to other points, effectively enhancing their mutual likeness while
diminishing their similarity to the remaining features in the augmented image.
To achieve this, we apply the softmax operation across all values in the
cosine similarity map, resulting in the final probability map
$\bm{S}^{i}_{\bm{c}_{j}}$. Correspondingly, the ground truth
$\bm{Y}^{i}_{\bm{c}_{j}}$ at scale $i$, is set as:

$\bm{Y}^{i}_{\bm{c}_{j}}(\bm{c})=\begin{cases}1,&\text{if}\
\bm{c}=round(\frac{\bm{c}^{\prime}_{j}}{2^{i}})\\\
0,&\text{otherwise}\end{cases}$ (5)

This ensures that all other points are set to 0, while the corresponding
position for $\bm{c}^{\prime}_{j}$ at scale $i$ is set to 1. We utilize the
Cross Entropy (CE) loss to align each $\bm{S}^{i}_{\bm{c}_{j}}$ with its
corresponding ground truth $\bm{Y}^{i}_{\bm{c}_{j}}$:

$L_{CE}=\sum_{i}-\bm{Y}^{i}_{\bm{c}_{j}}\cdot
log(\bm{S}^{i}_{\bm{c}_{j}})-(1-\bm{Y}^{i})\cdot
log(1-\bm{S}^{i}_{\bm{c}_{j}}).$ (6)

Eventually, we set the final loss function as:

$L_{SSL}=L_{MSE}+L_{CE}.$ (7)

Fig. 4: Structure of the inference stage in the Medical Localization Anything
Model (MedLAM). The process involves moving an agent from a randomly
initialized position toward the target landmark, guided by 3D relative
distance and multi-scale feature vectors. Fig. 5: Comparison between Whole-
Patch Localization (WPL) and Sub-Patch Localization (SPL) strategies. (a) WPL
generates a single minimal 3D bounding box ($\bm{B}$) by identifying the six
extreme points of the target anatomical structure. (b) SPL divides the target
structure into multiple segments, each with its own localized 3D bounding box
($\bm{B}_{i}$), resulting in a more accurate representation at each slice.

###  3.2 Inference of MedLSAM

In this section, we will elaborate on the process of utilizing MedLAM to
localize any point of interest within scans. Additionally, we further
elucidate the mechanism through which MedLAM crafts 3D bbox prompts for the
anatomical structures of interest. These 3D prompts are then sliced into 2D
bboxes corresponding to each layer of the structure. Each 2D bbox, acting as a
prompt for its respective layer, is combined with the corresponding image
slice and processed by the SAM model to produce the segmentation results.

####  3.2.1 Landmark Localization

As posited in Section 3.1.1, we assume that the same anatomical structure in
different individuals corresponds to the same latent coordinates. This allows
us to calculate the latent coordinates of any point within the query volume
and subsequently determine its relative distance to the target landmark by
computing the difference in their latent coordinates.

As shown in Fig. 4, we conceptualize the localization task as moving an agent
from a randomly initialized position in the query image toward the target
location. A patch is extracted from a random position within the query image,
and simultaneously, a support patch is extracted from the support image,
centered around the pre-specified landmark. Upon processing these two patches
through the MedLAM model, we obtain a 3D offset that represents the estimated
relative spatial displacement between the start and target positions and the
multi-scale feature vectors of the pre-specified landmark. By updating the
agent’s location based on this offset, we achieve a coarse localization of the
landmark within the query image.

To further refine the landmark localization, we employ the Multi-Scale
Similarity (MSS) component of MedLAM. Specifically, we extract a 3D patch of
size $D\times W\times H$ centered around the coarsely localized point in the
query image. Within the extracted patch, we compute multi-scale feature maps,
perform similarity calculations, and aggregate the resulting similarity maps.
This allows us to identify the location within the patch that exhibits the
highest feature similarity, thereby significantly enhancing the precision of
our landmark localization.

####  3.2.2 Sub-Patch Localization (SPL)

As shown in Fig. 5(a), for a target anatomical structure, we can generate a
single minimal 3D bounding box (bbox) $\bm{B}$ that completely encompasses the
organ by locating its six extreme points in three dimensions:
$[\bm{z}_{min},\bm{z}_{max},\bm{x}_{min},\bm{x}_{max},\bm{y}_{min},\bm{y}_{max}]$.
We call this localization approach as Whole-Patch Localization (WPL). However,
the cross-sectional area of this 3D bbox on each layer may significantly
exceed the actual cross-sectional area of the organ at that layer, resulting
in highly inaccurate bbox prompts.

In response, we introduce the Sub-Patch Localization (SPL) strategy. Rather
than using a single bounding box, SPL divides the target structure into
multiple segments, each segment having its own dedicated 3D bbox, denoted as
$\bm{B}_{i}$. This segmented approach, as visualized in Fig. 5(b), ensures
that each bbox is tailored to more accurately represent the organ’s actual
cross-sectional area at every layer.

Delving into the specifics, for any given target organ, SPL slices the organ
into segments at intervals of $n$ mm, resulting in $m$ distinct segments for
the organ. Consequently, MedLAM is then tasked with localizing these $m$
segments in the query volume, producing $m$ individual bounding boxes:
$[\bm{z}^{i}_{min},\bm{z}^{i}_{max},\bm{x}^{i}_{min},\bm{x}^{i}_{max},\bm{y}^{i}_{min},\bm{y}^{i}_{max}]$
for each segment $i$. This refined approach provides a much more accurate
spatial representation of the anatomical structure across its entirety.

####  3.2.3 SAM Segmentation

After generating the bbox prompts for each slice, we transition to the
segmentation stage. For this, we utilize both SAM and MedSAM, a specialized
variant of SAM that has been fine-tuned for medical image datasets. Both
models serve as the foundation for our segmentation tasks. The versatility of
SAM and the domain-specific adaptations of MedSAM help us provide robust
segmentation results, thereby adding to the overall efficacy of the MedSLAM
system.

##  4 Experiments

###  4.1 Dataset

Our MedLAM model is trained on an extensive set of 16 datasets, which
collectively comprise a total of 14,012 CT scans. These scans encompass
various regions of the human body, providing comprehensive anatomical
coverage. An overview of the training datasets is provided in Table 1. The
diverse and abundant training data ensures the robustness and generalizability
of our model across different medical imaging contexts.

To validate the effectiveness of our approach, we integrate MedLAM with two
segmentation backbones: SAM[27] and MedSAM[37]. We test these combined models
on two CT segmentation datasets: 1) StructSeg19 Task1 dataset
111https://structseg2019.grand-challenge.org/Home/ for the 22 Head-and-Neck
(HaN) organs with 50 scans; 2) the WORD dataset [36] for the 16 abdomen organs
with 120 scans.

Table 1: Detailed information of the 16 CT datasets for MedLAM training.

Dataset | Number | Anatomical Region
---|---|---
GLIA [6] | 1338 | HaN
ACRIN 6685[35] | 260 | HaN
OPC-Radiomics[28] | 606 | HaN
Head-Neck-PET-CT[50] | 298 | HaN
HNSCC [15] | 591 | HaN/Thorax/Abdomen
autoPET [14] | 1014 | Whole
MELA [9] | 770 | Thorax
LIDC-IDRI[2] | 1308 | Thorax
STOIC2021[48] | 2000 | Thorax
MSD-Lung[1] | 95 | Thorax
CBIS-DDSM[29] | 2620 | Thorax
AMOS 2022[25] | 500 | Thorax/Abdomen
Kits19[20] | 141 | Abdomen
MSD-Colon[1] | 190 | Abdomen
MSD-Pancreas[1] | 281 | Abdomen
FLARE2022[38] | 2000 | Abdomen
Total | 14,012 | Whole

###  4.2 Implementation Details

Our model was trained using four NVIDIA GTX 3090 Ti GPUs. We utilized the Adam
optimizer [26] with a batch size of 8, an initial learning rate of $10^{-3}$,
and a training duration of 250 epochs.

In terms of pre-processing for MedLAM’s training and inference, we rescaled
the voxel spacing to [3,3,3] mm and standardized the cropping patch sizes to
64$\times$64$\times$64 pixels. To ensure that the scanning range was fully
covered, we set the parameter $r$ as [1500, 600, 600].

In both the StructSeg HaN and WORD datasets, we start by randomly selecting 5
scans as support volumes. In the Whole-Patch Localization (WPL) approach, for
each organ in these scans, we compute the six extreme coordinates and then
average these coordinates and features across the 5 selected images. This
creates an average representation of latent coordinates and features for each
of the organ’s extreme points.

When employing Sub-Patch Localization (SPL), we deviate from this process.
Instead of focusing on the extreme coordinates of the organ, we divide each
target organ into a number of sub-patches, denoted as $i$. The value of $i$ is
determined by taking the minimum span of the target organ across all support
volumes and dividing it by the predefined patch distance $n$ mm. This ensures
that we have a consistent number of sub-patches for each target organ across
all support volumes. For each of these $i$ sub-patches, we crop, compute, and
integrate the latent coordinates and features across the 5 support volumes,
offering a more granular representation. As a result, we obtain the latent
coordinates and features for 6$i$ points.

Upon obtaining the latent coordinates and features for each target point, we
proceed to locate the corresponding 3D bounding boxes (bbox) in the query
volume, yielding a bbox for each slice, as elaborated in Sec. 3.2. To
accommodate for potential variations in organ size and shape, we extend each
slice’s bbox prompt by [10, 10] pixels in the $x$ and $y$ directions,
respectively. This allows us to ensure that the entire target structure is
adequately captured for subsequent processing.

Upon utilizing SAM and MedSAM, the original images are subjected to a separate
preprocessing routine in line with the standard procedures described in the
original MedSAM methodology. This includes adjusting the slice resolution to 3
× 1024 × 1024, normalizing the intensity.

To guarantee the robustness of our experimental results, we repeat this random
selection and averaging process five times. The final performance metrics are
obtained by averaging the outcomes of these iterations.

###  4.3 Evaluation Metrics

To evaluate the localization accuracy of MedLAM, we apply two widely utilized
metrics [23, 56, 31]: the Intersection over Union (IoU) and Wall Distance
(WD). The IoU overlap ratio is defined as:

$IoU=\frac{\bm{V}_{p}\cap\bm{V}_{gt}}{\bm{V}_{p}\cup\bm{V}_{gt}},$ (8)

where $V_{p}$ and $V_{gt}$ denote the volume of the predicted bbox and the
ground-truth bbox, respectively. The IoU metric (0-1, 0: mismatch; 1:
perfectly match) is employed to assess the relative accuracy of the predicted
bounding box with respect to the ground truth.

For Wall Distance (WD), it quantifies the average absolute distance difference
between the boundaries of the predicted bbox and the ground-truth bbox.
Specifically, it is defined as:

$WD=\frac{1}{6}\sum{|d_{p,i}-d_{gt,i}|},$ (9)

where $d_{p,i}$ and $d_{gt,i}$ are the distances of the six boundaries (top,
bottom, left, right, front, back) of the predicted and ground-truth bbox,
respectively. The pixel spacing is implicitly taken into account in the
distance measurement.

As for the segmentation metrics, we use Dice Similarity Coefficient (DSC)
score and 95% Hausdorff Distance ( $\text{HD}_{95}$ ) as evaluation metrics to
quantitatively evaluate our method. Dice score (0-1, 0: mismatch; 1: perfectly
match) measures the overlap of the prediction $\bm{A}$ and ground truth
$\bm{B}$, and is defined as:

$DSC(\bm{A},\bm{B})=\frac{2|\bm{A}\cap\bm{B}|}{|\bm{A}|+|\bm{B}|}\cdot 100\%$
(10)

The maximum Hausdorff Distance is the maximum distance of a set $\bm{A}$ to
the nearest point in the other set $\bm{B}$, defined as:

$d_{H}(\bm{A},\bm{B})=\max\\{\max\limits_{a\in\bm{A}}\min_{b\in\bm{B}}d(a,b),\max\limits_{b\in\bm{B}}\min\limits_{a\in\bm{A}}d(a,b)\\}$
(11)

HD95 is similar to maximum HD, and it is based on the calculation of the 95th
percentile of the distances between boundary points in $\bm{A}$ and $\bm{B}$.

###  4.4 Evaluation of MedLAM

####  4.4.1 Comparison with Other Methods

Table 2: MedLAM v.s. fully-supervised & zero-shot localization models on
StructSeg Head-and-Neck and WORD datasets.

(a) StructSeg Head-and-Neck

Organs |  IoU $\uparrow$ (mean$\pm$std %) |  WD $\downarrow$ (mean$\pm$std mm)
---|---|---
MedLAM | nnDetection | MIU-VL | MedLAM | nnDetection | MIU-VL
(5-shot) | (fully-supervised) | (zero-shot) | (5-shot) | (fully-supervised) | (zero-shot)
Brain Stem |  70.3$\pm$8.0 | $\bm{75.7\pm 6.4}$ |  0.1$\pm$0.0 | $\bm{2.4\pm 1.4}$ |  8.5$\pm$11.9 |  223.0$\pm$74.3
Eye L | $\bm{66.2\pm 11.5}$ |  29.0$\pm$35.8 |  0.0$\pm$0.0 | $\bm{1.7\pm 1.0}$ |  7.2$\pm$13.5 |  162.8$\pm$73.9
Eye R | $\bm{63.2\pm 11.2}$ |  42.7$\pm$28.4 |  0.0$\pm$0.0 | $\bm{1.8\pm 1.1}$ |  8.0$\pm$10.6 |  163.4$\pm$61.1
Lens L | $\bm{11.5\pm 11.0}$ |  10.2$\pm$16.0 |  0.0$\pm$0.0 | $\bm{1.9\pm 1.1}$ |  13.8$\pm$19.5 |  220.2$\pm$89.7
Lens R |  17.9$\pm$14.5 | $\bm{24.3\pm 22.2}$ |  0.0$\pm$0.0 | $\bm{1.8\pm 1.2}$ |  16.8$\pm$19.9 |  219.1$\pm$75.6
Opt Nerve L | $\bm{29.7\pm 27.0}$ |  15.4$\pm$15.6 |  0.0$\pm$0.0 | $\bm{2.0\pm 1.3}$ |  9.5$\pm$14.9 |  173.0$\pm$56.3
Opt Nerve R | $\bm{27.6\pm 25.4}$ |  23.6$\pm$17.1 |  0.0$\pm$0.0 | $\bm{2.1\pm 1.3}$ |  16.2$\pm$18.5 |  175.7$\pm$47.8
Opt Chiasma |  9.7$\pm$19.4 | $\bm{32.4\pm 9.4}$ |  0.5$\pm$0.0 | $\bm{2.8\pm 1.5}$ |  9.0$\pm$13.0 |  177.0$\pm$25.2
Temporal Lobes L | $\bm{70.9\pm 8.7}$ |  45.3$\pm$37.1 |  0.5$\pm$0.1 | $\bm{3.2\pm 1.8}$ |  17.0$\pm$21.0 |  163.3$\pm$33.9
Temporal Lobes R | $\bm{72.8\pm 10.3}$ |  26.8$\pm$33.8 |  0.0$\pm$0.2 | $\bm{3.1\pm 1.8}$ |  9.5$\pm$14.8 |  189.0$\pm$64.9
Pituitary |  8.9$\pm$18.5 | $\bm{25.6\pm 16.8}$ |  0.3$\pm$0.0 | $\bm{2.7\pm 1.6}$ |  24.4$\pm$19.5 |  229.2$\pm$84.7
Parotid Gland L | $\bm{63.1\pm 9.6}$ |  31.3$\pm$32.9 |  0.3$\pm$0.1 | $\bm{3.7\pm 1.9}$ |  11.8$\pm$15.5 |  162.7$\pm$40.2
Parotid Gland R | $\bm{62.7\pm 10.3}$ |  38.7$\pm$32.0 |  0.0$\pm$0.1 | $\bm{4.0\pm 2.1}$ |  13.6$\pm$20.8 |  170.1$\pm$44.6
Inner Ear L | $\bm{35.6\pm 8.2}$ |  34.1$\pm$25.2 |  0.0$\pm$0.0 | $\bm{2.5\pm 1.0}$ |  10.3$\pm$12.2 |  182.3$\pm$34.8
Inner Ear R | $\bm{33.7\pm 9.4}$ |  32.1$\pm$27.7 |  0.1$\pm$0.0 | $\bm{2.7\pm 1.1}$ |  15.3$\pm$16.9 |  185.4$\pm$28.0
Mid Ear L | $\bm{56.5\pm 12.9}$ |  31.0$\pm$32.9 |  0.1$\pm$0.1 | $\bm{3.4\pm 2.0}$ |  14.0$\pm$15.7 |  193.6$\pm$72.0
Mid Ear R | $\bm{56.5\pm 12.8}$ |  51.1$\pm$27.5 |  0.0$\pm$0.1 | $\bm{3.5\pm 2.1}$ |  15.0$\pm$17.1 |  204.6$\pm$77.5
TM Joint L | $\bm{40.7\pm 15.1}$ |  26.8$\pm$22.3 |  0.0$\pm$0.0 | $\bm{2.4\pm 1.3}$ |  11.3$\pm$13.9 |  188.8$\pm$57.6
TM Joint R | $\bm{38.7\pm 15.3}$ |  26.0$\pm$18.4 |  0.5$\pm$0.0 | $\bm{2.3\pm 1.2}$ |  21.4$\pm$21.4 |  200.2$\pm$64.7
Spinal Cord | $\bm{57.7\pm 16.7}$ |  10.4$\pm$14.8 |  0.9$\pm$0.2 | $\bm{2.7\pm 1.9}$ |  11.3$\pm$15.1 |  224.5$\pm$120.1
Mandible L | $\bm{80.9\pm 6.2}$ |  36.1$\pm$37.3 |  1.0$\pm$0.3 | $\bm{2.5\pm 1.6}$ |  13.8$\pm$17.7 |  164.2$\pm$73.1
Mandible R | $\bm{80.8\pm 5.9}$ |  34.3$\pm$40.0 |  1.0$\pm$0.4 | $\bm{2.6\pm 1.5}$ |  21.4$\pm$20.4 |  155.2$\pm$59.6
Average | $\bm{48.0\pm 13.1}$ |  31.9$\pm$25.0 |  0.2$\pm$0.1 | $\bm{2.6\pm 1.5}$ |  13.6$\pm$16.5 |  187.6$\pm$61.8

(b) WORD

Organs |  IoU $\uparrow$ (mean$\pm$std %) |  WD $\downarrow$ (mean$\pm$std mm)
---|---|---
MedLAM | nnDetection | MIU-VL | MedLAM | nnDetection | MIU-VL
(5-shot) | (fully-supervised) | (zero-shot) | (5-shot) | (fully-supervised) | (zero-shot)
Liver | $\bm{73.0\pm 11.6}$ |  52.7$\pm$10.2 |  10.6$\pm$2.1 | $\bm{10.5\pm 9.1}$ |  23.8$\pm$31.5 |  80.4$\pm$37.2
Spleen | $\bm{70.9\pm 13.3}$ |  38.2$\pm$17.5 |  1.7$\pm$1.0 | $\bm{5.7\pm 5.9}$ |  14.6$\pm$21.4 |  142.5$\pm$49.8
Kidney L | $\bm{71.0\pm 15.9}$ |  69.6$\pm$27.2 |  0.9$\pm$0.5 | $\bm{5.0\pm 8.4}$ |  13.9$\pm$22.1 |  164.3$\pm$69.2
Kidney R | $\bm{76.0\pm 13.8}$ |  74.2$\pm$11.5 |  0.8$\pm$0.2 | $\bm{3.6\pm 4.6}$ |  20.8$\pm$28.6 |  166.1$\pm$81.3
Stomach | $\bm{49.1\pm 14.3}$ |  36.3$\pm$23.3 |  4.7$\pm$2.1 |  17.7$\pm$12.8 | $\bm{17.0\pm 25.1}$ |  102.8$\pm$25.1
Gallbladder |  12.0$\pm$11.0 | $\bm{29.3\pm 18.2}$ |  0.2$\pm$0.2 | $\bm{16.9\pm 10.3}$ |  21.2$\pm$29.6 |  153.2$\pm$38.5
Esophagus |  44.2$\pm$17.9 | $\bm{58.5\pm 13.2}$ |  0.3$\pm$0.2 | $\bm{6.8\pm 6.7}$ |  16.8$\pm$18.9 |  153.8$\pm$22.7
Pancreas |  44.1$\pm$17.3 | $\bm{53.4\pm 17.6}$ |  1.6$\pm$0.6 | $\bm{12.2\pm 8.8}$ |  19.1$\pm$28.5 |  135.2$\pm$33.8
Duodenum | $\bm{44.5\pm 17.5}$ |  42.0$\pm$16.4 |  1.2$\pm$0.4 | $\bm{12.7\pm 9.1}$ |  20.5$\pm$26.1 |  139.8$\pm$31.0
Colon | $\bm{67.0\pm 13.0}$ |  28.5$\pm$10.9 |  17.9$\pm$4.4 | $\bm{15.6\pm 11.8}$ |  22.8$\pm$35.8 |  83.3$\pm$33.6
Intestine | $\bm{62.6\pm 11.0}$ |  24.6$\pm$10.7 |  12.2$\pm$4.6 | $\bm{15.4\pm 9.8}$ |  20.5$\pm$29.5 |  106.0$\pm$53.1
Adrenal |  40.4$\pm$17.7 | $\bm{57.3\pm 12.7}$ |  0.4$\pm$0.2 | $\bm{7.6\pm 5.5}$ |  16.1$\pm$26.6 |  160.2$\pm$58.0
Rectum |  49.3$\pm$15.3 | $\bm{62.6\pm 16.9}$ |  0.6$\pm$0.3 | $\bm{8.6\pm 6.1}$ |  17.4$\pm$24.6 |  203.1$\pm$146.3
Bladder |  55.4$\pm$16.9 | $\bm{63.0\pm 17.0}$ |  1.2$\pm$0.7 | $\bm{8.1\pm 7.6}$ |  24.1$\pm$33.2 |  167.7$\pm$96.9
Head of Femur L | $\bm{76.7\pm 16.7}$ |  20.0$\pm$26.8 |  1.6$\pm$2.4 | $\bm{5.3\pm 16.0}$ |  14.5$\pm$23.0 |  175.1$\pm$127.4
Head of Femur R | $\bm{69.4\pm 14.4}$ |  47.7$\pm$34.0 |  1.5$\pm$0.8 | $\bm{5.9\pm 6.9}$ |  23.8$\pm$41.5 |  163.4$\pm$117.5
Average | $\bm{56.6\pm 14.8}$ |  47.4$\pm$17.8 |  3.6$\pm$1.3 | $\bm{9.9\pm 8.7}$ |  19.2$\pm$27.9 |  143.6$\pm$63.8

Table 3: Ablation study of Unified Anatomical Mapping (UAM) and Multi Scale
Similarity (MSS) in MedLAM: IoU score (mean$\pm$std%) on the WORD dataset

Organs | UAM | ✗ | UAM | UAM | UAM | UAM | UAM | UAM | UAM
---|---|---|---|---|---|---|---|---|---
+ | + | + | + | + | + | + | + | +
✗ |  $\bm{F}^{2}$ \+ $\bm{F}^{1}$ \+ $\bm{F}^{0}$ | $\bm{F}^{2}$ | $\bm{F}^{1}$ | $\bm{F}^{0}$ |  $\bm{F}^{1}$ \+ $\bm{F}^{0}$ |  $\bm{F}^{2}$ \+ $\bm{F}^{0}$ |  $\bm{F}^{2}$ \+ $\bm{F}^{1}$ |  $\bm{F}^{2}$ \+ $\bm{F}^{1}$ \+ $\bm{F}^{0}$
Liver | 57.1$\pm$10.5 | 39.7$\pm$27.4 | 76.0$\pm$8.3 | 76.7$\pm$7.9 | 71.5$\pm$10.1 | 75.0$\pm$09.1 | $\bm{78.5\pm 7.7}$ | 76.6$\pm$7.7 | 73.0$\pm$11.6
Spleen | 40.1$\pm$7.7 | 49.3$\pm$34.2 | 63.0$\pm$15.7 | 72.1$\pm$16.0 | 69.1$\pm$21.7 | $\bm{74.7\pm 15.2}$ | 74.4$\pm$14.1 | 72.6$\pm$12.9 | 70.9$\pm$13.3
Kidney L | 45.9$\pm$11.4 | 39.9$\pm$30.7 | 60.5$\pm$10.4 | 58.9$\pm$19.4 | 52.1$\pm$23.4 | 54.5$\pm$22.9 | 64.8$\pm$16.4 | 63.7$\pm$14.8 | $\bm{71.0\pm 15.9}$
Kidney R | 40.4$\pm$9.3 | 40.6$\pm$34.6 | 62.7$\pm$12.1 | 68.2$\pm$14.4 | 54.4$\pm$27.8 | 73.0$\pm$13.1 | $\bm{76.3\pm 8.9}$ | 70.6$\pm$9.4 | 76.0$\pm$13.8
Stomach | 38.7$\pm$12.2 | 25.4$\pm$21.9 | 47.4$\pm$15.7 | 43.4$\pm$15.6 | 46.4$\pm$16.7 | 47.2$\pm$17.6 | 45.9$\pm$12.9 | 47.3$\pm$14.8 | $\bm{49.1\pm 14.3}$
Gallbladder | 10.2$\pm$12.9 | 4.1$\pm$9.5 | 10.1$\pm$9.5 | 7.4$\pm$7.1 | 7.3$\pm$8.1 | 9.3$\pm$13.1 | 9.2$\pm$6.8 | 8.9$\pm$7.0 | $\bm{12.0\pm 11.0}$
Esophagus | 22.0$\pm$9.2 | 1.5$\pm$3.4 | 41.0$\pm$15.6 | 39.5$\pm$18.4 | 46.4$\pm$20.4 | 43.6$\pm$19.2 | 44.2$\pm$20.5 | 41.0$\pm$18.5 | $\bm{44.2\pm 17.9}$
Pancreas | 42.9$\pm$13.5 | 13.6$\pm$15.5 | 39.8$\pm$11.5 | 44.0$\pm$18.8 | 34.1$\pm$21.6 | 42.5$\pm$17.4 | $\bm{45.0\pm 18.6}$ | 42.8$\pm$16.2 | 44.1$\pm$17.3
Duodenum | 32.2$\pm$10.5 | 31.4$\pm$17.8 | 45.8$\pm$16.2 | 35.8$\pm$20.3 | 31.6$\pm$18.2 | 39.3$\pm$17.6 | 42.8$\pm$15.4 | $\bm{51.0\pm 13.4}$ | 44.5$\pm$17.5
Colon | 60.9$\pm$13.4 | 42.8$\pm$25.6 | 63.7$\pm$12.9 | 65.3$\pm$12.1 | 61.0$\pm$11.1 | 65.2$\pm$11.8 | 64.5$\pm$11.7 | 63.2$\pm$12.5 | $\bm{67.0\pm 13.0}$
Intestine | 58.7$\pm$11.6 | 27.7$\pm$21.1 | 63.1$\pm$13.0 | 61.8$\pm$16.8 | 62.0$\pm$17.0 | 61.9$\pm$17.6 | $\bm{65.2\pm 13.1}$ | 63.9$\pm$14.0 | 62.6$\pm$11.0
Adrenal | 29.8$\pm$17.0 | 10.5$\pm$15.6 | 36.8$\pm$16.1 | 30.5$\pm$18.8 | 32.2$\pm$19.7 | 35.0$\pm$19.7 | 38.1$\pm$15.7 | 38.6$\pm$13.5 | $\bm{40.4\pm 17.7}$
Rectum | 34.6$\pm$10.5 | 47.9$\pm$18.2 | 49.1$\pm$12.4 | 47.5$\pm$11.6 | 32.1$\pm$19.4 | 47.9$\pm$12.7 | 51.8$\pm$12.7 | $\bm{53.3\pm 13.7}$ | 49.3$\pm$15.3
Bladder | 33.9$\pm$15.2 | 45.8$\pm$23.0 | 49.5$\pm$18.1 | 55.3$\pm$21.1 | 49.6$\pm$20.1 | 52.3$\pm$22.2 | 50.8$\pm$20.4 | $\bm{55.4\pm 16.9}$ | 55.4$\pm$16.9
Head of Femur L | 43.7$\pm$10.8 | 71.9$\pm$22.4 | 71.8$\pm$8.5 | 80.2$\pm$8.7 | $\bm{84.1\pm 8.9}$ | 83.1$\pm$8.9 | 81.6$\pm$9.1 | 77.9$\pm$8.5 | 76.7$\pm$16.7
Head of Femur R | 36.2$\pm$10.9 | 55.3$\pm$29.4 | 63.8$\pm$13.3 | 70.9$\pm$12.5 | $\bm{75.9\pm 13.7}$ | 74.0$\pm$14.0 | 70.2$\pm$14.2 | 68.3$\pm$13.9 | 69.4$\pm$14.4
Average | 39.2$\pm$11.7 | 34.2$\pm$21.9 | 52.8$\pm$13.1 | 53.6$\pm$15.0 | 50.6$\pm$17.4 | 54.9$\pm$15.8 | 56.5$\pm$13.6 | 55.9$\pm$13.2 | $\bm{56.6\pm 14.8}$

In this section, with the purpose of validating the performance of our
universal localization model MedLAM. We have conducted a comparative analysis
between MedLAM and two well-established methodologies in the medical image
detection field: 1) nnDetection [4], which serves as a highly robust fully-
supervised medical image detection baseline; 2) MIU-VL [44], a pioneer work
utilizing visual-language foundation model for zero-shot medical image
localization.

For nnDetection, we followed the automated data processing strategy and
performed five-fold cross-training approach to identify the optimal network
hyperparameters. Regarding MIU-VL, we follow the procedures outlined in the
original paper, using GLIP [32] as the foundation model for zero-shot
detection. In order to generate the required text prompts for MIU-VL, we
utilized ChatGPT [42] to generate prompts containing descriptions of [COLOR],
[SHAPE], [LOCATION], and [CLASS] for different organs. Since the MIU-VL
network is designed to directly handle 2D images, we performed the detection
process on 3D medical images slice by slice. Subsequently, we reconstructed
the obtained bounding boxes back into the 3D image space.

The final experimental results are presented in Table 2(b). MedLAM
significantly outperforms both nnDetection and MIU-VL on the StructSeg HaN and
WORD datasets. Specifically, on the StructSeg HaN dataset, MedLAM achieved an
average IoU of 48%, substantially surpassing nnDetection’s 16.1%. We also
observed that MIU-VL was only able to delineate the entire body area
regardless of the organ-specific prompt, highlighting a substantial gap
between foundation models pre-trained on natural images and their
applicability in the medical domain. These results also indicate that such
natural image foundation models may struggle to provide meaningful
localization in low-contrast medical images. In contrast, our MedLAM model,
which incorporates strong medical domain knowledge, is capable of localizing
any anatomical structure with just a few annotated examples, exceeding the
performance of fully supervised models.

To be more specific, in the StructSeg HaN dataset, the organ with the largest
average WD (Wall Distance) is the right parotid gland, which does not exceed
4mm. This suggests that MedLAM can perform highly accurate localization for
head and neck organs. On the other hand, for the WORD dataset, the abdominal
soft tissue organs have blurry boundaries and significant deformations,
resulting in larger average WD values. However, the WD for each organ is
generally within 15mm.

In summary, MedLAM exhibits robust localization performance and proves to be a
versatile foundational model for localizing any type of structure in 3D
medical imaging. For smaller organs, although the IoU scores are relatively
lower, the subsequent preprocessing step of expanding the 3D bounding box
ensures that these organs are adequately captured for segmentation. This makes
the model practically applicable across a broad spectrum of organ sizes. This
approach provides a pragmatic solution to the challenge of localizing smaller
organs with MedLAM, thus achieving a balanced performance across different
organ sizes.

Table 4: Comparison of segmentation performance between MedLAM-generated
bounding box prompts and manually annotated prompts on the StructSeg Head-and-
Neck dataset. Metrics include Dice Similarity Coefficient (DSC) and Hausdorff
Distance (HD95).

(a) StructSeg Head-and-Neck

Organs |  DSC $\uparrow$ (mean$\pm$std %) |  HD95 $\downarrow$ (mean$\pm$std mm)
---|---|---
MedLAM | Manual Prompt | MedLAM | Manual Prompt
SAM | MedSAM | SAM | MedSAM | SAM | MedSAM | SAM | MedSAM
Brain Stem |  63.5$\pm$6.3 | $\bm{73.3\pm 5.0}$ |  66.2$\pm$2.7 | $\bm{75.9\pm 2.7}$ |  8.4$\pm$1.6 | $\bm{6.2\pm 1.2}$ |  8.6$\pm$1.0 | $\bm{6.1\pm 1.1}$
Eye L |  64.5$\pm$6.5 | $\bm{69.3\pm 5.9}$ |  65.7$\pm$5.6 | $\bm{70.0\pm 4.6}$ |  8.0$\pm$1.5 | $\bm{5.8\pm 1.0}$ |  7.9$\pm$1.3 | $\bm{5.9\pm 1.1}$
Eye R |  67.3$\pm$5.8 | $\bm{69.4\pm 5.5}$ |  68.9$\pm$4.8 | $\bm{69.1\pm 5.2}$ |  7.6$\pm$1.9 | $\bm{5.7\pm 1.2}$ |  7.4$\pm$1.2 | $\bm{6.2\pm 1.1}$
Lens L |  15.9$\pm$7.8 | $\bm{16.0\pm 6.7}$ | $\bm{22.7\pm 5.6}$ |  19.2$\pm$3.5 | $\bm{8.0\pm 1.5}$ |  8.6$\pm$1.7 |  10.0$\pm$2.3 | $\bm{9.5\pm 1.3}$
Lens R |  13.8$\pm$8.8 | $\bm{14.0\pm 5.5}$ | $\bm{22.8\pm 7.7}$ |  17.3$\pm$4.6 | $\bm{9.3\pm 1.5}$ |  9.6$\pm$1.3 | $\bm{9.7\pm 2.1}$ |  10.9$\pm$2.3
Opt Nerve L | $\bm{23.7\pm 6.1}$ |  23.5$\pm$6.2 |  32.6$\pm$9.3 | $\bm{34.7\pm 5.4}$ |  9.0$\pm$1.7 | $\bm{8.6\pm 1.1}$ |  10.1$\pm$2.5 | $\bm{8.4\pm 1.3}$
Opt Nerve R | $\bm{27.8\pm 10.2}$ |  26.3$\pm$6.6 |  28.5$\pm$6.7 | $\bm{32.4\pm 5.2}$ |  9.2$\pm$3.2 | $\bm{9.0\pm 1.4}$ |  11.3$\pm$2.2 | $\bm{9.4\pm 1.0}$
Opt Chiasma |  11.4$\pm$10.6 | $\bm{14.4\pm 11.4}$ |  39.8$\pm$10.2 | $\bm{39.8\pm 8.1}$ |  10.2$\pm$1.4 | $\bm{8.8\pm 1.6}$ |  7.2$\pm$1.7 | $\bm{6.6\pm 1.2}$
Temporal Lobes L |  28.2$\pm$15.2 | $\bm{78.3\pm 3.5}$ |  36.8$\pm$16.6 | $\bm{83.5\pm 1.9}$ |  16.4$\pm$3.5 | $\bm{9.0\pm 2.1}$ |  13.7$\pm$3.5 | $\bm{6.6\pm 1.4}$
Temporal Lobes R |  24.1$\pm$17.4 | $\bm{78.0\pm 4.3}$ |  30.7$\pm$17.8 | $\bm{84.4\pm 1.4}$ |  18.2$\pm$4.2 | $\bm{10.1\pm 2.4}$ |  15.1$\pm$4.1 | $\bm{6.1\pm 2.5}$
Pituitary | $\bm{12.5\pm 10.7}$ |  10.2$\pm$9.2 | $\bm{36.6\pm 16.2}$ |  27.5$\pm$12.5 | $\bm{8.7\pm 2.2}$ |  8.8$\pm$1.8 | $\bm{7.4\pm 2.0}$ |  10.0$\pm$3.6
Parotid Gland L |  15.5$\pm$11.9 | $\bm{59.6\pm 6.5}$ |  29.5$\pm$7.7 | $\bm{64.4\pm 3.4}$ |  17.6$\pm$3.0 | $\bm{10.3\pm 1.9}$ |  16.0$\pm$1.9 | $\bm{9.3\pm 1.9}$
Parotid Gland R |  17.2$\pm$9.9 | $\bm{57.1\pm 6.8}$ |  29.6$\pm$6.5 | $\bm{64.9\pm 5.0}$ |  19.8$\pm$5.0 | $\bm{13.5\pm 3.5}$ |  14.0$\pm$2.1 | $\bm{9.5\pm 1.6}$
Inner Ear L |  40.4$\pm$11.8 | $\bm{42.3\pm 9.9}$ |  59.7$\pm$11.7 | $\bm{62.1\pm 9.7}$ |  9.4$\pm$1.7 | $\bm{7.6\pm 1.5}$ |  7.6$\pm$2.1 | $\bm{5.9\pm 1.8}$
Inner Ear R | $\bm{48.9\pm 9.5}$ |  45.9$\pm$11.2 | $\bm{67.3\pm 10.4}$ |  66.9$\pm$6.1 |  8.4$\pm$1.7 | $\bm{7.0\pm 1.7}$ |  6.8$\pm$1.9 | $\bm{5.4\pm 1.2}$
Mid Ear L | $\bm{64.6\pm 14.3}$ |  59.7$\pm$9.6 | $\bm{71.0\pm 11.7}$ |  67.5$\pm$6.3 |  7.9$\pm$2.6 | $\bm{7.7\pm 1.6}$ |  6.4$\pm$2.8 | $\bm{6.1\pm 1.1}$
Mid Ear R | $\bm{64.7\pm 13.1}$ |  59.3$\pm$11.2 | $\bm{73.3\pm 7.7}$ |  65.0$\pm$8.9 |  8.4$\pm$2.4 | $\bm{7.6\pm 1.4}$ | $\bm{6.2\pm 1.9}$ |  6.6$\pm$1.2
TM Joint L |  38.3$\pm$10.1 | $\bm{39.0\pm 10.9}$ |  59.7$\pm$12.8 | $\bm{61.2\pm 4.9}$ |  10.2$\pm$2.1 | $\bm{7.7\pm 1.6}$ |  6.8$\pm$1.9 | $\bm{6.0\pm 0.7}$
TM Joint R | $\bm{41.5\pm 10.0}$ |  38.3$\pm$9.5 |  59.2$\pm$17.8 | $\bm{60.0\pm 7.7}$ |  9.6$\pm$1.6 | $\bm{8.1\pm 1.4}$ |  8.2$\pm$1.9 | $\bm{6.3\pm 1.1}$
Spinal Cord |  27.9$\pm$8.3 | $\bm{34.7\pm 6.9}$ |  38.0$\pm$7.7 | $\bm{40.3\pm 6.1}$ |  15.5$\pm$2.7 | $\bm{13.3\pm 2.9}$ |  10.7$\pm$1.3 | $\bm{8.8\pm 0.9}$
Mandible L | $\bm{78.0\pm 4.9}$ |  66.7$\pm$6.1 | $\bm{86.0\pm 2.1}$ |  76.8$\pm$4.7 | $\bm{11.1\pm 2.8}$ |  11.8$\pm 2.1$ | $\bm{5.5\pm 1.7}$ |  9.0$\pm$4.5
Mandible R | $\bm{71.4\pm 4.0}$ |  66.0$\pm$4.7 | $\bm{81.2\pm 2.7}$ |  75.0$\pm$4.5 |  22.6$\pm$4.7 | $\bm{12.3\pm 2.6}$ |  21.6$\pm$5.4 | $\bm{9.6\pm 3.5}$
Average |  39.1$\pm$9.7 | $\bm{47.3\pm 7.4}$ |  50.3$\pm$9.2 | $\bm{57.2\pm 5.6}$ |  11.5$\pm$2.5 | $\bm{9.0\pm 1.8}$ |  9.9$\pm$2.2 | $\bm{7.6\pm 1.7}$

(b) WORD

Organs |  DSC $\uparrow$ (mean$\pm$std %) |  HD9 $\downarrow$ (mean$\pm$std mm)
---|---|---
MedLAM | Manual Prompt | MedLAM | Manual Prompt
SAM | MedSAM | SAM | MedSAM | SAM | MedSAM | SAM | MedSAM
Liver | $\bm{66.0\pm 10.1}$ |  23.8$\pm$8.6 | $\bm{84.2\pm 6.3}$ |  46.6$\pm$14.7 |  53.6$\pm$10.4 | $\bm{51.5\pm 8.6}$ | $\bm{21.8\pm 11.9}$ |  36.8$\pm$11.6
Spleen | $\bm{61.7\pm 14.4}$ |  36.3$\pm$13.2 | $\bm{85.3\pm 5.1}$ |  65.0$\pm$6.3 | $\bm{30.1\pm 16.1}$ |  30.5$\pm$14.4 | $\bm{10.8\pm 6.5}$ |  16.2$\pm$4.9
Kidney L | $\bm{82.1\pm 16.0}$ |  70.7$\pm$18.8 | $\bm{92.1\pm 1.6}$ |  84.1$\pm$5.2 |  19.2$\pm$18.6 | $\bm{18.5\pm 16.9}$ | $\bm{6.1\pm 1.9}$ |  8.2$\pm$1.6
Kidney R | $\bm{88.3\pm 4.8}$ |  77.3$\pm$6.2 | $\bm{92.9\pm 1.6}$ |  86.4$\pm$3.1 | $\bm{12.7\pm 5.7}$ |  13.4$\pm$4.1 | $\bm{6.0\pm 1.6}$ |  7.7$\pm$1.2
Stomach | $\bm{44.6\pm 15.0}$ |  37.2$\pm$14.7 |  77.1$\pm$10.6 | $\bm{80.3\pm 4.7}$ | $\bm{50.1\pm 27.0}$ |  51.7$\pm$27.6 |  12.7$\pm$4.2 | $\bm{11.5\pm 3.6}$
Gallbladder | $\bm{13.1\pm 17.3}$ |  10.2$\pm$13.4 | $\bm{72.7\pm 11.1}$ |  68.8$\pm$7.8 |  36.8$\pm$19.8 | $\bm{36.7\pm 20.4}$ | $\bm{6.0\pm 1.8}$ |  6.4$\pm$1.6
Esophagus | $\bm{36.6\pm 14.7}$ |  27.8$\pm$12.9 | $\bm{67.0\pm 6.4}$ |  63.1$\pm$7.7 |  21.1$\pm$12.7 | $\bm{19.8\pm 11.6}$ |  6.9$\pm$4.0 | $\bm{6.7\pm 3.2}$
Pancreas | $\bm{29.7\pm 14.1}$ |  21.4$\pm$9.7 | $\bm{64.4\pm 7.7}$ |  46.9$\pm$11.6 | $\bm{32.6\pm 13.5}$ |  33.1$\pm$10.8 | $\bm{15.9\pm 6.4}$ |  18.3$\pm$5.5
Duodenum | $\bm{26.0\pm 11.4}$ |  21.1$\pm$7.0 | $\bm{54.1\pm 13.2}$ |  51.0$\pm$11.9 |  31.1$\pm$9.7 | $\bm{30.9\pm 10.4}$ | $\bm{16.0\pm 4.6}$ |  17.1$\pm$5.9
Colon |  25.6$\pm$9.6 | $\bm{26.6\pm 8.9}$ |  41.8$\pm$6.8 | $\bm{44.1\pm 8.8}$ | $\bm{57.8\pm 17.8}$ |  60.6$\pm$17.8 |  49.4$\pm$15.3 | $\bm{49.1\pm 13.9}$
Intestine | $\bm{37.5\pm 7.6}$ |  34.1$\pm$8.4 | $\bm{61.4\pm 6.9}$ |  52.5$\pm$8.0 |  47.1$\pm$10.5 | $\bm{44.1\pm 10.0}$ |  21.7$\pm$4.2 | $\bm{21.0\pm 4.0}$
Adrenal |  3.3$\pm$3.9 | $\bm{10.0\pm 6.2}$ |  17.4$\pm$8.8 | $\bm{26.5\pm 5.9}$ |  30.8$\pm$8.7 | $\bm{29.9\pm 7.5}$ |  23.4$\pm$4.3 | $\bm{22.6\pm 2.7}$
Rectum | $\bm{50.1\pm 17.9}$ |  46.0$\pm$18.7 |  75.5$\pm$4.0 | $\bm{80.0\pm 3.6}$ |  18.6$\pm$7.0 | $\bm{17.4\pm 7.2}$ |  7.6$\pm$0.9 | $\bm{4.8\pm 0.6}$
Bladder | $\bm{65.3\pm 26.6}$ |  59.1$\pm$23.1 | $\bm{83.0\pm 15.5}$ |  82.9$\pm$8.0 |  20.5$\pm$10.2 | $\bm{19.8\pm 8.8}$ |  7.5$\pm$3.5 | $\bm{6.9\pm 2.0}$
Head of Femur L | $\bm{81.7\pm 3.9}$ |  71.5$\pm$4.1 | $\bm{90.5\pm 2.7}$ |  80.3$\pm$2.8 |  16.6$\pm$3.3 | $\bm{14.9\pm 1.6}$ | $\bm{9.2\pm 4.7}$ |  12.3$\pm$2.0
Head of Femur R | $\bm{80.1\pm 3.3}$ |  74.3$\pm$4.4 | $\bm{89.1\pm 3.4}$ |  83.2$\pm$2.7 |  19.5$\pm$2.6 | $\bm{13.4\pm 1.0}$ |  12.6$\pm$6.1 | $\bm{12.3\pm 7.7}$
Average | $\bm{49.5\pm 11.9}$ |  40.5$\pm$11.1 | $\bm{71.8\pm 7.0}$ |  65.1$\pm$7.0 |  31.1$\pm$12.1 | $\bm{30.4\pm 11.2}$ | $\bm{14.6\pm 5.1}$ |  16.1$\pm$4.5

Table 5: Comparison of DSC scores $\downarrow$ (mean $\pm$ std%) for MedLSAM
using different localization strategies on the StructSeg Head-and-Neck
dataset: Whole-Patch Localization (WPL) and Sub-Patch Localization (SPL) with
varying slice intervals $n$.

Organs | SAM | MedSAM
---|---|---
SPL | WPL | SPL | WPL
$n=6mm$ | $n=15mm$ | $n=30mm$ | ✗ | $n=6mm$ | $n=15mm$ | $n=30mm$ | ✗
Brain Stem | $\bm{63.5\pm 6.3}$ | 60.1$\pm$6.7 | 51.5$\pm$6.0 | 50.8$\pm$6.2 | $\bm{73.3\pm 5.0}$ | 71.9$\pm$4.8 | 67.4$\pm$4.2 | 66.6$\pm$4.4
Eye L | $\bm{64.5\pm 6.5}$ | 61.2$\pm$6.7 | 61.2$\pm$6.7 | 61.2$\pm$6.7 | $\bm{69.3\pm 5.9}$ | 66.4$\pm$5.7 | 66.4$\pm$5.7 | 66.4$\pm$5.7
Eye R | $\bm{67.3\pm 5.8}$ | 64.5$\pm$6.6 | 64.5$\pm$6.6 | 64.5$\pm$6.6 | $\bm{69.4\pm 5.5}$ | 67.8$\pm$6.0 | 67.8$\pm$6.0 | 67.8$\pm$6.0
Lens L | $\bm{15.9\pm 7.8}$ | 15.7$\pm$7.7 | 15.7$\pm$7.7 | 15.7$\pm$7.7 | $\bm{16.0\pm 6.7}$ | 15.8$\pm$6.7 | 15.8$\pm$6.7 | 15.8$\pm$6.7
Lens R | 13.8$\pm$8.8 | $\bm{14.1\pm 8.6}$ | 13.5$\pm$8.7 | 13.5$\pm$8.7 | $\bm{14.0\pm 5.5}$ | $\bm{14.0\pm 5.6}$ | 13.7$\pm$5.5 | 13.7$\pm$5.5
Opt Nerve L | 23.7$\pm$6.1 | $\bm{23.9\pm 5.9}$ | 23.7$\pm$6.1 | 23.7$\pm$6.1 | 23.5$\pm$6.2 | $\bm{23.6\pm 6.2}$ | 23.5$\pm$6.2 | 23.5$\pm$6.2
Opt Nerve R | $\bm{27.8\pm 10.2}$ | $\bm{27.8\pm 10.2}$ | $\bm{27.8\pm 10.2}$ | $\bm{27.8\pm 10.2}$ | $\bm{26.3\pm 6.6}$ | $\bm{26.3\pm 6.6}$ | $\bm{26.3\pm 6.6}$ | $\bm{26.3\pm 6.6}$
Opt Chiasma | $\bm{11.4\pm 10.6}$ | $\bm{11.4\pm 10.6}$ | $\bm{11.4\pm 10.6}$ | $\bm{11.4\pm 10.6}$ | $\bm{14.4\pm 11.4}$ | $\bm{14.4\pm 11.4}$ | $\bm{14.4\pm 11.4}$ | $\bm{14.4\pm 11.4}$
Temporal Lobes L | $\bm{28.2\pm 15.2}$ | 21.7$\pm$14.8 | 25.7$\pm$13.4 | 25.7$\pm$13.4 | $\bm{78.3\pm 3.5}$ | 73.1$\pm$3.5 | 71.0$\pm$3.6 | 71.0$\pm$3.6
Temporal Lobes R | $\bm{24.1\pm 17.4}$ | 18.2$\pm$17.7 | 18.5$\pm$16.4 | 18.5$\pm$16.4 | $\bm{78.0\pm 4.3}$ | 75.9$\pm$3.1 | 71.0$\pm$3.5 | 71.0$\pm$3.5
Pituitary | 12.5$\pm$10.7 | $\bm{12.6\pm 10.7}$ | $\bm{12.6\pm 10.7}$ | $\bm{12.6\pm 10.7}$ | 10.2$\pm$9.2 | $\bm{10.3\pm 9.4}$ | $\bm{10.3\pm 9.4}$ | $\bm{10.3\pm 9.4}$
Parotid Gland L | $\bm{15.5\pm 11.9}$ | 10.4$\pm$9.6 | 5.5$\pm$6.6 | 5.5$\pm$6.6 | $\bm{59.6\pm 6.5}$ | 56.4$\pm$6.6 | 48.1$\pm$7.3 | 48.1$\pm$7.3
Parotid Gland R | $\bm{17.2\pm 9.9}$ | 13.1$\pm$10.5 | 7.7$\pm$7.7 | 7.7$\pm$7.7 | $\bm{57.1\pm 6.8}$ | 54.6$\pm$6.8 | 45.3$\pm$7.8 | 45.3$\pm$7.8
Inner Ear L | $\bm{40.4\pm 11.8}$ | $\bm{40.4\pm 11.8}$ | $\bm{40.4\pm 11.8}$ | $\bm{40.4\pm 11.8}$ | $\bm{42.3\pm 9.9}$ | $\bm{42.3\pm 9.9}$ | $\bm{42.3\pm 9.9}$ | $\bm{42.3\pm 9.9}$
Inner Ear R | $\bm{48.9\pm 9.5}$ | $\bm{48.9\pm 9.5}$ | $\bm{48.9\pm 9.5}$ | $\bm{48.9\pm 9.5}$ | $\bm{45.9\pm 11.2}$ | $\bm{45.9\pm 11.2}$ | $\bm{45.9\pm 11.2}$ | $\bm{45.9\pm 11.2}$
Mid Ear L | $\bm{64.6\pm 14.3}$ | 62.3$\pm$14.7 | 62.3$\pm$14.7 | 62.3$\pm$14.7 | $\bm{59.7\pm 9.6}$ | 55.8$\pm$10.0 | 55.7$\pm$10.0 | 55.7$\pm$10.0
Mid Ear R | $\bm{64.7\pm 13.1}$ | 62.4$\pm$13.1 | 62.4$\pm$13.0 | 62.4$\pm$13.0 | $\bm{59.3\pm 11.2}$ | 53.6$\pm$12.1 | 53.4$\pm$11.9 | 53.4$\pm$11.9
TM Joint L | 38.3$\pm$10.1 | 38.3$\pm$10.1 | 38.3$\pm$10.1 | 38.3$\pm$10.1 | $\bm{39.0\pm 10.9}$ | $\bm{39.0\pm 10.9}$ | $\bm{39.0\pm 10.9}$ | $\bm{39.0\pm 10.9}$
TM Joint R | 41.5$\pm$10.0 | 41.5$\pm$10.0 | 41.5$\pm$10.0 | 41.5$\pm$10.0 | $\bm{38.3\pm 9.5}$ | $\bm{38.3\pm 9.5}$ | $\bm{38.3\pm 9.5}$ | $\bm{38.3\pm 9.5}$
Spinal Cord | 27.9$\pm$8.3 | 27.7$\pm$8.2 | 27.8$\pm$8.3 | 7.8$\pm$2.9 | $\bm{34.7\pm 6.9}$ | $\bm{34.7\pm 6.9}$ | $\bm{34.7\pm 6.9}$ | 11.7$\pm$3.9
Mandible L | $\bm{78.0\pm 4.9}$ | $\bm{78.0\pm 5.2}$ | 68.0$\pm$6.2 | 47.0$\pm$5.3 | $\bm{66.7\pm 6.1}$ | $\bm{66.7\pm 5.2}$ | 46.7$\pm$5.8 | 22.9$\pm$4.1
Mandible R | $\bm{71.4\pm 4.0}$ | 69.0$\pm$4.4 | 58.5$\pm$3.9 | 40.2$\pm$4.6 | $\bm{66.0\pm 4.7}$ | 64.7$\pm$4.9 | 48.2$\pm$5.0 | 25.9$\pm$3.2
Average | $\bm{39.1\pm 9.7}$ | 37.4$\pm$9.7 | 35.8$\pm$9.2 | 33.1$\pm$9.1 | $\bm{47.3\pm 7.4}$ | 46.0$\pm$7.4 | 43.0$\pm$7.5 | 39.8$\pm$7.2

Fig. 6: Visualization examples of segmentation results on WORD and StructSeg
Head-and-Neck datasets using pre-trained MedSAM and SAM, post landmark
localization with MedLAM.

####  4.4.2 Ablation Study of MedLAM

To assess the contributions of individual components within MedLAM, we conduct
a series of experiments on the WORD dataset. Specifically, we evaluate MedLAM
under the following conditions: 1) using only Unified Anatomical Mapping (UAM)
for coarse localization; 2) relying solely on Multi Scale Similarity (MSS) for
pinpointing the most similar point across the entire query image; 3) employing
RDR for initial coarse localization, followed by MSS with and without features
from each scale. The IoU scores obtained from these experiments are presented
in Table 3. As can be observed, when using either RDR or MSS independently,
the average IoU is below 50%, standing at 39.2% and 34.2%, respectively. This
could be attributed to the following reasons: 1) RDR operates under the
assumption of anatomical consistency, which may not hold true if the query
volume undergoes significant deformation; 2) MSS focuses on the similarity of
local features but lacks an understanding of global anatomical structures,
making it susceptible to interference from other areas in the image that are
similar to the query point. This results in the identification of false
positives, as evidenced by the high variance in the MSS detection results.

However, when initiating with a coarse localization point identified by RDR
and subsequently refining it with MSS, the detection accuracy is significantly
improved. As shown in Table 3, the average IoU can be increased from 39.2% to
56.6% when using multi-scale features for MSS-based refining. Furthermore,
using individual feature maps for $\bm{F}^{i}$ refining also improves the
average IoU to over 50%, validating the effectiveness of combining global
anatomical features with local pixel features.

###  4.5 Evaluation of MedLSAM

####  4.5.1 Effectness of MedLSAM

After validating the localization performance of MedLAM, we proceeded to
examine the segmentation performance of the proposed MedLSAM framework.
MedLSAM utilizes the localization information from MedLAM, together with
either SAM or MedSAM, to perform segmentation tasks.

In our evaluation, we also included manually-annotated bounding boxes to serve
as the performance upper bound for SAM and MedSAM on the WORD and StructSeg
HaN datasets. These were simulated based on the ground-truth masks, following
the same approach used for generating bboxes in MedSAM. During inference, a
bbox prompt was generated from each ground-truth mask, with a random
perturbation of 0-20 pixels introduced to mimic the potential inaccuracies
that would be present in manually-drawn bboxes.

It can be observed that MedLSAM demonstrates performance closely approximating
that achieved with manual prompts for many organs. For instance, in the case
of the brain stem, MedLSAM attains a DSC score of 73.3%, while the manual
prompt reaches 75.9%. However, for certain abdominal organs with substantial
shape variability, such as the pancreas, the performance gap between MedLSAM
and manual prompts is more significant, with a DSC score difference reaching
up to 35%. Fig. 6 visually presents the results of using MedLSAM for
localization and segmentation tasks on both datasets.

Overall, MedLSAM performs more comparably to manual prompts in the case of
head and neck organs, representing a pioneering and effective attempt towards
fully automating the use of SAM in medical imaging.

####  4.5.2 Ablation study of SPL

To validate the effectiveness of Sub-Patch Localization (SPL), we conducted
experiments on the StructSeg HaN dataset, comparing the performance of SPL at
different sub-patch intervals $n$ mm against the Whole-Patch Localization
(WPL) approach. As described in Sec. 4.2, the WPL method relies on the extreme
coordinates of the entire organ for localization. On the other hand, SPL
employs a more granular approach by segmenting the organ into multiple sub-
patches at intervals of $n$ mm.

As shown in Table 5, it is evident that the SPL approach consistently
outperforms WPL. Notably, for organs with significant morphological
variations, such as the brain stem and spinal cord, the DSC score improves by
more than 7% and 20%, respectively. Moreover, when $n$ is relatively small,
SPL enables finer-grained localization by concentrating on smaller, more
specific portions of the organ. This results in bounding boxes that better
approximate the organ’s actual shape at each slice, thereby enhancing the
accuracy of the subsequent segmentation process.

##  5 Discussion & Conclusions

In this study, we have introduced MedLSAM, the first completely automated,
two-stage medical adaptation of the SAM model, designed to significantly
alleviate the annotation workload in the segmentation of medical images. The
first stage, MedLAM, serves as a pioneering localization foundation model,
capable of precisely identifying anatomical structures in 3D medical images.
The second stage leverages the strength of SAM and its medical adaptations for
meticulous image segmentation. Together, these components form a fully
automated pipeline that significantly reduces the reliance on manual
annotations and annotated data.

This endeavor was fueled by the observation that the spatial distributions of
organs across different patients maintain strong similarities. Consequently,
MedLAM was designed to project every part of the scans onto a shared 3D latent
coordinate system, accurately localizing target anatomical parts within the
body. Coupling this approach with SAM’s segmentation capabilities led to an
efficient and accurate process for image segmentation.

Moreover, MedLSAM demonstrated its effectiveness across two 3D datasets
covering 38 different organs, providing robust evidence of its versatility.
Importantly, this automated approach is not burdened by an increased
annotation workload as data size increases. It also holds promise for direct
integration with potential future 3D SAM models in the medical field, which
could further enhance its performance and utility.

## Acknowledgments

## References

  * Antonelli et al. [2022] Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., et al., 2022\.  The medical segmentation decathlon.  Nature communications 13, 4128\.
  * Armato III et al. [2011] Armato III, S.G., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P., Zhao, B., Aberle, D.R., Henschke, C.I., Hoffman, E.A., et al., 2011\.  The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans.  Medical physics 38, 915–931.
  * Azizi et al. [2022] Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., MacWilliams, P., Mahdavi, S.S., Wulczyn, E., et al., 2022\.  Robust and efficient medical imaging with self-supervision.  arXiv preprint arXiv:2205.09723 .
  * Baumgartner et al. [2021] Baumgartner, M., Jäger, P.F., Isensee, F., Maier-Hein, K.H., 2021\.  nndetection: a self-configuring method for medical object detection, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24, Springer. pp. 530–539.
  * Bilic et al. [2023] Bilic, P., Christ, P., Li, H.B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G.E.H., Chartrand, G., et al., 2023.  The liver tumor segmentation benchmark (lits).  Medical Image Analysis 84, 102680\.
  * Bo et al. [2021] Bo, Z.H., Qiao, H., Tian, C., Guo, Y., Li, W., Liang, T., Li, D., Liao, D., Zeng, X., Mei, L., et al., 2021\.  Toward human intervention-free clinical diagnosis of intracranial aneurysm via deep neural network.  Patterns 2, 100197\.
  * Butoi et al. [2023] Butoi, V.I., Ortiz, J.J.G., Ma, T., Sabuncu, M.R., Guttag, J., Dalca, A.V., 2023\.  Universeg: Universal medical image segmentation.  arXiv preprint arXiv:2304.06131 .
  * Caron et al. [2021] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A., 2021.  Emerging properties in self-supervised vision transformers, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650–9660.
  * Chen [2022] Chen, C., 2022.  Miccai 2022 mela challenge: Mediastinal lesion analysis.  URL: https://mela.grand-challenge.org/Dataset/.
  * Chen et al. [2021] Chen, X., Lian, C., Deng, H.H., Kuang, T., Lin, H.Y., Xiao, D., Gateno, J., Shen, D., Xia, J.J., Yap, P.T., 2021.  Fast and accurate craniomaxillofacial landmark detection via 3d faster r-cnn.  IEEE Transactions on Medical Imaging 40, 3867–3878.
  * Cheng et al. [2023] Cheng, D., Qin, Z., Jiang, Z., Zhang, S., Lao, Q., Li, K., 2023.  Sam on medical images: A comprehensive study on three prompt modes.  arXiv preprint arXiv:2305.00035 .
  * Dai et al. [2016] Dai, J., Li, Y., He, K., Sun, J., 2016.  R-fcn: Object detection via region-based fully convolutional networks.  Advances in neural information processing systems 29\.
  * Dosovitskiy et al. [2020] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al., 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929 .
  * Gatidis et al. [2022] Gatidis, S., Hepp, T., Früh, M., La Fougère, C., Nikolaou, K., Pfannenberg, C., Schölkopf, B., Küstner, T., Cyran, C., Rubin, D., 2022\.  A whole-body fdg-pet/ct dataset with manually annotated tumor lesions.  Scientific Data 9, 601\.
  * Grossberg et al. [2018] Grossberg, A.J., Mohamed, A.S., Elhalawani, H., Bennett, W.C., Smith, K.E., Nolan, T.S., Williams, B., Chamchod, S., Heukelom, J., Kantor, M.E., et al., 2018\.  Imaging and clinical data archive for head and neck squamous cell carcinoma patients treated with radiotherapy.  Scientific data 5, 1–10.
  * He et al. [2017] He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017\.  Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969.
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * He et al. [2023a] He, S., Bao, R., Li, J., Grant, P.E., Ou, Y., 2023a.  Accuracy of segment-anything model (sam) in medical image segmentation tasks.  arXiv preprint arXiv:2304.09324 .
  * He et al. [2023b] He, S., Bao, R., Li, J., Stout, J., Bjornerud, A., Grant, P.E., Ou, Y., 2023b.  Computer-vision benchmark segment-anything model (sam) in medical images: Accuracy in 12 datasets.  arXiv preprint arXiv:2304.09324 .
  * Heller et al. [2020] Heller, N., Isensee, F., Maier-Hein, K.H., Hou, X., Xie, C., Li, F., Nan, Y., Mu, G., Lin, Z., Han, M., et al., 2020.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge.  Medical Image Analysis , 101821.
  * Huang et al. [2023a] Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al., 2023a.  Segment anything model for medical images?  arXiv preprint arXiv:2304.14660 .
  * Huang et al. [2023b] Huang, Z., Wang, H., Deng, Z., Ye, J., Su, Y., Sun, H., He, J., Gu, Y., Gu, L., Zhang, S., et al., 2023b.  Stu-net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training.  arXiv preprint arXiv:2304.06716 .
  * Humpire-Mamani et al. [2018] Humpire-Mamani, G.E., Setio, A.A.A., Van Ginneken, B., Jacobs, C., 2018\.  Efficient organ localization using multi-label convolutional neural networks in thorax-abdomen ct scans.  Physics in Medicine & Biology 63, 085003.
  * Hussain et al. [2021] Hussain, M.A., Hamarneh, G., Garbi, R., 2021.  Cascaded regression neural nets for kidney localization and segmentation-free volume estimation.  IEEE Transactions on Medical Imaging 40, 1555–1567.
  * Ji et al. [2022] Ji, Y., Bai, H., Yang, J., Ge, C., Zhu, Y., Zhang, R., Li, Z., Zhang, L., Ma, W., Wan, X., et al., 2022.  Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.  arXiv preprint arXiv:2206.08023 .
  * Kingma and Ba [2014] Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Kirillov et al. [2023] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al., 2023\.  Segment anything.  arXiv preprint arXiv:2304.02643 .
  * Kwan et al. [2018] Kwan, J.Y.Y., Su, J., Huang, S.H., Ghoraie, L.S., Xu, W., Chan, B., Yip, K.W., Giuliani, M., Bayley, A., Kim, J., et al., 2018.  Radiomic biomarkers to refine risk models for distant metastasis in hpv-related oropharyngeal carcinoma.  International Journal of Radiation Oncology* Biology* Physics 102, 1107–1116.
  * Lee et al. [2017] Lee, R.S., Gimenez, F., Hoogi, A., Miyake, K.K., Gorovoy, M., Rubin, D.L., 2017\.  A curated mammography data set for use in computer-aided detection and diagnosis research.  Scientific data 4, 1–9.
  * Lei et al. [2019] Lei, W., Wang, H., Gu, R., Zhang, S., Zhang, S., Wang, G., 2019.  Deepigeos-v2: Deep interactive segmentation of multiple organs from head and neck images with lightweight cnns, in: Large-Scale Annotation of Biomedical Data and Expert Label Synthesis and Hardware Aware Learning for Medical Imaging and Computer Assisted Intervention: International Workshops, LABELS 2019, HAL-MICCAI 2019, and CuRIOUS 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 13 and 17, 2019, Proceedings 4, Springer. pp. 61–69.
  * Lei et al. [2021] Lei, W., Xu, W., Gu, R., Fu, H., Zhang, S., Zhang, S., Wang, G., 2021\.  Contrastive learning of relative position regression for one-shot object localization in 3d medical images, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24, Springer. pp. 155–165.
  * Li et al. [2022] Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., et al., 2022.  Grounded language-image pre-training, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965–10975.
  * Liu et al. [2023] Liu, J., Zhang, Y., Chen, J.N., Xiao, J., Lu, Y., Landman, B.A., Yuan, Y., Yuille, A., Tang, Y., Zhou, Z., 2023.  Clip-driven universal model for organ segmentation and tumor detection.  arXiv preprint arXiv:2301.00785 .
  * Liu et al. [2022] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S., 2022.  A convnet for the 2020s, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976–11986.
  * Lowe et al. [2019] Lowe, V.J., Duan, F., Subramaniam, R.M., Sicks, J.D., Romanoff, J., Bartel, T., Yu, J.Q.M., Nussenbaum, B., Richmon, J., Arnold, C.D., et al., 2019\.  Multicenter trial of [18f] fluorodeoxyglucose positron emission tomography/computed tomography staging of head and neck cancer and negative predictive value and surgical impact in the n0 neck: results from acrin 6685.  Journal of Clinical Oncology 37, 1704\.
  * Luo et al. [2022] Luo, X., Liao, W., Xiao, J., Chen, J., Song, T., Zhang, X., Li, K., Metaxas, D.N., Wang, G., Zhang, S., 2022.  Word: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from ct image.  Medical Image Analysis 82, 102642\.
  * Ma and Wang [2023] Ma, J., Wang, B., 2023.  Segment anything in medical images.  arXiv preprint arXiv:2304.12306 .
  * Ma et al. [2022] Ma, J., Zhang, Y., Gu, S., An, X., Wang, Z., Ge, C., Wang, C., Zhang, F., Wang, Y., Xu, Y., et al., 2022\.  Fast and low-gpu-memory abdomen ct organ segmentation: the flare challenge.  Medical Image Analysis 82, 102616\.
  * Mazurowski et al. [2023] Mazurowski, M.A., Dong, H., Gu, H., Yang, J., Konz, N., Zhang, Y., 2023.  Segment anything model for medical image analysis: an experimental study.  arXiv preprint arXiv:2304.10517 .
  * Navarro et al. [2022] Navarro, F., Sasahara, G., Shit, S., Sekuboyina, A., Ezhov, I., Peeken, J.C., Combs, S.E., Menze, B.H., 2022\.  A unified 3d framework for organs-at-risk localization and segmentation for radiation therapy planning, in: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), IEEE. pp. 1544–1547.
  * Navarro et al. [2020] Navarro, F., Sekuboyina, A., Waldmannstetter, D., Peeken, J.C., Combs, S.E., Menze, B.H., 2020\.  Deep reinforcement learning for organ localization in ct, in: Medical Imaging with Deep Learning, PMLR. pp. 544–554.
  * Ouyang et al. [2022] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al., 2022.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems 35, 27730–27744.
  * Qin et al. [2022] Qin, Z., Yi, H., Lao, Q., Li, K., 2022.  Medical image understanding with pretrained vision language models: A comprehensive study.  arXiv preprint arXiv:2209.15517 .
  * Qin et al. [2023] Qin, Z., Yi, H.H., Lao, Q., Li, K., 2023.  Medical image understanding with pretrained vision language models: A comprehensive study, in: The Eleventh International Conference on Learning Representations.  URL: https://openreview.net/forum?id=txlWziuCE5W.
  * Radford et al. [2021] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al., 2021\.  Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR. pp. 8748–8763.
  * Redmon et al. [2016] Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016\.  You only look once: Unified, real-time object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788.
  * Ren et al. [2015] Ren, S., He, K., Girshick, R., Sun, J., 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems 28\.
  * Revel et al. [2021] Revel, M.P., Boussouar, S., de Margerie-Mellon, C., Saab, I., Lapotre, T., Mompoint, D., Chassagnon, G., Milon, A., Lederlin, M., Bennani, S., et al., 2021\.  Study of thoracic ct in covid-19: the stoic project.  Radiology 301, E361–E370.
  * Sun et al. [2013] Sun, Y., Luebbers, H.T., Agbaje, J.O., Schepers, S., Vrielinck, L., Lambrichts, I., Politis, C., 2013.  Validation of anatomical landmarks-based registration for image-guided surgery: an in-vitro study.  Journal of Cranio-Maxillofacial Surgery 41, 522–526.
  * Vallieres et al. [2017] Vallieres, M., Kay-Rivest, E., Perrin, L.J., Liem, X., Furstoss, C., Aerts, H.J., Khaouam, N., Nguyen-Tan, P.F., Wang, C.S., Sultanem, K., et al., 2017\.  Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer.  Scientific reports 7, 10117\.
  * Vorontsov et al. [2023] Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Liu, S., Mathieu, P., Eck, A.v., Lee, D., Viret, J., et al., 2023\.  Virchow: A million-slide digital pathology foundation model.  arXiv preprint arXiv:2309.07778 .
  * Wan et al. [2023] Wan, K., Li, L., Jia, D., Gao, S., Qian, W., Wu, Y., Lin, H., Mu, X., Gao, X., Wang, S., et al., 2023\.  Multi-target landmark detection with incomplete images via reinforcement learning and shape prior embedding.  Medical Image Analysis , 102875.
  * Wang et al. [2023] Wang, G., Wu, J., Luo, X., Liu, X., Li, K., Zhang, S., 2023.  Mis-fm: 3d medical image segmentation using foundation models pretrained on a large-scale unannotated dataset.  arXiv preprint arXiv:2306.16925 .
  * Wang et al. [2018] Wang, G., Zuluaga, M.A., Li, W., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al., 2018\.  Deepigeos: a deep interactive geodesic framework for medical image segmentation.  IEEE transactions on pattern analysis and machine intelligence 41, 1559–1572.
  * Wu et al. [2023] Wu, J., Fu, R., Fang, H., Liu, Y., Wang, Z., Xu, Y., Jin, Y., Arbel, T., 2023\.  Medical sam adapter: Adapting segment anything model for medical image segmentation.  arXiv preprint arXiv:2304.12620 .
  * Xu et al. [2019] Xu, X., Zhou, F., Liu, B., Fu, D., Bai, X., 2019\.  Efficient multiple organ localization in ct image using 3d region proposal network.  IEEE transactions on medical imaging 38, 1885–1898.
  * Yan et al. [2018] Yan, K., Wang, X., Lu, L., Summers, R.M., 2018.  Deeplesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning.  Journal of medical imaging 5, 036501–036501.
  * Yao et al. [2021] Yao, Q., Quan, Q., Xiao, L., Kevin Zhou, S., 2021.  One-shot medical landmark detection, in: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24, Springer. pp. 177–188.
  * Zhang et al. [2018] Zhang, G., Jiang, S., Yang, Z., Gong, L., Ma, X., Zhou, Z., Bao, C., Liu, Q., 2018.  Automatic nodule detection for lung cancer in ct images: A review.  Computers in biology and medicine 103, 287–300.
  * Zhang et al. [2022] Zhang, H., Zhang, P., Hu, X., Chen, Y.C., Li, L., Dai, X., Wang, L., Yuan, L., Hwang, J.N., Gao, J., 2022\.  Glipv2: Unifying localization and vision-language understanding.  Advances in Neural Information Processing Systems 35, 36067–36080.
  * Zhang and Liu [2023] Zhang, K., Liu, D., 2023.  Customized segment anything model for medical image segmentation.  arXiv preprint arXiv:2304.13785 .
  * Zhang and Metaxas [2023] Zhang, S., Metaxas, D., 2023\.  On the challenges and perspectives of foundation models for medical image analysis.  arXiv preprint arXiv:2306.05705 .
  * Zhang and Jiao [2023] Zhang, Y., Jiao, R., 2023.  How segment anything model (sam) boost medical image segmentation?  arXiv preprint arXiv:2305.03678 .
  * Zhou et al. [2023] Zhou, Y., Chia, M.A., Wagner, S.K., Ayhan, M.S., Williamson, D.J., Struyven, R.R., Liu, T., Xu, M., Lozano, M.G., Woodward-Court, P., et al., 2023.  A foundation model for generalizable disease detection from retinal images.  Nature , 1–8.