# CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor
Segmentation

Zhongzhen Huang1,2, Yankai Jiang2, Rongzhao Zhang2,
Shaoting Zhang1,222footnotemark: 2, Xiaofan Zhang1,222footnotemark: 2
1Qing Yuan Research Institute, Shanghai Jiao Tong University 2Shanghai AI
Laboratory
<EMAIL_ADDRESS>
{jiangyankai, zhangrongzhao<EMAIL_ADDRESS>
###### Abstract

Existing promptable segmentation methods in the medical imaging field
primarily consider either textual or visual prompts to segment relevant
objects, yet they often fall short when addressing anomalies in medical
images, like tumors, which may vary greatly in shape, size, and appearance.
Recognizing the complexity of medical scenarios and the limitations of textual
or visual prompts, we propose a novel dual-prompt schema that leverages the
complementary strengths of visual and textual prompts for segmenting various
organs and tumors. Specifically, we introduce CAT, an innovative model that
Coordinates Anatomical prompts derived from 3D cropped images with Textual
prompts enriched by medical domain knowledge. The model architecture adopts a
general query-based design, where prompt queries facilitate segmentation
queries for mask prediction. To synergize two types of prompts within a
unified framework, we implement a ShareRefiner, which refines both
segmentation and prompt queries while disentangling the two types of prompts.
Trained on a consortium of 10 public CT datasets, CAT demonstrates superior
performance in multiple segmentation tasks. Further validation on a
specialized in-house dataset reveals the remarkable capacity of segmenting
tumors across multiple cancer stages. This approach confirms that coordinating
multimodal prompts is a promising avenue for addressing complex scenarios in
the medical domain.

##  1 Introduction

Advanced prompt engineering [1, 2] has augmented large language models [3, 4,
5] with emerging capabilities. However, these paradigms have not yet been
successfully applied to vision tasks, primarily due to the ever-changing and
unpredictable natures of computer vision [6, 7]. This is particularly evident
in medical domains, where variations in imaging protocols, noise/artifacts,
and patient-specific pathologies pose significant challenges [8, 9, 10]. To
tackle these challenges, there have been some efforts within the community to
develop promptable models for segmenting objects in medical images: 1) One
type of such effort focuses on textual-prompted models [11, 12, 13], which
show profound competencies in segmenting a specific organ or tumor referenced
by arbitrary text phrases. These approaches involve distilling knowledge from
language models like CLIP [14] or BERT [15] to facilitate the alignment
between visual and textual representations; 2) Another direction in promptable
segmentation research aims to visual-prompted models [16, 17, 18, 19, 20, 21],
which rely on visual examples or visual landmarks (e.g., boxes and points).
Recently, the trend toward visual-prompted segmentation models in the medical
domain mainly focuses on fine-tuning SAM [22] with lightweight, plug-and-play
adapters or modifying SAM into 3D-based architecture.

Figure 1: Left: Long-tailed curve of the category and the number of available
cases that can be obtained in the medical field. Right: Tumors in different
cancer staging with diverse shapes and sizes.

Despite the promising advancements brought by the prompt-enhanced segmentation
paradigm, those segmentation models still face significant limitations. The
distribution of datasets often follows a long-tail pattern, where an increase
in the diversity of detectable anomalies corresponds with a sharp decline in
the number of available cases, as illustrated in the left part of Figure 1.
Textual prompts could provide a comprehensive way to describe each rare
anomaly with domain knowledge. However, such data scarcity introduced by the
long-tailed distribution hinders the effective learning of alignments between
textual and visual features. Conversely, visual prompts are not constrained to
the need for cross-modal alignment, providing a more intuitive and direct
method to enhance the segmentation process. Nonetheless, visual prompts fail
to convey the general concept of each object, leading to a performance drop
when confronted with various scenarios in medical domains, especially for
tumors. For instance, colon tumors range from different cancer stages [23] and
have significant inter- and intra-patient in tumor sizes and shapes, as
demonstrated in the right part of 1. Moreover, tumors at the same stage can
exhibit varying densities. Given the inherent diversity in tumors, there is a
necessity to provide comprehensive knowledge of each tumor type via textual
descriptions.

In this work, we strive to develop a promptable segmentation model that
utilizes the strengths of both visual and textual prompts without human
interaction, aiming at a fully automatic model for medical professionals. We
propose a new dual-perspective prompting scheme. On the one hand, we directly
employ the cropped volumes derived from the anatomical structure as our visual
prompts. We refer to such prompts as anatomical prompts, intending to
represent target objects in a more intuitive and visually coherent manner. On
the other hand, we enhance the textual prompts with more comprehensive
knowledge. With the proposed prompting scheme, we introduce CAT, a model
towards comprehensive segmentation that harnesses a dual-prompting mechanism
to Coordinate Aatomical and Textual prompts.

CAT follows the general query-based design with two extra parallel encoders
dedicated to processing anatomical and textual prompts into prompt queries.
Inspired by previous segmentation models [24, 25, 26, 27], we adopt query
embeddings to perform mask predictions. A ShareRefiner is utilized to refine
segmentation queries and prompt queries by attending them to image features
from the backbone. We employ two distinct feature assignment strategies for
prompt queries, ensuring that these prompt queries are disentangled for more
versatile representations. Following the refinement process, PromptRefer
updates segmentation queries by integrating both types of prompt queries. The
mask prediction module then generates binary masks by performing a simple dot-
product of segmentation query embeddings with the pixel embedding map obtained
from the backbone. To further facilitate the coordination, we augment CAT with
cross-modal alignment between anatomical and textual prompts. By training on
an assembly of $10$ public CT datasets in the abdomen, CAT demonstrates strong
segmentation capabilities on these datasets and achieves remarkable results on
an in-house dataset encompassing four cancer stages. Further ablation analysis
shows that anatomical and textual prompts serve complementary roles. To
summarize, our contributions are threefold:

  * •

We design a new prompting scheme that utilizes both complementary strengths of
anatomical prompts and textual prompts for medical image segmentation.

  * •

We build CAT, a fully automatic and promptable model that coordinates
anatomical prompts derived from 3D cropped volumes with textual prompts
enriched by medical domain knowledge, enabling strong flexibility for various
segmentation tasks.

  * •

Extensive experiments demonstrate the benefits of coordinating anatomical
prompts and textual within one model. CAT consistently achieves state-of-the-
art performance on multiple segmentation tasks and has generalization
capability to diverse tumor types.

##  2 Related Work

###  2.1 Textual-prompted Segmentation in Medical Imaging

Prompt engineering has [1, 2] achieved remarkable progress in natural language
processing and shown great potential in general visual perception [28, 29, 30,
24, 7, 31]. By leveraging pre-trained vision-language foundation models [14],
textual-prompted methods demonstrate impressive capabilities in open-
vocabulary segmentation [31, 32, 33, 34, 35, 36]. These advancements in the
natural image field have significantly propelled the field of 2D medical image
segmentation [37], where the innovative model architecture is used to
recognize a variety of visual contexts through textual prompts. Recently, some
efforts tried to train a textual-prompted universal model for segmenting
various organs and tumors in 3D volumes [13, 12, 11]. A pre-trained text
encoder is adopted to encode the injected prompts and guide the target
grounding process. Moreover, the latest attempts [12] focused on extending the
short textual phrases with knowledge. However, the reliance on texts struggles
with medical image segmentation due to the potential misalignment between
textual descriptions and complex visual patterns. To solve the limitations
caused by linguistic ambiguity, we integrate visual inputs (i.e., anatomical
prompts) for more accurate and comprehensive image perception.

###  2.2 Visual-prompted Segmentation in Medical Imaging

Beyond textual-prompted methods, the field has seen a notable shift towards
incorporating visual prompts to enhance accuracy and context sensitivity [16].
Different from textual prompts, visual prompts introduce more flexibility and
context-awareness for models. The innovative Segment Anything [22] introduces
a promptable model for generalized image segmentation. Building upon the
foundation model, the field has seen a substantial shift towards applying this
paradigm to the medical domain. One line adapts SAM to general medical image
segmentation with fine-tuning [17, 19, 20, 21, 18]. Some works like SAM-Med2D
[19] and 3DSAM-adapter [20] utilize adapters [38] to transfer the capabilities
of SAM to medical images with a few number of trainable parameters. Other line
[39] trains 3D models from scratch straightforwardly. SAM-Med3D [40] and
SegVol [39] developed 3D SAM models from scratch by reforming the 2D model and
training with a large scale of CT scans. Incorporated with a higher annotated
ratio, a 3D point-promptable model CT-SAM3D [41] is introduced with a
progressively and spatially aligned prompt encoding technique. Nonetheless,
due to the substantial disparities among anomalies in the medical domain,
visual prompts could not provide a generic concept for each type. Our work
resembles the textual and anatomical prompts to support the multi-organ and
tumor segmentation.

##  3 Method

In this paper, we focus on applying anatomical prompts (i.e., cropped 3D
medical volumes) and textual prompts for generic segmentation tasks in the
abdomen involving organs and tumors. CAT employs a popular query-based
encoder-decoder architecture with a sophisticated interaction paradigm between
queries and prompts, aiming at predicting $N$ categories in the abdomen, as
shown in Figure 2. Our model integrates four main components: i) Vision
Backbone, designed to extract image features and construct pixel embedding
maps, ii) Prompt Encoders, used to encode anatomical and textual prompts
provided by users, respectively, iii) ShareRefiner, utilized to refine
segmentation queries and prompt queries, and iv) PromptRefer, generating
target queries for prediction.

Figure 2: (a) CAT follows the query-based segmentation architecture. 3D
cropped volumes according to the anatomical structure are utilized as
anatomical prompts. Texts enhanced by professional knowledge are adopted as
textual prompts. Learnable queries and both prompts are utilized for the final
prediction via ShareRefiner and PromptRefer. (b) The case of colon tumor in
Stage-IV invading the intestine. (c) Attention masks in PromptRefer for
assigning specific prompts to queries.

###  3.1 Vision Backbone and Prompt Encoders

Backbone. Following other unified models [26, 42] for segmentation tasks, we
perform mask classification for all organs and tumors. We adopt a key idea
from Mask2Former [42, 12] to construct a pixel embedding map, which is
obtained from the backbone encoder features. As shown in Figure 2, the vision
backbone comprises a vision encoder $\mathbf{Enc}$ responsible for extracting
multi-scale visual features $\mathbf{V}$ and a vision decoder represented as
$\mathbf{Dec}$, which gradually upsamples visual features to the high-
resolution pixel embedding map $O$. Given an input 3D volume
$I\in\mathbb{R}^{H\times\ W\times D}$, the process can be formally represented
as follows:

$\mathbf{V}=\mathbf{Enc}(I),O=\mathbf{Dec}(\mathbf{V}),$ (1)

where $\mathbf{V}=\\{V^{i}\\}^{L}_{i=1}$, $V^{i}\in\mathbb{R}^{H^{i}\times
W^{i}\times D^{i}\times C^{i}}$, and $L$ is the number of layers of the vision
encoder. Here, $H^{i}$, $W^{i}$, $D^{i}$ and $C^{i}$ denote the height, width,
depth and channel dimension of $V^{i}$, respectively. $O\in\mathbb{R}^{H\times
W\times D\times C_{o}}$ is obtained by fusing feature maps from the
$\mathbf{Enc}$, $C_{o}$ is the dimension of $O$.

Prompt Encoders. Prompts have been widely used in segmentation tasks recently.
Textual prompts provide comprehensive concepts for the target object but bring
some ambiguities, such as the fact that both the intestines and the colon have
a “tubular shape”. Conversely, visual prompts are more helpful in
disambiguating the user’s intent when textual prompts fail to identify the
correct object. However, these visual prompts have limited robustness in
handling large variations in medical images [41]. Our method enhances the
segmentation process via the coordination of anatomical and textual prompts.
Specifically, the anatomical prompt is a referred region from another CT exam,
which can be a particular tumor or a cropped volume of the same semantic
concept. Given $N$ cropped volumes $P_{Aj}\in\mathbb{R}^{H_{A}\times
W_{A}\times D_{A}},j\in\\{1,2,...,N\\}$, we encode these volumes through a
pre-trained encoder $\mathbf{Enc}_{A}$ and obtain anatomical prompt embeddings
$\mathbf{E}_{A}$. To convey the comprehensive notion of targets with detailed
information, we utilize long textual descriptions as our textual prompts.
Following previous work [12], we incorporate textual prompts with medical
domain knowledge. We employ the text encoder $\mathbf{Enc}_{T}$ [43] to encode
these descriptions and use the [CLS] token output as the textual prompt
embedding, denoted as $\mathbf{E}_{T}$. The process can be denoted as follows:

$\displaystyle\mathbf{E}_{A}=\mathbf{Enc}_{A}(\mathbf{P}_{A}),\mathbf{E}_{T}=\mathbf{Enc}_{T}(\mathbf{P}_{T})$
(2)
$\displaystyle\mathbf{Q}_{A}=\operatorname{Linear}(\mathbf{E}_{A}),\mathbf{Q}_{T}=\operatorname{Linear}(\mathbf{E}_{T}),$
(3)

where $\mathbf{Q}_{A},\mathbf{Q}_{T}\in\mathbb{R}^{N\times C}$ are prompt
queries and $C$ is the channel dimension.

###  3.2 ShareRefiner

CAT predicts $N$ masks based on the learnable segmentation queries
$\mathbf{Q}_{S}$ and the pixel embedding map $O$. Our initial steps involve
feeding both segmentation queries $\mathbf{Q}_{S}$ and prompts queries
$\mathbf{Q}_{A},\mathbf{Q}_{T}$ into the proposed ShareRefiner to enable the
queries to interact with multi-scale features. In our approach, queries will
selectively attend to local image features for self-refinements. Specifically,
the ShareRefiner consists of a series of cross-attention blocks where queries
perform cross-attention with respect to the target visual feature $V^{i}$. To
disentangle these queries, we adopt soft and hard assignments for
$[\mathbf{Q}_{S},\mathbf{Q}_{T}]$ and $\mathbf{Q}_{A}$, respectively. A hard
cross-attention layer is applied to extract anatomical prompt features from
the multi-scale feature maps, ensuring that each anatomical query gathers
discriminative visual regions without overlaps. For the $i$-th layer of
ShareRefiner, the hard assignment similarity matrix of $\mathbf{Q}^{i}_{A}$ is
computed via $\operatorname{Gumbel-Softmax}$ [44, 45] as follows:

$\displaystyle S^{i}=\mathbf{Q}^{i}_{A}\operatorname{Flatten}(V^{i})^{T}$ (4)
$\displaystyle
S^{i}_{\text{gumbel}}=\operatorname{Softmax}\left(\left(S^{i}+G^{i}\right)/\tau\right),$
(5) $\displaystyle
S^{i}_{\text{onehot}}=\operatorname{Onehot}\left(\operatorname{argmax}_{N}\left(S^{i}_{\text{gumbel}}\right)\right).$
(6)

Where $S^{i},G^{i}\in\mathbb{R}^{N\times H^{i}W^{i}D^{i}}$, $G^{i}$ are i.i.d
random samples drawn from the $Gumbel(0,1)$ distribution and $\tau$ is a
learnable coefficient. The one-hot operation of the $\operatorname{argmax}$ is
performed over $S^{i}_{\text{gumbel}}$ for hard assigning. Since the
straightforward hard assignment (i.e., one-hot) is not differentiable, the
straight-through trick in [46, 47] is used to compute the assignment
similarities ${S^{\prime}}^{i}$ of one-hot value:

${S^{\prime}}^{i}=\left(S^{i}_{\text{onehot}}\right)^{\top}+S^{i}_{\text{gumbel}}-\operatorname{sg}\left(S^{i}_{\text{gumbel}}\right),$
(7)

where $\operatorname{sg}$ denotes the stop gradient operator. Following the
feature grouping process, we use a self-attention layer to regulate the
relationships among queries and a feed-forward layer for projection. Formally,
the aforementioned process is represented as:

$\displaystyle\mathbf{Q^{\prime}}_{S},\mathbf{Q^{\prime}}_{T}=\mathbf{ShareRefiner}([\mathbf{Q}_{S},\mathbf{Q}_{T}],\mathbf{V},\text{hard}=\text{False}),$
(8)
$\displaystyle\mathbf{Q^{\prime}}_{A}=\mathbf{ShareRefiner}(\mathbf{Q}_{A},\mathbf{V},\text{hard}=\text{True}).$
(9)

###  3.3 PromptRefer

In practice, when segmenting target objects, more attention needs to be paid
to the relevant context. Localizing the typical tumor requires being aware of
the anomalous features in the relevant organ, and even identifying organs
requires focusing on the anatomical structures involved. For instance, as
shown in Figure 2(b), colon tumors of Stage-IV would invade the adjacent
organs like intestines. Directly combining prompt queries for mask prediction
is suboptimal since there is still an inherent gap between the two types of
prompt embeddings even though we have unified them into prompt queries. The
typical objective is to classify $\mathbf{Q^{\prime}}_{S}$ into respective
regions with the guidance of prompts. To this end, we introduce
$\mathbf{PromptRefer}$ implemented by the cross-attention mechanism with
carefully crafted attention masks in Figure 2(c). Here, a group of prompt
queries $\\{{Q^{\prime}}_{Ai},\dots,{Q^{\prime}}_{Tj},\dots\\}$ is employed to
a specific segmentation query. To ensure the distinction of queries,
segmentation queries are required to only interact with queries in the group.
This is illustrated in the following equation:

$\mathbf{O}_{S}=\mathbf{PromptRefer}([\mathbf{Q^{\prime}}_{S},\mathbf{Q^{\prime}}_{A},\mathbf{Q^{\prime}}_{T}],\text{mask}).$
(10)

Here, $\mathbf{O}_{S}$ represents the decoded segmentation query features for
predicting masks.

To further integrate two types of prompts and push segmentation queries
closely to the referenced prompt, we employ query-level contrastive learning
in our model. Specifically, given the decoded segmentation query features and
refined prompt queries, we define two types of losses and calculate the
InfoNCE loss [14] as follows:

$\mathcal{L}_{s2p}=-\frac{1}{N}\sum_{i=1}^{N}\frac{\mbox{exp}(\tilde{O}_{Si}\cdot\tilde{Q^{\prime}}_{\\{A/T\\}i})}{\sum_{j=0}^{N}\mbox{exp}(\tilde{O}_{Si}\cdot\tilde{Q^{\prime}}_{\\{A/T\\}j})},\mathcal{L}_{p2p}=-\frac{1}{N}\sum_{i=1}^{N}\frac{\mbox{exp}(\tilde{Q^{\prime}}_{Ai}\cdot\tilde{Q^{\prime}}_{Ti})}{\sum_{j=0}^{N}\mbox{exp}(\tilde{Q^{\prime}}_{Ai}\cdot\tilde{Q^{\prime}}_{Tj})}.$
(11)

Where
$\mathbf{\tilde{O}}_{S},\mathbf{\tilde{Q^{\prime}}}_{A},\mathbf{\tilde{Q^{\prime}}}_{T}$
are derived via linear projection layers. The contrastive alignment can be
regarded as a distillation process, whereby each prompt query contributes to
the segmentation queries. It also benefits the knowledge exchange between
queries from two modalities.

###  3.4 Training Objective and Strategy

Training Objective. The binary mask proposals $\mathbf{M}\in[0,1]^{N\times
H\times W\times D}$ are calculated through the multiplication operation
between the decoded segmentation query features $\mathbf{O}_{S}$ and the high-
resolution pixel embedding map $O$ followed by a Sigmoid. We employ the dice
loss for mask prediction. For classification loss, following [12, 13], we
adopt the cross-entropy loss that measures the difference between predicted
objects and the categories. Moreover, the contrastive loss is applied to the
similarities among queries. The final loss takes the following form:

$\mathcal{L}_{total}=\mathcal{L}_{dice}+\mathcal{L}_{cls}+\mathcal{L}_{s2p}+\mathcal{L}_{p2p}.$
(12)

Anatomical prompt training strategy. For anatomical prompts, we leverage the
bounding box derived from masks in the public dataset and relevant anatomical
structures to crop a set of prompt volumes for each category and unify these
volumes into the same size. To alleviate the domain shift, we use a pre-
trained encoder [48] for feature extraction. During training, we randomly
sample an instance from the set of prompt volumes for each category, excluding
the identical one.

Textual prompt training strategy. For the textual prompt, we identify the
category names in the abdomen. We call GPT-4 [49] to generate descriptions
with medical domain knowledge for each category more than 20 times. We recruit
one board-certified physician to rewrite the description according to the
results. Besides long descriptions for each category, we also construct
several short textual templates. We use the long description for the positive
category and randomly sample short phrases for those negative categories. More
details can be seen in the Appendix A.1.

##  4 Experiments

Dataset and Settings. CAT is trained on the curated dataset from 10 public
datasets [50, 51, 52, 53, 54, 55, 56, 57, 58, 59], which contain multiple
organs and tumors in the abdomen. Following settings in [13], we assemble this
dataset with the data pre-processing to reduce the domain gap among various
datasets. In the testing phase, two public datasets and one in-house dataset
are used for evaluation. The in-house test dataset contains $80$ 3D CT volumes
with colon tumor masks, ranging from Stage-I to Stage-IV. For evaluating the
accuracy of organ segmentation, we employ FLARE22 [60] as the external test
set. For tumor segmentation, we utilize the combination of MSD dataset [59]
and the private dataset. Details of datasets are in Appendix A.2.

Implementation Details and Evaluation Metrics. Our model follows the query-
based segmentation architecture [42, 26]. That is, we use Swin UNETR [61] as
the backbone and Clinical-Bert [43] as the text-encoder. We follow the
experiment settings of previous work [13, 12]. Details are in Appendix A.3.
For the evaluation, the Dice Similarity Coefficient (DSC) and 95% Hausdorff
Distance (HD95) are utilized to gauge the performance of organ/tumor
segmentation.

###  4.1 Main Results

Organ Segmentation. In this study, we explore the organ segmentation
performance of our model on the external testing set of FLARE22. We present
the detailed organ-wise segmentation results in Table 1. Compared to models
[17, 19] adapting SAM [22] to the medical fields, our approach yields
substantially better performance across all 12 organs. For instance, CAT
significantly surpasses MedSAM [17] by a large margin of $21\%$ DSC points on
the adrenal gland segmentation and $15\%$ DSC points on the esophagus
segmentation. Such results show the gap in applying 2D SAM-based models for 3D
volumes, especially for objects with small sizes or intricate shapes. When
compared to 3D SAM models [41, 39, 40], our model performs significantly
better than models [39, 40], which take in visual prompts like SAM. For
example, we surpass SAM-Med3D [40] and SegVol [39] by $30\%$ points and $20\%$
points in terms of the average score. We can observe that these models also
struggle with the problem of handling some difficult organs. In addition, just
with one prompt for each category, CAT achieves comparable results with CT-
SAM3D [41], which utilizes spatially aligned prompts progressively in multi-
rounds. Owing to the PromptRefer module, our model demonstrates superior
performance in organs where tumors frequently occur. Specifically, CAT leads
by $2\%$ points in the liver and by $6\%$ points in the pancreas. CAT also
demonstrates superior performance compared with recent textual-prompted models
[13, 12]. Although the gap of organ segmentation is only $2\%$ points, we
observed that they do not generalize well on the tumor segmentation task
without more intuitive visual prompts, as described in the following part.
More details are shown in Table 6.

Table 1: Organ segmentation performance on FLARE22. The results(%) are
evaluated by DSC. Scores of SAM-based are adopted from the CT-SAM-Med3D [41].
${\dagger}$ denotes obtained via the official pre-trained weights. $*$ means
implemented from the official code and trained on the same dataset.
Abbreviations: “Liv.”-Liver, “R_Kid.”-Right Kidney, “Spl.”-Spleen,
“Pan.”-Pancreas, “Aor.”-Aorta, “IVC”-Inferior Vena Cava, “RAG”-Right Adrenal
Gland, “LAG”-Left Adrenal Gland, “Gal.”-Gallbladder, “Eso.”-Esophagus,
“Sto.”-Stomach, “Duo.”-Duodenum, “ L_Kid.”-Inferior Vena Cava.

Methods | Liv. | R_Kid. | Spl. | Pan. | Aor. | IVC | RAG | LAG | Gal. | Eso. | Sto. | Duo. | L_Kid. | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
SAM [22] | 86.0 | 87.6 | 84.5 | 53.4 | 77.5 | 44.5 | 19.4 | 33.9 | 52.4 | 35.2 | 68.0 | 44.4 | 82.6 | 59.2
MedSAM [17] | 93.0 | 90.0 | 89.1 | 73.5 | 82.5 | 76.5 | 36.0 | 48.7 | 56.4 | 64.7 | 84.0 | 53.9 | 89.7 | 72.2
SAM-Med2D [19] | 91.4 | 83.7 | 83.9 | 58.8 | 60.6 | 18.6 | 10.6 | 27.1 | 32.9 | 28.1 | 72.9 | 45.4 | 86.0 | 53.8
SAM-Med3D [40] | 85.4 | 84.2 | 84.7 | 46.9 | 60.4 | 44.5 | 32.6 | 35.3 | 56.0 | 32.6 | 46.9 | 27.4 | 84.9 | 55.5
SegVol [39] | 83.9 | 71.7 | 75.9 | 69.4 | 83.1 | 80.3 | 42.1 | 49.7 | 55.6 | 69.6 | 81.1 | 55.6 | 75.1 | 68.7
CT-SAM3D [41] | 95.6 | 95.0 | 96.1 | 83.6 | 94.5 | 91.8 | 78.4 | 82.5 | 88.4 | 82.9 | 92.3 | 73.2 | 94.8 | 88.4
Universal† [13] | 97.4 | 95.5 | 96.4 | 73.7 | 84.9 | 84.4 | 72.9 | 73.4 | 86.0 | 76.8 | 88.5 | 74.5 | 96.9 | 84.7
ZePT∗ [12] | 96.7 | 95.6 | 96.6 | 84.3 | 90.0 | 84.4 | 67.2 | 66.8 | 79.6 | 74.2 | 85.2 | 59.1 | 97.2 | 82.8
CAT | 97.7 | 96.3 | 97.1 | 89.2 | 90.5 | 88.0 | 73.6 | 74.3 | 83.0 | 80.1 | 88.2 | 73.4 | 97.3 | 86.8

Table 2: Segmentation performance (%) of tumors on MSD [59] and In-house
dataset. We compare our method with traditional and promptable methods.
${\dagger}$ denotes obtained via the official pre-trained weights. $*$ means
implemented from the official code and trained on the same dataset.

Method | MSD Dataset (Tumor in Abdomen) | In-house Data (Colon Tumor) |
---|---|---|---
Liver | Pancreas | Hepatic Vessel | Colon | T1 | T2 | T3 | T4 | Avg.
DSC$\uparrow$ | HD95$\downarrow$ | DSC$\uparrow$ | HD95$\downarrow$ | DSC$\uparrow$ | HD95$\downarrow$ | DSC$\uparrow$ | HD95$\downarrow$ | DSC$\uparrow$ | DSC$\uparrow$ | HD95$\downarrow$
nnUNet∗ [62] | 66.42 | 42.29 | 43.50 | 25.80 | 66.90 | 47.59 | 41.41 | 153.06 | 19.51 | 45.06 | 44.87 | 45.54 | 43.00 | 150.48
Swin UNETR∗ [48] | 68.67 | 42.54 | 41.77 | 22.87 | 63.32 | 44.02 | 39.35 | 161.26 | 21.40 | 33.32 | 46.11 | 52.72 | 45.92 | 168.25
SAM-Med3D† [40] | 44.78 | - | 40.05 | - | 44.86 | - | 39.23 | - | 34.28 | 42.65 | 50.20 | 42.65 | 47.11 | -
SegVol† [39] | 66.20 | - | 46.36 | - | 68.57 | - | 60.63 | - | 36.93 | 42.63 | 60.17 | 49.83 | 50.28 | -
Universal† [13] | 65.68 | 63.31 | 45.72 | 16.58 | 66.31 | 51.47 | 42.26 | 115.40 | 7.11 | 43.28 | 46.52 | 53.08 | 47.14 | 140.28
ZePT∗ [12] | 68.58 | 43.23 | 44.39 | 19.47 | 68.12 | 33.94 | 40.38 | 113.07 | 23.87 | 34.64 | 50.81 | 51.09 | 46.28 | 155.83
CAT | 72.73 | 34.64 | 49.67 | 15.56 | 70.11 | 33.44 | 48.31 | 108.26 | 30.62 | 45.61 | 55.85 | 57.37 | 53.35 | 80.96

Tumor Segmentation. We do the evaluation on MSD dataset [59] and an in-house
dataset to validate its tumor segmentation capability. Table 2 shows the
comparison results of four tumor categories. We do not report HD95 scores for
SAM-based methods since this comparison may not be entirely equitable. Our CAT
outperforms baselines [62, 48] significantly across all tumor subtypes,
achieving at least a $4\%$ improvement in DSC. For the comparison of SAM-based
methods [40, 39], it is worth noting that only one point derived from the
ground truths is utilized as the prompt during the inference time since tumors
of small sizes are difficult to identify but can be easily segmented with the
provided prompts. Without using information from labels, CAT still achieves
better performance except for the colon tumor segmentation, as evidenced by a
$5\%$ improvement in the DSC of the pancreas tumor. These results validate the
proposed PromptRefer module (i.e., attending to specific organs when
segmenting tumors). Moreover, we compare CAT with textual-prompted methods.
CAT surpasses the previously across four tasks by $4\%$ in the average DSC. As
our model is an extension of textual-prompted methods, these results
demonstrate that incorporating the anatomical prompts can produce higher-
quality masks for tumors. More details are shown in Table 7.

We further conduct the evaluation on an in-house colon tumor dataset, with
tumors ranging from Stage-I (T1) to Stage-IV (T4) according to the Cancer
Staging System [23]. As shown in Table 2, CAT showcases strong capabilities in
dealing with tumors of diverse sizes and shapes. To provide deeper insight, we
present the average score of each subtype. It is noted that the higher the
number after the “T”, the larger the tumor or the more extensively it has
grown into nearby tissues. We can observe that SAM-derived methods perform
well in segmenting relatively small T1 tumors with point prompts. However,
SegVol [39], which achieves the best results in MSD colon tumor segmentation,
struggles to handle tumors that invade nearby organs or tissues. It is evident
from the performance gap between T3 and T4 tumors. In these cases, tumors in
each instance can present with varying densities and show distinct shapes at
different locations. Only using visual prompts without comprehensive
descriptions to cover all abnormal parts proves challenging. Such scenarios
are vital in the medical field, highlighting the need for further research in
applying visual prompts for tumor segmentation. In this paper, we coordinate
anatomical and textual prompts along with the predefined attention mask to
alleviate this problem. CAT outperforms other models by at least absolute
$7\%$ DSC in T4 and $3\%$ DSC on average, demonstrating much better
generalizability and robustness. This result suggests that leveraging
anatomical prompts together with medical domain knowledge is an alternative
way to address intricate scenarios in the medical domain.

Qualitative Comparison. To give more intuitive comparisons, we present the
qualitative results in Figure 5. In the first row, we observe that current
methods may struggle with segmenting tubular-shaped organs, particularly SAM-
derived methods like SegVol [39], since visual prompts can not convey the
general concept of such tubular-shaped objects. The results in the second and
third rows reveal that textual-prompted models are prone to misclassification
and overlook key details due to the lack of intuitive context, underscoring
the importance of visual prompts. However, these only relying on visual
prompts struggle with tumors of intricate shapes without the support of
medical domain knowledge, as shown in the fourth row, where SegVol exhibits a
high number of false positives on normal CT scans. When tumors invade other
tissues, existing models often fail due to segmentation target incompleteness
and misclassification of normal regions as tumors. In contrast, CAT can
precisely identify most abnormal regions and consistently generate results
that are more consistent with the ground truth compared to all other models.

Figure 3: Qualitative visualizations of the proposed model and other prompting
methods on organ/tumor segmentation. The segmentation results presented from
rows one to five correspond, in order, to the duodenum, liver tumors, pancreas
tumors, colon tumors, and colon tumors in Stage-IV.

###  4.2 Ablation Studies

To fully investigate the contribution of our introduced components, we
conducted ablation studies to compare each part in both organ and tumor
segmentation tasks. We selected a subset of organ and tumor categories that
are particularly challenging for models, facilitating a more intuitive
comparison.

Ablation of Two Prompts. We start from a basic query-based segmentation
architecture without any extra prompts, and then we ablate the effectiveness
of using different prompts for different tasks. As demonstrated in Table 3
(the first four rows), eliminating the prompt schema leads to a substantial
performance drop on both organ and tumor segmentation ($64.08\rightarrow
54.39$ in Duodenum and $72.49\rightarrow 66.37$ in Liver tumor). The observed
declines in the third row indicate that textual features, which are rich in
semantics but lack appearance features, may not generalize well to objects
with intricate structures or small sizes. It is evidenced by a performance
reduction of $6\%$ in Esophagus segmentation and $4\%$ in Hepatic Vessel tumor
segmentation. Moreover, we can observe from the second row that the tumor
segmentation capability is inadequate when relying solely on anatomical
prompts. This issue stems from the diversity and variance of tumors. For
instance, every example the model encounters is drastically different as it
tries to identify the location and nature of pancreas tumor. Although tumors
can be regarded as anomalies, the lack of consistent context poses a challenge
for the model in establishing a general concept solely with the cropped
volumes. The coordination of anatomical and textual prompts improves the
performance of the promptable model. This confirms the efficacy of our joint
prompts schema, which is designed to help the model form more stable and
generalizable masks in the medical field.

Table 3: Ablation studies of two prompts and model designs on organ and tumor
segmentation dataset.

Variant | Organ (%) | Tumor (%)
---|---|---
AP | TP | Hard | Mask | Pan. | RAG | LAG | Eso. | Duo. | Liver | Pancreas | HepVes. | Colon | T4
|  |  |  | 78.18 | 69.63 | 69.08 | 76.99 | 54.39 | 66.37 | 42.05 | 62.20 | 39.85 | 51.17
✓ |  |  |  | 83.55 | 72.80 | 71.65 | 79.31 | 60.45 | 64.82 | 45.08 | 68.72 | 43.84 | 53.91
| ✓ |  |  | 80.62 | 71.02 | 70.34 | 72.81 | 57.31 | 69.13 | 44.31 | 65.18 | 40.16 | 52.32
✓ | ✓ |  |  | 83.50 | 72.71 | 69.96 | 78.99 | 64.08 | 72.49 | 44.55 | 69.40 | 44.50 | 55.84
| ✓ |  | ✓ | 86.74 | 72.41 | 69.00 | 77.68 | 59.99 | 69.12 | 43.23 | 67.75 | 41.32 | 54.33
✓ | ✓ | ✓ |  | 87.36 | 74.46 | 74.02 | 75.39 | 70.80 | 72.64 | 48.49 | 69.02 | 47.29 | 53.67
✓ | ✓ |  | ✓ | 88.49 | 73.24 | 74.51 | 80.76 | 70.26 | 72.18 | 46.46 | 69.97 | 46.65 | 58.49
✓ | ✓ | ✓ | ✓ | 88.28 | 74.42 | 72.50 | 79.30 | 71.26 | 70.95 | 45.52 | 69.51 | 46.07 | 56.41
✓ | ✓ | ✓ | ✓ | 89.24 | 73.69 | 74.63 | 80.10 | 73.46 | 72.73 | 49.67 | 70.11 | 48.31 | 57.37

Effectiveness of Module Designs. As presented in the last five rows in Table
3, we evaluate the impacts of the module designs in our ShareRefiner and
PromptRefer. The comparison of results between the seventh and last rows
demonstrates that employing the hard assignment for refining the anatomical
queries can boost performances, especially for tumor segmentation. However,
applying the paradigm to the other two queries (i.e., the ‘Hard’ column is
marked as ✓) will decrease the performance of segmentation. This decline can
be attributed to the entanglement of the three query types. Such a phenomenon
further reinforces that anatomical and textual prompts contribute from
different perspectives. Using a one-hot hard assignment ensures that each
query feature remains exclusive to the others, allowing queries to focus on
distinct visual regions without overlap. We hypothesize that anatomical
queries could attend to features and distinguish between different categories
based on inherent appearance characteristics. Additionally, we remove the mask
mechanism in the PromptRefer and utilize a vanilla cross-attention layer to
update $\mathbf{Q^{\prime}}_{S}$. As illustrated in Table 3, this variant
leads to reduced performance in tumors that invade other organs (e.g., Stage-
IV). The result verifies that guiding queries to focus on relevant objects is
an effective strategy for achieving more robust segmentation in medical
scenarios.

Figure 4: Left: T-SNE visualization of the distribution of two prompt
embedding before and after refinement. (1-9: right kidney, left kidney, liver,
pancreas, colon, kidney tumor, liver tumor, pancreas tumor, colon tumor).
Right: Heatmaps of two samples for analyzing the effectiveness of two prompts.

Visualizations. We select nine categories (five organs and four tumors) and
use t-SNE [63] to visualize the distribution of prompt features
($\mathbf{E}_{A}$ and $\mathbf{E}_{T}$) and updated prompts queries
($\mathbf{Q^{\prime}}_{A}$ and $\mathbf{Q^{\prime}}_{T}$) in Figure 4. Our
findings indicate that the initial prompt features are not well-separated in
the feature space. For instance, the anatomical features of the right kidney
and left kidney (categories 1 and 2) are entangled. Therefore, different
refined paradigms need to be applied to update both queries simultaneously.
With our ShareRefiner, prompt features are separated in the feature space.
Moreover, we can observe that the distribution of refined anatomical prompt
queries is more structured than textual prompts, which further verifies that
the anatomical prompt is more intuitive. Additionally, we visualize the
heatmaps derived from experiments in the second and third rows of Table 3. As
shown in Figure 4, solely using textual prompts fails to cover all regions,
while only relying on anatomical prompts results in a high false positive
rate. These visual examples underscore the importance of combining both types
of prompts to guide medical image segmentation effectively.

##  5 Conclusion and Discussion

Conclusion. We present CAT, a promising attempt towards comprehensive medical
segmentation via coordinating anatomical-textual prompts. Apart from
performing generic organ segmentation, CAT can identify varying tumors without
human interaction. To effectively integrate two prompt modalities into a
single model, we design ShareRefiner to refine latent prompt queries with
different strategies and introduce PromptRefer with specific attention masks
to assign prompts to segmentation queries for mask prediction. Extensive
experiments indicate that the coordination of these two prompt modalities
yields competitive performance on organ and tumor segmentation benchmarks.
Further studies revealed the robust generalization capabilities to segment
tumors in different cancer stages. We hope our early exploration of the
complementary advantages between anatomical and textual prompts could bring
new insights into the field of the community.

Limitation and Impact. We hope our model can support professionals in the
arduous clinical diagnosis process. Despite the integration of anatomical and
textual prompts showing competitive performance in the segmentation tasks, the
lack of general anatomical prompts encoders raises challenges, as indicated by
the messy distribution of initial anatomical embeddings. Therefore, further
research into improving the CT foundation models is essential. Moreover, there
may also be mistakes in the segmentation results, especially when the test
sample contains rare types of lesions or undergoes radical resection surgeries
that cause large variations in the anatomical structures. Therefore, before
integrating these AI-based algorithms into clinical practice, legislation
needs to be developed and implemented to ensure that there are clear
guidelines and standards for their use.

## References

  * [1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [2] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in neural information processing systems, 35:24824–24837, 2022.
  * [3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [5] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [6] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao.  Multimodal foundation models: From specialists to general-purpose assistants.  arXiv preprint arXiv:2309.10020, 1(2):2, 2023.
  * [7] Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Chunyuan Li, Jianwei Yang, et al.  Visual in-context prompting.  arXiv preprint arXiv:2311.13601, 2023.
  * [8] Ehab A AlBadawy, Ashirbani Saha, and Maciej A Mazurowski.  Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing.  Medical physics, 45(3):1150–1158, 2018.
  * [9] Mohammad Hesam Hesamian, Wenjing Jia, Xiangjian He, and Paul Kennedy.  Deep learning techniques for medical image segmentation: achievements and challenges.  Journal of digital imaging, 32:582–596, 2019.
  * [10] Shaoting Zhang and Dimitris Metaxas.  On the challenges and perspectives of foundation models for medical image analysis.  Medical Image Analysis, page 102996, 2023.
  * [11] Ziheng Zhao, Yao Zhang, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.  One model to rule them all: Towards universal segmentation for medical images with text prompts.  arXiv preprint arXiv:2312.17183, 2023.
  * [12] Yankai Jiang, Zhongzhen Huang, Rongzhao Zhang, Xiaofan Zhang, and Shaoting Zhang.  Zept: Zero-shot pan-tumor segmentation via query-disentangling and self-prompting.  arXiv preprint arXiv:2312.04964, 2023.
  * [13] Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou.  Clip-driven universal model for organ segmentation and tumor detection.  arXiv preprint arXiv:2301.00785, 2023.
  * [14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [16] Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca.  Universeg: Universal medical image segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21438–21451, 2023.
  * [17] Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang.  Segment anything in medical images.  Nature Communications, 15(1):654, 2024.
  * [18] Cheng Chen, Juzheng Miao, Dufan Wu, Zhiling Yan, Sekeun Kim, Jiang Hu, Aoxiao Zhong, Zhengliang Liu, Lichao Sun, Xiang Li, et al.  Ma-sam: Modality-agnostic sam adaptation for 3d medical image segmentation.  arXiv preprint arXiv:2309.08842, 2023.
  * [19] Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al.  Sam-med2d.  arXiv preprint arXiv:2308.16184, 2023.
  * [20] Shizhan Gong, Yuan Zhong, Wenao Ma, Jinpeng Li, Zhao Wang, Jingyang Zhang, Pheng-Ann Heng, and Qi Dou.  3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation.  arXiv preprint arXiv:2306.13465, 2023.
  * [21] Kaidong Zhang and Dong Liu.  Customized segment anything model for medical image segmentation.  arXiv preprint arXiv:2304.13785, 2023.
  * [22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  * [23] National Cancer Institute.  Cancer staging, 2022.  Accessed: 2024-05-21.
  * [24] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee.  Segment everything everywhere all at once.  Advances in Neural Information Processing Systems, 36, 2024.
  * [25] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum.  Mask dino: Towards a unified transformer-based framework for object detection and segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3041–3050, 2023.
  * [26] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022.
  * [27] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  Max-deeplab: End-to-end panoptic segmentation with mask transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5463–5474, 2021.
  * [28] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  arXiv preprint arXiv:2104.13921, 2021.
  * [29] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [30] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955–2966, 2023.
  * [31] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang.  A simple framework for open-vocabulary segmentation and detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1020–1031, 2023.
  * [32] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai.  Decoupling zero-shot semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583–11592, 2022.
  * [33] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061–7070, 2023.
  * [34] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Scaling open-vocabulary image segmentation with image-level labels.  In European Conference on Computer Vision, pages 540–557. Springer, 2022.
  * [35] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai.  A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model.  In European Conference on Computer Vision, pages 736–753. Springer, 2022.
  * [36] Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al.  Freeseg: Unified, universal and open-vocabulary image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19446–19455, 2023.
  * [37] Shiyi Du, Xiaosong Wang, Yongyi Lu, Yuyin Zhou, Shaoting Zhang, Alan Yuille, Kang Li, and Zongwei Zhou.  Boosting dermatoscopic lesion segmentation via diffusion models with visual and textual prompts.  arXiv preprint arXiv:2310.02906, 2023.
  * [38] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [39] Yuxin Du, Fan Bai, Tiejun Huang, and Bo Zhao.  Segvol: Universal and interactive volumetric medical image segmentation.  arXiv preprint arXiv:2311.13385, 2023.
  * [40] Haoyu Wang, Sizheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng, Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, et al.  Sam-med3d.  arXiv preprint arXiv:2310.15161, 2023.
  * [41] Heng Guo, Jianfeng Zhang, Jiaxing Huang, Tony CW Mok, Dazhou Guo, Ke Yan, Le Lu, Dakai Jin, and Minfeng Xu.  Towards a comprehensive, efficient and promptable anatomic structure segmentation model using 3d whole-body ct scans.  arXiv preprint arXiv:2403.15063, 2024.
  * [42] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  Advances in neural information processing systems, 34:17864–17875, 2021.
  * [43] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical bert embeddings.  arXiv preprint arXiv:1904.03323, 2019.
  * [44] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  In ICLR, 2017.
  * [45] Chris J Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  In ICLR, 2017.
  * [46] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [47] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134–18144, 2022.
  * [48] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh.  Self-supervised pre-training of swin transformers for 3d medical image analysis.  In CVPR, pages 20730–20740, 2022.
  * [49] OpenAI.  Gpt-4 technical report, 2023.
  * [50] Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, T Langerak, and Arno Klein.  Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge.  In Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge, volume 5, page 12, 2015.
  * [51] Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al.  Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.  Advances in Neural Information Processing Systems, 35:36722–36732, 2022.
  * [52] A Emre Kavur, N Sinem Gezer, Mustafa Barış, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Duc Duy Pham, Soumick Chatterjee, Philipp Ernst, Savaş Özkan, et al.  Chaos challenge-combined (ct-mr) healthy abdominal organ segmentation.  Medical Image Analysis, 69:101950, 2021.
  * [53] Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L Rubin.  Ct-org, a new dataset for multiple organ segmentation in computed tomography.  Scientific Data, 7(1):381, 2020.
  * [54] Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, et al.  Abdomenct-1k: Is abdominal organ segmentation a solved problem.  IEEE TPAMI, 2021.
  * [55] Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers.  Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part I 18, pages 556–564. Springer, 2015.
  * [56] Xiangde Luo, Wenjun Liao, Jianghong Xiao, Jieneng Chen, Tao Song, Xiaofan Zhang, Kang Li, Dimitris N Metaxas, Guotai Wang, and Shaoting Zhang.  Word: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from ct image.  Medical Image Analysis, 82:102642, 2022.
  * [57] Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al.  The liver tumor segmentation benchmark (lits).  Medical Image Analysis, 84:102680, 2023.
  * [58] Nicholas Heller, Sean McSweeney, Matthew Thomas Peterson, Sarah Peterson, Jack Rickman, Bethany Stai, Resha Tejpaul, Makinna Oestreich, Paul Blake, Joel Rosenberg, et al.  An international challenge to use artificial intelligence to define the state-of-the-art in kidney and kidney tumor segmentation in ct imaging., 2020.
  * [59] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.  The medical segmentation decathlon.  Nature communications, 13(1):4128, 2022.
  * [60] Jun Ma, Yao Zhang, Song Gu, Cheng Ge, Shihao Ma, Adamo Young, Cheng Zhu, Kangkang Meng, Xin Yang, Ziyan Huang, Fan Zhang, Wentao Liu, YuanKe Pan, Shoujin Huang, Jiacheng Wang, Mingze Sun, Weixin Xu, Dengqiang Jia, Jae Won Choi, Natália Alves, Bram de Wilde, Gregor Koehler, Yajun Wu, Manuel Wiesenfarth, Qiongjie Zhu, Guoqiang Dong, Jian He, the FLARE Challenge Consortium, and Bo Wang.  Unleashing the strengths of unlabeled data in pan-cancer abdominal organ quantification: the flare22 challenge.  arXiv preprint arXiv:2308.05862, 2023.
  * [61] Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu.  Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images.  In International MICCAI Brainlesion Workshop, pages 272–284. Springer, 2021.
  * [62] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature methods, 18(2):203–211, 2021.
  * [63] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [64] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.

##  Appendix A Appendix / supplemental material

###  A.1 Textual Prompts Construction

The instruction for GPT-4 Short phrase templates

    Please describe the provided organs and tumors. Descriptions
    should include the location, its anatomical structure, common
    pathological findings, and common tumors (for organs). Answers
    must be based on observations from CT imaging.

A computerized tomography of a [CLS]. A photo of a [CLS]. There is [CLS] in
this computerized tomography. [CLS]

Table 4: The strategy of constructing textual prompts.

We call GPT-4 to generate long descriptions for each category and use
templates to construct some short textual prompts, as shown in Table 4. We use
the long description for the positive category and randomly sample short
phrases for those negative categories. Positive categories indicate labeled
categories in the case.

###  A.2 Dataset Details

The overall categories used in our paper consist of $23$ organs and $6$
anomalies. The test set of MSD is the same as the official code of [13].
Details of utilized datasets are shown as follows:

(1) The BTCV dataset [50] consists of 30 abdominal CT scans from 13 annotated
organs, conducted under the supervision of radiologists at Vanderbilt
University Medical Center.

(2) The CT-ORG dataset [53] encompasses 140 CT images, featuring six organ
classes. These images primarily display liver lesions, including both benign
and malignant types.

(3) The AbdomenCT-1K dataset [54] comprises 1,112 CT scans pooled from five
datasets, annotated for the liver, kidney, spleen, and pancreas.

(4) CHAOS [52] offers 40 CT scans of healthy abdominal organs for multi-organ
segmentation, specifically excluding any pathological abnormalities such as
tumors or metastasis.

(5) AMOS22 [51], a multi-modality abdominal multi-organ segmentation challenge
of 2022, includes 500 CT scans with voxel-level annotations of 15 abdominal
organs.

(6) The WORD dataset [56] collects 150 CT scans from patients prior to
radiation therapy at a single center, with each volume consisting of 159 to
330 slices and comprehensively annotated for 16 anatomical organs.

(7) Pancreas-CT [55] contains 82 contrast-enhanced abdominal CT volumes
focused solely on the pancreas, annotated by an experienced radiologist and
excluding any pancreatic tumors.

(8) The LiTS dataset [57] comprises 201 contrast-enhanced abdominal CT scans
(131 for training and 70 for testing), acquired across six clinical sites
using various scanners and protocols, with a resolution range from 0.55 to 1.0
mm and slice spacing from 0.45 to 6.0 mm.

(9) KiTS [58] includes 300 CT scans (210 for training and 90 for testing) with
annotations provided by the University of Minnesota Medical Center, each
featuring one or more kidney tumors.

(10) The Medical Segmentation Decathlon (MSD) [59] comprises $947$ CT scans
targeting liver, lung, pancreas, colon, hepatic vessels, and spleen,
encompassing a total of four organs and five tumors.

(11) Our in-house dataset contains 80 CT scans of patients diagnosed with
colon cancer, annotated by an experienced gastroenterologist and verified by a
senior radiologist. All scans maintain a consistent in-plane dimension of
$512\times 512$ pixels, while the z-axis dimension ranges from $36$ to $146$,
with a median of $91$. Note that there are some cases where the cancer stage
can not be indentified.

Datasets | #Scans | Annotated categories
---|---|---
BTCV [50] | 30 | Spl, RKid, LKid, Gall, Eso, Liv, Sto, Aor, IVC, R&SVeins, Pan, RAG, LAG
CT-ORG [53] | 140 | Lung, Liver, Kidneys and Bladder
AbdomenCT-1K [54] | 1000 | Spleen, Kidney, Liver, Pancreas
CHAOS [52] | 40 | Liver, Left Kidney, Right Kidney, Spleen
AMOS22 [51] | 500 | Spl, RKid, LKid, Gall, Eso, Liv, Sto, Aor, IVC, Pan, RAG, LAG, Duo, Bla, Pro/UTE
WORD [56] | 150 | Spl, RKid, LKid, Gall, Eso, Liv, Sto, Pan, RAG, Duo, Col, Int, Rec, Bla, LFH, RFH
Pancreas-CT [55] | 82 | Pancreas
LiTS [57] | 201 |  Liver, Liver Tumor
KiTS [58] | 300 |  Kidney, Kidney Tumor
MSD CT Tasks [59] | 947 |  Spl, Liver, Liver Tumor, Lung Tumor, Colon Tumor, Pancreas and Pancreas Tumor, Hepatic Vessel and Hepatic Vessel Tumor
In-house dataset | 80 |  Colon Tumor with four subtypes

Table 5: The information for all datasets.

###  A.3 Implementation Details

AdamW optimizer [64] is utilized as the optimizer. The default initial
learning rate is $1\times 10^{-4}$, and we decay it following the learning
rate scheduling strategy of [61]. Each volume is cropped into patches with a
size of $96\times 96\times 96$. Random shift, zoom and scale, and scaling are
applied on-the-fly to improve the generalization. Our framework is implemented
using PyTorch and all experiments are conducted on $8$ NVIDIA A$100$ GPUs. For
the evaluation, the Dice Similarity Coefficient (DSC) and 95% Hausdorff
Distance (HD95) are utilized to gauge the performance of organ/tumor
segmentation.

###  A.4 Results Details

Table 6: Organ segmentation performance on FLARE22. The results(%) are
evaluated by DSC. Scores of SAM-based are adopted from the CT-SAM-Med3D [41].
${\dagger}$ denotes obtained via the official pre-trained weights. $*$ means
implemented from the official code and trained on the same dataset.
Abbreviations: “Liv.”-Liver, “R_Kid.”-Right Kidney, “Spl.”-Spleen,
“Pan.”-Pancreas, “Aor.”-Aorta, “IVC”-Inferior Vena Cava, “RAG”-Right Adrenal
Gland, “LAG”-Left Adrenal Gland, “Gal.”-Gallbladder, “Eso.”-Esophagus,
“Sto.”-Stomach, “Duo.”-Duodenum, “ L_Kid.”-Inferior Vena Cava.

Methods | Liv. | R_Kid. | Spl. | Pan. | Aor. | IVC | RAG | LAG | Gal. | Eso. | Sto. | Duo. | L_Kid.
---|---|---|---|---|---|---|---|---|---|---|---|---|---
SAM [22] | 86.0$\pm$5.5 | 87.6$\pm$8.7 | 84.5$\pm$8.9 | 53.4$\pm$10.6 | 77.5$\pm$16.1 | 44.5$\pm$16.3 | 19.4$\pm$14.0 | 33.9$\pm$15.0 | 52.4$\pm$16.4 | 35.2$\pm$6.8 | 68.0$\pm$11.2 | 44.4$\pm$12.4 | 82.6$\pm$11.3
MedSAM [17] | 93.0$\pm$3.1 | 90.0$\pm$5.3 | 89.1$\pm$11.0 | 73.5$\pm$9.6 | 82.5$\pm$19.7 | 76.5$\pm$19.4 | 36.0$\pm$23.6 | 48.7$\pm$22.6 | 56.4$\pm$27.1 | 64.7$\pm$19.9 | 84.0$\pm$13.0 | 53.9$\pm$11.7 | 89.7$\pm$7.9
SAM-Med2D [19] | 91.4$\pm$5.8 | 83.7$\pm$17.3 | 83.9$\pm$15.2 | 58.8$\pm$18.7 | 60.6$\pm$22.5 | 18.6$\pm$10.4 | 10.6$\pm$9.7 | 27.1$\pm$12.4 | 32.9$\pm$21.6 | 28.1$\pm$13.3 | 72.9$\pm$16.6 | 45.4$\pm$19.8 | 86.0$\pm$16.8
SAM-Med3D [40] | 85.4$\pm$13.2 | 84.2$\pm$9.5 | 84.7$\pm$11.8 | 46.9$\pm$14.3 | 60.4$\pm$10.7 | 44.5$\pm$13.4 | 32.6$\pm$20.9 | 35.3$\pm$18.3 | 56.0$\pm$19.4 | 32.6$\pm$16.4 | 46.9$\pm$19.8 | 27.4$\pm$13.6 | 84.9$\pm$6.9
SegVol [39] | 83.9$\pm$25.3 | 71.7$\pm$30.6 | 75.9$\pm$28.8 | 69.4$\pm$16.1 | 83.1$\pm$12.1 | 80.3$\pm$13.9 | 42.1$\pm$13.3 | 49.7$\pm$22.6 | 55.6$\pm$31.1 | 69.6$\pm$8.4 | 81.1$\pm$20.7 | 55.6$\pm$19.8 | 75.1$\pm$22.6
CT-SAM3D [41] | 95.6$\pm$2.0 | 95.0$\pm$1.8 | 96.1$\pm$4.4 | 83.6$\pm$12.0 | 94.5$\pm$2.8 | 91.8$\pm$4.7 | 78.4$\pm$18.0 | 82.5$\pm$4.0 | 88.4$\pm$8.1 | 82.9$\pm$18.1 | 92.3$\pm$4.4 | 73.2$\pm$16.8 | 94.8$\pm$1.4
Universal† [13] | 97.4$\pm$1.4 | 95.5$\pm$6.7 | 96.4$\pm$1.45 | 73.7$\pm$29.6 | 84.9$\pm$10.2 | 84.4$\pm$9.4 | 72.9$\pm$19.5 | 73.4$\pm$14.3 | 86.0$\pm$8.8 | 76.8$\pm$17.4 | 88.5$\pm$16.0 | 74.5$\pm$11.3 | 96.9$\pm$0.8
ZePT∗ [12] | 96.7$\pm$2.0 | 95.6$\pm$7.7 | 96.6$\pm$2.9 | 84.3$\pm$2.5 | 90.0$\pm$9.0 | 84.4$\pm$2.5 | 67.2$\pm$20.4 | 66.8$\pm$10.1 | 79.6$\pm$9.5 | 74.2$\pm$17.3 | 85.2$\pm$11.1 | 59.1$\pm$11.8 | 97.2$\pm$1.0
CAT | 97.7$\pm$1.3 | 96.3$\pm$6.1 | 97.1$\pm$1.1 | 89.2$\pm$7.6 | 90.5$\pm$9.1 | 88.0$\pm$2.9 | 73.6$\pm$18.3 | 74.3$\pm$10.0 | 83.0$\pm$10.0 | 80.1$\pm$9.2 | 88.2$\pm$12.3 | 73.4$\pm$8.8 | 97.3$\pm$0.7

Table 7: Segmentation performance of tumors on MSD [59] and In-house dataset.
We compare our method with traditional and promptable methods. ${\dagger}$
denotes obtained via the official pre-trained weights. $*$ means implemented
from the official code and trained on the same dataset.

Method | MSD Dataset (Tumor in Abdomen) | In-house Data (Colon Tumor)
---|---|---
Liver | Pancreas | Hepatic Vessel | Colon | T1 | T2 | T3 | T4 | Avg.
nnUNet∗ [62] | 66.42$\pm$26.5 | 43.50$\pm$34.1 | 66.90$\pm$21.1 | 41.41$\pm$27.8 | 19.51$\pm$33.8 | 45.06$\pm$33.9 | 44.87$\pm$32.4 | 45.54$\pm$29.9 | 43.00$\pm$31.9
Swin UNETR∗ [48] | 68.67$\pm$23.1 | 41.77$\pm$32.9 | 63.32$\pm$25.5 | 39.35$\pm$28.6 | 21.40$\pm$22.2 | 33.32$\pm$30.4 | 46.11$\pm$25.3 | 52.72$\pm$22.8 | 45.92$\pm$26.5
SAM-Med3D† [40] | 44.78$\pm$38.7 | 40.05 $\pm$18.3 | 44.86$\pm$26.3 | 39.23$\pm$19.1 | 34.28$\pm$23.4 | 42.65$\pm$19.1 | 50.20$\pm$16.9 | 42.65$\pm$18.8 | 47.11$\pm$19.2
SegVol† [39] | 66.20$\pm$26.0 | 46.36$\pm$28.8 | 68.57$\pm$28.2 | 60.63$\pm$24.1 | 36.93$\pm$33.5 | 42.63$\pm$27.8 | 60.17$\pm$27.2 | 49.83$\pm$28.1 | 50.28$\pm$28.7
Universal† [13] | 65.68$\pm$27.2 | 45.72$\pm$31.9 | 66.31$\pm$21.5 | 42.26$\pm$30.4 | 7.11$\pm$12.3 | 43.28$\pm$29.6 | 46.52$\pm$28.6 | 53.08$\pm$26.9 | 47.14$\pm$29.1
ZePT∗ [12] | 68.58$\pm$24.3 | 44.39$\pm$34.2 | 68.12$\pm$20.6 | 40.38$\pm$29.9 | 23.87$\pm$23.8 | 34.64$\pm$32.2 | 50.81$\pm$28.2 | 51.09$\pm$25.5 | 46.28$\pm$28.5
CAT | 72.73$\pm$23.3 | 49.67$\pm$32.7 | 70.11$\pm$19.7 | 48.31$\pm$27.3 | 30.62$\pm$31.5 | 45.61$\pm$30.8 | 55.85$\pm$27.7 | 57.37$\pm$23.4 | 53.35$\pm$27.0

Further results are depicted in Figure 5. Our model demonstrates the
capability to segment all structures of the aorta, including the arcus aortae.
This comprehensive segmentation is advantageous in real-world scenarios,
although it slightly lowers the Dice score.

Figure 5: Visualization of segmentation result of CAT. The left part is the
ground truth and the right part is ours. The first five rows are results of
organ segmentation results, and the others are tumor segmentation results.
From the top to down are liver tumors, pancreas tumors, Hepatic Vessel tumors,
colon tumors, and colon tumors in T4. For each tumor type, we present two
cases.