# Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision

Zhaoqing Wang1,4 Xiaobo Xia1 Ziye Chen2 Xiao He4 Yandong Guo4
Mingming Gong2,3 Tongliang Liu1
1The University of Sydney 2The University of Melbourne
3Mohamed bin Zayed University of Artificial Intelligence 4AI2Robotics
https://github.com/DerrickWang005/Uni-OVSeg.pytorch

###### Abstract

Contemporary cutting-edge open-vocabulary segmentation approaches commonly
rely on image-mask-text triplets, yet this restricted annotation is labour-
intensive and encounters scalability hurdles in complex real-world scenarios.
Although some methods are proposed to reduce the annotation cost with only
text supervision, the incompleteness of supervision severely limits the
versatility and performance. In this paper, we liberate the strict
correspondence between masks and texts by using independent image-mask and
image-text pairs, which can be easily collected respectively. With this
unpaired mask-text supervision, we propose a new weakly-supervised open-
vocabulary segmentation framework (Uni-OVSeg) that leverages confident pairs
of mask predictions and entities in text descriptions. Using the independent
image-mask and image-text pairs, we predict a set of binary masks and
associate them with entities by resorting to the CLIP embedding space.
However, the inherent noise in the correspondence between masks and entities
poses a significant challenge when obtaining reliable pairs. In light of this,
we advocate using the large vision-language model (LVLM) to refine text
descriptions and devise a multi-scale ensemble to stablise the matching
between masks and entities. Compared to text-only weakly-supervised methods,
our Uni-OVSeg achieves substantial improvements of 15.5% mIoU on the ADE20K
datasets, and even surpasses fully-supervised methods on the challenging
PASCAL Context-459 dataset.

Figure 1: The proposed Uni-OVSeg framework learns open-vocabulary segmentation
with unpaired mask-text supervision. Compared to the labour-intensive image-
mask-text annotations, independent image-mask and image-text pairs are easier
to collect. With one suite of weights, given different visual prompts (_e.g_.,
points and boxes), Uni-OVSeg can segment and categorise various objects and
stuff from an open set of vocabulary in the real world.

##  1 Introduction

Open-vocabulary segmentation refers to the segmentation and categorisation of
objects from an expansive and unrestricted vocabulary, even though the object
categories within the vocabulary are not encountered during training [35, 16,
70]. Compared to traditional closed-vocabulary segmentation [83, 13], which
depends on predefined training categories and cannot recognise absent
categories, open-vocabulary segmentation can segment any category with
arbitrary text descriptions. This innovative segmentation paradigm has
garnered considerable attention [67, 39] and opened up numerous potential
applications [87, 88, 72].

Cutting-edge approaches in open-vocabulary segmentation typically leverage
supervision with triplet annotations that are composed of images, masks, and
corresponding texts [63, 76]. It is worth noting that strict alignment between
each mask and text results in an expensive annotation cost. To mitigate this,
some weakly-supervised methods propose using only text supervision [64, 6,
65]. However, learning with this supervision, the model falls short in
capturing complex spatial details, which is suboptimal for dense prediction.
Furthermore, this type of supervision lacks positional information, making the
model difficult to distinguish different instances with the same semantic
class. These issues severely limit the versatility and segmentation
performance of existing weakly-supervised methods.

Motivated by that, in this paper, we propose an advanced weakly-supervised
open-vocabulary segmentation framework, named Uni-OVSeg, to reduce the
annotation expense while significantly enhancing performance. In essence, we
liberate the strict correspondence between masks and texts, by using
independent image-mask and image-text pairs. These two types of pairs can be
easily collected from different sources, as illustrated in Fig. 1, such as
SA-1B [34] for image-mask pairs and CC3M [50] for image-text pairs. By
resorting to independent image-mask and image-text pairs, our Uni-OVSeg has a
strong segmentation ability to group semantically similar pixels, and align
mask-wise embeddings with entity embeddings of texts into the same space,
achieving open-vocabulary segmentation.

Technically, when presented with an image-mask pair, we train a visual prompt
encoder, a pixel decoder, and a mask decoder to generate a set of binary
masks. The collected image-text pair often contains certain texts that does
not match the image [52], leading to an incorrect correspondence between masks
and entities. We employ the LLaVa model [42] to refine the quality of text
descriptions, coupled with a ChatGPT-based parser for precise entity
extraction. For mask-text alignment, we introduce a mask-text bipartite
matching to exploit confident pairs of predicted masks and entities. A multi-
scale feature adapter is designed to enhance the visual embeddings of
predicted masks, which are further aligned with the text embeddings of
corresponding entities. As masks and text are allowed to be unpaired in Uni-
OVSeg, a multi-scale ensemble is introduced to improve the quality of visual
embeddings and tackle inherent noise in the correspondence between masks and
entities effectively, thereby stablising the matching process. During the
inference phase, a zero-shot classifier, built by embedding target dataset
category names, assigns categories to predicted masks, enabling the system to
segment objects across an open vocabulary. We summarise our contributions in
the following aspects:

  1. 1.

We introduce a new framework Uni-OVSeg and offer an innovative solution that
significantly reduces the need for costly triplet annotations. This makes
open-vocabulary segmentation more accessible and scalable.

  2. 2.

Uni-OVSeg employs a visual prompt encoder, pixel decoder, and mask decoder for
binary mask segmentation. An inclusion of refinement of text descriptions,
multi-scale ensemble enhances segmentation performance by alleviating inherent
noise in mask-text correspondence.

  3. 3.

With one suit of weights, we achieve 32.6% mIoU on the ADE20K dataset, and
even surpassing the state-of-the-art FC-CLIP [76] on the challenging PASCAL
Context-459 dataset. Comprehensive ablation studies and discussions are also
provided.

##  2 Related works

Generic segmentation. Given an image, segmentation of specific visual concepts
has remained an ongoing research topic in computer vision, as indicated by the
extensive literature on it [44, 23, 33]. Generic segmentation mainly includes
semantic segmentation [44, 79, 77, 12], instance segmentation [23, 4, 7], and
panoptic segmentation [33, 11, 56, 11], related to different levels of
granularity. In more detail, semantic segmentation [20, 12, 28, 78, 38] aims
to assign a label to each pixel of the input image, according to their
respective semantic classes. In addition, instance segmentation [53, 59, 54]
attempts to distinguish different object instances of the same semantic class.
Panoptic segmentation [57, 75, 74, 9] combines the characteristics of semantic
segmentation and instance segmentation. Following a close-vocabulary
assumption, previous works only can predict a predefined set of object
categories. In this paper, we aim to build an advanced open-vocabulary
segmentation system, which can categorise objects and stuff from an open set
of vocabulary in the real world.

Vision foundation models. Recent advancements in visual foundation models have
diversified optimisation techniques across various learning paradigms. These
developments range from vision-only pretraining [24, 25, 62, 2, 61] to joint
vision-language pre-training [48, 30, 73], and extend to multi-modal
frameworks that integrate visual prompting [1]. A prime example of this
evolution is SAM [34], which shows the potential of extensive training for
general segmentation, offering impressive generalisability and scalability.
Despite its impressive capabilities, SAM cannot categorise predicted masks
into different semantic classes, which is limited by the supervision of the
image-mask pairs. More recently, Semantic-SAM [36] unifies different sources
of human-annotated segmentation datasets and augments SAM by adding semantic
labels and increased levels of granularity. In our work, our aim is to develop
a more flexible vision foundation model, which can be trained with unpaired
mask-text supervision (_e.g_., independent image-mask and image-text pairs)
and can be easily adapted to different segmentation tasks.

Open-vocabulary segmentation. Open-vocabulary segmentation counters the
constraints of closed-vocabulary segmentation by allowing the segmentation of
a diverse range of classes, even those unseen during training [21, 67, 80,
68]. Existing works [17, 76, 66] leverage the pretrained vision-language
models (_e.g_., CLIP [48] and ALIGN [30]) to perform open-vocabulary
segmentation. Most open-vocabulary segmentation methods commonly utilise
human-annotated supervision (_i.e_., the image-mask-text triplets) to
generalise the capability of vision-language models from the image level to
the pixel level. To reduce the dependency on this labour-intensive
supervision, some weakly-supervised methods are proposed to use only text
supervisions [46, 65]. They learn to group image regions into shaped segments,
but struggle to distinguish different instances with the same semantic class
and the segmentation performance is unsatisfactory [85, 64]. This dilemma
drives our pursuit of more advanced open-vocabulary segmentation framework,
where the aim is to enjoy as low annotation cost as possible and
simultaneously achieve significant performance.

Figure 2: Overview of the Uni-OVSeg framework. This framework consists of
feature extraction, mask generation, and mask-text alignment. A frozen CLIP
model and prompt encoder are used for image and text feature extraction and
prompt encoding, respectively. We employs a mask and pixel decoder for binary
mask prediction. A mask-text bipartite matching is designed to exploit
confident mask-entity pairs. Visual prompts using boxes are omitted for
simplicity.

##  3 Method

In this section, we first define the problem of open-vocabulary segmentation
in Sec. 3.1. We then introduce a straightforward baseline in Sec. 3.2.
Finally, we present our proposed Uni-OVSeg framework in Sec. 3.3, including an
overview, mask generation, mask-text alignment, and open-vocabulary inference.

###  3.1 Problem definition

Given an image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$, where $H$ and
$W$ denote the height and width of the image respectively, open-vocabulary
segmentation aims to segment the image into a set of masks with associated
semantic classes:

$\\{\mathbf{y}_{i}\\}_{i=1}^{k}=\\{(\mathbf{m}_{i},\mathbf{c}_{i})\\}_{i=1}^{k}.$
(1)

The $k$ masks $\mathbf{m}_{i}\in\\{0,1\\}^{H\times W}$ include the associated
ground truth class $\mathbf{c}_{i}$ [76]. Unlike traditional image
segmentation tasks [44, 23, 33], open-vocabulary segmentation is more
challenging because inference classes are not observed during training. During
the evaluation, the test categories $\mathbf{C}_{\text{test}}$ are different
from $\mathbf{C}_{\text{train}}$, which contain novel categories not seen in
training, _i.e_., $\mathbf{C}_{\text{train}}\neq\mathbf{C}_{\text{test}}$.
Different from previous works [64, 66], in our setting, no paired human-
annotated mask and semantic category is provided in advance for any training
image. We assume that the category names of $\mathbf{C}_{\text{test}}$ are
available, which are represented in the form of natural language.

###  3.2 Baseline

We introduce a straightforward baseline using the knowledge of image-text and
image-mask pairs. Specifically, we employ a CLIP model as the visual and text
encoder, which is trained on a large amount of image-text pairs. Afterward, we
use the image-mask pairs to obtain a branch of mask generation, predicting a
set of binary masks. To perform open-vocabulary segmentation, we crop and pool
the CLIP image features based on these predicted masks, which are further
classified by the CLIP text embeddings. Although this straightforward baseline
enables open-vocabulary segmentation, it exhibits an obvious knowledge gap
between the image-level and pixel-level tasks.

###  3.3 Uni-OVSeg framework

Overview. An overview of our framework Uni-OVSeg, for weakly-supervised open-
vocabulary segmentation is illustrated in Fig. 2. On a macro level, Uni-OVSeg
contains a CLIP model to extract features of both images and text
descriptions. With the image-mask pairs, a branch of mask generation,
including a visual prompt encoder, a pixel decoder and a mask decoder, is
employed to predict a set of binary masks of an input image. With the image-
text pairs, mask-text bipartite matching is used to exploit confident pairs
between predicted masks and entities in text descriptions. Afterward, we adopt
a multi-scale feature adapter to enhance the mask-wise visual embeddings,
which are further aligned with associated entity embeddings based on the
confident pairs. Finally, we perform open-vocabulary segmentation with the
above-mentioned parts. More details can be found in Appendix A.

Mask generation. Given an input image $\mathbf{I}$, mask generation aims to
predict a set of binary masks $\mathbf{m}$. These masks represent groups of
semantically similar pixels on various locations of the image. To effectively
associate these masks with visual prompts (_e.g_., points and boxes), we adopt
an architecture inspired by [13, 34], employing a visual prompt encoder, a
pixel decoder and a query-based mask decoder. Both point $(x,y)$ and box
$(x,y,w,h)$ prompts are transformed into unified anchor boxes. Each point is
converted into an anchor box with a predefined width $w^{{}^{\prime}}=4$ and
height $h^{{}^{\prime}}=4$, effectively approximating the location of points.
To address the challenge of predicting masks with varying granularities, as
depicted in Fig. 2, we encode each point into two positional embeddings
$\mathbf{q}^{loc}_{n}\in\mathbb{R}^{2\times dim}$ using sine encoding [55],
and combine them with $M$ distinct mask embeddings
$\mathbf{q}^{msk}\in\mathbb{R}^{M\times dim}$. Each mask embedding is a
learnable vector corresponding to a specific granularity level. For box
prompts, the process involves encoding into two position embeddings followed
by concatenation with a singular mask embedding, as each box is linked to a
unique object. Afterward, we represent $N$ input location prompts as a set of
query embeddings $\mathbf{Q}={\mathbf{q}_{1},...,\mathbf{q}_{n}}$. For the
$i$-th query,

$\mathbf{q}_{i}=Concat(\mathbf{q}^{loc}_{i};\mathbf{q}^{msk})+\mathbf{q}^{type}+\mathbf{q}^{feat},$
(2)

where $\mathbf{q}^{type}\in\mathbb{R}^{dim}$ denotes the query type, chosen
from either the point or the box. $\mathbf{q}^{feat}\in\mathbb{R}^{dim}$ is
the content embedding sampled from visual features. For box prompts, we sample
visual features based on the center of each box.

Subsequently, to capture the multi-scale information, a pixel decoder equipped
with multi-scale deformable attention [86] is used and outputs enhanced pixel
features $\mathbf{x}^{p}$. These features, in conjunction with the query
embeddings, are fed into the query-based mask decoder to predict binary masks.
Each layer of the mask decoder comprises a masked cross-attention mechanism
[13], a self-attention layer [55], and a feed-forward network. The binary
masks for each visual prompt $\mathbf{m}_{n}$ are derived through a matrix
multiplication between the query embeddings and pixel features,

$\mathbf{m}_{n}=Sigmoid(\mathbf{q}^{msk}_{n}\cdot\mathbf{x}^{p}),$ (3)

where $\mathbf{m}_{n}\in\mathbb{R}^{M\times H\times W}$ and $Sigmoid(\cdot)$
is the sigmoid function to normalize the mask values into $[0,1]$. An input
point is likely to exist in different granularity masks simultaneously. To
capture this property and accurately predict multi-granularity masks, unlike
SAM [34], we devise a many-to-many matching strategy to automatically
associate the ground-truth masks with the mask predictions of each point,
allowing a more comprehensive use of image-mask pairs. Once matching, we
compute the segmentation loss for each mask prediction $\mathbf{m}_{i}$ with
the corresponding ground-truth mask $\mathbf{y}^{msk}_{j}$,

$\begin{split}\mathcal{L}_{seg}(\mathbf{m}_{i},\mathbf{y}^{msk}_{j})=&\lambda_{bce}\cdot\mathcal{L}_{bce}(\mathbf{m}_{i},\mathbf{y}^{msk}_{j})+\\\
&\lambda_{dice}\cdot\mathcal{L}_{dice}(\mathbf{m}_{i},\mathbf{y}^{msk}_{j}),\end{split}$
(4)

where $\lambda_{bce}$ and $\lambda_{dice}$ are two hyper-parameters to balance
the binary cross-entropy loss $\mathcal{L}_{bce}$ and the dice loss
$\mathcal{L}_{dice}$. Note that we apply no penalty to the remaining unmatched
mask predictions, ensuring that multiple granularity masks can be predicted
for each point.

Mask-text alignment. To enable the model categorising the predicted masks from
an open set of vocabulary, given the image-text pairs, we build the
correspondence between objects in the image and entities in the text
description. Once a set of binary masks $\mathbf{m}$ are generated from the
input image, we obtain the region embeddings $\mathbf{r}_{i}$ by employing a
mask pooling layer $P$ and the CLIP visual projector $F_{v}$,

$\mathbf{r}_{i}=F{v}(P(\mathbf{x}^{v},\textbf{m}_{i})),$ (5)

where $\mathbf{x}^{v}$ denotes the visual features extracted from the CLIP
visual encoder. However, the collected image-text pairs often contain
mismatches between visual and textual information. Directly aligning these
incorrect correspondence leads to suboptimal segmentation performance. To
alleviate that, we adopt a large vision-language model, LLaVa [42], to refine
the text descriptions. For each text description, traditional natural language
processing tools (such as NLTK) struggle to extract reliable entities due to
the complex structure and diversity of natural language. Recently, ChatGPT has
demonstrated strong capabilities in contextual processing, thereby more
accurately identifying noun phrases. Therefore, we utilise the ChatGPT-based
parser to extract reliable entities 111https://openai.com/chatgpt. Afterwards,
each entity combined with the prompts used in [22] is fed into the CLIP text
encoder to obtain the text embedding $t_{k}$. we calculate a cost matrix
$\mathbf{\Delta}$ between the region embeddings and text embeddings,

$\mathbf{\delta}_{i,j}=\frac{\exp(\mathbf{\delta}^{{}^{\prime}}_{i,j})}{\sum_{j}\exp(\delta^{{}^{\prime}}_{i,j})},\quad\mathbf{\delta}^{{}^{\prime}}_{i,k}=1-\frac{\mathbf{r}_{i}\cdot\mathbf{t}_{k}}{\|\mathbf{r}_{i}\|_{2}\|\mathbf{t}_{k}\|_{2}},$
(6)

where $\mathbf{\delta}^{{}^{\prime}}_{i,k}$ denotes the reverse cosine
similarity between the $i$-th region embedding and the $k$-th text embedding.
The scale of objects within images often varies significantly. To enhance the
quality of matching, we extract region embeddings at different scales by
utilising multi-resolution images. This approach allows us to integrate cost
matrices from different scales to derive the final cost matrix,

$\mathbf{\Delta}^{{}^{\prime}}=\frac{1}{S}\sum_{s=1}^{S}\mathbf{\Delta}_{s},$
(7)

where $S$ denotes the number of scales and $\mathbf{\Delta}_{s}$ is the cost
matrix at the $s$-th scale. With this matrix, we apply the bipartite matching
algorithm [31] to obtain the confident pairs. Finally, we fed the multi-scale
visual features and the CLIP visual features into a multi-scale feature
adapter to obtain the advanced region embeddings
$\mathbf{r}^{{}^{\prime}}_{i}$, and compute the cosine similarity loss for
each paired region and text embedding,

$\mathcal{L}_{align}(\mathbf{r}^{{}^{\prime}}_{i},\mathbf{t}_{k})=1-\frac{\mathbf{r}^{{}^{\prime}}_{i}\cdot\mathbf{t}_{k}}{\|\mathbf{r}^{{}^{\prime}}_{i}\|_{2}\|\mathbf{t}_{k}\|_{2}}.$
(8)

Open-vocabulary inference. During inference, given the test categories
$\mathbf{C}_{\text{test}}$, we conduct prompt engineering [21] and use the
CLIP text encoder to extract text embeddings for open-vocabulary segmentation.
For each input image, we input an uniform point grid as the visual prompt to
predict a set of binary masks, and compute the cosine similarity with the text
embeddings to predict the category with the maximum similarity as the label of
the corresponding mask.

##  4 Experiments

###  4.1 Implementation details

Datasets. During training, we randomly sample the 30% subset from the SA-1B
dataset [34], which contains $\sim 3$ million images and $\sim 0.3$ billion
masks. Although this supervision provides diverse binary masks, it lacks the
semantic class for each mask. In addition, following Chen et al. [8], we
collect about 1.3 million image-text pairs and use a large vision-language
model to refine them. Afterward, we use the ChatGPT-based parser to extract
entities with descriptive words from these text descriptions.

Table 1:  Open-vocabulary semantic segmentation performance. We mainly compare with the fully-supervised and weakly-supervised methods. “COCO S.”, “COCO P.” and “COCO C.” denote the COCO stuff, panoptic and caption datasets. “O365” denotes the Object 365 dataset. “M. 41M” denotes the merged 41M image dataset. We report mIoU for all datasets.  Method | Training Data | A-847 | PC-459 | A-150 | PC-59 | PAS-21 | PAS-20 | Citys
---|---|---|---|---|---|---|---|---
mIoU (%)
Fully-supervised method (image-text-mask pair)
SimBaseline [67] | COCO S. | - | - | 15.3 | - | 74.5 | - | -
ZegFormer [16] | COCO S. | - | - | 16.4 | - | 73.3 | - | -
LSeg+ [35] | COCO S. | 3.8 | 7.8 | 18.0 | 46.5 | - | - | -
OVSeg [39] | COCO S. | 9.0 | 12.4 | 29.6 | 55.7 | - | 94.5 | -
SAN [68] | COCO S. | 13.7 | 17.1 | 33.3 | 60.2 | - | 95.5 | -
OpenSeg [21] | COCO P. + C. | 6.3 | 9.0 | 21.1 | 42.1 | - | - | -
ODISE [66] | COCO P. | 11.1 | 14.5 | 29.9 | 57.3 | 84.6 | - | -
X-Decoder [87] | COCO P. + C. | - | - | 25.0 | - | - | - | 47.3
OpenSEED [80] | COCO P. + O365 | - | - | 22.9 | - | - | - | 46.1
MaskCLIP [17] | COCO P. | 8.2 | 10.0 | 23.7 | 45.9 | - | - | -
FC-CLIP [76] | COCO P. | 14.8 | 18.2 | 34.1 | 58.4 | 81.8 | 95.4 | 56.2
Weakly-supervised method (image-text pair or image-text & image-mask pair)
GroupViT [64] | GCC + YFCC | 4.3 | 4.9 | 10.4 | 23.4 | 52.3 | 79.7 | 18.5
TCL [6] | GCC | - | - | 14.9 | 30.3 | 51.2 | 77.5 | 23.5
OVSeg [65] | CC4M | - | - | 5.6 | - | 53.8 | - | -
SegCLIP [46] | CC3M + COCO C. | - | - | 8.7 | - | 52.6 | - | -
CLIPpy [49] | HQITP-134M | - | - | 13.5 | - | 52.2 | - | -
MixReorg [5] | CC12M | - | - | 10.1 | 25.4 | 50.5 | - | -
SAM-CLIP [58] | M. 41M | - | - | 17.1 | 29.2 | 60.6 | - | -
Uni-OVSeg (Ours) | 30% SA1B + M. 1.3M | 13.5 | 18.5 | 32.6 | 50.9 | 78.9 | 91.9 | 39.7

Training configuration. We adopt the ConvNext-Large CLIP model [43, 48] from
OpenCLIP [29]. For mask segmentation training, we mainly follow Cheng et al.
[13] and adopt a similar training recipe and losses without any special
design. The image input size is $896\times 896$ and the batch size is 56. The
model is optimised with AdamW [32, 45] with a learning rate of $1\times
10^{-4}$ and a weight decay of $0.05$. We train the model for 60k iterations
and update the learning rate by a multi-step decay schedule. Besides, we adopt
the same training recipe for mask-text alignment. The whole training is
conducted on 56 V100-32G for $\sim 2.5$ days. $\lambda_{bce}$ and
$\lambda_{dice}$ are set to 2 and 1, respectively.

Evaluation & metrics. We evaluate our model mainly on three tasks, including
open-vocabulary semantic segmentation, open-vocabulary panoptic segmentation,
and promptable segmentation. Following previous work [76], we adopt prompt
engineering from [21, 66] and prompt templates from [22, 37]. For open-
vocabulary semantic segmentation, we zero-shot evaluate the model on the COCO
[40], ADE20K [84], PASCAL [18] datasets. The open-vocabulary semantic
segmentation results are evaluated with the mean Intersection-over-Union
(mIoU). For open-vocabulary panoptic segmentation, we evaluate the model on
the COCO, ADE20K, and Cityscapes [15] datasets. We report the panoptic quality
(PQ), semantic quality (SQ), and recognition quality (RQ) for open-vocabulary
panoptic segmentation. For promptable segmentation, we report the 1-Point and
1-Box IoU (Oracle) on a wide range of datasets. Oracle denotes that we select
the output mask with the max IoU by calculating the IoU between the prediction
and target mask. More details can be found in Appendix B.

###  4.2 Main results

Table 2:  Open-vocabulary panoptic segmentation performance. We mainly compare with the fully-supervised and unsupervised methods. “COCO P.” denotes the COCO panoptic datasets. “COCO” denotes the COCO image dataset. “IN 1K” denotes the ImageNet-1K image dataset. We report PQ, SQ and RQ for all datasets.  |  | COCO | ADE20K | Cityscapes
---|---|---|---|---
Method | Training Data | PQ | SQ | RQ | PQ | SQ | RQ | PQ | SQ | RQ
Fully-supervised method (image-text-mask pair)
MaskCLIP [17] | COCO P. | 30.9 | - | - | 15.1 | 70.5 | 19.2 | - | - | -
ODISE [66] | COCO P. | 55.4 | - | - | 22.6 | - | - | 23.9 | 75.3 | 29.0
FC-CLIP [76] | COCO P. | 54.4 | 83.0 | 64.8 | 26.8 | 71.6 | 32.3 | 44.0 | 75.4 | 53.6
OPSNet [10] | COCO P. | 57.9 | 84.1 | 68.2 | 19.0 | 52.4 | 23.0 | 41.5 | 67.5 | 50.0
Unsupervised method (Unlabelled image)
CutLER+STEGO [60] | IN 1K + COCO | 12.4 | 64.9 | 15.5 | - | - | - | 12.4 | 36.1 | 15.2
U2Seg [47] | IN 1K + COCO | 16.1 | 71.1 | 19.9 | - | - | - | 17.6 | 52.7 | 21.7
Weakly-supervised method (image-text & image-mask pair)
Uni-OVSeg (Ours) | 30% SA1B + M. 1.3M | 18.0 | 72.6 | 24.3 | 14.1 | 66.1 | 19.0 | 17.5 | 65.2 | 23.5
Figure 3: Point-promptable segmentation performance. We compare our method
with SAM-ViT/L [34] on a wide range of datasets. Given a $20\times 20$ point
grid as visual prompt, we select the output masks with max IoU by calculating
the IoU with the ground-truth masks. We report 1-pt IoU for all datasets.

Open-vocabulary semantic segmentation. As demonstrated in Tab. 1, our study
presents a comprehensive comparison of our Uni-OVSeg against previous works
across a range of benchmarks. These include ADE20K (encompassing both 150 and
847 class variants), PASCAL Context (459 and 59 class variants), PASCAL VOC
(with 20 and 21 class categories), and Cityscapes. Compared to weakly-
supervised methods, Uni-OVSeg exhibits remarkable performance improvements
across all evaluated datasets. Specifically, in the more challenging datasets
of PASCAL Context-459, Uni-OVSeg not only surpasses its weakly-supervised
counterparts but also outperforms the cutting-edge fully-supervised methods,
_e.g_., FC-CLIP. This is indicative of Uni-OVSeg’s superior capability in
categorizing a diverse array of semantic classes. Furthermore, in the PASCAL
VOC benchmarks (20 and 21 classes), Uni-OVSeg demonstrates a substantial
enhancement over state-of-the-art weakly-supervised methods, achieving
improvements of 18.3% and 12.2% mIoU, respectively, which demonstrates our
Uni-OVSeg captures fine-grained spatial structures. These results elevate the
practical applicability of weakly-supervised open-vocabulary segmentation to
new heights.

Open-vocabulary panoptic segmentation. For open-vocabulary panoptic
segmentation, we zero-shot evaluate our model on the COCO, ADE20K, and
Cityscapes datasets, shown in Tab. 2. Existing weakly-supervised methods only
use text supervision, which causes a challenge to discriminate different
instances with the same semantic class. To the best of our knowledge, we are
the first to learn open-vocabulary panoptic segmentation with weak
supervision. Compared to unsupervised methods, we obviously outperform U2Seg
by 1.9% PQ, 1.5%SQ, and 4.4% RQ on COCO datasets. Unfortunately, our used
image-mask pairs contain multiple granularity masks, such as object-wise and
part-wise masks, which is different with the panoptic segmentation datasets.
This discrepancy leads to a number of false positive results, limiting our
performance.

Promptable segmentation. To evaluate the segmentation efficacy of Uni-OVSeg in
scenarios involving interactive point and box prompts, we conduct comparative
analyses with the SAM-ViT/L model across a range of datasets from diverse
domains. For visual prompts, we implement a uniform 20$\times$20 point grid as
the interactive point prompt and utilise the actual bounding boxes as box
prompts. The segmentation performance is measured using the 1-pt IoU (Oracle)
metric across all datasets. The results, as demonstrated in Fig. 3, indicate
that Uni-OVSeg exceeds SAM in most datasets when employing point and box
prompts. In particular, Uni-OVSeg significantly surpasses SAM in terms of IoU
by 31.1%, 20.1%, and 43.2% on the TimberSeg, COCO instance, and PIDray
datasets, respectively. Additionally, we provide a visualisation of the
segmentation results of both SAM and Uni-OVSeg in Fig. 4. The visualisation
reveals that SAM tends to miss larger masks and erroneously categorise areas
with unclear boundaries. In contrast, Uni-OVSeg demonstrates exceptional
precision in segmenting objects of varying scales and different stuff. More
experimental results are visualisation can be found in the Appendix B and C.

Figure 4: Visualisaton of point-promptable automatic mask generation. We adopt
a $20\times 20$ point grid as a visual prompt and select the output masks with
max IoU by calculating the IoU with the ground truth masks.

Table 3:  Ablation study on mask-text alignment. “Refine.” denotes the text refinement by the LVLM. “Parser.” denotes the text parser, which extracts entities from text descriptions. “NLTK” and “GPT” denote the natural language toolkit and ChatGPT-based parser. “M.S.” denotes the multi-scale ensemble strategy.  Refine. | Parser. | M.S. | COCO | PC-59
---|---|---|---|---
PQ (%) | mIoU (%) | mIoU (%)
baseline | 13.2 | 33.2 | 39.7
| NLTK |  | 14.1 | 34.5 | 40.9
✓ | NLTK |  | 15.6 | 37.3 | 44.5
✓ | GPT |  | 16.2 | 40.4 | 48.2
✓ | GPT | ✓ | 18.0 | 42.7 | 50.9

Table 4:  Mask classification performance. we first pool the region features based on the provided ground truth masks. These pooled features are then projected into the CLIP embedding space, where they are classified using text embeddings. We report the Top-1 accuracy (%) and time (sec. / sample).  Input size | ADE20K | COCO | Time
---|---|---|---
768 | 45.2 | 49.0 | 0.10
896 | 48.3 | 53.4 | 0.13
1024 | 50.0 | 55.5 | 0.16
768, 896 | 49.2 | 55.2 | 0.20
896, 1024 | 51.3 | 58.3 | 0.26
768, 896, 1024 | 51.6 | 58.4 | 0.35

###  4.3 Ablation study

We conduct an extensive ablation study to demonstrate the contribution of each
component of our framework.

Mask-text alignment. Compared to the straightforward baseline, as shown in
Tab. 4, our proposed Uni-OVSeg achieves significant gains of 4.8% PQ and 9.5%
mIoU on the COCO dataset, and 11.2% mIoU on the PASCAL Context-59 dataset.
This demonstrates our method effectively align objects in images and entities
in text descriptions, generalising the CLIP embedding space from the image
level to pixel level. By resorting to the refinement of text descriptions, new
texts are more correlated with the corresponding images, improving the mIoU
from 34.5% to 37.3% on the COCO dataset. Compared to the traditional NLP
toolkit (NLTK) [3], ChatGPT-based parser extracts more reliable entities from
text descriptions, which achieves obvious improvements of 3.1% and 3.7% mIoU
on the COCO and PASCAL Context-59 datasets, respectively. Finally, the
proposed multi-scale ensemble strategy that leverages the multi-scale
information of objects within the images, stabilise the mask-text matching,
which achieves a performance gain of 1.8% PQ on the COCO datasets.

Multi-scale ensemble in mask-text matching. The quality of correspondence
between masks and entities is an essential part of mask-text matching. To
investigate the impact of multi-scale information on this correspondence, as
illustrated in Tab. 4, we use masks and semantic classes from the ADE20K and
COCO datasets, reporting the Top1 accuracy and forward time per sample. We
first resize input images to multiple resolutions and extract visual features
via the clip visual encoder. Given ground-truth masks, regional features are
pooled from CLIP visual features and projected into the clip embedding space.
Each regional embedding is classified by text embeddings. Taking into account
the trade-off between performance and latency, we adopt the sizes of
$869\times 896$ and $1024\times 1024$ as default.

##  5 Conclusion

In conclusion, this paper proposes an innovative framework for weakly-
supervised open-vocabulary segmentation, named Uni-OVSeg. Using independent
image-text and image-mask pairs, Uni-OVSeg effectively reduces the dependency
on labour-intensive image-mask-text triplets, meanwhile achieving impressive
segmentation performance in open-vocabulary settings. Using the LVLM to refine
text descriptions and multi-scale ensemble to enhance the quality of region
embeddings, we alleviate the noise in mask-text correspondences, achieving
substantial performance improvements. Notably, Uni-OVSeg significantly
outperforms previous state-of-the-art weakly-supervised methods and even
surpasses the cutting-edge fully-supervised method on the Challenging PASCAL
Context-459 dataset. This impressive advancement demonstrates the superiority
of our proposed framework and paves the way for further research.

##  6 Broader impacts

The Uni-OVSeg framework represents a significant advancement in open-
vocabulary segmentation by reducing the dependency on labour-intensive image-
mask-text triplet annotations. This innovation has the potential to
democratise access to cutting-edge vision perception systems, offering
substantial benefits across various sectors, such as medical imaging and
autonomous vehicles. The development of a more efficient and accurate vision
perception system contributes to the community, potentially leading to more
innovative applications and research in machine learning, computer vision, and
related areas. As with any AI model, there is a risk of bias in the data used
for training. Efforts must be made to ensure that the datasets are diverse and
representative to avoid perpetuating or amplifying biases.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _NeurIPS_ , 35:23716–23736, 2022.
  * Bai et al. [2022] Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu.  Rsa: Reducing semantic shift from aggressive augmentations for self-supervised learning.  _NeurIPS_ , 35:21128–21141, 2022.
  * Bird et al. [2009] Steven Bird, Ewan Klein, and Edward Loper.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  ” O’Reilly Media, Inc.”, 2009.
  * Bolya et al. [2019] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation.  In _ICCV_ , pages 9157–9166, 2019.
  * Cai et al. [2023] Kaixin Cai, Pengzhen Ren, Yi Zhu, Hang Xu, Jianzhuang Liu, Changlin Li, Guangrun Wang, and Xiaodan Liang.  Mixreorg: Cross-modal mixed patch reorganization is a good mask learner for open-world semantic segmentation.  In _ICCV_ , pages 1196–1205, 2023.
  * Cha et al. [2023] Junbum Cha, Jonghwan Mun, and Byungseok Roh.  Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs.  In _CVPR_ , pages 11165–11174, 2023.
  * Chen et al. [2019] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al.  Hybrid task cascade for instance segmentation.  In _CVPR_ , pages 4974–4983, 2019.
  * Chen et al. [2023a] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.  Sharegpt4v: Improving large multi-modal models with better captions.  _arXiv preprint arXiv:2311.12793_ , 2023a.
  * Chen et al. [2023b] Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J Fleet.  A generalist framework for panoptic segmentation of images and videos.  In _ICCV_ , pages 909–919, 2023b.
  * Chen et al. [2023c] Xi Chen, Shuang Li, Ser-Nam Lim, Antonio Torralba, and Hengshuang Zhao.  Open-vocabulary panoptic segmentation with embedding modulation.  _ICCV_ , 2023c.
  * Cheng et al. [2020] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen.  Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation.  In _CVPR_ , pages 12475–12485, 2020.
  * Cheng et al. [2021] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  In _NeurIPS_ , pages 17864–17875, 2021.
  * Cheng et al. [2022] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In _CVPR_ , pages 1290–1299, 2022.
  * Ciampi et al. [2021] Luca Ciampi, Carlos Santiago, João Paulo Costeira, Claudio Gennaro, and Giuseppe Amato.  Domain adaptation for traffic density estimation.  In _VISIGRAPP (5: VISAPP)_ , pages 185–195, 2021.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _CVPR_ , pages 3213–3223, 2016.
  * Ding et al. [2022] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai.  Decoupling zero-shot semantic segmentation.  In _CVPR_ , pages 11583–11592, 2022.
  * Ding et al. [2023] Zheng Ding, Jieke Wang, and Zhuowen Tu.  Open-vocabulary universal image segmentation with maskclip.  2023\.
  * Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International Journal of Computer Vision_ , 88:303–338, 2010.
  * Fortin et al. [2022] Jean-Michel Fortin, Olivier Gamache, Vincent Grondin, François Pomerleau, and Philippe Giguère.  Instance segmentation for autonomous log grasping in forestry operations.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6064–6071. IEEE, 2022.
  * Fu et al. [2019] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In _CVPR_ , pages 3146–3154, 2019.
  * Ghiasi et al. [2022] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Scaling open-vocabulary image segmentation with image-level labels.  In _ECCV_ , pages 540–557, 2022.
  * Gu et al. [2021] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  _arXiv preprint arXiv:2104.13921_ , 2021.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _ICCV_ , pages 2961–2969, 2017.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _CVPR_ , pages 9729–9738, 2020.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _CVPR_ , pages 16000–16009, 2022.
  * Hong et al. [2020] Jungseok Hong, Michael Fulton, and Junaed Sattar.  Trashcan: A semantically-segmented dataset towards visual detection of marine debris.  _arXiv preprint arXiv:2007.08097_ , 2020.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Huang et al. [2022] Yixiang Huang, Zhaoqing Wang, Xin Jiang, Ming Wu, Chuang Zhang, and Jun Guo.  Pointshift: Point-wise shift mlp for pixel-level cloud type classification in meteorological satellite imagery.  In _IGARSS_ , pages 607–610. IEEE, 2022.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, 2021.  If you use this software, please cite it as below.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_ , pages 4904–4916, 2021.
  * Karp et al. [1990] Richard M Karp, Umesh V Vazirani, and Vijay V Vazirani.  An optimal algorithm for on-line bipartite matching.  In _STOC_ , pages 352–358, 1990.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirillov et al. [2019] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár.  Panoptic segmentation.  In _CVPR_ , pages 9404–9413, 2019.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _ICCV_ , 2023.
  * Li et al. [2022a] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl.  Language-driven semantic segmentation.  In _ICLR_ , 2022a.
  * Li et al. [2023] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao.  Semantic-sam: Segment and recognize anything at any granularity.  _arXiv preprint arXiv:2307.04767_ , 2023.
  * Li et al. [2022b] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu.  Panoptic segformer: Delving deeper into panoptic segmentation with transformers.  In _CVPR_ , pages 1280–1289, 2022b.
  * Liang et al. [2022] Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang.  Gmmseg: Gaussian mixture based generative semantic segmentation models.  In _NeurIPS_ , pages 31360–31375, 2022.
  * Liang et al. [2023] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  In _CVPR_ , pages 7061–7070, 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , pages 740–755, 2014.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _CVPR_ , pages 2117–2125, 2017.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * Liu et al. [2022] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In _CVPR_ , pages 11976–11986, 2022.
  * Long et al. [2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _CVPR_ , pages 3431–3440, 2015.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Luo et al. [2023] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li.  Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation.  In _ICML_ , pages 23033–23044, 2023.
  * Niu et al. [2023] Dantong Niu, Xudong Wang, Xinyang Han, Long Lian, Roei Herzig, and Trevor Darrell.  Unsupervised universal image segmentation.  _arXiv preprint arXiv:2312.17243_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , pages 8748–8763, 2021.
  * Ranasinghe et al. [2023] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens.  Perceptual grouping in contrastive vision-language models.  In _ICCV_ , pages 5571–5584, 2023.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _ACL_ , pages 2556–2565, 2018.
  * Snyder and Do [2019] Corey Snyder and Minh Do.  Streets: A novel camera network dataset for traffic flow.  _NeurIPS_ , 32, 2019.
  * Thomee et al. [2016] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  _Communications of the ACM_ , 59(2):64–73, 2016.
  * Tian et al. [2020] Zhi Tian, Chunhua Shen, and Hao Chen.  Conditional convolutions for instance segmentation.  In _ECCV_ , pages 282–298, 2020.
  * Tian et al. [2022] Zhi Tian, Bowen Zhang, Hao Chen, and Chunhua Shen.  Instance and panoptic segmentation using conditional convolutions.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(1):669–680, 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _NeurIPS_ , 30, 2017.
  * Wang et al. [2020] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  Axial-deeplab: Stand-alone axial-attention for panoptic segmentation.  In _ECCV_ , pages 108–126, 2020.
  * Wang et al. [2021a] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  Max-deeplab: End-to-end panoptic segmentation with mask transformers.  In _CVPR_ , pages 5463–5474, 2021a.
  * Wang et al. [2023a] Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari.  Sam-clip: Merging vision foundation models towards semantic and spatial understanding.  _arXiv preprint arXiv:2310.15308_ , 2023a.
  * Wang et al. [2021b] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Solo: A simple framework for instance segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(11):8587–8601, 2021b.
  * Wang et al. [2023b] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra.  Cut and learn for unsupervised object detection and instance segmentation.  In _CVPR_ , pages 3124–3134, 2023b.
  * Wang et al. [2022a] Zhaoqing Wang, Ziyu Chen, Yaqian Li, Yandong Guo, Jun Yu, Mingming Gong, and Tongliang Liu.  Mosaic representation learning for self-supervised visual pre-training.  In _ICLR_ , 2022a.
  * Wang et al. [2022b] Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu.  Exploring set similarity for dense self-supervised representation learning.  In _CVPR_ , pages 16590–16599, 2022b.
  * Wang et al. [2022c] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu.  Cris: Clip-driven referring image segmentation.  In _CVPR_ , pages 11686–11695, 2022c.
  * Xu et al. [2022a] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In _CVPR_ , pages 18134–18144, 2022a.
  * Xu et al. [2023a] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie.  Learning open-vocabulary semantic segmentation models from natural language supervision.  In _CVPR_ , pages 2935–2944, 2023a.
  * Xu et al. [2023b] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In _CVPR_ , pages 2955–2966, 2023b.
  * Xu et al. [2022b] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai.  A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model.  In _ECCV_ , pages 736–753, 2022b.
  * Xu et al. [2023c] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai.  Side adapter network for open-vocabulary semantic segmentation.  In _CVPR_ , pages 2945–2954, 2023c.
  * Yang et al. [2021a] Lei Yang, Yan Zi Wei, Yisheng He, Wei Sun, Zhenhang Huang, Haibin Huang, and Haoqiang Fan.  ishape: A first step towards irregular shape instance segmentation.  _arXiv preprint arXiv:2109.15068_ , 2021a.
  * Yang et al. [2021b] Shuo Yang, Peize Sun, Yi Jiang, Xiaobo Xia, Ruiheng Zhang, Zehuan Yuan, Changhu Wang, Ping Luo, and Min Xu.  Objects in semantic topology.  In _ICLR_ , 2021b.
  * Yogamani et al. [2019] Senthil Yogamani, Ciarán Hughes, Jonathan Horgan, Ganesh Sistu, Padraig Varley, Derek O’Dea, Michal Uricár, Stefan Milz, Martin Simon, Karl Amende, et al.  Woodscape: A multi-task, multi-camera fisheye dataset for autonomous driving.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9308–9318, 2019.
  * You et al. [2023] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang.  Ferret: Refer and ground anything anywhere at any granularity.  _arXiv preprint arXiv:2310.07704_ , 2023.
  * Yu et al. [2022a] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022a.
  * Yu et al. [2022b] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  Cmt-deeplab: Clustering mask transformers for panoptic segmentation.  In _CVPR_ , pages 2560–2570, 2022b.
  * Yu et al. [2022c] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  k-means mask transformer.  In _ECCV_ , pages 288–307, 2022c.
  * Yu et al. [2023] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen.  Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip.  _NeurIPS_ , 2023.
  * Yuan et al. [2021] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang.  Ocnet: Object context for semantic segmentation.  _International Journal of Computer Vision_ , 129(8):2375–2398, 2021.
  * Zhang et al. [2022a] Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al.  Segvit: Semantic segmentation with plain vision transformers.  In _NeurIPS_ , pages 4971–4982, 2022a.
  * Zhang et al. [2018] Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal.  Context encoding for semantic segmentation.  In _CVPR_ , pages 7151–7160, 2018.
  * Zhang et al. [2023a] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang.  A simple framework for open-vocabulary segmentation and detection.  In _ICCV_ , pages 1020–1031, 2023a.
  * Zhang et al. [2022b] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi.  Fine-grained egocentric hand-object segmentation: Dataset, model, and applications.  In _European Conference on Computer Vision_ , pages 127–145. Springer, 2022b.
  * Zhang et al. [2023b] Libo Zhang, Lutao Jiang, Ruyi Ji, and Heng Fan.  Pidray: A large-scale x-ray benchmark for real-world prohibited item detection.  _International Journal of Computer Vision_ , 131(12):3170–3192, 2023b.
  * Zhang et al. [2021] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy.  K-net: Towards unified image segmentation.  _NeurIPS_ , 34:10326–10338, 2021.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In _CVPR_ , pages 633–641, 2017.
  * Zhou et al. [2022] Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  In _ECCV_ , pages 696–712, 2022.
  * Zhu et al. [2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In _ICLR_ , 2020.
  * Zou et al. [2023a] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al.  Generalized decoding for pixel, image, and language.  In _CVPR_ , pages 15116–15127, 2023a.
  * Zou et al. [2023b] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee.  Segment everything everywhere all at once.  _arXiv preprint arXiv:2304.06718_ , 2023b.

##  Appendix A Framework details

Inputs. For input images, we initially apply a random horizontal flip to each
image. Subsequently, the image is randomly scaled to a resolution within the
range of $716\times 716$ to $1075\times 1075$ resolutions. Finally, a crop of
$896\times 896$ resolutions is extracted from the scaled image to serve as the
input. Regarding the category names, we construct a sentence using a prompt
and tokenise it using a lower-cased byte pair encoding (BPE). For the visual
prompt, we create a uniform grid of points with dimensions $h\times w$, which
is aligned with the center of the pixels.

CLIP encoder. In general, the CLIP encoder can be any architecture. Motivated
by the scalability of different input resolutions, we employ a ConvNext-based
CLIP model to serve as both the image and text encoders. The image encoder is
configured as a ConvNext-Large model, comprising four stages. Each stage
contains a different number of blocks: 3 in the first, 3 in the second, 27 in
the third, and 3 in the fourth. In contrast, the text encoder is structured as
a 16-layer transformer, each layer being 768 units wide and featuring 12
attention heads. We harness the power of multi-scale features extracted by the
image encoder. These features are represented as feature maps of varying
widths and scales: a 192-wide feature map downscaled by a factor of 4, a
384-wide map downscaled by 8, a 768-wide map downscaled by 16, and a 1536-wide
map downscaled by 32.

Pixel decoder. Following Mask2Former [13], we incorporate a lightweight pixel
decoder based on the widely used Feature Pyramid Network (FPN) architecture
[41]. We first adopt a 6-layer multi-scale deformable transformer on the
multi-scale features to aggregate contextual information. Afterward, we
upscale the low-resolution feature map in the decoder by a factor of 2 and
then combine it with the corresponding resolution feature map from the
backbone, which has been projected to match channel dimensions. This
projection is achieved through a 1$\times$1 convolution layer, followed by
Group Normalization (GroupNorm). Subsequently, the merged features undergo
further processing with an additional 3$\times$3 convolution layer,
complemented by GroupNorm and ReLU activation. This procedure is applied
iteratively, beginning with the 32$\times$ downscaled feature map, until we
attain a final feature map that is 4$\times$ downscaled. To generate pixel-
wise embeddings, a single 1$\times$1 convolution layer is applied at the end.
Throughout the pixel decoder, all feature maps maintain a consistent dimension
of 256 channels.

Visual prompt encoder. We have two types of visual prompts, including points
and bounding boxes. Each type of prompts is mapped to 256-wide embeddings as
follows. A point is first converted into a small bounding box. We represent a
bounding box by an embedding pair, including the sine position embedding of
its top-left and bottom-right corners. Afterward, we use two extra learnable
embeddings to indicate these two corners.

Figure 5:  Architecture of the mask decoder layer. This decoder layer updates
both visual prompt embeddings and pixel features by the cross-attention
layers. The self-attention layer is used to update visual prompts. At each
attention layer, positional encodings are added to the pixel features, and the
entire original visual prompts (including position encoding) are added to the
updated visual prompts.

Mask decoder. Following Mask2Former, we use a similar transformer decoder
design. Each visual prompt embedding within our framework is coupled with a
sine positional embedding. As depicted in Fig. 5, our approach utilizes six
mask decoder layers, and we apply the same loss function after each layer.
Moreover, every decoder layer includes a cross-attention layer. This layer is
crucial as it ensures that the final pixel features are enriched with
essential geometric information, such as point coordinates and bounding boxes.
Besides, in each decoder layer, we use a cross-attention layer, which ensures
that the final pixel features have access to critical geometric information
(_e.g_., point coordinates, and boxes). Finally, visual prompt embeddings are
fed to a layer normalization, followed by processing through a multiple
perception layer (MLP). These processed embeddings are then dot-producted with
the pixel features, culminating in the generation of mask predictions.

Figure 6:  Architecture of the multi-scale feature adapter. Each LoRA adapter
is used to update a specific downscaled features.

Multi-scale feature adapter. The multi-scale feature adapter illustrated in
Fig. 6 is designed to refine and enhance pixel features at various scales.
This adapter is composed of multiple Low-rank Adapters (LoRA adapters) [27],
each specifically tailored to update features at a particular scale. Each LoRA
adapter incorporates two linear layers, denoted as A and B, with a non-linear
activation function placed in between. The linear layers A and B serve to
transform the input data linearly, while the activation function introduces
non-linearity, allowing for the modeling of more complex relationships in the
data. Each adapter is responsible for handling features at a specific scale,
suggesting a hierarchical approach to feature refinement. This modular design
allows for a focused and specialized treatment of features depending on their
resolution and semantic complexity, which can be particularly advantageous for
dense prediction tasks.

Figure 7: Box-promptable segmentation performance. We compare our method with SAM-ViT/L [34] on a wide range of datasets. Given a ground-truth box as the visual prompt, we select the output masks with max IoU by calculating the IoU with the ground-truth masks. We report 1-pt IoU for all datasets. Figure 8: Point-promptable segmentation performance. We compare our method with SAM-ViT/L [34] on the SegInW datasets [87]. Given a $20\times 20$ point grid as a visual prompt, we select the output masks with max IoU by calculating the IoU with the ground-truth masks. We report 1-pt IoU for all datasets. Figure 9: Box-promptable segmentation performance. We compare our method with SAM-ViT/L [34] on the SegInW datasets [87]. Given a ground-truth box as the visual prompt, we select the output masks with max IoU by calculating the IoU with the ground-truth masks. We report 1-pt IoU for all datasets. Table 5:  Segmentation datasets used to evaluate promptable segmentation with point and box prompts. The 11 datasets cover a broad range of domains, which are illustrated in “image type”.  Dataset | Image type | Mask type | Description
---|---|---|---
Egocentric Hand-Object Segmentation (EgoHOS) [81] | Egocentric | Instance |  Fine-grained egocentric hand-object segmentation dataset. Dataset contains mask annotations for existing datasets.
TimberSeg 1.0 (TimberSeg) [19] | Logs | Instance |  Segmentation masks of individual logs in piles of timber in various environments and conditions. Images are taken from an operator’s point-of-view.
STREETS [51] | Traffic camera | Instance | Segmentation masks of cars in traffic camera footage.
iShape [69] | Irregular shapes | Instance |  Segmentation masks of irregular shapes like antennas, logs, shapes fences, and hangers.
COCO [40] | Scenes | Instance |  Segmentation masks of complex everyday scenes containing common objects in their natural context.
ADE20K [84] | Scenes | Instance |  Object and part segmentation masks for images from SUN and Places datasets.
Night and Day Instance Segmented Park (NDISPart) [14] | Parking lots | Instance |  Images of parking lots from video footage taken at day and night during different weather conditions and camera angles for vehicle segmentation.
WoodScape [71] | Fisheye driving | Instance |  Fisheye driving dataset with segmentation masks. Images are driving taken from four surround-view cameras.
PIDray [82] | X-ray | Instance |  Segmentation masks of prohibited items in X-ray images of baggage.
TrashCan [26] | Underwater | Instance |  Segmentation masks of trash in images taken by underwater ROVs. Images are sourced from the J-EDI dataset.
Segmentation in the wild (SegInW) [87] | Multiple domain | Instance |  This dataset consists of 25 free public Segmentation datasets, crowd-sourced on roboflow.com.

##  Appendix B Promptable segmentation.

Evaluation details. We perform a prompt segmentation evaluation on a wide
range of datasets in various domains. For the point prompt, we adopt a uniform
point grid $h\times w$ as input prompts (_e.g_., 20 $\times$ 20). For the box
prompt, we use ground-truth bounding boxes as input prompts. 1-pt IoU denotes
the oracle performance of one point by evaluating the intersection-over-union
(IoU) of the predicted masks that best match ground truth. 1-box IoU denotes
is similar to 1-pt IoU. More evaluation results are reported in Fig. 7, Fig. 8
and Fig. 9.

Dataset details. A description of each dataset is given in Tab. 5. The iShape
dataset has 6 subsets, including antenna, branch, fence, hanger, log and wire.

##  Appendix C Visualisation

We illustrate a wide range of visualisations of promptable segmentation and
open-vocabulary segmentation across multiple datasets.

Figure 10: Visualisation of open-vocabulary segmentation between the baseline
and ours Uni-OVSeg. Figure 11: Visualisation of open-vocabulary segmentation
between the baseline and ours Uni-OVSeg. Figure 12: Visualisation of open-
vocabulary segmentation between the baseline and ours Uni-OVSeg. Figure 13:
Visualisation of promptable segmentation between SAM-ViT/L and ours Uni-OVSeg.
Figure 14: Visualisation of promptable segmentation between SAM-ViT/L and ours
Uni-OVSeg. Figure 15: Visualisation of promptable segmentation between SAM-
ViT/L and ours Uni-OVSeg. Figure 16: Visualisation of promptable segmentation
between SAM-ViT/L and ours Uni-OVSeg. Figure 17: Visualisation of promptable
segmentation between SAM-ViT/L and ours Uni-OVSeg.