# UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via
Segment Anything Model

Zhenghao Zhang∗, Zhichao Wei∗, Shengfan Zhang, Zuozhuo Dai and Siyu Zhu
Alibaba Group
Equal contribution<EMAIL_ADDRESS>
###### Abstract

Unsupervised video object segmentation has made significant progress in recent
years, but the manual annotation of video mask datasets is expensive and
limits the diversity of available datasets. The Segment Anything Model (SAM)
has introduced a new prompt-driven paradigm for image segmentation, unlocking
a range of previously unexplored capabilities. In this paper, we propose a
novel paradigm called UVOSAM, which leverages SAM for unsupervised video
object segmentation without requiring video mask labels. To address SAM’s
limitations in instance discovery and identity association, we introduce a
video salient object tracking network that automatically generates
trajectories for prominent foreground objects. These trajectories then serve
as prompts for SAM to produce video masks on a frame-by-frame basis. Our
experimental results demonstrate that UVOSAM significantly outperforms current
mask-supervised methods. These findings suggest that UVOSAM has the potential
to improve unsupervised video object segmentation and reduce the cost of
manual annotation.

##  1 Introduction

Video object segmentation (VOS) is a crucial task [1; 25; 24; 55; 50; 49] in
computer vision that involves segmenting primary objects in a video sequence.
This task has numerous applications, such as video editing, autonomous driving
and robotics. VOS tasks can be broadly classified into semi-supervised VOS
[25; 36; 4; 43; 5; 27] and unsupervised VOS [49; 20; 7; 16; 28; 35; 48; 52].
Semi-supervised VOS utilizes the segmentation mask of the first frame during
inference to initialize the model, with the aim of tracking and segmenting
specific objects throughout the entire sequence. In contrast, unsupervised VOS
relies on the model to discover and extract the masks of the most salient
objects without any prior information. Unsupervised VOS is gaining popularity
due to its ability to automatically perform segmentation without the need for
manual annotation, making it suitable for real-time applications. However,
current state-of-the-art unsupervised VOS methods [7; 16; 28; 22; 54; 19]
still require a large-scale manually annotated video dataset [29; 44; 31; 9]
for training. The annotation of video object segmentation is prohibitively
expensive, which needs to provide both the mask and trajectory. Even
annotating masks based on coarse polygons takes several times longer than
annotating video bounding boxes [6]. The high cost of annotating masks makes
it challenging to scale existing VOS. Hence, a mask-free setting [14] may be
necessary to handle this task more efficiently.

Recent advancements in segmentation foundation models [15; 39; 57] have led to
significant improvements in image segmentation. One of the most notable models
is the Segment Anything Model (SAM) [15], which has been trained on over 1
billion masks and has shown exceptional capabilities in accurately creating
object masks based on different types of prompts such as bounding boxes and
points. SAM’s superior performance in this area has gained widespread
recognition. Naturally, we consider the following question: Can we employ SAM
in unsupervised VOS by providing simple prompts to liberate the labor-
expensive video annotation. This approach could potentially make this task
more accessible and cost-effective, while still achieving high-quality
results.

The main challenge is that SAM inherently loses the capability to discover
specific instances [53] and associate the same identities over the sequence
[46]. A simple way is manually locating the same target object in different
poses frame by frame, and then activate SAM with prompt for segmentation.
Although good performance can be achieved in this manner, it is not only time-
consuming and labor-intensive, but also deviates from the core principles of
unsupervised video object segmentation. We therefore revisit the need for
automatically generating video prompts on SAM.

To this end, we propose UVOSAM, a mask-free approach based on SAM for
unsupervised video object segmentation. The proposed UVOSAM consists of SAM
and a novel video salient object tracking (VSOT) model, which inherits the
most architecture of an advanced video instance segmentation methods IDOL [42]
while removing the mask prediction branch. This approach provides more
discriminative object features with better temporal consistency and guarantees
more accurate association results. More specifically, VSOT is applied firstly
to discovery the salient objects and provide the complete trajectories of them
by leveraging both the spatial and temporal correspondence. Then, SAM is
employed to obtain precise object mask results frame by frame. Our main
contributions can be concluded as follows: 1) we first extend the SAM
applications to the unsupervised VOS. Rather than using SAM to directly
process each video frame, we propose a new pipeline which first detects
salient objects and generates stable trajectories to prompt the SAM. 2) Our
proposed method shows superior performance in complex video scenes and
outperforms current mask-supervised unsupervised VOS methods by a large
margin.

##  2 Related Works

###  2.1 Unsupervised Video Object Segmentation

Unsupervised VOS aims to segment class-agnostic salient objects without any
manual guidance. Early works [11; 37; 45] consider that the pixels of objects
share same motion patterns across consecutive frames and utilize motion cues
captured by optical flow to guide segmentation process. For example, MP-Net
[37] based on encoder-decoder architecture takes the optical flow as input for
moving object segmentation. One limitation to these methods is that their
performances are heavily affected by the quality of optical flow, especially
in occlusion and fast motion scenarios. Some works [56; 52; 35; 48; 7] also
learn to attend to appearance features of objects. MATNet [56] employs the
MAT-block to transform appearance features into motion-attentive
representations. RTNet [35] reciprocally transforms the appearance features
and the motion features to discover primary objects. AMC-Net [48] introduces
multi-modality co-attention gate to promote deep collaboration of appearance
and motion information for accurate segmentation. By combining motion cues and
appearance cues, these methods are able to capture prominent regions of
salient objects and achieve more accurate segmentation.

For multiple objects segmentation, most methods employ the track-by-detect
paradigm where object proposals are generated by instance segmentation models
and then tracked by re-identification models. UnOVOST [22] first groups
segments into short tracklets and then merges them into long-term object
tracks based on their similarity. Target-Aware [54] employs a target-aware
adaptive tracking framework to associate proposals across frames, achieving
more robust matching results. Different from these methods which require
annotation-expensive segmentation datasets for training, our mask-free UVOSAM
achieves accurate segmentation by supplying simple prompts to vision
foundation models, making the challenging Unsupervised VOS more accessible.

###  2.2 Vision Foundation Models

Recently, foundation models have arisen significant interest for their
impressive generalization capability on various downstream tasks. In natural
language processing, the milestone works are GPT series [33; 34; 3], which
demonstrate strong ability on many language generation tasks. Motivated by
this, some models were proposed to further explore the potential of zero-shot
generalization, including GLM [51], PaLM [8], LLaMA [38] and so on.

In the field of computer vision, research of foundation models are still in
its infancy. The pioneering work CLIP [32], trained on web-scale image-text
pairs, exhibits promising zero-shot capability and is widely adopted in many
multi-modal tasks for feature alignment. BLIP [18; 17] performs multi-modal
pre-training with a dataset bootstrapped from web-scale noisy data,
facilitating zero-shot performance on text-to-video retrieval and VQA. To
build a universal detection framework, UniDetector [40] utilizes images of
multiple sources and heterogeneous label spaces for training, whose zero-shot
performance surpasses the supervised baselines with large margins. More
recently, SAM [15], the first promptable foundation model for segmentation
tasks, was released and has gained massive attention for its ability to
segment any object in any scenarios with various prompts, such as box, point
and so on. Concurrent and share the similar idea with SAM, SegGPT [39] and
SEEM [57] were proposed as generalist models to perform arbitrary segmentation
tasks with different prompt types. Our method focuses on employing vision
foundation models to perform mask-free Unsupervised VOS, enhancing
segmentation performance in a more efficient way.

##  3 Methods

We introduce a novel paradigm, termed as UVOSAM, which is based on SAM for
unsupervised video object segmentation without requiring any video mask
labels. More specifically, we first train a video salient object tracking
network in a mask-free setting, which is employed to generate the trajectories
of the targets automatically. The trajectories serve as prompts to SAM, which
produce video masks on a frame-by-frame basis. In Section 3.1, we first
revisit SAM, followed by the methodology of the video salient object tracking
network in Section 3.2. Lastly, in Section 3.3, we present the overall
pipeline of UVOSAM.

###  3.1 A revisit of SAM

SAM is a fundamental model that has been trained using more than one billion
masks and showcases remarkable zero-shot proficiency. Its main components
comprise an image encoder (i.e. $E_{i}$), a prompt encoder (i.e. $E_{p}$), and
a lightweight mask decoder (i.e. $D_{m}$). To be more precise, SAM takes an
image $I\in\mathbb{R}^{H\times W\times 3}$ and its corresponding prompts $P$
(e.g., points, bounding boxes, text, or masks) as inputs. These two stream
inputs are then embedded by the image encoder and prompt encoder,
respectively, which can be formulated as follows:

$\displaystyle F_{i}=E_{i}(I)\in\mathbb{R}^{h\times w\times
c}~{}~{}~{}~{}~{}F_{p}=E_{p}(P)\in\mathbb{R}^{k\times c}$ (1)

where $h$, $w$ and $c$ denote the height, width and channel number of the
feature map, and $k$ represents the length of the prompt tokens. After that
$F_{i}$ and $F_{p}$ are sent to $D_{m}$, there are several learnable tokens
$F_{t}$ concatenated to $F_{p}$ as prefix for flexible prompting , these two
embeddings are then interacted via cross-attention and the zero-shot mask is
obtained by decoding the interactive features, which can be described as:

$\displaystyle
M=D_{m}(~{}Attn(F_{i},concat(F_{p},F_{t}))~{})\in\mathbb{R}^{H\times W}$ (2)

###  3.2 VSOT

Our VSOT builds upon the advancements of the VIS method, IDOL. As an online
VIS method, IDOL leverages an additional association head to instance
segmentation models. By employing contrastive loss, it can extract ReID
features that allow for accurate instance association. Since our VSOT does not
require mask prediction, we have discarded the original mask head. VSOT
focuses on two main components: salient object detection and object
association.

Salient object detection. VSOT employs DeformableDETR as the fundamental
detector. Specifically speaking, a CNN backbone is first utilized to extract
multi-scale feature maps from the input frame of a video. These feature maps,
along with fixed positional encodings and $N$ learnable object queries, are
then provided to the Deformable DETR module. The object queries are
transformed into output embeddings by the transformer decoder, which are then
decoded using the 3-layer feed-forward network into box coordinates and class
labels. We only generate a binary category result to separate the foreground
objects from the background.

Object Association. To distinguish objects across multiple frames, VSOT
utilizes contrastive learning between frames to distinct representations.
Furthermore, for each key frame, it dynamically selects the most relevant
positive and negative examples from the reference frame to minimize false
positives and further enhance the quality of the embedding. During inference,
VSOT utilizes a memory bank-based association strategy to improve the
association quality. Assuming $N$ predicted objects with $N$ ReID embeddings
$\left\\{d_{i}^{T}\right\\}_{i=1}^{N}\in R^{C}$ and $M$ trajectories that
contain a group of $T-1$ embeddings
$\left\\{d_{j}^{t}\right\\}_{j=1,t=0}^{j=M,t=T-1}\in R^{C}$ from previous
$T-1$ frames, the association of frame $T$ is established. To obtain a global
embedding for each trajectory, a temporally weighted softmax score is first
utilized, which can be defined as:

$\displaystyle
e_{j}=\frac{{\textstyle\sum_{t=1}^{T-1}}d_{j}^{t}\times(\tau+(T-1)/t)}{\sum_{t=1}^{T-1}\tau+(T-1)/t}$
(3)

Where $\tau$ represents the time constant, Then the similarity is computed
between $\left\\{d_{i}^{T}\right\\}_{i=1}^{N}$ and $e_{j}$ to perform the
association.

###  3.3 The overall of proposed UVOSAM

Figure 1: The overall illustration of our proposed UVOSAM, mainly consisting
of the VSOT and SAM. The VSOT is trained only using the bounding box
annotations to generate the trajectories of video salient objects, which are
then served as the prompts for SAM. By freezing the entire SAM, we aim to
retain its valuable pre-trained knowledge while also ensuring that our
pipeline achieves mask-free training for unsupervised VOS approaches.

Figure 1 illustrates the proposed UVOSAM approach for unsupervised video
object segmentation, which generates high-quality masks without the need for
manual annotations. UVOSAM is a two-stage paradigm that consists of two main
components: VSOT and SAM. In the VSOT stage, the backbone extracts per-frame
features that are fed into the transformer to refine the object queries of
each frame. The object queries are decoded into objectness score, location,
and ReID embedding using the class head, box head, and ReID head,
respectively. The decoded outputs are then sent to the tracking head, which
matches existing tracklets and identifies new ones. The SAM stage involves the
processing of the input frames and the trajectories obtained from the
preceding VSOT stage. More specifically, the images are fed into the image
encoder, whereas the trajectories are passed to the prompt encoder. The final
video object segmentation outputs are then produced by the mask decoder.

##  4 Experiment

###  4.1 Experimental Setup

Datasets. We evaluate UVOSAM on two popular video segmentation datasets:
DAVIS2017-unsupervised [30] and Youtube-VIS 2019 [47]. The
DAVIS2017-unsupervised dataset consists of 120 high-quality video in total.
These videos are further split into 60 for train, 30 for val and 30 for test-
dev. The Youtube-VIS 2019 dataset consists of 2238 training, 302 validation,
and 343 test videos. Each video has been manually annotated at pixel level and
the semantic category number is 40. Since our VSOT only produces the category-
agnostic trajectories, it needs provide this label to SAM for evaluating the
performance on video instance segmentation. We apply the split set as valid
set from Youtube-VIS 2019 training datasets following IDOL so that we can
obtain the specific category from ground truth.

Evaluation Metrics. For DAVIS2017-unsupervised, we employ the official
evaluation measures including region similarity $\mathcal{J}$, boundary
accuracy $\mathcal{F}$ and the overall metric $\mathcal{J}\&\mathcal{F}$. For
Youtube-VIS 2019, we adopt Average Precision (AP) and Average Recall (AR) to
evaluate the performance of our model. In addition, the Intersection over
Union (IoU) computation is also performed in the whole video.

Implements Details. We employ pre-trained SAM with the ViT-H [10] image
encoder as the base segmentation model. For VSOT, we select ResNet-50 [12] as
the default backbone network. All hyper-parameters about the network
architecture are the same as the official IDOL, we change the network to a
binary classification output, only identifying whether the object is
foreground. We use the AdamW [21] optimizer with a base learning rate of
$5\times 10^{-5}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, and weight decay
$=10^{-4}$. VSOT is trained on 4 NVIDIA A100 GPUs with 4 pairs of frames per
GPU.

###  4.2 Main Results

Table 1 illustrates the results of all compared methods on
DAVIS2017-unsupervised valid set. Our UVOSAM achieves
$\mathcal{J}\&\mathcal{F}$ scores of 78.9, which outperforms current mask-
supervised methods by a large margin. We also produce trajectories with
bounding boxes using the ground truth for each object, and then leverage these
carefully curated bounding boxes as prompts in SAM , which we refer to as
"Human-prompts". It can be clearly seen that when the prompts are sufficiently
accurate, SAM can produce ideal segmentation results despite complex video
scenes.

Methods | $\mathcal{J}\&\mathcal{F}$ |  $\mathcal{J}$-$Mean$ |  $\mathcal{F}$-$Mean$
---|---|---|---
MATNet [56] | 58.6 | 56.7 | 60.4
STEm-Seg [2] | 64.7 | 61.5 | 67.8
UnOVOST [22] | 67.9 | 66.4 | 69.3
Target-Aware [54] | 65.0 | 63.7 | 66.2
Propose-Reduce [19] | 68.3 | 65.0 | 71.6
INO [26] | 72.5 | 68.7 | 76.3
UVOSAM | 78.9 | 75.5 | 82.0
Human-prompts + SAM | 87.3 | 83.5 | 91.0
Table 1: The performance comparison of our method and state-of-the-art methods
on DAVIS2017-unsupervised valid set.

We report our comparisons on YoutubeVIS-19 split set in Table 2. The results
of IFC, SeqFormer and IDOL have been provided the ground truth ID for a fair
comparision. Without any masks in training, our UVOSAM provides an absolute
gain of 2.3 AP over the baseline IDOL. By visualizing outputs on
YoutubeVIS-2019 in Figure 2, we observe that SAM masks are often accurate with
crisper boundaries. As a result, we hypothesize that the lower AP scores of
UVOSAM are largely attributed to the relatively poor quality of annotations in
this dataset.

Figure 2: The Mask quality from Ground truth and UVOSAM on YoutubeVIS-19 split set. The annotations of YoutubeVIS are not always accurate in identifying the foreground object, as they sometimes label the background areas near the edge of the object as foreground, which are highlighted with red dotted region. In contrast, the masks generated by UVOSAM are often accurate. Best viewed zoomed in. Methods | $\rm AP$ | $\rm AP_{50}$ | $\rm AP_{75}$
---|---|---|---
IFC [13] | 51.8 | - | -
SeqFormer [41] | 57.0 | - | -
IDOL [42] | 56.1 | - | -
UVOSAM | 58.4 | 87.9 | 62.1
Human-prompts + SAM | 64.5 | 95.8 | 70.9
Table 2: The performance comparison of our method and state-of-the-art methods
on YoutubeVIS-19 split set.

We also compare the performance of our UVOSAM with start-of-the-art method,
INO [26], using the DAVIS2017-unsupervised dataset. The results are presented
in Figure 3, which demonstrates that UVOSAM outperforms INO in various
challenging video sequences. More specifically, in the dog-jump sequence, the
UVOSAM demonstrates remarkable robustness to occlusions and scale changes,
making it a reliable toolkit for video object segmentation. Moreover, it is
capable of generating finely detailed boundary masks even in challenging
background situations that closely resemble the target object, as illustrated
in the goat sequences.

Figure 3: Qualitative comparison between our UVOSAM and start-of-the-art
method. The five frames in each row are from the same video.

###  4.3 Ablation Study

All the ablation experiments are conducted on the challenging valid set of
DAVIS2017-unsupervised with the same implementation details as described in
Section 4.1 if not specified.

#### Tracking framework.

We conduct experiments to study the effect of different tracking framework,
including track-by-detect multi-object tracking (MOT) framework and the
proposed video salient object tracking (VSOT) framework. For the
implementation of the former, we adopt YOLO-v8111https://ultralytics.com/ and
DeepOCSORT [23] for detection and association, respectively. As shown in Table
3, the VSOT-based model surpasses the MOT-based counterpart by large margins
on all evaluation metrics, which demonstrates that our approach can generate
more complete trajectories by effectively exploiting spatial-temporal
correlations, and thus achieve more precise segmentation results.

Methods | $\mathcal{J}\&\mathcal{F}$ |  $\mathcal{J}$-$Mean$ |  $\mathcal{F}$-$Mean$
---|---|---|---
MOT | 61.1 | 59.5 | 62.7
VSOT | 78.9 | 75.5 | 82.0
Table 3: The performance comparison of different tracking framework on
DAVIS2017 valid set.

#### Prompt types for SAM.

We investigate the effect of different prompt settings for SAM on performance,
including box, point and box&point, as shown in Table 4. For simplicity, the
box prompt and point prompt are generated as the bounding rectangle of the
foreground object and randomly sampled points of the foreground object,
respectively. From the first two rows of Table 4, we can see that the box
prompt largely improves the $\mathcal{J}\&\mathcal{F}$ score,
$\mathcal{J}$-$Mean$ and $\mathcal{F}$-$Mean$ by 2.9, 2.1 and 3.5 respectively
compared to the point prompt. This demonstrates that box prompt could provide
more structural information of the objects which is helpful for SAM to
generate more accurate masks. From the last row of Table 4, the combination of
box and point prompts further boosts the performance, which exhibits a rough
upper bound of our model.

Prompt type | $\mathcal{J}\&\mathcal{F}$ |  $\mathcal{J}$-$Mean$ |  $\mathcal{F}$-$Mean$
---|---|---|---
box | 87.3 | 83.5 | 91.0
point | 84.4 | 81.4 | 87.5
box&point | 87.8 | 84.1 | 91.4
Table 4: The performance comparison of different prompt settings for SAM on
DAVIS2017 valid set.

#### Different settings of point prompt.

To investigate the effect of point number in point prompt on performance, we
vary it as {1, 3, 5, 10}. As shown in Table 5, the model performance is
enhanced consistently with more points in point prompt. Specially, when the
point number increases from 1 to 3, the model enjoys an significant gain of
29.6 on $\mathcal{J}\&\mathcal{F}$ score. This is because using more points
can help the SAM eliminate semantic ambiguity and thus locate objects more
accurately. These results motivate us to explore the combination of box with
more points, as discuss below.

Point number | $\mathcal{J}\&\mathcal{F}$ |  $\mathcal{J}$-$Mean$ |  $\mathcal{F}$-$Mean$
---|---|---|---
1 | 50.1 | 44.9 | 55.4
3 | 79.7 | 76.5 | 82.9
5 | 83.1 | 79.9 | 86.2
10 | 84.4 | 81.4 | 87.5
Table 5: The performance comparison of different point settings of point
prompt for SAM on DAVIS2017 valid set.

#### Prompt combination.

In Table 6, we study the effect of different prompt combination for SAM. It
can be seen that the model performance increases slowly with more points and
saturates with 5 points. This demonstrates that a small set of points can
provide more structural information about objects, while overmuch points may
make the model confused to differentiate different objects.

Combination | $\mathcal{J}\&\mathcal{F}$ |  $\mathcal{J}$-$Mean$ |  $\mathcal{F}$-$Mean$
---|---|---|---
box & 1 point | 87.7 | 84.0 | 91.1
box & 3 points | 88.1 | 84.4 | 91.8
box & 5 points | 88.3 | 84.6 | 91.9
box & 10 points | 87.8 | 84.1 | 91.4
Table 6: The performance comparison of different prompt combination for SAM on
DAVIS2017 valid set.

##  5 Limitations

Although UVOSAM has demonstrated outstanding performance under the mask-free
setting, there are two significant limitations that hinder its ability to
understand complex videos. Firstly, it struggles with detecting slender
objects, as depicted in Figure 4 (a). The bounding box for such objects, like
ropes, is consistently uncertain and inaccurate, causing SAM to receive
erroneous prompts. Additionally, regular objects may also appear slender due
to occlusion and rapid motion, exacerbating the issue. Secondly, there are
instances of trajectory drift or tracking disruption when objects obstruct
each other, or their scale changes (please refer to Figure 4 (b)). These
failure cases indicate that our VSOT has scope for improvement.

Figure 4: Visualisation of the failed cases.

##  6 Conclusion

In this paper, to tackle the challenge of label-expensive for Unsupervised
VOS, we propose a simple yet effective framework called UVOSAM. UVOSAM is a
two-stage paradigm that first leverages a video salient object tracking model
to automatically discovery the main objects and generate the trajectories. It
then employs SAM to sequentially generate the segmentation results with the
prompting of trajectories. We evaluate our approach on two popular video
segmentation datasets and the results show that UVOSAM outperforms the state-
of-the-art mask-supervised methods. We hope that our work will inspire future
research on label-efficient Unsupervised VOS by prompting the vision
fundamental models.

## References

  * [1] Araslanov, N., Schaub-Meyer, S., Roth, S.: Dense unsupervised learning for video segmentation. Advances in Neural Information Processing Systems 34, 25308–25319 (2021)
  * [2] Athar, A., Mahadevan, S., Osep, A., Leal-Taixé, L., Leibe, B.: Stem-seg: Spatio-temporal embeddings for instance segmentation in videos. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16. pp. 158–177. Springer (2020)
  * [3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [4] Chen, Y., Hao, C., Liu, A.X., Wu, E.: Appearance-consistent video object segmentation based on a multinomial event model. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15(2), 1–15 (2019)
  * [5] Chen, Y., Hao, C., Liu, A.X., Wu, E.: Multilevel model for video object segmentation based on supervision optimization. IEEE Transactions on Multimedia 21(8), 1934–1945 (2019)
  * [6] Cheng, B., Parkhi, O., Kirillov, A.: Pointly-supervised instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2617–2626 (2022)
  * [7] Cho, S., Lee, M., Lee, S., Park, C., Kim, D., Lee, S.: Treating motion as option to reduce motion dependency in unsupervised video object segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 5140–5149 (2023)
  * [8] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)
  * [9] Ding, H., Liu, C., He, S., Jiang, X., Torr, P.H., Bai, S.: Mose: A new dataset for video object segmentation in complex scenes. arXiv preprint arXiv:2302.01872 (2023)
  * [10] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T.: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [11] Fragkiadaki, K., Arbelaez, P., Felsen, P., Malik, J.: Learning to segment moving objects in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4083–4090 (2015)
  * [12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [13] Hwang, S., Heo, M., Oh, S.W., Kim, S.J.: Video instance segmentation using inter-frame communication transformers. Advances in Neural Information Processing Systems 34, 13352–13363 (2021)
  * [14] Ke, L., Danelljan, M., Ding, H., Tai, Y.W., Tang, C.K., Yu, F.: Mask-free video instance segmentation. arXiv preprint arXiv:2303.15904 (2023)
  * [15] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [16] Lee, M., Cho, S., Lee, S., Park, C., Lee, S.: Unsupervised video object segmentation via prototype memory network. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 5924–5934 (2023)
  * [17] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [18] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900. PMLR (2022)
  * [19] Lin, H., Wu, R., Liu, S., Lu, J., Jia, J.: Video instance segmentation with a propose-reduce paradigm. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1739–1748 (2021)
  * [20] Liu, D., Yu, D., Wang, C., Zhou, P.: F2net: Learning to focus on the foreground for unsupervised video object segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 2109–2117 (2021)
  * [21] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: 7th International Conference on Learning Representations, ICLR 2019 (2019)
  * [22] Luiten, J., Zulfikar, I.E., Leibe, B.: Unovost: Unsupervised offline video object segmentation and tracking. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 2000–2009 (2020)
  * [23] Maggiolino, G., Ahmad, A., Cao, J., Kitani, K.: Deep oc-sort: Multi-pedestrian tracking by adaptive re-identification. arXiv preprint arXiv:2302.11813 (2023)
  * [24] Miao, J., Wei, Y., Yang, Y.: Memory aggregation networks for efficient interactive video object segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10366–10375 (2020)
  * [25] Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Video object segmentation using space-time memory networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9226–9235 (2019)
  * [26] Pan, X., Li, P., Yang, Z., Zhou, H., Zhou, C., Yang, H., Zhou, J., Yang, Y.: In-n-out generative learning for dense unsupervised video segmentation. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 1819–1827 (2022)
  * [27] Park, K., Woo, S., Oh, S.W., Kweon, I.S., Lee, J.Y.: Per-clip video object segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1352–1361 (2022)
  * [28] Pei, G., Shen, F., Yao, Y., Xie, G.S., Tang, Z., Tang, J.: Hierarchical feature alignment network for unsupervised video object segmentation. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIV. pp. 596–613. Springer (2022)
  * [29] Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 724–732 (2016)
  * [30] Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 (2017)
  * [31] Qi, J., Gao, Y., Hu, Y., Wang, X., Liu, X., Bai, X., Belongie, S., Yuille, A., Torr, P.H., Bai, S.: Occluded video instance segmentation: A benchmark. International Journal of Computer Vision 130(8), 2022–2039 (2022)
  * [32] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [33] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)
  * [34] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [35] Ren, S., Liu, W., Liu, Y., Chen, H., Han, G., He, S.: Reciprocal transformations for unsupervised video object segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15455–15464 (2021)
  * [36] Seong, H., Hyun, J., Kim, E.: Kernelized memory network for video object segmentation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16. pp. 629–645. Springer (2020)
  * [37] Tokmakov, P., Alahari, K., Schmid, C.: Learning motion patterns in videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3386–3394 (2017)
  * [38] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [39] Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., Huang, T.: Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 (2023)
  * [40] Wang, Z., Li, Y., Chen, X., Lim, S.N., Torralba, A., Zhao, H., Wang, S.: Detecting everything in the open world: Towards universal object detection. arXiv preprint arXiv:2303.11749 (2023)
  * [41] Wu, J., Jiang, Y., Bai, S., Zhang, W., Bai, X.: Seqformer: Sequential transformer for video instance segmentation. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII. pp. 553–569. Springer (2022)
  * [42] Wu, J., Liu, Q., Jiang, Y., Bai, S., Yuille, A., Bai, X.: In defense of online models for video instance segmentation. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII. pp. 588–605. Springer (2022)
  * [43] Xie, H., Yao, H., Zhou, S., Zhang, S., Sun, W.: Efficient regional memory network for video object segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1286–1295 (2021)
  * [44] Xu, N., Yang, L., Fan, Y., Yue, D., Liang, Y., Yang, J., Huang, T.Y.V.: A large-scale video object segmentation benchmark. arXiv preprint (2018)
  * [45] Yang, C., Lamdouar, H., Lu, E., Zisserman, A., Xie, W.: Self-supervised video object segmentation by motion grouping. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7177–7188 (2021)
  * [46] Yang, J., Gao, M., Li, Z., Gao, S., Wang, F., Zheng, F.: Track anything: Segment anything meets videos. arXiv preprint arXiv:2304.11968 (2023)
  * [47] Yang, L., Fan, Y., Xu, N.: Video instance segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5188–5197 (2019)
  * [48] Yang, S., Zhang, L., Qi, J., Lu, H., Wang, S., Zhang, X.: Learning motion-appearance co-attention for zero-shot video object segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1564–1573 (2021)
  * [49] Yang, Y., Loquercio, A., Scaramuzza, D., Soatto, S.: Unsupervised moving object detection via contextual information separation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 879–888 (2019)
  * [50] Yang, Z., Wei, Y., Yang, Y.: Collaborative video object segmentation by foreground-background integration. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V. pp. 332–348. Springer (2020)
  * [51] Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al.: Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 (2022)
  * [52] Zhang, K., Zhao, Z., Liu, D., Liu, Q., Liu, B.: Deep transport network for unsupervised video object segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8781–8790 (2021)
  * [53] Zhang, R., Jiang, Z., Guo, Z., Yan, S., Pan, J., Dong, H., Gao, P., Li, H.: Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048 (2023)
  * [54] Zhou, T., Li, J., Li, X., Shao, L.: Target-aware object discovery and association for unsupervised video multi-object segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6985–6994 (2021)
  * [55] Zhou, T., Porikli, F., Crandall, D.J., Van Gool, L., Wang, W.: A survey on deep learning technique for video segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [56] Zhou, T., Wang, S., Zhou, Y., Yao, Y., Li, J., Shao, L.: Motion-attentive transition for zero-shot video object segmentation. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 13066–13073 (2020)
  * [57] Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718 (2023)