#  NOVIS: A Case for End-to-End Near-Online
Video Instance Segmentation

Tim Meinhardt  Matt Feiszli  Yuchen Fan  Laura Leal-Taixé  Rakesh Ranjan

###### Abstract

Until recently, the Video Instance Segmentation (VIS) community operated under
the common belief that offline methods are generally superior to a frame by
frame online processing. However, the recent success of online methods
questions this belief, in particular, for challenging and long video
sequences. We understand this work as a rebuttal of those recent observations
and an appeal to the community to focus on dedicated near-online VIS
approaches. To support our argument, we present a detailed analysis on
different processing paradigms and the new end-to-end trainable NOVIS (Near-
Online Video Instance Segmentation) method. Our transformer-based model
directly predicts spatio-temporal mask volumes for clips of frames and
performs instance tracking between clips via overlap embeddings. NOVIS
represents the first near-online VIS approach which avoids any handcrafted
tracking heuristics. We outperform all existing VIS methods by large margins
and provide new state-of-the-art results on both YouTube-VIS (2019/2021) and
the OVIS benchmarks.

video object segmentation, near-online, transformers

##  1 Introduction

The computer vision community directs much of its attention to building robust
video and scene understanding models. Teaching machines to mimic or even
surpass human capabilities opens vast possibilities for applications in
robotics and automation such as autonomous driving. This work focuses on the
Video Instance Segmentation (VIS) task, which performs pixel level
segmentation and tracking of object instances within a given RGB video input.

(a) Offline frame generalization

(b) Near-online processing

(c) NOVIS with overlap embeddings

Figure 1:  A frame processing analysis and comparison of our near-online NOVIS
approach with comparable online and offline baselines. We evaluate on a 2-fold
training and the official validation split of OVIS. Figure (a) illustrates the
training to test frame generalization gap of offline methods and the
superiority of online methods recently observed by the VIS community. The
near-online variants shown in (b) do not suffer from the gap and outperform
online baseline. In Figure (c), we demonstrate how NOVIS outperforms online
and offline approaches and benefits from computing _overlap embeddings_.

Since its introduction in (Yang et al., 2019), the general consensus has been
that the complexity of the VIS task justifies offline processing of a given
video sequence. Offline methods (Athar et al., 2020; Bertasius & Torresani,
2020; Lin et al., 2021; Li et al., 2021; Liu et al., 2021a; Wu et al., 2022a;
Hwang et al., 2021; Heo et al., 2022; Cheng et al., 2021) process the entire
sequence at once and are hence expected to provide more consistent track
management and mask quality over time. This is usually achieved by training
models end-to-end on multiple frames at once. However, the training as well as
inference comes with considerable computational costs, which scale with the
sequence length and often render an offline processing of long sequences
infeasible. More recently, online (Wu et al., 2022b; Huang et al., 2022)
approaches, which segment and track instances on a frame-by-frame basis, have
shown great potential, even surpassing existing offline methods. These methods
usually exhibit a smaller computational footprint which allows for the
application on real-time data streams. However, the limited temporal
information (predictions must not include future information) makes occlusion
handling and consistent mask predictions far more challenging. To mitigate
these issues, most online methods rely on handcrafted tracking heuristic.
Near-online methods (Bertasius & Torresani, 2020; Athar et al., 2020; Caelles
et al., 2022; Hwang et al., 2021), also referred to as sliding window, process
clips of frames and use heuristic instance tracking between clips. They
combine the best of the online and offline paradigms, while avoiding their
respective downsides. Processing multiple frames at once improves mask quality
and track management, and clip overlaps facilitate the clip-to-clip tracking
substantially.

The flawed case against (existing) offline methods. The original VIS benchmark
only focused on short sequences which allowed offline methods to excel. More
recent benchmarks such as OVIS (Qi et al., 2022) include very long sequences
that most offline methods are not able to process. This limitation alone could
explain the recent resurgence of online approaches for VIS. However, the
analysis of the IDOL (Wu et al., 2022b) paper goes a step further by
questioning the general efficiency of existing offline methods. In a series of
oracle experiments, they show diminishing performance with increasing sequence
length for offline as well as online processing. However, the performance drop
for offline methods (Wu et al., 2022a; Hwang et al., 2021) is larger than for
comparable online approaches. This observation runs against the general
intuition that processing all frames at once yields more robust mask
predictions. The IDOL analysis and benchmark results presented in (Wu et al.,
2022b; Huang et al., 2022) represent a broader argument against the efficiency
of existing offline methods advocating for a frame-to-frame online processing.
In this work, we draw a different conclusion by identifying the true reason
for their suboptimal performance: The generalization gap between the number of
train and test frames.

Why near-online is the future of VIS. In Figure 1(a), we evaluate two
comparable offline (Cheng et al., 2021) and online (Huang et al., 2022) VIS
approaches, both based on the (Cheng et al., 2022) architecture. In contrast
to IDOL, we do not perform oracle experiments but first split the OVIS
training sequences into a train and validation set and then split each
validation sequence into sub-sequences of length $\mathcal{T}$. While the
original version of Mask2Former for VIS (Cheng et al., 2021) is only trained
on two frames, we train additional versions on 4 and 8 frames. The superiority
of the online MinVIS approach for longer sequences agrees with the findings of
IDOL. However, for sequences similar in length to the number of offline
training frames this gap closes, _e.g_., $\mathcal{T}=8$ and Mask2Former
trained on 8 frames. Furthermore, our results show that training on more
frames improves performance on longer sequences. The experiments of IDOL did
not consider the generalization gap between the number of training and test
frames. More specifically, their offline candidates (Wu et al., 2022a; Hwang
et al., 2021) are trained on 3 and 5 frames, respectively, but expected to
generalize to evaluations with up to 36 frames. Therefore, instead of
renouncing offline methods entirely, we advocate for models that predict
multiple frames at once but avoid the frame generalization gap via near-online
processing. In Figure 1(b), we show comparable near-online versions. Their
saturation/drop for increasing clip size coincides with observations of
previous near-online (Athar et al., 2020; Bertasius & Torresani, 2020)
approaches. However, for longer sequences ($\mathcal{T}>4$) all presented
near-online versions perform as well or better than the online counterpart.

To support our argument, we present the transformer-based _Near-Online Video
Instance Segmentation_ (NOVIS) method. Our approach combines offline and
online aspects by performing _spatio-temporal instance segmentation_ and
tracking with _overlap embeddings_ jointly in an end-to-end trainable model.
Given a clip of frames, our model directly predicts spatio-temporal mask
volumes and their corresponding object class. To this end, we compute cross-
attention between instance queries and the entire three dimensional feature
volume. The model design allows us to bridge short-term object occlusions
within a mask volume. To avoid expensive and handcrafted tracking heuristics,
we match instances in adjacent clips via cosine similarity of query embeddings
(Huang et al., 2022; Wu et al., 2022b). Figure 1(c) illustrates the relation
between clip length and stride/overlap. For long clips with small overlap,
adjacent instance queries will not match if an object trajectory changes
significantly. To avoid large and computationally expensive overlaps, we
compute overlap embeddings via cross-attention between each instance query and
the features of the clip overlap.

NOVIS is the first end-to-end near-online method that relies not on explicit,
handcrafted tracking heuristics like (Hwang et al., 2021; Caelles et al.,
2022) and outperforms comparable online (MinVIS) and offline (IDOL, VITA)
approaches by a large margin. We surpass previous methods by +3.0 and +3.7
points on the YouTube-VIS 2019/2021 benchmarks. For the challenging OVIS
dataset, we obtain a new state-of-the-art with an improvement of +2.5 points.

##  2 Related work

We differentiate Video Instance Segmentation (VIS) approaches by their
respective frame processing paradigms:

Offline. Since its inception, the VIS community focused on offline approaches
which promise superior track consistency and mask quality over time by
processing a given sequence all at once. The SeqMask-RCNN (Lin et al., 2021)
generates sequence-wide instance proposals based on multiple key frames. A
post-processing removes redundant proposals via intersection over union (IoU).
The authors of (Li et al., 2021) improve on single-frame processing via a
temporal fusion of adjacent frames. Initiated by VisTR (Wang et al., 2021),
the community has shifted towards transformer-based methods. VisTR extends the
DETR (Carion et al., 2020) object detector to the temporal domain computing
attention between all pixels in a spatio-temporal feature volume. The IFC
(Hwang et al., 2021) method reduces expensive attention computations via
inter-frame tokens and (Wu et al., 2022a; Heo et al., 2022; Cheng et al.,
2021) share temporal information on the object/instance query level. In
particular, VITA (Heo et al., 2022) decouples the per-frame segmentation and
instance association entirely which allows them to run very long sequences.
The TeViT (Yang et al., 2022) approach benefits from an early fusion of
temporal information via messenger shift tokens in the backbone.

Except from (Cheng et al., 2021), existing offline methods share temporal
information across multiple frames but refrain from treating VIS as a single
spatio-temporal mask prediction problem. For example, VisTR computes spatio-
temporal attention but relies on a single image mask prediction head.
Furthermore, the high memory costs limit the number of training frames which
results in _diminishing test-time performance for increasing sequence length_.
Running inference on very _long sequences is infeasible_ for most offline
methods. Our near-online model incorporates an end-to-end trainable temporal
information exchange and directly predicts spatio-temporal VIS masks without
suffering from the aforementioned memory limitations.

Online. A frame-to-frame online processing is essential for many real-time
applications. During evaluation, these methods must generate tracks on the
current frame without including any future information. Early VIS methods
(Yang et al., 2019; Cao et al., 2020; Liu et al., 2021a) generate and connect
per-frame mask predictions with a tracking head. The additional head computes
instance embeddings and matches them to a memory/buffer of previous
embeddings. The authors of (Yang et al., 2021) and (Fu et al., 2021) improve
on the embedding matching with a crossover learning scheme and feature
aggregation via attention, respectively. The VISOLO (Han et al., 2022) method
propagates information stored in a track memory via grid-based space-time
aggregation. Generally, online methods are expected to be inferior to an
offline processing of the entire sequence at once. However, recently IDOL (Wu
et al., 2022b) and MinVIS (Huang et al., 2022) surpassed all previous online
and offline methods by segmenting and tracking objects via Transformer
instance queries. IDOL further improves query matching with a contrastive loss
between frames.

It is common for online methods to rely on a _heuristic track memory/buffer_.
These buffers recover tracks from shot-term occlusions and reduce the overall
noise between frames. We interpret our near-online processing with clip
overlap as an end-to-end trainable version of such a track memory.

Near-online. Near-online methods process clips of multiple frames in a sliding
window fashion and connect clips via instance tracking. A significant overlap
between clips allows for a more consistent tracking generally superior to
online methods. It is common for online methods to track instances via
embedding distance. Existing near-online approaches, on the other hand, rely
on handcrafted heuristics, _e.g_., mask IoU or object class. Adding such an
instance tracking enables near-online processing with any offline model, see
OVIS benchmark results of (Hwang et al., 2021; Wu et al., 2022a; Cheng et al.,
2021; Yang et al., 2022). _However, the VIS community is missing specifically
designed end-to-end trainable near-online approaches._ In other words,
existing near-online methods merely combine offline processing with a
heuristic instance tracking. MaskProp (Bertasius & Torresani, 2020) extends
(Yang et al., 2019) with a clip-wide mask propagation branch. The authors of
STEm-Seg (Athar et al., 2020) predict pixel embeddings with Gaussian variances
to model object instances in spatio-temporal volumes. The DeVIS (Caelles et
al., 2022) approach presents an efficient version of VisTR by computing
deformable attention.

We deem our work as the first near-online VIS method which avoids handcrafted
heuristics by performing instance tracking via end-to-end trainable
embeddings. To this end, we compute overlap embeddings via cross attention
between instance queries and clip overlaps.

##  3 Problem statement

Figure 2:  Overview of our near-online NOVIS model architecture and clip
processing pipeline. Given a video with length $\mathcal{T}$ and clips with
overlap $\tau=[t+S,\dots,t+T]$, our model first computes $F$ per-frame feature
scales via deformable attention in the _pixel decoder_. The feature scales of
a clip are concatenated ($\boldsymbol{\oplus}$) to spatio-temporal volumes
($TH_{f}W_{f}$). Our _video instance decoder_ computes cross-attention between
instance queries and feature volumes over $L$ decoding layer. The resulting
output embeddings directly compute the object class and segmentation mask via
a linear layer and dot product ($\boldsymbol{\otimes}$) with the $F$-th
feature scale, respectively. We track instances between two clips by matching
output embeddings via cosine similarity. To improve results for objects with
large trajectory changes, we match overlap embeddings computed on the shared
frame features ($\tau H_{F-1}W_{F-1}$).

Given a video sequence $\mathbf{X}\in\mathbb{R}^{\mathcal{T}\times H\times
W\times 3}$ of arbitrary length $\mathcal{T}$ consisting of RGB color frames,
the Video Instance Segmentation (VIS) task requires the segmentation and
temporal consistent tracking of object instances. In contrast to VOS (Xu et
al., 2018), the number of object instances within a sequence is unknown. A
single instance prediction $i$ is defined by its object class
$c_{i}\in\\{1,\dots,K\\}$ and spatio-temporal foreground-background mask
$\mathbf{Y}_{i}\in\\{0,1\\}^{\mathcal{T}\times H\times W}$. The VIS task does
not evaluate any background/stuff classes. The instance mask $\mathbf{Y}_{i}$
includes occlusions and measures object identity preservation via Intersection
over Union (IoU) to other instance masks.

##  4 NOVIS

To process a given video sequence most VIS methods either follow an online or
offline approach. Online methods predict object instances one frame at a time
and are hence prone to suffer from temporally inconsistent mask and object
identities. Offline methods mitigate these issues by processing the entire
video at once but the associated computational costs limit their applicability
to long sequences.

Near-online methods, on the other hand, avoid the respective downsides by
processing a sequence in clips of frames. To demonstrate their superiority, we
present _Near-Online Video Instance Segmentation_ (NOVIS), an end-to-end
trainable near-online model free from any handcrafted tracking heuristics.
This is achieved by combining offline and online aspects into two processing
steps: (i) spatio-temporal instance segmentation, and (ii) instance tracking
between clips with overlap embeddings.

###  4.1 Spatio-temporal instance segmentation

Given an input clip of size $T\leq\mathcal{T}$, our model directly predicts
spatio-temporal instance masks and their object class. As previous VIS methods
(Huang et al., 2022; Heo et al., 2022; Cheng et al., 2021), we build upon the
all-purpose mask prediction model introduced in (Cheng et al., 2022).
Following the recent trend of image detection and instance segmentation
approaches, (Cheng et al., 2022) segments objects via Transformer (Vaswani et
al., 2017) self- and cross-attention between queries and image features.

Similar to (Cheng et al., 2021), we extend this concept to the temporal domain
by segmenting three-dimensional object tubes within a clip
$\mathbf{X}_{t:t+T}\in\mathbb{R}^{T\times H\times W\times 3}$ of adjacent
input frames. The backbone and pixel decoder, illustrated in Figure 2, extract
object features per frame and on multiple feature scales. To this end, the
pixel decoder computes deformable attention (Zhu et al., 2021) between pixels
and across $F$ feature scales, _e.g_., $1/32$, $1/16$, $1/8$ and $1/4$ for a
ResNet-50 (He et al., 2016) backbone. Given the extracted features, it is the
responsibility of the _video instance decoder_ to segment individual object
trajectories within the clip.

Video instance decoder. Most existing Transformer-based offline VIS approaches
(Wu et al., 2022a; Wang et al., 2021; Hwang et al., 2021; Heo et al., 2022)
segment objects over time by temporally aligning sets of _per-frame_ instance
queries. In NOVIS, we advocate for a more holistic temporal approach by
directly predicting three dimensional object masks with a single set of _video
instance queries_. To this end, we compute cross-attention between a set of
$N$ queries $\mathbf{Q}\in\mathbb{R}^{N\times C}$ with hidden dimension $C$
and entire spatio-temporal feature volumes with a (flattened) dimension of
$TH_{f}W_{f}$. More specifically, our instance decoder consists of $L$
decoding layers alternating between instance query self-attention and cross-
attention with multi-frame features. In addition to the common spatial
encoding, we add a temporal encoding to each feature volume. This allows
queries to discriminate between frames and build temporally consistent three-
dimensional object tubes. By iterating over the $F$ feature scales, a single
instance query computes cross-attention $L/F$ times with each scale. Following
(Cheng et al., 2022), we apply masked cross-attention based on the predictions
of each previous layer. The spatio-temporal attention for layer $l$ is masked
via the re-scaled binary mask prediction
$\mathbf{M}_{l-1}\in\\{0,1\\}^{N\times TH_{l-1}W_{l-1}}$ of the previous
($l-1$)-th instance decoder layer. Ignoring the zero entries of
$\mathbf{M}_{l-1}$ alleviates the foreground-background pixel imbalance
allowing the model to focus on the object instances.

The resulting instance output embeddings $\mathbf{Q}_{L}\in\mathbb{R}^{N\times
C}$ directly predict the necessary VIS task outputs $c_{i}$ and
$\mathbf{Y}_{i}$ without relying on any additional post-processing. The class
predictions $c_{i}$ are obtained by projecting each embedding into $K+1$
dimensional class space. The additional background class allows the model to
predict an unknown number of objects within a clip. We obtain the foreground-
background mask logits by computing the dot-product between $\mathbf{Q}_{L}$
and the $F$-th feature volume. For more details on the underlying segmentation
architecture, we refer to (Cheng et al., 2022).

Class and multi-frame segmentation losses. The loss for a given clip of frames
consists of a cross-entropy class $\mathcal{L}_{class}$ and two segmentation
losses, namely, a binary mask cross-entropy $\mathcal{L}_{mask}$ and dice
$\mathcal{L}_{dice}$ loss. To compute each loss, we match our model
predictions to the ground truth objects present in (at least one frame of) the
given clip. Previous methods (Cheng et al., 2021; Wu et al., 2022a; Hwang et
al., 2021; Wang et al., 2021; Heo et al., 2022) compute both segmentation
losses per-frame. We found a volumetric version of $\mathcal{L}_{dice}$, which
computes a single loss for the entire clip, to be superior. However, a
volumetric version of the per-pixel $\mathcal{L}_{mask}$ loss suffers from a
foreground-background pixel imbalance for objects with several occluded
frames. Hence, we compute the mask loss separately for each frame.

In contrast to MinVIS and VITA, our multi-frame segmentation loss includes
fully occluded frames. This allows our model to predict object occlusions as
missing part of the three dimensional mask and perform short-term re-
identification for windows smaller than $T$. In other words, our model
performs like an end-to-end trainable track buffer/memory commonly applied by
online methods (Yang et al., 2019; Cao et al., 2020; Liu et al., 2021a; Wu et
al., 2022b; Han et al., 2022).

###  4.2 Instance tracking with embeddings

The number of training and inference frames within a clip is limited by the
computational resources. As discussed in Figure 1 for offline methods, clip
models yield suboptimal performance for sequences significantly longer
($\mathcal{T}\gg T$) than the number of frames it was trained on. This
generalization issue is comparable to training and testing on vastly different
image input resolutions.

Hence, we advocate for a near-online instance tracking in a sliding window
fashion with stride $1\leq S\leq T$. For $S=1$ the method operates in a pure
online fashion. Averaging mask predictions for overlapping frames further
boosts the temporal mask consistency and overall quality. Finding the optimal
clip length $T$ to stride $S$ ratio requires a balancing of two near-online
processing effects. On the one hand, a smaller stride, _i.e_., larger overlap
between two adjacent clips, improves and stabilizes instance tracking and mask
quality. On the other, a near-online processing with minimum overlap larger
than zero ($S=T-1$) increases the potential re-identification window between
two clips up to $2T-1$. In our experiments, we will show ablations on optimal
clip length to stride ratio.

Existing near-online VIS approaches rely on handcrafted tracking heuristics,
_e.g_., mask IoU or object class, to connect instances between clips. However,
for online VIS approaches (Huang et al., 2022; Wu et al., 2022b), a cosine
distance matching between decoder queries of adjacent frames has shown
superior results to the common heuristics. Our video instance decoder allows
us to apply a similar matching between instance queries of two adjacent clips.
However, for objects with large visual or trajectory changes and clips with
small overlap the risk of matching wrong instance queries increases. Figure 3
shows that in particular long clips, _e.g_., for $T=6$ on OVIS (Qi et al.,
2022), suffer from this effect. To minimize the necessary overlap and maximize
the benefits from potential shot-term re-identifications between two clips, we
compute _overlap embeddings_ specifically designed for a near-online instance
tracking.

###  4.3 Clip overlap embeddings

In Figure 2, we illustrate the integration of overlap embeddings into the
overall model architecture. At its core, the clip-to-clip tracking is
performed via cosine similatity but we do not directly match output embeddings
$\mathbf{Q}^{t}_{L}=\mathbf{Q}^{t:t+T}_{L}\in\mathbb{R}^{N\times C}$ and
$\mathbf{Q}^{t+S}_{L}=\mathbf{Q}^{t+S:t+S+T}_{L}\in\mathbb{R}^{N\times C}$ of
two adjacent clips with length $T$. Instead, we compute the representation of
each embedding with respect to the overlapping frames $\tau=[t+S,\dots,t+T]$
at the start and end of their respective clip and perform inter-clip matching
only based on these shared frames. Performing near-online instance tracking
only based on overlapping frames is a common practice for heuristic approaches
and often achieved by computing a volumetric IoU (Bertasius & Torresani, 2020;
Athar et al., 2020; Caelles et al., 2022).

To obtain the instance representation $\mathbf{Q}^{t\rightarrow\tau}_{L}$ of
the subset of frames $\tau$ of the clip starting at $t$, we compute cross-
attention between each output embedding and the respective frame features.
More specifically, we re-compute the last instance decoder layer $L$ with the
following attention mask for the $n$-th query:

$\displaystyle\mathbf{\hat{M}}_{L}(n,t,x,y)=\left\\{\begin{array}[]{ll}\mathbf{M}_{L}(n,t,x,y)&\text{if~{}}t\in\tau\\\
0&\text{otherwise}\end{array}\right..$ (3)

If a query predicts no masks within the overlap, _e.g_., due to an object
occlusion, the equation above results in
$\mathbf{\hat{M}}_{L}(n,t)=0~{}\forall~{}t$. In that case, we refrain from
computing the overlap attention and use the original output embedding for an
instance matching based on the entire clip.

The video tracking module allows for a computation of instance representations
on the edges of each clip more suitable for embedding-based matching.
Exploiting the clip overlap allows us to avoid additional contrastive learning
between embeddings as done in IDOL or VITA.

Method | Data | $T$ / $S$ | YouTube-VIS 2019 (Yang et al., 2019) | YouTube-VIS 2021 (Yang et al., 2019) |
---|---|---|---|---|---
FPS | AP | AP50 | AP75 | AR1 | AR10 | AP | AP50 | AP75 | AR1 | AR10 |
ResNet-50 | MaskTrack (Yang et al., 2019) | VIS | ON | 26.1 | 30.3 | 51.1 | 32.6 | 31.0 | 35.5 | 28.6 | 48.9 | 29.6 | 26.5 | 33.8 |
VisTR (Wang et al., 2021) | VIS | OFF | 69.9 | 36.2 | 59.8 | 36.9 | 37.2 | 42.4 | – | – | – | – | – |
IFC (Hwang et al., 2021) | VIS | 5 / 1 | 46.5 | 39.0 | 60.4 | 42.7 | 41.7 | 51.6 | – | – | – | – | – |
IFC (Hwang et al., 2021) | VIS | OFF | 107.1 | 41.2 | 65.1 | 44.6 | 42.3 | 49.6 | 35.2 | 57.2 | 37.5 | – | – |
DeVIS (Caelles et al., 2022) | VIS | 6 / 4 | 18.4 | 44.4 | 66.7 | 48.6 | 42.4 | 51.6 | 43.1 | 66.8 | 46.6 | 38.0 | 50.1 |
SeqFormer (Wu et al., 2022a) | VIS | OFF | 72.3 | 45.1 | 66.9 | 50.5 | 45.6 | 54.6 | – | – | – | – | – |
IDOL (Wu et al., 2022b) | VIS | ON | 30.6 | 46.4 | 70.7 | 51.9 | 44.8 | 54.9 | 43.9 | 68.0 | 49.6 | 38.0 | 50.9 |
Mask2Former (Cheng et al., 2021) | VIS | OFF | – | 46.4 | 68.0 | 50.5 | – | – | 40.6 | 60.9 | 41.8 | – | – |
TeViT (Yang et al., 2022) | VIS | OFF | – | 46.6 | 71.3 | 51.6 | 44.9 | 54.3 | 37.9 | 61.2 | 42.1 | 35.1 | 44.6 |
SeqFormer (Wu et al., 2022a) | CC+VIS | OFF | 72.3 | 47.4 | 69.8 | 51.8 | 45.5 | 54.8 | 40.5 | 62.4 | 43.7 | 36.1 | 48.1 |
MinVIS (Huang et al., 2022) | VIS | ON | - | 47.4 | 69.0 | 52.1 | 45.7 | 55.7 | 44.2 | 66.0 | 48.1 | 39.2 | 51.7 |
IDOL (Wu et al., 2022b) | CC+VIS | ON | 30.6 | 49.5 | 74.0 | 52.9 | 47.7 | 58.7 | – | – | – | – | – |
VITA (Heo et al., 2022) | CC+VIS | OFF | – | 49.8 | 72.6 | 54.5 | 49.4 | 61.0 | 45.7 | 67.4 | 49.5 | 40.9 | 53.6 |
NOVIS | VIS | 4 / 2 | 19.9 | 51.4 | 74.9 | 55.2 | 48.7 | 60.1 | 47.2 | 69.4 | 50.0 | 41.3 | 54.4 |
CC+VIS | 4 / 2 | 19.9 | 52.8 | 75.7 | 56.9 | 50.3 | 60.6 | – | – | – | – | – |
Swin-L | SeqFormer (Wu et al., 2022a) | VIS | OFF | 27.7 | – | – | – | – | – | 51.8 | 74.6 | 58.2 | 42.8 | 58.1 |
DeVIS (Caelles et al., 2022) | VIS | 6 / 4 | – | 57.1 | 80.8 | 66.3 | 50.8 | 61.0 | 54.4 | 77.7 | 59.8 | 43.8 | 57.8 |
SeqFormer (Wu et al., 2022a) | CC+VIS | OFF | 27.7 | 59.3 | 82.1 | 66.4 | 51.7 | 64.4 | – | – | – | – | – |
IDOL (Wu et al., 2022b) | VIS | ON | 17.6 | 61.5 | 84.2 | 69.3 | 53.3 | 65.6 | 56.1 | 80.8 | 63.5 | 45.0 | 60.1 |
VITA (Heo et al., 2022) | CC+VIS | OFF | – | 63.0 | 86.9 | 67.9 | 56.3 | 68.1 | – | – | – | – | – |
IDOL (Wu et al., 2022b) | CC+VIS | ON | 17.6 | 64.3 | 87.5 | 71.0 | 55.6 | 69.1 | – | – | – | – | – |
NOVIS | VIS | 4 / 2 | 10.0 | 63.8 | 86.6 | 69.3 | 55.9 | 68.7 | 59.8 | 82.0 | 66.5 | 47.9 | 64.4 |
CC+VIS | 4 / 2 | 10.0 | 65.7 | 87.8 | 72.2 | 56.3 | 70.3 | – | – | – | – | – |

Table 1:  Benchmark results on the YouTube-VIS 2019 and 2021 validation sets.
We sort results by the YouTube-VIS 2019 average precision (AP) and distinguish
methods between online (ON), offline (OFF) and near-online (clip length $T$
and stride $S$). Additional training data by simulating clips via COCO image
augmentations is denoted with CC+VIS.

##  5 Experiments

###  5.1 Benchmarks

YouTube-VIS 2019 and 2021. The YouTube-VIS (Yang et al., 2019) datasets
provide instance ground truth masks for 40 object categories on a total of
2883 and 3859 sequences, respectively. Although less severe for the 2021
version, the YouTube-VIS benchmark only includes comparatively short videos
(average of 27.4 / 39.7 frames for the validation sets). Furthermore, both
versions provide comparatively few object instances (average of 1.69 / 2.10
per video) and include little to no object occlusions and movement. In
particular the short sequences allowed offline approaches to flourish and
dominate the VIS community.

OVIS. The Occluded VIS (OVIS) (Qi et al., 2022) benchmark follows the same
data structure as (Yang et al., 2019) but includes significantly longer
sequences (average of 62.7 for the validation set) crowded with more object
instances (average of 5.80 per video). The benchmarks labels 25 object
categories and currently represents the most challenging public VIS benchmark.
To test the limits of our NOVIS approach and near-online processing in
general, we perform ablations on (Qi et al., 2022).

Metrics. Following (Yang et al., 2019), all VIS benchmarks evaluate average
precision (AP) and recall (AR). The ground truth matching is performed via
volumetric IoU matching with varying thresholds (0.5 and 0.75).

###  5.2 Implementation and training details

Our NOVIS method builds on top of the Mask2Former (Cheng et al., 2022) image
segmentation model. If not otherwise specified, we follow their hyperparameter
settings. Each model is pre-trained on COCO (Lin et al., 2014) as in (Cheng et
al., 2022) and then fine-tuned on the respective VIS dataset. For YouTube-VIS
2019 it is common to additionally report results from a joint COCO and VIS
(CC+VIS) training. To this end, we follow the augmentation pipeline of (Heo et
al., 2022) to simulate clips from COCO images. We train with a batch size of 8
distributed over 8 $\times$ 32 GB GPUs and apply random clip frame reversal,
horizontal flipping, cropping and rescaling (shortest edge between 228 and 512
pixels) augmentations. Following the VIS convention, the minimum shortest edge
size at test-time (MST) is 360 pixels. For OVIS, we also report results
augmented from 480 to 800 and evaluated with 720 pixels. We ablate with a
ResNet-50 (He et al., 2016) backbone on a 2-fold OVIS training set split. All
reported results are averaged over three runs with different random seeds. For
more details and benchmark specific hyperparameters, we refer to the
supplementary.

###  5.3 Benchmark results

Method |  |  $T$ / $S$ | MST |  | AP | AP50 | AP75 | AR1 | AR10
---|---|---|---|---|---|---|---|---|---
ResNet-50 | MaskTrack (Yang et al., 2019) |  | ON | - |  | 10.8 | 25.3 | 8.5 | 7.9 | 14.9
IFC (Wu et al., 2022a) |  | 10 / 1 | - |  | 13.1 | 27.8 | 11.6 | 9.4 | 23.9
SeqFormer (Wu et al., 2022a) |  | 10 / 1 | 720 |  | 15.1 | 31.9 | 13.8 | 10.4 | 27.1
Mask2Former (Cheng et al., 2021) |  | 30 / 30 | 360 |  | 17.3 | 37.3 | 15.1 | 10.5 | 23.5
TeViT (Yang et al., 2022) |  | 36 / - | 360 |  | 17.4 | 34.9 | 15.0 | 11.2 | 21.8
VITA (Heo et al., 2022) ∗∗ |  | OFF | 360 |  | 19.6 | 41.2 | 17.4 | 11.7 | 26.0
DeVIS (Caelles et al., 2022) |  | 6 / 4 | 360 |  | 23.8 | 48.0 | 20.8 | – | –
MinVIS (Huang et al., 2022) |  | ON | 360 |  | 25.0 | 45.5 | 24.0 | 13.9 | 29.7
IDOL (Wu et al., 2022b) |  | ON | 720 |  | 30.2 | 51.3 | 30.0 | 15.0 | 37.5
NOVIS |  | 4 / 2 | 360 |  | 30.8 | 54.4 | 31.0 | 15.2 | 35.3
| 4 / 2 | 720 |  | 32.7 | 56.2 | 32.6 | 15.7 | 37.1
Swin-L | Mask2Former (Cheng et al., 2021) |  | 30 / 30 | 360 |  | 25.8 | 46.5 | 24.4 | 13.7 | 32.2
VITA (Heo et al., 2022) ∗∗ |  | OFF | 360 |  | 27.7 | 51.9 | 24.9 | 14.9 | 33.0
DeVIS (Caelles et al., 2022) |  | 6 / 4 | 360 |  | 34.6 | 58.7 | 36.8 | – | –
MinVIS (Huang et al., 2022) |  | ON | 360 |  | 39.4 | 61.5 | 41.3 | 18.1 | 43.3
IDOL (Wu et al., 2022b) |  | ON | 720 |  | 42.6 | 65.7 | 45.2 | 17.9 | 49.6
NOVIS |  | 4 / 2 | 360 |  | 43.0 | 66.9 | 44.5 | 18.9 | 46.3
|  | 4 / 2 | 720 |  |  ∗43.5 | 68.3 | 43.8 | 19.4 | 46.9

Table 2:  Benchmark results on the OVIS validation set. We indicate the
minimum size at test-time (MST) of the shortest frame edge and distinguish
between online (ON), offline (OFF) and near-online (clip length $T$ and stride
$S$). Our model with ∗ was only evaluated but not trained on larger (720)
input resolutions. With ∗∗ we denote a joint training on simulated clips from
COCO. Mask2Former and SeqFormer/IFC results are from the MinVis and IDOL
papers.

YouTube-VIS 2019 and 2021. The presented NOVIS method achieves state-of-the-
art results on both YouTube-VIS (Yang et al., 2019) benchmarks surpassing all
previous methods by large margins. In Table 1, we present results for our best
performing near-online configuration trained only on YouTube-VIS or jointly
with COCO (CC+VIS). We differentiate between online (ON), offline (OFF) and
near-online with clip length $T$ and stride $S$. We outperform IDOL, the
previous best online method, by +4.5 and +3.7 points on the 2019 and 2021
benchmarks, respectively. To demonstrate the superiority of near-online
processing, we want to focus on the offline (VITA / Mask2Former) and online
(MinVIS) methods which also build on top of (Cheng et al., 2022). Most
notably, our approach outperforms both offline methods (+3.0 and +4.5 on
YouTube-VIS 2019) as they mitigate their computational limitations by either
avoiding any temporal pixel-level information exchange (VITA) or training on
far less frames (only 2) than required during test-time (Mask2Former). While
offline methods which are able to process all frames of short sequences at
once generally obtain higher runtimes, our NOVIS approach still achieves 19.9
FPS. The superiority of our approach also translates to the top-performance
evaluation setting with a Swin-L (Liu et al., 2021b) backbone.

OVIS. To further strengthen the argument for near-online approaches, we
present state-of-the-art results on the challenging OVIS (Qi et al., 2022)
benchmark in Table 2. For an increased train and test resolution (MST), we
outperform the runner up IDOL method by +2.5 points. The significantly longer
sequences and increased object occlusions present in OVIS only exacerbate the
aforementioned issues of existing offline methods. Hence, we outperform VITA
by +11.2 points. The second strongest near-online approach, DeVIS, suffers
from its heuristic instance tracking which, as we will show in our ablations,
is substantially inferior to our embedding-based tracking. However, the
increased input resolution of IDOL highlights a weakness that our approach
inherits from offline methods. For a Swin-L backbone and $T$$=$$4$ our method
exceeds the computational limits during training. Hence, we present a version
only evaluated on the higher input resolution.

Figure 3:  Analysis on clip length and stride/overlap evaluated on training
set splits of YouTube-VIS 2019 and OVIS. Instance tracking between clips is
performed via cosine distance matching. Computing cross-attention between
queries and clip overlaps improves performance for larger clip sizes.

###  5.4 Ablation studies

In order to predict instance segmentations over time, our NOVIS method treats
individual frames as a single spatio-temporal mask volume. Our baseline with
$20.9$ AP in Table 3 represents a naive application of an image segmentation
method (Cheng et al., 2022) to such a pixel volume. The heuristic tracking
incorporates multiple cues based on volumetric IoU and object class. Adding a
temporal encoding allows instance queries to discriminate between frames and
boosts results by $+0.8$. For objects occluded in several frames, the
volumetric mask loss exacerbates the foreground-background pixel imbalance.
Hence, our final model only computes volumetric versions of the dice loss.
Replacing the handcrafted and expensive heuristics with embedding matching
allows for an implicit multi-cue tracking improving results by $+1.1$ points.
We evaluate results with comparatively large clip overlaps. Hence, a
computation of overlap embeddings yields only $+0.3$ points. For an in-depth
analysis of the relation between clip length, stride and overlap, we refer to
the next section.

| volumetric loss |  |  |  |
---|---|---|---|---|---
TE | dice | mask | tracking | AP | AP75 | AR1
|  |  | heuristic | 20.9 | 19.7 | 12.0
$\times$ |  |  | heuristic | 21.7 | 21.4 | 12.2
$\times$ |  | $\times$ | heuristic | 21.4 | 21.1 | 12.0
$\times$ | $\times$ |  | heuristic | 21.9 | 21.4 | 12.3
$\times$ | $\times$ |  | embed | 23.0 | 23.2 | 12.8
$\times$ | $\times$ |  | overlap embed | 23.3 | 23.3 | 13.0

Table 3:  Ablation of our near-online  NOVIS method on a OVIS training set
split. We motivate several design decisions from a naive application of
Mask2Former without temporal encoding (TE) to our final model evaluated with
clip length $T$$=$$4$ and stride $S$$=$$2$.

###  5.5 Clip analysis

To further understand the benefits and limitations of our near-online
approach, we evaluate varying clip configurations for OVIS and YouTube-VIS
2019. The long sequences of OVIS render a direct comparison with offline
methods infeasible. Only (Heo et al., 2022) is able to process OVIS in a pure
offline fashion. However, their benchmark results in Table 2 are inferior to
MinVis which corresponds to the online variant _without_ track buffer in
Figure 3.

Online with track buffer. Adding a track buffer which matches embeddings of
the current frame with the average of previous frame embeddings improves the
vanilla online version significantly. This track buffer is comparable to what
existing online approaches (Wu et al., 2022b; Han et al., 2022) apply.
Interestingly, the optimal buffer length for OVIS coincides with our optimal
clip length $T=4$. Nevertheless, our near-online approach outperforms the
_online + buffer_ version on both benchmarks by large margins. This is due to
the buffer‘s inability to improve the object mask quality over time. Only a
(near-online) multi-frame processing provides improved track management _and_
more consistent mask predictions without relying on more complex, handcrafted
heuristics.

Overlap embeddings. For the more challenging OVIS dataset, the cross-attention
between instance embeddings and overlapping frames yields substantial
improvements. Large visual or trajectory changes between two clips limit the
matchability of their instance queries. Hence, we see the largest gains of up
to 2 points for long clip sizes with small overlap, _e.g_., $T$$=$$8$ and clip
overlap of 1.

Long clip sizes. Intuitively, we expect longer clip sizes $T$ to yield better
performance as this allows the model to bypass longer occlusions and increase
the temporal consistency of object masks. However, due to the increased object
movement and occlusions present in OVIS, we observe degrading performance for
clips longer than 4. To explain this, one has to understand that for single
image segmentation (Cheng et al., 2022) the notion of an object instance
emerges from the pixel decoder. The instance queries merely organize and rank
the provided instances. To predict spatio-temporal instances for challenging
object trajectories, the pixel decoder is therefore required to (efficiently)
compute temporal connections. We deem temporal deformable attention as in
(Caelles et al., 2022; Zhou et al., 2022) very promising but leave the
exploration of further improvements for near-online open for future research.

Nevertheless, for short clip sizes we see significant improvements over the
presented online variants. This indicates that a few frames are already enough
to capture the bulk of advantages expected from multi-frame (offline)
processing. We obtain the final near-online configuration ($T$$=$$4$ and
$S$$=$$2$) for our benchmark results by evaluating the top configurations of
Figure 3 on the OVIS validation set.

##  6 Conclusion

We have shown that the recent trend of online methods for VIS must not be
understood as an argument against the efficiency of existing offline methods.
In fact, the analysis provided in this work and our state-of-the-art benchmark
results make a strong case for multi-frame processing. Hence, we advocate for
near-online methods, which combine the best of online and offline approaches,
while avoiding their downsides. To this end, we presented NOVIS which directly
predicts spatio-temporal temporal mask volumes and performs tracking between
clips via overlap embeddings. Our approach represents the first end-to-end
trainable near-online method which avoids handcrafted tracking heuristics. We
hope this work fosters future research on dedicated near-online models and
marks a valuable rebuttal on the recent online vs. offline debate within the
VIS community.

## References

  * Athar et al. (2020) Athar, A., Mahadevan, S., Ošep, A., Leal-Taixé, L., and Leibe, B.  Stem-seg: Spatio-temporal embeddings for instance segmentation in videos.  In _Eur. Conf. Comp. Vis._ , 2020.
  * Bertasius & Torresani (2020) Bertasius, G. and Torresani, L.  Classifying, segmenting, and tracking object instances in video with mask propagation.  In _IEEE Conf. Comp. Vis. Pattern Recog._ , 2020.
  * Caelles et al. (2022) Caelles, A., Meinhardt, T., Brasó, G., and Leal-Taixé, L.  DeVIS: Making Deformable Transformers Work for Video Instance Segmentation.  _arXiv:2207.11103_ , 2022.
  * Cao et al. (2020) Cao, J., Anwer, R. M., Cholakkal, H., Khan, F. S., Pang, Y., and Shao, L.  Sipmask: Spatial information preservation for fast image and video instance segmentation.  In _Eur. Conf. Comp. Vis._ , 2020.
  * Carion et al. (2020) Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S.  End-to-end object detection with transformers.  In _Eur. Conf. Comp. Vis._ , 2020.
  * Cheng et al. (2021) Cheng, B., Choudhuri, A., Misra, I., Kirillov, A., Girdhar, R., and Schwing, A. G.  Mask2former for video instance segmentation.  _arXiv:2112.10764_ , 2021.
  * Cheng et al. (2022) Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Girdhar, R.  Masked-attention mask transformer for universal image segmentation.  In _IEEE Conf. Comp. Vis. Pattern Recog._ , 2022.
  * Fu et al. (2021) Fu, Y., Yang, L., Liu, D., Huang, T. S., and Shi, H.  Compfeat: Comprehensive feature aggregation for video instance segmentation.  In _AAAI_ , 2021.
  * Han et al. (2022) Han, S. H., Hwang, S., Oh, S. W., andHyunwoo Kim, Y. P., Kim, M., and Kim, S. J.  VISOLO: grid-based space-time aggregation for efficient online video instance segmentation.  In _IEEE Conf. Comp. Vis. Pattern Recog._ , 2022.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _IEEE Conf. Comp. Vis. Pattern Recog._ , 2016.
  * Heo et al. (2022) Heo, M., Hwang, S., Oh, S. W., Lee, J.-Y., and Kim, S. J.  Vita: Video instance segmentation via object token association.  In _Adv. Neural Inform. Process. Syst._ , 2022.
  * Huang et al. (2022) Huang, D.-A., Yu, Z., and Anandkumar, A.  Minvis: A minimal video instance segmentation framework without video-based training.  In _Adv. Neural Inform. Process. Syst._ , 2022.
  * Hwang et al. (2021) Hwang, S., Heo, M., Oh, S. W., and Kim, S. J.  Video instance segmentation using inter-frame communication transformers.  In _Adv. Neural Inform. Process. Syst._ , 2021.
  * Kirillov et al. (2020) Kirillov, A., Wu, Y., He, K., and Girshick, R.  PointRend: Image segmentation as rendering.  _IEEE Conf. Comp. Vis. Pattern Recog._ , 2020.
  * Li et al. (2021) Li, M., Li, S., Li, L., and Zhang, L.  Spatial feature calibration and temporal fusion for effective one-stage video instance segmentation.  In _IEEE Conf. Comp. Vis. Pattern Recog._ , 2021.
  * Lin et al. (2021) Lin, H., Wu, R., Liu, S., Lu, J., and Jia, J.  Video instance segmentation with a propose-reduce paradigm.  In _Int. Conf. Comp. Vis._ , 2021.
  * Lin et al. (2014) Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P.  Microsoft coco: Common objects in context.  _arXiv:1405.0312_ , 2014.
  * Liu et al. (2021a) Liu, D., Cui, Y., Tan, W., and Chen, Y.  Sg-net: Spatial granularity network for one-stage video instance segmentation.  In _IEEE Conf. Comp. Vis. Pattern Recog._ , 2021a.
  * Liu et al. (2021b) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Int. Conf. Comp. Vis._ , 2021b.
  * Qi et al. (2022) Qi, J., Gao, Y., Hu, Y., Wang, X., Liu, X., Bai, X., Belongie, S., Yuille, A., Torr, P., and Bai, S.  Occluded video instance segmentation: A benchmark.  In _Int. J. Comp. Vis._ , 2022.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I.  Attention is all you need.  In _Adv. Neural Inform. Process. Syst._ , 2017.
  * Wang et al. (2021) Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., and Xia, H.  End-to-end video instance segmentation with transformers.  In _IEEE Conf. Comp. Vis. Pattern Recog._ , 2021.
  * Wu et al. (2022a) Wu, J., Jiang, Y., Bai, S., Zhang, W., and Bai, X.  Seqformer: Sequential transformer for video instance segmentation.  In _Eur. Conf. Comp. Vis._ , 2022a.
  * Wu et al. (2022b) Wu, J., Liu, Q., Jiang, Y., Bai, S., Yuille, A., and Bai, X.  In defense of online models for video instance segmentation.  In _Eur. Conf. Comp. Vis._ , 2022b.
  * Xu et al. (2018) Xu, N., Yang, L., Fan, Y., Yue, D., Liang, Y., Yang, J., and Huang, T. S.  Youtube-vos: A large-scale video object segmentation benchmark.  _arXiv:1809.03327_ , 2018.
  * Yang et al. (2019) Yang, L., Fan, Y., and Xu, N.  Video instance segmentation.  In _Int. Conf. Comp. Vis._ , 2019.
  * Yang et al. (2021) Yang, S., Fang, Y., Wang, X., Li, Y., Fang, C., Shan, Y., Feng, B., and Liu, W.  Crossover learning for fast online video instance segmentation.  In _Int. Conf. Comp. Vis._ , October 2021.
  * Yang et al. (2022) Yang, S., Wang, X., Li, Y., Fang, Y., Fang, J., Liu, Zhao, X., and Shan, Y.  Temporally efficient vision transformer for video instance segmentation.  In _IEEE Conf. Comp. Vis. Pattern Recog._ , 2022.
  * Zhou et al. (2022) Zhou, Q., Li, X., He, L., Yang, Y., Cheng, G., Tong, Y., Ma, L., and Tao, D.  TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers.  _arXiv:2201.05047_ , 2022.
  * Zhu et al. (2021) Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J.  Deformable detr: Deformable transformers for end-to-end object detection.  In _Int. Conf. Learn. Represent._ , 2021.

##  Appendix A Implementation and training details

We only mention the hyperparameters of our NOVIS approach which differ from
the default Mask2Former (Cheng et al., 2022) settings. Table 4 summarizes the
total number of training iterations, learning rate drops and top-k evaluation
for the three benchmarks presented in this work. Since NOVIS is designed for
near-online processing, its instance decoder is not required to generalize to
arbitrary sequence lengths. This allows us to combine a common sinusoidal
spatial with a learned/parametric temporal encoding. As mentioned in the main
paper, we train our model on normal (228 to 512) and large (480 to 800) input
resolution scales. The former is only evaluated for a comparison to IDOL (Wu
et al., 2022b) on OVIS (Qi et al., 2022). We adjust the number of PointRend
(Kirillov et al., 2020) training points for the two scale ranges to 3136 and
6272, respectively. All Swin-L (Liu et al., 2021b) models are evaluated with a
minimum frame size at test-time (MST) of 480 pixels.

To train a model for a specific clip size, we augment the input data by
sampling $T$ random frames from a given sequence. This is achieved by sampling
$T-1$ frames (without replacement) in a range of $\pm 5$ frames around a
random reference frame. Considering the OVIS (Qi et al., 2022) training set
contains sequences of up to 500 frames, the sampling range is quite small.
Larger ranges, _i.e_., more challenging training clips, generally result in
worse performance.

##  Appendix B Mask quality ablation studies

To further demonstrate the superiority of multi-frame processing, _i.e_.,
offline and near-online, over frame-by-frame online methods, we evaluate the
image segmentation performance of models trained for varying clip sizes in
Figure 4. To this end, we process the same video data (2-fold split on the
OVIS (Qi et al., 2022) training set) either as single images or in clips of
size $T$ and evaluate the mask average precision for each frame separately.
This allows us to isolate and analyse the mask quality without considering the
track consistency or effects of the embedding matching. Interestingly, we
observe diminishing _per-image_ segmentation performance with increasing clip
size coinciding with our evaluations on full length sequences shown in the
main paper. Training on longer clips (or with a large frame sampling range)
has a detrimental effect on the model. As discussed in the main paper, we
consider the inability of the pixel decoder to model large object movements as
the main reason for the diminishing performance.

However, for clip sizes up to $T=6$, we observe a clear advantage of multi-
frame processing. Evaluating a model on clips of frames provides more
consistent and less noisy objects masks. It should be noted, that for $T=4$,
our top-performing final benchmark configuration, the image segmentation
performance without multi-frame processing is inferior to an (online) model
trained for clip size one. However, for an evaluation on full sequences the
advantages of a multi-frame processing outweigh the detrimental effect on the
isolated per-image performance. Hence, we see great potential in addressing
the diminishing performance of our approach by improving its temporal
modelling capacities.

##  Appendix C Qualitative benchmark results

We present qualitative results (including failure cases) for the YouTube-VIS
2019/2021 (Yang et al., 2019) and OVIS (Qi et al., 2022) benchmarks. For
visualization purposes, we do not show all top-k results but filter instance
predictions with a classification score smaller than 5%. Furthermore, the
selected examples are chosen with data privacy in mind and do no contain full
person objects or visible faces.

###  C.1 YouTube-VIS 2019 and 2021

In Figure 5, we show results for a range of YouTube-VIS sequences for
different object classes. Our approach is able to segment and track most
object instances and provides detailed masks across multiple frames. The first
5 rows demonstrates the ability of NOVIS to track various object scales and
movements for low (less than 3 object instances) crowdedness. In particular,
row 5 (hand and lizard) shows a successful object-object occlusion handling
between different object classes. Row 7 and 8 illustrate examples for
significantly more crowded scenes. For small object movements, our multi-frame
processing with clip overlap is very robust and only rarely produces
noisy/inconsistent masks as in the last frame of row 8 (cows). The last row
(bear) demonstrates the limited long-term (more than 60 frames in this
example) occlusion handling capabilities of our approach. However, an
additional appearance-based re-identification step or more long-term frame
processing could mitigate these issues.

###  C.2 OVIS

The OVIS sequences presented in Figure 6 represent significantly more
challenging VIS scenarios with large object movements, high crowdedness and
strong object occlusions. Our model is able to discriminate several instances
even for object classes with similar instance appearance. In particular, row 4
(fishes) demonstrates strong object tracking capabilities in a scenario
challenging even for human observers. Row 2 (cats) and row 5 (dogs) allow for
a comparison of crowded scenes with either small or large object movements,
respectively. As seen in row 2 of Figure 5, our model is generally able to
track large object movements. However, the combination of crowdedness and
object movement in row 5 generates several false positive segmentations. These
are caused by the aforementioned temporal modelling limitations of the pixel
decoder.

| Iterations |
---|---|---
Benchmark | LR drops | Total | Top-K
YouTube-VIS 2019 (Yang et al., 2019) | 4000, 8000 | 12000 | 10
YouTube-VIS 2021 (Yang et al., 2019) | 6000, 10000 | 14000 | 10
OVIS (Qi et al., 2022) | 8000, 12000 | 16000 | 30

Table 4:  Summary of  NOVIS hyperparameters for several VIS benchmarks. In
contrast to (Cheng et al., 2022), we drop the learning rate (LR) twice. The
OVIS (Qi et al., 2022) benchmark contains sequences with more than 20 objects,
hence we increase the number of top-k predictions at test-time.  Figure 4:
Analysis on mask prediction quality for different clip lengths $T$. Each model
is trained and evaluated on a 2-fold split on the OVIS (Qi et al., 2022)
training set. We evaluate per-frame mask prediction performance by processing
video data with the same model either as single images or clips of size $T$.

Figure 5:  Example qualitative results from the YouTube-VIS 2019/2021
validation sets. We show outputs from our NOVIS model with the top-performing
Swin-L (Liu et al., 2021b) backbone for 4 frames uniformly selected over the
given sequence.

Figure 6:  Example qualitative results from the OVIS validation set. We show
outputs from our NOVIS model with the top-performing Swin-L (Liu et al.,
2021b) backbone for 4 frames uniformly selected over the given sequence. The
model is evaluated with 360 pixels minimum scale at test-time (MST).


k\delta({\bf r})\mbox{Im}\\{{\bf p}\times{\bf m}^{*}\\}_{i},$ (64)

and

$\displaystyle\mbox{Im}\\{\bm{\mathcal{F}}_{mcd}\\}_{i}=-\frac{1}{2}\mbox{Im}\\{{m}_{j}\partial_{i}{B}_{j}^{*}\\}-2\pi
k\delta({\bf r})\mbox{Re}\\{{\bf p}\times{\bf
m}^{*}\\}_{i},\,\,\,\,(i,j=1,2,3)$ (65)

At this stage we should remark that for a Gilbert dipole model the Maxwell
equation is no longer $\nabla\times{\bf E}=ik{\bf B}$, but $\nabla\times{\bf
E}=ik{\bf B}-\frac{c}{4\pi}{\bf J}_{mc}$, where ${\bf J}_{mc}=-i\omega{\bf
m}\delta({\bf r})$, as seen above. So instead of Eq. (52), valid for an
electric dipole without magnetic charges, the equation for the complex force
on the electric dipole if one assumes the existence of magnetic charges is

$\displaystyle(\bm{\mathcal{F}})_{i}=\frac{1}{2}{p}_{j}^{*}\partial_{i}{E}_{j}+2i\pi
k\delta({\bf r})({\bf m}\times{\bf p}^{*})_{i}.$ (66)

Whose real and imaginary parts are

$\displaystyle<\bm{\mathcal{F}}>_{i}=\frac{1}{2}\mbox{Re}\\{{p}_{j}\partial_{i}{E}_{j}^{*}\\}-2\pi
k\delta({\bf r})\mbox{Im}\\{{\bf p}\times{\bf m}^{*}\\}_{i}.$ (67)

and

$\displaystyle\mbox{Im}\\{\bm{\mathcal{F}}\\}_{i}=-\frac{1}{2}\mbox{Im}\\{{p}_{j}\partial_{i}{E}_{j}^{*}\\}-2\pi
k\delta({\bf r})\mbox{Re}\\{{\bf p}\times{\bf
m}^{*}\\}_{i},\,\,\,\,(i,j=1,2,3).$ (68)

###  E.3 ELECTRIC AND MAGNETIC DIPOLE

Therefore, we conclude that if magnetic charges are assumed, the real and
imaginary parts of the resulting electric and magnetic dipole forces are:

$\displaystyle<\bm{\mathcal{F}}>_{i}+<\bm{\mathcal{F}}_{mcd}>_{i}=<\bm{F}>_{i}+<\bm{\mathcal{F}}_{cld}>_{i}\,=\frac{1}{2}\mbox{Re}\\{{p}_{j}\partial_{i}{E}_{j}^{*}\\}+\frac{1}{2}\mbox{Re}\\{{m}_{j}\partial_{i}{B}_{j}^{*}\\},$
(69)

and our proposed expression:

$\displaystyle\mbox{Im}\\{\bm{\mathcal{F}}\\}_{i}+\mbox{Im}\\{\bm{\mathcal{F}}_{mcd}\\}_{i}=-\frac{1}{2}\mbox{Im}\\{{p}_{j}\partial_{i}{E}_{j}^{*}\\}-\frac{1}{2}\mbox{Im}\\{{m}_{j}\partial_{i}{B}_{j}^{*}\\}-4\pi
k\delta({\bf r})\mbox{Re}\\{{\bf p}\times{\bf m}^{*}\\}_{i}.$ (70)

While in the model in which no magnetic charges are present, we obtain for the
resulting reactive force

$\displaystyle\mbox{Im}\\{\bm{F}\\}_{i}+\mbox{Im}\\{\bm{\mathcal{F}}_{cld}\\}_{i}=-\frac{1}{2}\mbox{Im}\\{{p}_{j}\partial_{i}{E}_{j}^{*}\\}-\frac{1}{2}\mbox{Im}\\{{m}_{j}\partial_{i}{B}_{j}^{*}\\}.$
(71)

Which in contrast with the time-averaged force which is the same for the $cld$
amd $mcd$ models, (71) differs from (70) by $4\pi k\delta({\bf
r})\mbox{Re}\\{{\bf p}\times{\bf m}^{*}\\}_{i}$.

This procedure does not yield for the RLF (69) the interference term [cf. Eq.
(47)]: $-\frac{k^{4}}{3}\mbox{Re}\\{{\bf p}\times{\bf m}^{*}\\}$ nieto1 ;
chaumet  of the electric and magnetic dipoles. Therefore it is likely that the
ILF so obtained, although valid for pure electric or magnetic dipoles, be not
valid for magnetoelectric dipolar particles.

From Eq. (2) we guess for the reactive spin torque:

$\displaystyle\bm{\Xi}^{spin}=-\frac{1}{5}\mbox{Im}[{\bf p}\times{\bf
E}^{*}+{\bf m}\times{\bf B}^{*}]+\frac{k}{c}\int_{V}d^{3}r\,({\bf
L}_{e}^{O}-{\bf L}_{m}^{O}).$ (72)

Notice that, again, the diagonal terms of the complex MST do not contribute to
the recoil, or scattering, component nieto1  of $\bm{\Xi}^{spin}$.

Or in terms of the above quoted the electric and magnetic angular momenta:
${\mathscr{F}}_{e}$ and ${\mathscr{F}}_{m}$, we write

$\displaystyle\bm{\Xi}^{spin}=-\frac{k}{c}\\{\frac{1}{5}(\alpha_{e}^{I}{\mathscr{F}}_{e}+\alpha_{m}^{I}{\mathscr{F}}_{m})-\int_{V}d^{3}r\,({\bf
L}_{e}^{O}-{\bf L}_{m}^{O})\\}.$ (73)

Of course the angular orbital momenta, like we saw above for the orbital
momenta, store power of the propagating plane wave components of the
wavefields.

##  Appendix F  On the calculation of the $\bm{ROM}$

Both $<\bm{\mathcal{F}}>$ and $\mbox{Im}\\{\bm{\mathcal{F}}\\}$ are calculated
from the complex Lorentz integral in $V_{0}$, Eq.(6). The induced densities
$\rho$ and ${\bf j}$ are obtained from the constitutive relations on the total
field ${\bf E}={\bf E}^{(i)}+{\bf E}^{(s)}$, ${\bf B}={\bf B}^{(i)}+{\bf
B}^{(s)}$ via the first and fourth Maxwell equations, respectively.

In our numerical method, the total field is obtained using the commercial
software package “FDTD Solutions” (Lumerical, Inc.). The simulation region is
$0.22\times 0.22\times 0.22$ $\mu m^{3}$, and a uniform mesh with grid size of
$5nm$ was used. We compute the complex Lorentz force by the expression:
$\bm{\mathcal{F}}=\frac{1}{2}\int_{V}[\rho{\bf E}({\bf r})+\frac{\bf
j^{*}}{c}\times{\bf B}({\bf r})]\,d^{3}r$, determining $\rho$ and ${\bf j}$
via the procedure described in lumerics . Then we evaluate the IMST across the
surface of a cube that encloses the spherical dipolar particle of volume
$V_{0}$. Let the volume of this cube be denoted $V_{q}$. The volume of the
four corners between $V_{q}$ and $V_{0}$ is $V_{4c}$, its surface being
$\partial V_{4c}$

Obviously since $\rho=0$ and ${\bf j}=0$ outside $V_{0}$, we have from the
total field ${\bf E}={\bf E}^{(i)}+{\bf E}^{(s)}$, ${\bf B}={\bf B}^{(i)}+{\bf
B}^{(s)}$ the following:

$\displaystyle 0=\mbox{Im}\\{\bm{\mathcal{F}}\\}_{4c}=\int_{\partial
V_{4c}}d^{2}r\,\mbox{Im}\\{T_{ij}\\}n_{j}+i\omega\int_{V_{4c}}d^{3}r\,[{\bf
P}_{m}^{O}-{\bf P}_{e}^{O}]_{i}\,,\,\,\,$ (74)

I.e.

$\displaystyle\int_{\partial
V_{4c}}d^{2}r\,\mbox{Im}\\{T_{ij}\\}n_{j}=-i\omega\int_{V_{4c}}d^{3}r\,[{\bf
P}_{m}^{O}-{\bf P}_{e}^{O}]_{i}\,.\,\,\,$ (75)

But

$\displaystyle\int_{\partial
V_{4c}}d^{2}r\,\mbox{Im}\\{T_{ij}\\}n_{j}=\int_{\partial V_{q}-\partial
V_{0}}d^{2}r\,T_{ij}n_{j}\,.\,\,\,$ (76)

Then from (75):

$\displaystyle i\omega\int_{V_{q}}d^{3}r\,[{\bf P}_{m}^{O}-{\bf
P}_{e}^{O}]_{i}=i\omega\int_{V_{0}}d^{3}r\,[{\bf P}_{m}^{O}-{\bf
P}_{e}^{O}]_{i}-\int_{\partial V_{4c}}d^{2}r\,\mbox{Im}\\{T_{ij}\\}n_{j}\,.$
(77)

And from (76) and (77) one derives:

$\displaystyle i\omega\int_{V_{q}}d^{3}r\,[{\bf P}_{m}^{O}-{\bf
P}_{e}^{O}]_{i}=i\omega\int_{V_{0}}d^{3}r\,[{\bf P}_{m}^{O}-{\bf
P}_{e}^{O}]_{i}+\int_{\partial V_{0}-\partial
V_{q}}d^{2}r\,\mbox{Im}\\{T_{ij}\\}n_{j}\,.$ (78)

Which taking into account (57) for $V=V_{0}$, leads to

$\displaystyle i\omega\int_{V_{q}}d^{3}r\,[{\bf P}_{m}^{O}-{\bf
P}_{e}^{O}]_{i}=\mbox{Im}\\{\bm{\mathcal{F}}_{i}\\}-\int_{\partial
V_{q}}d^{2}r\,\mbox{Im}\\{T_{ij}\\}n_{j}\,.$ (79)

Equation (79) is the procedure we use to calculate the ROM integrated in the
volume $V_{q}$ from the two terms of its right side previously computed as
described above.

## References

  * (1) J.D. Jackson, Classical Electrodynamics, 2nd edn. J. Wiley (New York, 1975).
  * (2) M. Born and E. Wolf, Principles of Optics, Cambridge University Press, Cambridge (1995).
  * (3) L. Mandel and E.Wolf, Optical Coherence and Quantum Optics, Cambridge University Press, Cambridge, 1995).
  * (4) M. Nieto-Vesperinas, J.J. Saenz, R. Gomez-Medina, and L. Chantada, “Optical forces on small magnetodielectric particles,” Opt. Express 18, 11428 (2010).
  * (5) M. Nieto-Vesperinas, R. Gomez-Medina and J.J.Saenz, Angle-suppressed scattering and optical forces on submicrometer dielectric particles, J. Opt. Soc. Am. A 28, 54 (2011).
  * (6) K. Y. Bliokh, A. Y. Bekshaev and F. Nori, Dual electromagnetism: helicity, spin, momentum and angular momentum, New Journal of Physics 15, 033026 (2013).
  * (7) T.H. Boyer, The force on a magnetic dipole, Am. J. Phys. 56, 688 (1988).
  * (8) P. A. M. Dirac, Quantised singularities in the electromagnetic field, Proc. R. Soc. Lond. A 133, 60 (1931); P. A. M. Dirac, The theory of magnetic poles, Phys. Rev. 74, 817 (1948).
  * (9) N. Cabibbo and E. Ferrari, Quantum electrodynamics with Dirac monopoles, Nuovo Cimento 23 1147 (1962).
  * (10) J. Schwinger, Magnetic charge and quantum field theory, Phys. Rev. 144, 1087 (1966).
  * (11) R. A. Brandt, F. Neri and D. Zwanziger, Lorentz invariance from classical particle paths in quantum field theory of electric and magnetic charge, Phys. Rev. D 19, 1153 (1979).
  * (12) K. Li, W-. J. Chen and C. Naon, Classical Electromagnetic Field Theory in the Presence of Magnetic Sources, Chin. Phys. Lett. 20, 321 (2003).
  * (13) L. D. Landau and E.M. Lifshitz, The Classical Theory of Fields (Pergamon Press, N.Y. 1994).
  * (14) M. Nieto-Vesperinas, Scattering and Diffraction in Physical Optics, 2nd ed. (World Scientific, Singapore, 2006).
  * (15) P.C. Chaumet and A. Rahmani, Electromagnetic force and torque on magnetic and negative index scatterers. Opt Express 17 2224 (2009).
  * (16) L. Vaidman, Torque and force on a magnetic dipole, Am. J. Phys. 56 978 (1990).
  * (17) K. T. McDonald, Forces on Magnetic Dipoles (2018). https://www.physics.princeton.edu/ mcdonald/examples/neutron.pdf.
  * (18) W. Shockley and R.P. James, “Try Simplest Cases” Discovery of “Hidden Momentum” Forces on “Magnetic Currents”, Phys. Rev. Lett. 18, 876 (1967).
  * (19) Ansys/Lumerical, Inc., https://support.lumerical.com/hc/en-us/articles/360042214594-Methodology-for-optical-force-calculations.