# Impact of electron shell excitations on the energy spectrum
of $\beta$-electrons in neutrinoless double-$\beta$ decay

M. I. Krivoruchenko, K. S. Tyrin  National Research Centre ”Kurchatov
Institute”, Pl. Akademika Kurchatova 1, 123182 Moscow, Russia  F. F. Karpeshin
D. I. Mendeleev All-Russian Research Institute of Metrology (VNIIM)
Moskovsky Ave. 19, 190005 Saint Petersburg

###### Abstract

The electron shell of the daughter atoms often appears excited in the
double-$\beta$ decays, which causes a change in the energy taken away by
$\beta$-electrons. The average value and variance of the excitation energy of
the electron shell of the daughter atom are calculated for the double-$\beta$
decay of germanium
${}_{32}^{76}\mathrm{Ge}\rightarrow_{34}^{76}\mathrm{Se}^{*}+2\beta^{-}(+~{}2\bar{\nu_{e}})$
in both the Thomas–Fermi model and the relativistic Dirac–Hartree–Fock theory.
Using the results obtained, a two-parameter model of the energy spectrum of
$\beta$-electrons in the neutrinoless mode is constructed, taking into account
energy redistribution in the decay channels. The shift in total energy of
$\beta$-electrons is found to be under 50 eV at a confidence level of 90% .
The average excitation energy, on the other hand, is an order of magnitude
higher and equal to $\sim 400$ eV, while the square root of the variance is
equal to $\sim 2900$ eV, which is presumably explained by the contribution of
the core electrons to the energy characteristics of the process. The
probability is nearly saturated with excitations with a small amount of
released energy, which is common for the outermost electrons. The distortion
of the peak shape of the neutrinoless double-$\beta$ decay should be taken
into consideration when analyzing data from detectors with a resolution of
$\sim 100$ eV or higher.

Neutrinos are one of the most promising tools for investigating new physics
beyond the Standard Model. The study of $\beta$-processes sensitive to
neutrino mixing, mass, and nature (Dirac/Majorana) presents an attractive
approach in the quest for extensions of the Standard Model.

Experimental observation of neutrinoless double-$\beta$ decay would
significantly narrow the range of possibilities for the generalization. Such
an observation would prove the existence of a Majorana neutrino mass, as well
as a violation of the conservation of the total lepton number (for a review
see, e.g., Bilenky:2018 ). The neutrinoless double-$\beta$ decay is registered
by a narrow peak at the kinetic energy of $\beta$-electrons, $T$, equal to the
decay energy, $Q$. Channels with the electron shell of the daughter atom in
the ground state do not always dominate in $\beta$ processes Bambynek:1977 ;
Krivoruchenko:2020 , which leads, in particular, to an increased rate of
neutrinoless double-electron capture in non-resonant modes Karpeshin:2020 ;
Karpeshin:2022 ; Karpeshin:2023 . Excitation of electron shells and chemical
shifts of atomic levels, which depend on the aggregate state of the substance
Lindgren:2004 , affect the energy spectrum in the neutrinoless double-$\beta$
decay. The energy peak at $T=Q$ is used as an indicator of the neutrinoless
double-$\beta$ decay, and the search for it is being carried out by the
collaborations CUORE Alfonso:2015 , EXO Anton:2019 , and KamLAND-Zen
Gando:2016 . The SuperNEMO Arnold:2015  collaboration can observe
multielectron modes. The GERDA collaboration GERDA:2017  determined an upper
limit on the half-life
${}_{32}^{76}\mathrm{Ge}\rightarrow_{34}^{76}\mathrm{Se}^{*}+2\beta^{-}$ of
$5.3\times 10^{25}$ years with a confidence level of 90%. To interpret the
results in terms of the mass of the Majorana neutrino, the uncertainties in
the axial coupling constants of nucleons and the nuclear matrix elements
should be controlled Suhonen:2017 ; Simkovic:2008 . In this paper, the effect
of electron shell excitations on the energy spectrum of $\beta$-electrons in
the double-$\beta$ decays is investigated.

The nucleus of a $\beta$-decaying atom experiences a charge change of $\Delta
Z=\pm 1,\pm 2$. The change in charge acts on the electron shell as a sudden
perturbation, causing the electrons of the daughter atom to move with an
appropriate probability to higher discrete levels (shake-up) or to the
continuous spectrum (shake-off). The energy spectrum of $\beta$-electrons is
determined by a decay channel, where all electrons maintain their initial
quantum numbers, and decay channels with electron shell excitation. In the
second case, the effective reaction energy decreases, modifying the energy
spectrum of $\beta$-electrons.

To calculate the energy spectrum, it is necessary to sum over all the
multiparticle channels with their individual amplitudes, taking into account
the modified reaction energy. In a configuration corresponding to a specific
excitation of the shell, the wave function of the parent-atom electrons is
projected onto the wave function of the daughter-atom electrons. As a rule,
each such amplitude needs a complex multiparticle numerical simulation. The
Coulomb problem is also unique in that there are an infinite number of decay
channels.

The computation of the average excitation energy of the daughter atomic shell
and its variance, however, is quite straightforward within the Thomas–Fermi
model and the relativistic Dirac–Hartree–Fock (DHF) formalism. The results
obtained with these methods can be used to construct a simple two-parameter
probability distributions of the energy taken away by $\beta$-electrons.

The non-relativistic Hamiltonian for an atom with a nuclear charge of $Z$ and
$N$ electrons has the form

$\hat{H}_{Z,N}=\sum_{i=1}^{N}\left(\frac{1}{2}\mathbf{p}_{i}^{2}-\frac{Z}{|\mathbf{r}_{i}|}\right)+\sum_{i<j}^{N}\frac{1}{|\mathbf{r}_{i}-\mathbf{r}_{j}|},$
(1)

where $\mathbf{p}_{i}$ are the momenta and $\mathbf{r}_{i}$ are the
coordinates of the electrons; hereafter, the system of atomic units is used:
$\hbar=m=e=1$, $c=137$, where $m$ is the mass of the electron, $e$ is the
charge of the proton, and $c$ is the speed of light. The ground state is
denoted by $|Z,N\rangle$. The ground-state binding energy of the electrons,
$E_{Z,N}$, is the eigenvalue of $\hat{H}_{Z,N}$.

In neutrinoless double-$\beta$ decay, the nuclear charge increases by two
units, and the parent-atom ground state becomes a superposition of energy
eigenstates of the daughter-ion Hamiltonian:

$\hat{H}_{Z+2,Z}=\hat{H}_{Z,Z}-2\sum_{i=1}^{Z}\frac{1}{{r}_{i}},$ (2)

where $r_{i}=|\mathbf{r}_{i}|$. The second term acts as a sudden perturbation.
In the daughter ion, the average energy of the $Z$ electrons that make up the
parent atomic shell is equal to

$\langle Z,Z|\hat{H}_{Z+2,Z}|Z,Z\rangle=E_{Z,Z}+2Z^{-1}E_{Z,Z}^{\mathrm{C}},$

where $E_{Z,Z}^{\mathrm{C}}$ is the Coulomb interaction energy of $Z$
electrons with the nucleus:

$\displaystyle E_{Z,Z}^{\mathrm{C}}=-Z\sum_{i=1}^{Z}\langle
Z,Z|\frac{1}{r_{i}}|Z,Z\rangle.$ (3)

The average excitation energy of the electron shell becomes

$\displaystyle\mathcal{M}=E_{Z,Z}+2Z^{-1}E_{Z,Z}^{\mathrm{C}}-E_{Z+2,Z}.$ (4)

The binding energies of neutral atoms $E_{Z,Z}$ are tabulated in Refs. Lu:1971
; Desclaux:1973 ; Clementi:1974 ; HUANG:1976  and can be computed with the use
of computer programs that determine the electron shell structures of atoms
(see, e.g., Dyall:1989 ). The value of $E_{Z+2,Z}$ differs from the total
binding energy of electrons in the neutral atom $E_{Z+2,Z+2}$ by the
ionization energe of two valence electrons, which does not exceed $\sim 20$
eV. The values of the Coulomb energy $E_{Z,Z}^{\mathrm{C}}$ obtained with the
use of the DHF method are reported in Ref. HUANG:1976 .

The variance of excitation energy reads

$\displaystyle\mathcal{D}=\langle Z,Z|\hat{H}_{Z+2}^{2}|Z,Z\rangle-\langle
Z,Z|\hat{H}_{Z+2}|Z,Z\rangle^{2}.$ (5)

Given Eq. (2) and the fact that $E_{Z,N}$ is the eigenvalue of
$\hat{H}_{Z,N}$, we obtain

$\displaystyle\frac{1}{4}\mathcal{D}=\sum_{i=1}^{Z}\sum_{j=1}^{Z}\langle
Z,Z|\frac{1}{r_{i}}\frac{1}{r_{j}}|Z,Z\rangle-\langle
Z,Z|\sum_{i=1}^{Z}\frac{1}{r_{i}}|Z,Z\rangle^{2}.$

In the first term, we neglect the exchange contributions and factorize the
matrix elements for $i\neq j$:

$\langle Z,Z|\frac{1}{r_{i}r_{j}}|Z,Z\rangle\approx\langle
Z,Z|\frac{1}{r_{i}}|Z,Z\rangle\langle Z,Z|\frac{1}{r_{j}}|Z,Z\rangle.$

The cancellation of non-diagonal terms caused by the factorization allows to
give the variance in the form

$\displaystyle\frac{1}{4}\mathcal{D}\approx\sum_{i=1}^{Z}\langle
Z,Z|\frac{1}{r_{i}^{2}}|Z,Z\rangle-\sum_{i=1}^{Z}\langle
Z,Z|\frac{1}{r_{i}}|Z,Z\rangle^{2}.$ (6)

The electron wave functions of the Roothaan-Hartree-Fock formalism
Clementi:1974  can be used to estimate the exchange effects and demonstrate
that the exchange contribution to the variance is under 10%.

Equations (2) – (6) are also valid in the relativistic framework using the
Dirac-Coulomb Hamiltonian as a zero-order approximation and the Breit
potential to describe the electron-electron interaction in the order of
$1/c^{2}$ Grant:2008 .

The total binding energy and Coulomb interaction energy in the Thomas–Fermi
model are given by the equations $E_{Z,Z}=-0.764Z^{7/3}$ and
$E_{Z,Z}^{\mathrm{C}}=7E_{Z,Z}/3$, respectively (see, e.g., LL:1987 ). The
virial theorem establishes the relationship between $E_{Z,Z}$ and
$E_{Z,Z}^{\mathrm{C}}$. The average excitation energy is equal to

$\mathcal{M}=\left(1+\frac{14}{3Z}\right)E_{Z,Z}-E_{Z+2,Z}.$ (7)

In the double-$\beta$ decay of germanium, each of the two missing 4$p$
electrons of selenium is bound at 9.752 eV, while the sum of the first two
selenium ionization energies is 30.948 eV Kramida:2022 . Taking into account
the correction, we find $\mathcal{M}=382$ eV.

The electron density is expressed in terms of the wave function of atomic
shells according to

$n(\mathbf{r})=\sum_{i=1}^{Z}\langle
Z,Z|\delta(\mathbf{r}-\mathbf{r}_{i})|Z,Z\rangle.$ (8)

The first term in Eq. (6) can obviously be reduced to the form $\int
d\mathbf{r}r^{-2}n(\mathbf{r})$. The parametrization of shielding function for
neutral atoms of Ref. Mason:1964  can be utilized to calculate the integral.
The integral diverges as $\sim dr/r^{3/2}$ at short distances. The semi-
classical approximation has a distance restriction $1/Z\lesssim r$. The
electron density is taken to be constant and equal to $n(r=1/Z)$ for numerical
calculations in the range $0\leq r\leq 1/Z$.

To evaluate the second term in Eq. (6), we apply the well-known inequality,
according to which the arithmetic mean does not exceed the quadratic mean:

$\displaystyle\frac{1}{Z}\sum_{i=1}^{Z}\langle
Z,Z|\frac{1}{{r}_{i}}|Z,Z\rangle^{2}\geq\left(\frac{1}{Z}\sum_{i=1}^{Z}\langle
Z,Z|\frac{1}{{r}_{i}}|Z,Z\rangle\right)^{2}$
$\displaystyle=\left(\frac{1}{Z}\int
d\mathbf{r}\frac{1}{r}n(\mathbf{r})\right)^{2}.~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$
(9)

The right side equals $(Z^{-2}E_{Z,Z}^{\mathrm{C}})^{2}$. The inequality
determines the upper bound of the variance:

$\frac{1}{4}\mathcal{D}\leq\frac{1}{4}\bar{\mathcal{D}}=\int
d\mathbf{r}\frac{1}{{r}^{2}}n(\mathbf{r})-Z^{-1}\left(\int
d\mathbf{r}\frac{1}{{r}}n(\mathbf{r})\right)^{2}.$ (10)

The estimated value for germanium double-$\beta$ decay is
$\bar{\mathcal{D}}^{1/2}$ = $2160$ eV.

The first negative moment in the germanium atom has a value of $\langle
1/r\rangle=4.99$ Lu:1971 . The relativistic DHF approach is also implemented
in the RAINE software package RAINE ; Band:1986 , which calculates the
position of the selenium ground-state energy with respect to the germanium
ground-state energy. The average excitation energy $\mathcal{M}=300$ eV is
found by substituting the values obtained into Eq. (6), which is qualitatively
compatible with the Thomas-Fermi result. The values of $E_{Z,Z}$ and
$E_{Z,Z}^{\mathrm{C}}$ of Huang et al. HUANG:1976  result in $\mathcal{M}=400$
eV.

The variance (6) is calculated in terms of the first and second negative
moments of the electron distribution in the germanium atom. Using the values
of the second moments of the electron distribution in atoms given by Desclaux
Desclaux:1973  one finds $\mathcal{D}^{1/2}=2870$ eV. A qualitative
consistency is also seen when the Thomas–Fermi value of $2160$ eV is compared
to the obtained value.

The values of $\mathcal{M}$ and $\mathcal{D}$ are substantially greater than
expected for channels associated with valence shell excitations. The shake-up
contribution of the outer shells to $\mathcal{M}$ and $\mathcal{D}$ does not
exceed $\sim 10$ eV. The binding energy determines the convergence of the
integral over a continuous energy spectrum, so the shake-off contribution to
$\mathcal{M}$ and $\mathcal{D}$ is comparable to the shake-up contribution. At
the same time, valence shell excitations dominate the probability, because a
change in the nuclear charge $\Delta Z=2$ results in a considerable, in
relative units, change in the screened potential at the atomic border.

Due to their lower energy, the core electrons have a small probability of
being excited. They may considerably contribute, however, to both the average
excitation energy and the variance. The change in the nuclear charge $\Delta
Z$ is much less than $Z$, allowing perturbation theory to be used to find the
probability of the K–electron shake-off Feinberg:1941 ; Migdal:1941  (see also
LL:1987 ). In the leading order, the probability, the average excitation
energy, and the average square of excitation energy are equal
$\Delta\mathcal{P}=0.65\Delta Z^{2}/Z^{2}$, $\Delta\mathcal{M}=0.66\Delta
Z^{2}$ and $\Delta\mathcal{M}_{2}=1.87\Delta Z^{2}Z^{2}$, respectively. In the
double-$\beta$ decay of germanium, $\Delta\mathcal{P}=2.6\times 10^{-3}$,
$\Delta\mathcal{M}=72$ eV and
$\Delta\mathcal{D}\equiv\Delta\mathcal{M}_{2}-\Delta\mathcal{M}^{2}=(2380$
eV$)^{2}$. $\Delta\mathcal{M}$ and $\Delta\mathcal{D}$ are comparable to
$\mathcal{M}$ and $\mathcal{D}$. The average and variance of the excitation
energy are therefore strongly influenced by the K-electron shake-off.

Figure 1:  The probability distribution function of the kinetic energy of
$\beta$-electrons, $T$, taking into account the overlap factor, $K_{Z}$, of
the electron shells of atoms according to Eq. (13). Lines 1 and 2 correspond
to the average excitation energy of the electron shell, $\mathcal{M}=300$ eV
and $400$ eV, respectively, and the square root of the variance,
$\mathcal{D}^{1/2}=2870$ eV. Numerical values characterize germanium
double-$\beta$ decay, with $Q=2039.061(7)$ keV being the reaction energy
GERDA:2017 .

In $\beta$ processes, the parent-atom electron shell evolves into the
daughter-ion electron shell in the ground state with a finite probability that
is not equal to unity Bambynek:1977 ; Krivoruchenko:2020 ; Ge:2021 . To
determine the overlap amplitude
$K_{Z}=\langle\mathrm{Se~{}III}|\mathrm{Ge}\rangle$ in germanium
double-$\beta$ decay, the electron wave functions of the selenium ion
$|\mathrm{Se~{}III}|\rangle$ in the ground state and the electron wave
functions of the germanium atom $|\mathrm{Ge}\rangle$ in the ground state are
constructed using the Grasp-2018 software package, which implements the
relativistic DHF method Dyall:1989 ; Grant:2008 . The value of $K_{Z}$ is
sensitive to the approximations used due to the noticeable difference in
binding energies of valence electrons with the same quantum numbers in
$|\mathrm{Ge}\rangle$ and $|\mathrm{Se~{}III}|\rangle$. Calculations based on
Grasp-2018 result in $K_{Z}=0.575$.

In the neighborhood of $T=Q$, the kinetic energy of $\beta$-electrons with
account of the energy loss for the excitation and ionization of the daughter
atom is described by the probability distribution function

$\displaystyle dF(T)$ $\displaystyle=$
$\displaystyle\left(K^{2}_{Z}\delta(T-Q)\right.$ (11) $\displaystyle+$
$\displaystyle\left.(1-K^{2}_{Z})w(1-T/Q)/Q)\right)dT,$

where $K^{2}_{Z}$ is the probability of the electron shell of the parent atom
to evolve into the ground state of the daughter ion, and $w(x)$ is the
probability density function of the excitation energy $\epsilon=Q-T$, measured
in units of $Q$. The second term leads to a smearing of the peak.

The average value, the variance, and the overlap amplitude, $K_{Z}$, can be
used to construct a simple two-parameter model for the energy spectrum of
$\beta$-electrons. A wide range of random processes is well described by the
beta distribution Korolyuk:1985

$w(x)=\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}$
(12)

for a random variable $x$ taking values on the interval $(0,1)$;
$\alpha,\beta>0$ are free parameters. In the case under consideration,
$x=\epsilon/Q$. The averages of $x$ and $x^{2}$ equal
$m={\alpha}/(\alpha+\beta)$ and
$m_{2}=\alpha(\alpha+1)/((\alpha+\beta)(\alpha+\beta+1))$, respectively. The
equations $(1-K^{2}_{Z})m=\mathcal{M}/Q$ and
$(1-K^{2}_{Z})m_{2}=(\mathcal{D}+\mathcal{M}^{2})/Q^{2}$ allow to determine
the parameters $\alpha$ and $\beta$ in terms of $\mathcal{M}$, $\mathcal{D}$,
and $K_{Z}$. As an example, let us consider the germanium-76 isotope used in
the GERDA collaboration experiments. For $\mathcal{M}=300$ eV and $400$ eV and
the variance $\mathcal{D}^{1/2}=2870$ eV we find $\alpha=0.016$, $\beta=74$
and $\alpha=0.029$, $\beta=99$, respectively. The two-parameter gamma
distribution Korolyuk:1985 , describing random quantities on the half-axis
$(0,+\infty)$, is also suitable for the modeling, since the condition
$Q\gg\mathcal{M},\mathcal{D}^{1/2}$ allows to extend the integration by
$x=\epsilon/Q$ to the half-axis $(0,+\infty)$. In the physically interesting
region $T\sim Q$, the beta and gamma distributions practically coincide
because of $\beta\gg 1$.

The probability density is singular, but integrable at $T=Q$. Figure 1 shows
the distribution function

$F(T)=K_{Z}^{2}+(1-K_{Z}^{2})\int_{T}^{Q}w(1-T^{\prime}/Q)dT^{\prime}/Q,$ (13)

which determines the probability of $\beta$-electrons to have an energy that
differs from $Q$ by no more than $Q-T>0$. For $\mathcal{M}=300$ eV with a
probability of 90% , the energy of $\beta$-electrons deviates from $Q$ by no
more than 1 eV. For $\mathcal{M}=400$ eV with the probability of 90%, the
deviation is less than 50 eV. The probability increases with $Q-T$ roughly
logarithmically, i.e., quite slowly. The modern detectors measure the energy
released in double-$\beta$ decay with a resolution of $\sim 1$ keV, which
significantly complicates the observation of effects associated with the
excitation of electron shells. The processes of shaking K–electrons are
accompanied by the energy release of $\sim 10$ keV, however, these processes
are rare.

To summarize, the electron shell of the daughter atom in double-$\beta$ decay
is excited with high probability. In $\beta$ processes, transitions of the
valence electrons to discrete states or to a continuous spectrum dominate the
probability. The average excitation energy of a daughter atom’s electron
shell, $\mathcal{M}$, and its variance, $\mathcal{D}$, are estimated for
germanium $0\nu 2\beta$ decay using the Thomas–Fermi model and the
relativistic Dirac–Hartree–Fock formalism. The reported values
$\mathcal{M}\sim 400$ eV and $\mathcal{D}^{1/2}\sim 2900$ eV greatly exceed
the values typical for valence electron processes. The analysis of the energy
characteristics reveals that less probable K–electron shake-off processes
contribute significantly to the average excitation energy and its variance.
The distortion of the peak shape in the neutrinoless double-$\beta$ decay
caused by electron shell excitations becomes important for the interpretation
of observational data when the resolution of detectors increases to $\sim 100$
eV.

The work was supported by the Russian Science Foundation under grant No.
23-22-00307.

## References

  * (1) S. Bilenky, Introduction to the Physics of Massive and Mixed Neutrinos, 2nd ed., Lecture Notes in Physics Vol. 947 (Springer-Verlag, Berlin 2018).
  * (2) W. Bambynek, H. Behrens, M. H. Chen, B. Crasemann, M. L. Fitzpatrick, K. W. D. Ledingham, H. Genz, M. Mutterer, and R. L. Intemann, Orbital electron capture by the nucleus, Rev. Mod. Phys. 49, 77 (1977).
  * (3) M. I. Krivoruchenko, K. S. Tyrin, Overlap of electron shells in $\beta$ and double-$\beta$ decays, Eur. Phys. J. A 56, 16 (2020).
  * (4) F. F. Karpeshin, M. B. Trzhaskovskaya, and L. F. Vitushkin, Nonresonance Shake Mechanism in Neutrinoless Double Electron Capture, Yad. Fiz. 83, 344 (2020). [Phys. At. Nucl. 83, 608 (2020)].
  * (5) F. F. Karpeshin, M. B. Trzhaskovskaya, Effect of shake-off on the rate of a neutrinoless double electronic capture in 164Er, Yad. Fiz. 85, 387 (2022). [Phys. At. Nucl. 85 (2020)].
  * (6) F. F. Karpeshin and M. B. Trzhaskovskaya, Shake-off in the Er164 neutrinoless double-electron capture and the dark matter puzzle, Phys. Rev. C 107, 045502 (2023).
  * (7) I. Lindgren, Chemical shifts in x-ray and photo-electron spectroscopy: a historical review, Journal of Electron Spectroscopy and Related Phenomena 137-140, 59-71 (2004).
  * (8) K. Alfonso et al., [CUORE Collaboration], Search for neutrinoless double-beta decay of 130Te with CUORE-0. Phys. Rev. Lett. 115, 102502 (2015).
  * (9) G. Anton et al. (EXO-200 Collaboration), Search for Neutrinoless Double-$\beta$ Decay with the Complete EXO-200 Dataset, Phys. Rev. Lett. 123, 161802 (2019).
  * (10) A. Gando et al. [KamLAND-Zen Collaboration], Search forMajorana neutrinos near the inverted mass hierarchy region with KamLAND-Zen, Phys. Rev. Lett. 117, 082503 (2016) Addendum: [Phys. Rev. Lett. 117, 109903 (2016)]
  * (11) R. Arnold et al., [NEMO-3 Collaboration], Results of the search for neutrinoless double-$\beta$ decay in 100Mo with the NEMO-3 experiment, Phys. Rev. D 92, 072011 (2015)
  * (12) The GERDA Collaboration, Background-free search for neutrinoless double-$\beta$ decay of 76Ge with GERDA, Nature 544, 47 - 52 (2017).
  * (13) J. T. Suhonen, Value of the axial-vector coupling strength in $\beta$ and $\beta\beta$ decays: a review, Front. Phys. 5, 55 (2017). 123
  * (14) F. Šimkovic, Amand Faessler, V. Rodin, P. Vogel, J. Engel, Anatomy of nuclear matrix elements for neutrinoless double-beta decay, Phys. Rev. C 77, 045503 (2008).
  * (15) C. C. Lu, T. A. Carlson, F. B. Malik, T. C. Tucker, and C. W. Nestor, Jr., Relativistic Hartree-Fock-Slater eigenvalues, radial expectation values, and potentials for atoms, $2\leq Z\leq 126$, At. Data Nucl. Data Tables 3, 1 (1971).
  * (16) J. P. Desclaux, Relativistic Ditac-Fock expectation values for atoms with $Z=1$ to $Z=120$, At. Data Nucl. Data Tables 12, 31l-406 (1973).
  * (17) E. Clementi and C. Roetti, Roothaan-Hartree-Fock Atomic Wavefunctions. Basis Functions and Their Coefficients for Ground and Certain Excitex States of Neutral and Ionized Atoms, $Z\leq 54$, At. Data Nucl. Data Tables 14, 177-478 (1974).
  * (18) Ken-Ning Huang, Michio Aoyagi, Mau Hsiung Chen, Bern Grasemann, Hans Mark, Neutral-atom electron binding energies from relaxed-orbital relativistic Hartree-Fock-Slater calculations, At. Data Nucl. Data Tables 18, 243-29 (1976).
  * (19) K. G. Dyall, I. P. Grant, C. T. Johnson, F. A. Parpia, and E. P. Blummer, GRASP: A general-purpose relativistic atomic structure program, Comput. Phys. Commun. 55, 425-456 (1989).
  * (20) I. P. Grant, Relativistic Quantum Theory of Atoms and Molecules: Theory and Computation, (Springer Science+Business Media, New York 2007).
  * (21) L. D. Landau, E. M. Lifschitz, Quantum Mechanics: Non-relativistic Theory. Course of Theoretical Physics. Vol. 3 (3rd ed.), (London: Pergamon1977).
  * (22) A. Kramida, Yu. Ralchenko, J. Reader and NIST ASD Team (2022). NIST Atomic Spectra Database (ver. 5.10), https://physics.nist.gov/asd. National Institute of Standards and Technology, Gaithersburg, MD. DOI: https://doi.org/10.18434/T4W30F
  * (23) J. C. Mason, Rational approximations to the ordinary Thomas-Fermi function and its derivatives, Math. Proc. Cambridge Philos. Soc. 84, 357-360 (1964).
  * (24) I. M. Band, M. B. Trzhaskovskaya, C. W. Nestor Jr., P. O. Tikkanen, and S. Raman, Dirac-Fock Internal Conversion Coefficients, At. Data Nucl. Data Tables 81, 1 (2002).
  * (25) I. M. Band and M. B. Trzhaskovskaya, Electron-wave-function expansion amplitudes near the origin calculated in the Dirac-Fock-Slater and Dirac-Fock potentials, At. Data Nucl. Data Tables 35, 1 (1986).
  * (26) E. L. Feinberg, Ionization of the atom due to $\beta$-decay, J. Phys. (USSR) 4, 423 (1941).
  * (27) A. Migdal, Ionizacia atomov v $\alpha$\- i $\beta$\- raspade, ZhETF 11, 207-212 (1941) [A. Migdal, Ionization of atoms accompanying $\alpha$\- and $\beta$-decay, J. Phys. Acad. Sci. USSR 4 (1-6), 449-453 (1941)].
  * (28) Z. Ge, T. Eronen, K. S. Tyrin, J. Kotila, J. Kostensalo, D. A. Nesterenko, O. Beliuskina, R. de Groote, A. de Roubin, S. Geldhof, W. Gins, M. Hukkanen, A. Jokinen, A. Kankainen, A. Koszorus, M. I. Krivoruchenko, S. Kujanpaa, I. D. Moore, A. Raggio, S. Rinta-Antila, J. Suhonen, V. Virtanen, A. P. Weaver, and A. Zadvornaya, 159Dy Electron-Capture: A New Candidate for Neutrino Mass Determination, Phys. Rev. Lett. 127, 272301 (2021).
  * (29) V. S. Korolyuk, N. I. Portenko, A. V. Skorokhod, A. F. Turbin, Handbook of Probability Theory and Mathematical Statistics (Moscow, Nauka 1985). 640 pp.


11institutetext: University of Illinois at Urbana-Champaign 11email:
<EMAIL_ADDRESS>https://anwesachoudhuri.github.io/OpenWorldVISCap/

# OW-VISCap: Open-World Video Instance Segmentation and Captioning

Anwesa Choudhuri  Girish Chowdhary  Alexander G. Schwing

###### Abstract

Open-world video instance segmentation is an important video understanding
task. Yet most methods either operate in a closed-world setting, require an
additional user-input, or use classic region-based proposals to identify never
before seen objects. Further, these methods only assign a one-word label to
detected objects, and don’t generate rich object-centric descriptions. They
also often suffer from highly overlapping predictions. To address these
issues, we propose Open-World Video Instance Segmentation and Captioning (OW-
VISCap), an approach to jointly segment, track, and caption previously seen or
unseen objects in a video. For this, we introduce open-world object queries to
discover never before seen objects without additional user-input. We generate
rich and descriptive object-centric captions for each detected object via a
masked attention augmented LLM input. We introduce an inter-query contrastive
loss to ensure that the object queries differ from one another. Our
generalized approach matches or surpasses state-of-the-art on three tasks:
open-world video instance segmentation on the BURST dataset, dense video
object captioning on the VidSTG dataset, and closed-world video instance
segmentation on the OVIS dataset.

Figure 1: OW-VISCap is able to simultaneously detect, track and caption
objects in the given video frames. The first example (top row) shows a road
scene with a previously unseen trailer truck and cars which are seen during
training. The second example (bottom row) shows a person on a lawn mower, and
a dog on the grass. The lawn mower isn’t part of the training set. We generate
meaningful object-centric captions even for objects never seen during
training. The captions for unseen objects are underlined.

##  1 Introduction

Open-world video instance segmentation (OW-VIS) involves detecting, segmenting
and tracking previously seen or unseen objects in a video. This task is
challenging because the objects are often never seen during training, are
occasionally partly or entirely occluded, the appearance and position of these
objects changes over time, and because the objects may leave the scene only to
re-appear at a later time. Addressing these challenges to obtain an accurate
method for OW-VIS that works online is crucial in fields like autonomous
systems, and augmented as well as virtual reality, among others.

Some recent methods based on abstract object queries perform remarkably well
for closed-world video instance segmentation [7, 50, 18, 13]. These works
assume a fixed set of object categories during training and evaluation.
However, it is unrealistic to assume that all object categories are seen
during training. For example, in Fig. 1, the trailer truck (top row)
highlighted in yellow, and the lawn mower (bottom row) highlighted in green,
are never seen before during training.

For this reason, open-world video instance segmentation (OW-VIS) has been
proposed [27, 2, 44, 28, 39, 10]. Current works on OW-VIS suffer from the
following three main issues. Firstly, they often require a prompt, _i.e_.,
additional input from the user, ground-truth or another network. The prompts
can be in the form of points, bounding boxes or text. These methods only work
when the additional inputs are available, making them less practical in the
real-world. Prompt-less OW-VIS methods [27, 2, 44, 28, 39, 10] sometimes rely
on classic region-based object proposals [27, 2, 44, 28], or only operate on
one kind of object query for both the open- and the closed-world [39, 10],
which may result in sub-optimal results (shown later in Tab. 5). Secondly, all
methods on video instance segmentation, closed- or open-world, assign a one-
word label to the detected objects. However, a one word label is often not
sufficient to describe an object. The ability to generate rich object-centric
descriptions is important, especially in the open-world setting. DVOC-DS [58]
jointly addresses the task of closed-world object detection and object-centric
captioning in videos. However, it is not clear how DVOC-DS [58] can be
extended to an open-world setting. Besides, the features from only the
individual object trajectories are used for object-centric captioning in DVOC-
DS [58], so the overall context from the entire video frames may be lost in
this method. DVOC-DS [58] also struggles with very long videos, and cannot
caption multiple action segments within a single object trajectory because the
method produces a single caption for the entire object trajectory. Thirdly,
some of the aforementioned works [8, 7, 13, 18] suffer from multiple similar
object queries resulting in repetitive predictions. Non-maximum suppression,
or other post-processing techniques may be necessary to suppress the
repetitions and highly overlapping false positives.

We address the three aforementioned issues through our Open-World Video
Instance Segmentation and Captioning (OW-VISCap) approach: it simultaneously
detects, segments and generates object-centric captions for objects in a
video. Fig. 1 shows two examples in which our method successfully detects,
segments and captions both closed- and open-world objects.

To address the first issue, our OW-VISCap combines the advantages of both
prompt-based and prompt-less methods. We introduce open-world object queries,
in addition to closed-world object queries used in prior work [8]. This
encourages discovery of never before seen open-world objects without
compromising the closed-world performance much. Notably, we do not require
additional prompts from the ground truth or separate networks. Instead, we use
equally spaced points distributed across the video frames as prompts and
encode them to form open-world object queries, which enables discovery of new
objects. The equally spaced points incorporate information from different
spatial regions of the given video-frames. We also introduce a specifically
tailored open-world loss to train the open-world object queries to discover
new objects.

To address the second issue, OW-VISCap includes a captioning head to produce
an object-centric caption for each object query, both open- and closed-world.
We use masked cross attention in an object-to-text transformer in the
captioning head to generate object-centric text queries, that are then used by
a frozen large language model (LLM) to produce an object-centric caption.
Note, masked attention has been used for closed-world object segmentation [8,
7]. However, to our best knowledge it has not been used for object captioning
before. The masked cross attention helps focus on the local object features,
whereas the self attention in the object-to-text transformer incorporates
overall context by looking at the video-frame features. Moreover, unlike DVOC-
DS [58], we are able to handle long videos and multiple action segments within
a single object trajectory because we process short video clips sequentially
and combine the clips using CAROQ [13].

To address the third issue, we introduce an inter-query contrastive loss for
both open- and closed-world object queries. It encourages the object queries
to differ from one another. This prevents repetitive predictions and
encourages novel object discovery in the open-world. Note that this
contrastive loss also helps in closed-world video instance segmentation by
automatically encouraging non-maximum suppression, and by removing highly
overlapping false positive predictions.

To demonstrate the efficacy of our OW-VISCap on open-world video instance
segmentation and captioning, we evaluate this approach on three diverse and
challenging tasks: open-world video instance segmentation (OW-VIS), dense
video object captioning (Dense VOC), and closed-world video instance
segmentation (VIS). We achieve a performance improvement of $\sim 6\%$ on the
previously unseen (uncommon) categories in the BURST [2] dataset for OW-VIS,
and a $\sim 7\%$ improvement on the captioning accuracy for detected objects
on the VidSTG [57] dataset for the Dense VOC task, while performing similar to
the state-of-the-art on the closed-world VIS task on the OVIS data (our AP
score is $25.4$ as compared to a score of $25.8$ for a recent VIS SOTA, CAROQ
[13]).

Figure 2: The left figure shows an overview of our OW-VISCap (Sec. 3.1). We
introduce open-world object queries $q_{\mathrm{ow}}$ (Sec. 3.2) and a
captioning head (Sec. 3.3). The open-world object queries are generated by
encoding a grid of points along the image-feature dimensions via a prompt
encoder (shown in purple). The right figure details the captioning head (Sec.
3.3) for object-centric captioning. We use masked attention in the object-to-
text transformer of the captioning head.

##  2 Related Work

###  2.1 Open-world Video Instance Segmentation

Prompt-less methods. Several works have explored the open-world video instance
segmentation problem without requiring initial prompts [28, 44, 39, 10, 2].
New objects are discovered based on an objectness score. Some works [28, 2]
rely on classic region-based object proposals. However, recent query-based
methods [39, 10] have shown to outperform those methods for object detection
and segmentation. OW-VISFormer [39] proposes a feature enrichment mechanism
and a spatio-temporal objectness module to distinguish the foreground objects
from the background. Different from our work, they operate on one type of
object query. Also orthogonal to our work is DEVA [10], which develops a
class-agnostic temporal propagation approach to track objects detected by a
segmentation network. UVO [44] and BURST [2] provide novel datasets and
region-proposal-based baselines for the task of open-world video instance
segmentation. In this work, we evaluate our approach on the BURST [2] dataset
as it is more diverse and allows to separately evaluate common and uncommon
classes.

Prompt-based methods. Prompt-based methods rely on prompts, _i.e_., prior
knowledge, to segment objects in videos. Video object segmentation (VOS) [51,
43, 56, 17, 19, 35, 11, 40, 20] is a task where the ground truth segmentations
of objects (belonging to any category) in the first frame are available, and
act as prompts to track the objects throughout the video. CLIPSeg [31] uses
words as prompts, by projecting the words onto the image space following CLIP
[38], and calculating a similarity matrix to obtain a segmentation mask for
the given word prompt. Prompt-based methods work when one has some prior
knowledge on what or where to look for. However, in a true open-world setting,
such prior knowledge may not be available. We operate in such a setting.

In this work, we combine the advantages of both prompt-based and prompt-less
methods. Notably, we do not require additional prompts from the ground truth
or separate networks. Instead, we use equally spaced points distributed over
the video frames and encode them as prompts to discover new objects. Also
note, that all the aforementioned methods, prompt-less or prompt-based, assign
only a one-word label to the detected object. In contrast, we are interested
in generating a rich object-centric caption for the given object in this work.

###  2.2 Dense Video Object Captioning

Dense video object captioning involves detecting, tracking, and captioning
trajectories of objects in a video. For this task, DVOC-DS [58] extends the
image-based GRiT [47] and trains it with a mixture of disjoint tasks. In
addition, existing video grounding datasets VidSTG [57] and VLN [41] are re-
purposed for this task. However, DVOC-DS [58] cannot caption multiple action
segments within a single object trajectory because the method produces a
single caption for the entire object trajectory. In addition, similar to many
other video models [1, 48, 45], DVOC-DS [58] struggles with very long videos
and only processes up to 200 frames. The method is further constrained to
handle only known object categories and it is unclear how the method extends
to an open-world setting.

Unlike DVOC-DS [58], we use open-world object queries to tackle open-world
video instance segmentation. Differently, in this work, we also process video-
frames sequentially using a short temporal context. Hence, our method is able
to process long videos, as well as handle multiple action segments within a
single object trajectory. Finally, we leverage masked attention for dense
video object captioning, which has not been explored before.

###  2.3 Contrastive Loss for Object Queries

Many recent approaches have used a contrastive loss to help in video instance
segmentation. OWVISFormer [39] uses a contrastive loss in the open-world
setting: it ensures that assigned foreground objects are similar to each other
while being different from the background objects. IDOL [50] works in the
closed-world setting, and uses an interframe contrastive loss to ensure object
queries belonging to the same object across frames are similar, and object
queries of different instances across frames differ. In contrast, in this
work, for both the closed- and open-world setting, we use a contrastive loss
to ensure that no two object queries in the foreground are similar to each
other, even in the same frame.

###  2.4 Generalized Video Understanding

Recently, there has been progress in unifying different video related tasks.
TubeFormer [23], Unicorn [53], and CAROQ [13] unify different video
segmentation tasks in the closed world. DVOC-DS [58] unifies the tasks of
detecting closed-world objects in videos and captioning those closed-world
objects. In this work, we explore the task of detecting or segmenting both
closed- and open-world objects in videos, and captioning these objects.

Video understanding starts from a strong generalized image understanding.
Several works generalize multiple image related tasks. Some methods [9, 8, 21]
combine different image segmentation methods, and provide a baseline for many
different video understanding tasks [7, 13, 18, 50]. X-Decoder [59] unifies
different image segmentation tasks along with the task of referring image
segmentation. SAM [24] introduces a vision foundation model, that primarily
performs prompt-based open-world image segmentation, and can be used for many
downstream tasks. Different from these works, we develop a generalized method
for videos that tackles segmentation and object-centric captioning for both
open- and closed-world objects.

###  2.5 Closed-World Video Instance Segmentation

Closed-world video instance segmentation involves simultaneously segmenting
and tracking objects from a fixed category-set in a video. Some works [54, 36,
4, 3, 55, 12, 5, 16, 32, 42, 52] rely on classical region-based proposals.
Recent works [46, 50, 18, 13, 7, 33, 49] rely on query-based proposals and
perform significantly better at discovering closed-world objects. Differently,
in this work, we explore query-based proposals for both the closed- and open-
world setting.

##  3 Approach

Given a video, our goal is to jointly detect, segment and caption object
instances present in the video. Importantly, note that object instance
categories may not be part of the training set (_e.g_., the parachutes shown
in Fig. 3 (top row)), placing our goal in an open-world setting. To achieve
this goal, a given video is first broken into short clips, each consisting of
$T$ frames. Each clip is processed using our approach OW-VISCap. We discuss
merging of the results of each clip in Sec. 4.

We provide an overview of OW-VISCap to process each clip in Sec. 3.1. We then
discuss our contributions: (a) introduction of open-world object queries in
Sec. 3.2, (b) use of masked attention for object-centric captioning in Sec.
3.3, and (c) use of inter-query contrastive loss to ensure that the object
qeries are different from each other in Sec. 3.4. In Sec. 3.5, we discuss the
final training objective.

###  3.1 Overview

Fig. 2 (left) provides an overview of our approach OW-VISCap. To deal with the
open-world setting, _i.e_., to ensure that we can detect, segment and caption
never before seen object instances, we introduce open-world object queries,
$q_{\mathrm{ow}}\in\mathbb{R}^{N_{\mathrm{obj,ow}}\times C}$. These open-world
object queries are in addition to closed-world object queries
$q_{\mathrm{cw}}\in\mathbb{R}^{N_{\mathrm{obj,cw}}\times C}$, commonly used in
prior work [7, 8, 13, 50, 18]. Here, $N_{\mathrm{obj,ow}}$ and
$N_{\mathrm{obj,cw}}$ are the total number of open-world and closed-world
queries, and $C$ refers to the channel dimension for each object query. The
open-world object queries are created by encoding a grid of points, which act
as prompts, through a prompt encoder (shown in purple in Fig. 2 (left)). Note,
that by using open-world object queries, we aim to discover new open-world
objects without needing additional prompts from the user, ground-truth or a
different network. This allows us to combine the prompt-based and prompt-less
methods.

Both open- and closed-world object queries are processed by our specifically
designed captioning head which yields an object-centric caption, a
classification head which yields a category label, and a detection head which
yields either a segmentation mask or a bounding-box.

We compute both sets of object queries (open- and closed-world respectively)
from initial embeddings
$e_{\mathrm{ow}}\in\mathbb{R}^{N_{\mathrm{obj,ow}}\times C}$ and
$e_{\mathrm{cw}}\in\mathbb{R}^{N_{\mathrm{obj,cw}}\times C}$. Concretely, to
generate meaningful open- and closed-world object queries $q_{\mathrm{ow}}$
and $q_{\mathrm{cw}}$, these initial embeddings are continuously modulated in
the object transformer (shown in green) by combining them with multi-level
image features via masked attention following [8, 7]. The closed-world
embeddings $e_{\mathrm{cw}}$ are trainable. We discuss the generation of open-
world embeddings $e_{\mathrm{ow}}$, the computation of open-world object
queries $q_{\mathrm{ow}}$ from open-world embeddings $e_{\mathrm{ow}}$ and
their training in Sec. 3.2.

Fig 2 (right) illustrates our captioning head. It consists of an object-to-
text transformer (shown in blue) and a frozen LLM (shown in gray). We use
$q_{\mathrm{obj}}=[q_{\mathrm{ow}},q_{\mathrm{cw}}]$ to denote all the object
queries obtained from the object transformer. The $i^{\mathrm{th}}$ object
query $q^{i}_{\mathrm{obj}}\in\mathbb{R}^{1\times C}$ is concatenated with
learnt text embeddings
$e_{\mathrm{text}}\in\mathbb{R}^{N_{\mathrm{text}}\times C}$, where
$N_{\mathrm{text}}$ is the total number of text queries. The concatenated
object query and text embeddings are continuously modulated in the object-to-
text transformer by combining them with image features so as to generate
meaningful text queries
$q^{i}_{\mathrm{text}}\in\mathbb{R}^{(N_{\mathrm{text}}+1)\times C}$ for the
$i^{\mathrm{th}}$ object. Note, $q^{i}_{\mathrm{text}}$ is used by the frozen
LLM to generate object-centric captions. The segmentation mask for the
$i^{\mathrm{th}}$ object generated in the detection head is used to mask the
attention in the object-to-text transformer, as described in Sec. 3.3. We find
this design to enable the LLM to generate more object-centric captions.

We introduce an inter-query contrastive loss to ensure that the object queries
are encouraged to differ from each other. We provide details in Sec. 3.4. For
closed world objects, this loss helps in removing highly overlapping false
positives. For open-world objects, it helps in the discovery of new objects.

Finally, we provide the full training objective in Sec. 3.5.

###  3.2 Open-World Object Queries

To help discover new objects, we introduce open-world object queries
$q_{\mathrm{ow}}$ in addition to the commonly used closed world object
queries. Our open-world object queries are generated from open-world
embeddings $e_{\mathrm{ow}}$, which are continuously modulated in the object
transformer by combining them with image features via masked attention,
following [8, 7]. In the following, we discuss how we compute open-world
embeddings $e_{\mathrm{ow}}$ via a prompt encoder, and how we design the
training loss so as to encourage discovery of new objects.

Prompt encoder. The open-world embeddings $e_{\mathrm{ow}}$ are generated in
the prompt encoder. An illustration is provided in Fig. 2 (left). We simply
encode a grid of equally spaced points along the height and width of the
frames of the clip, using the prompt encoder employed in SAM [24]. The use of
equally spaced points encourages the open-world object queries to focus on
different regions of the video frames, encouraging object discovery throughout
the frames. This also encourages the open-world object queries to be diverse
from one another.

Open-world loss. To encourage discovery of new objects, we introduce an open-
world loss $\mathcal{L}_{\mathrm{ow}}$. It differs in one key-aspect from a
classic closed-world loss [7]: we don’t penalize false positive predictions.
We elaborate on details and efficacy of this design next.

We first match the ground truth objects with the open-world predictions by
minimizing a matching cost using the Hungarian algorithm [34]. The optimal
matching is then used to calculate the final open-world loss.

Concretely, consider a set of ground truth objects, padded with a no-object
category $\varnothing$ to equal the number of open-world predictions
$N_{\mathrm{obj,ow}}$. Let us use $\hat{\sigma}_{\mathrm{ow}}$ to represent
the optimal matching between the ground truth objects and the open-world
predictions.

The open-world loss is given by

$\displaystyle\mathcal{L}_{\mathrm{ow}}=\sum_{i}{\mathds{1}_{c^{i}\neq\varnothing}[\mathcal{L}_{\mathrm{cls}}(i,\hat{\sigma}_{\mathrm{ow}}^{i})}+\mathcal{L}_{\mathrm{det}}(i,\hat{\sigma}_{\mathrm{ow}}^{i})],$
(1)

where $i$ is the object index.
$\mathcal{L}_{\mathrm{cls}}(i,\hat{\sigma}_{\mathrm{ow}}^{i})$ is the binary
classification loss between the ground truth object with index $i$ and a
prediction with index $\hat{\sigma}_{\mathrm{ow}}^{i}$. This loss helps in
classifying a prediction as a foreground object or as background.
$\mathcal{L}_{\mathrm{det}}(i,\hat{\sigma}_{\mathrm{ow}}^{i})$ is the
detection loss (mask loss or bounding box loss) between the ground truth
object with index $i$ and a prediction with index
$\hat{\sigma}_{\mathrm{ow}}^{i}$. $c^{i}$ is the class prediction, which is
foreground ($c^{i}\neq\varnothing$) or background $c^{i}=\varnothing$ for the
open-world setting.

Differently, in the classic closed-world setting, the false positive
predictions that are matched with the ground truth no-object category are also
penalized. Concretely, there is an additional component
$\sum_{i}\mathds{1}_{c^{i}=\varnothing}[\mathcal{L}_{\mathrm{cls}}(i,\hat{\sigma}_{\mathrm{cw}}^{i})]$
in the closed-world loss, where $\hat{\sigma}_{\mathrm{cw}}$ refers to the
optimal matching between the ground truth objects and the closed-world
predictions and $\mathcal{L}_{\mathrm{cls}}(i,\hat{\sigma}_{\mathrm{cw}}^{i})$
is the binary classification loss between the ground truth object with index
$i$ and a prediction with index $\hat{\sigma}_{\mathrm{cw}}^{i}$. This differs
from the aforementioned open-world setting, where the false positives are
important as they act as newly discovered objects.

###  3.3 Captioning Head

To obtain an object-centric caption, object queries
$q_{\mathrm{obj}}=[q_{\mathrm{ow}},q_{\mathrm{cw}}]$, concatentating the open-
world object queries discussed in the previous section and classic closed-
world object queries, are processed via our captioning head. Fig 2 (right)
illustrates the captioning head. It consists of an object-to-text transformer
and a frozen LLM.

To compute a caption, the $i^{\mathrm{th}}$ object query
$q^{i}_{\mathrm{obj}}$ is concatenated with learnt text embeddings
$e_{\mathrm{text}}$. The concatenated result is continuously modulated in the
object-to-text transformer by combining it with image features to generate
meaningful text queries $q^{i}_{\mathrm{text}}$ for the $i^{\mathrm{th}}$
object. Subsequently, $q^{i}_{\mathrm{text}}$ is used by the frozen LLM to
generate object-centric captions. The segmentation mask for the
$i^{\mathrm{th}}$ object generated in the detection head is used to mask the
attention in the object-to-text transformer. We find this to enable the LLM to
generate more object-centric captions. We provide details next.

Masked attention for object-centric captioning. Masked attention involves
attending within the foreground region of the predicted mask for each object
query. Concretely, for each object $i$, we restrict the cross attention in the
object-to-text transformer by using the segmentation mask of the object
generated by the detection head to generate object-centric text queries
$q^{i}_{\mathrm{text}}$. Intuitively, this enables the model to focus on local
object-centric features, which are sufficient to update the text queries.
Importantly, note that context information from the video frames can be
gathered through the self-attention layers. The proposed design hence doesn’t
take away any information. It rather provides the same information in clearly
separated layers.

Formally, for the $i^{\mathrm{th}}$ object we compute the query features
$X^{\mathrm{cap},i}_{l}\in\mathbb{R}^{(N_{\mathrm{text}}+1)\times C}$ obtained
from the $l^{\mathrm{th}}$ object-to-text transformer layer via

$\displaystyle
X^{\mathrm{cap},i}_{l}=\mathrm{softmax}(\mathcal{M}^{i}+Q^{i}_{l}K^{T}_{l})V_{l}+X^{\mathrm{cap},i}_{l-1}.$
(2)

Here, $\mathcal{M}^{i}\in\\{0,-\infty\\}^{1\times HWT}$ is the attention mask
in the object-to-text transformer such that at feature location $(x,y)$,
$\mathcal{M}^{i}(x,y)=0$, if $M^{i}(x,y)=1$ and
$\mathcal{M}^{i}(x,y)=-\infty$, if $M^{i}(x,y)=0$. $M^{i}$ is the binary mask
obtained from the detection head. Moreover,
$K_{l},V_{l}\in\mathbb{R}^{HWT\times C}$ are the linearly transformed image
features. To initialize, we let
$X^{\mathrm{cap},i}_{0}=[q^{i}_{\mathrm{obj}},e_{\mathrm{text}}]$, where
$q^{i}_{\mathrm{obj}}\in\mathbb{R}^{1\times C}$ is the $i^{\mathrm{th}}$
object query obtained from the object transformer and
$e_{\mathrm{text}}\in\mathbb{R}^{N_{\mathrm{text}}\times C}$ are the learnt
text embeddings shown in Fig. 2 and introduced in Sec. 3.1. $C$ is the channel
dimension, $H,W$ are the spatial dimensions of the image features and $T$ is
the clip-length. $N_{\mathrm{text}}$ is the total number of text queries.

We’ll show in Sec. 4.3 that the masked attention helps in generating more
object-centric captions.

###  3.4 Inter-Query Contrastive Loss

We introduce an inter-query contrastive loss $\mathcal{L}_{\mathrm{cont}}$ to
ensure that the object queries are different from each other, as described
next. For closed-world objects, this loss helps in removing highly overlapping
false positives. For open-world objects, it helps in the discovery of new
objects. Formally,

$\displaystyle\mathcal{L}_{\mathrm{cont}}=-\sum_{i,j}{L(q^{i}_{\mathrm{obj}},q^{j}_{\mathrm{obj}})},$
(3)

where $q^{i}_{\mathrm{obj}}$ and $q^{j}_{\mathrm{obj}}$ are the
$i^{\mathrm{th}}$ and the $j^{\mathrm{th}}$ objects and $i\neq j$. $L$ refers
to the L1 distance. Via this loss, we _maximize_ the L1 distance between the
object queries, _i.e_., we encourage that the object queries differ from each
other.

###  3.5 Training

Our total training loss is

$\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{ow}}+\mathcal{L}_{\mathrm{cont}}+\mathcal{L}_{\mathrm{cap}}+\mathcal{L}_{\mathrm{cw}}.$

Here, $\mathcal{L}_{\mathrm{ow}}$ is the open-world loss as discussed in Sec.
3.2 and $\mathcal{L}_{\mathrm{cont}}$ is the inter-query contrastive loss
discussed in Sec. 3.4. $\mathcal{L}_{\mathrm{cap}}$ is the standard captioning
loss introduced by DVOC-DS [58]. $\mathcal{L}_{\mathrm{cw}}$ is the closed-
world loss following prior work [8]. In order to compute
$\mathcal{L}_{\mathrm{cw}}$, the ground truth objects are first matched with
the closed-world objects; the optimal matching is used to compute the final
closed-world loss $\mathcal{L}_{\mathrm{cw}}$. This closed-world loss involves
penalizing the false positive predictions, which is different from
$\mathcal{L}_{\mathrm{ow}}$. Note that in our setting, the ground truth
objects are matched twice, once with the open-world object queries to compute
$\mathcal{L}_{\mathrm{ow}}$, and once with the closed-world object queries to
compute $\mathcal{L}_{\mathrm{cw}}$.

##  4 Experiments

We evaluate our proposed approach on the diverse tasks of open-world video
instance segmentation (OW-VIS), dense video object captioning (Dense VOC), and
closed-world video instance segmentation (VIS). Note that there is no
dedicated dataset for the task of open-world video instance segmentation and
captioning. Hence we use the three aforementioned tasks and evaluate the three
different aspects of our approach: open-world capability, video object
captioning, and video instance segmentation. In the following subsections, we
first discuss the datasets and evaluation metrics used in our evaluation in
Sec. 4.1. We then compare our performances to baselines in Sec. 4.2. We
demonstrate how each of our contributions results in better performance
through an ablation study in Sec. 4.3. Finally we show some qualitative
results in Sec. 4.4.

Table 1: Open-world tracking accuracy (OWTA) on the BURST validation and test sets for all, common (comm.) and uncommon (unc.) categories of objects. Onl. refers to online frame-by-frame processing. The best scores are highlighted in bold font, and the second-best scores are underlined. Method | Validation (OWTA) |  Test (OWTA)
---|---|---
| all | comm. | unc. | all | comm. | unc.
OWTB [28] | 55.8 | 59.8 | 38.8 | 56.0 | 59.9 | 38.3
Mask2Former [8]+STCN [11] | 64.6 | 71.0 | 25.0 | 57.5 | 62.9 | 23.9
Mask2Former [8]+DEVA [10] (onl.) | 69.5 | 74.6 | 42.3 | 70.1 | 75.0 | 44.1
EntitySeg [37]+DEVA [10] (onl.) | 68.8 | 72.7 | 49.6 | 69.5 | 72.9 | 53.0
OW-VISCap (ours) + DEVA [10] (onl.) | 69.0 | 73.5 | 55.2 | 69.2 | 72.4 | 57.2
Table 2: Dense video object captioning results on the VidSTG [57] dataset. Off. indicates offline methods and onl. refers to online methods. Method | Mode | CHOTA | DetA | AssA | CapA | APM
---|---|---|---|---|---|---
DVOC-DS (joint training) [58] | off. | 51.6 | 65.5 | 56.9 | 36.8 | 69.3
DVOC-DS (disjoint training) [58] | off. | 28.0 | 45.9 | 48.0 | 10.0 | 39.8
OW-VISCap (ours) | onl. | 53.0 | 60.1 | 54.0 | 43.9 | 62.6
Table 3: Results on the OVIS [36] validation data. All methods use the ResNet-50 backbone. The best performing methods are highlighted in bold font. The second best methods are underlined. Method | AP | AP50 | AP75 | AR1 | AR10
---|---|---|---|---|---
MaskTrack [54] | 10.8 | 25.3 | 8.5 | 7.9 | 14.9
DeVIS [6] | 23.7 | 47.6 | 20.8 | 12.0 | 28.9
MinVIS [18] | 25.0 | 45.5 | 24.0 | 13.9 | 29.7
VMT [22] | 16.9 | 36.4 | 13.7 | 10.4 | 22.7
InstanceFormer [25] | 20.0 | 40.7 | 18.1 | 12.0 | 27.1
VITA [15] | 19.6 | 41.2 | 17.4 | 11.7 | 26.0
CAROQ [13] | 25.8 | 47.9 | 25.4 | 14.2 | 33.9
OW-VISCap (w/o $\mathcal{L}_{\mathrm{cont}}$) | 23.2 | 45.2 | 21.7 | 13.5 | 30.1
OW-VISCap (ours) | 25.4 | 48.8 | 22.8 | 14.3 | 32.8

Table 4: Ablation on the BURST [2] validation data. ‘w/o $q_{\mathrm{ow}}$’ refers to without open-world object queries; ‘w/o $\mathcal{L}_{\mathrm{cont}}$’ refers to without contrastive loss. Method | OWTA
---|---
| all | comm. | unc.
Ours | 55.5 | 58.2 | 43.8
w/o $q_{\mathrm{ow}}$ | 54.3 | 57.9 | 41.2
w/o $\mathcal{L}_{\mathrm{cont}}$ | 53.5 | 56.1 | 41.6

Table 5: Ablation on the VidSTG [57] data. ‘w/o m.a.’ refers to without masked attention. ‘bb. cap.’ and ‘en. bb. cap.’ refers to bounding box captioning and enlarged bounding box captioning. Method | CHOTA | DetA | AssA | CapA
---|---|---|---|---
Ours | 51.0 | 56.1 | 54.0 | 43.9
w/o m.a. | 39.5 | 56.1 | 54.0 | 20.3
bb. cap. | 48.9 | 56.1 | 54.0 | 38.6
en. bb. cap. | 49.4 | 56.1 | 54.0 | 40.4

###  4.1 Datasets and Evaluation Metrics

We evaluate our approach on the OW-VIS, Dense VOC and VIS tasks. For OW-VIS,
we use the challenging BURST dataset [2]. For the Dense VOC task, we use the
VidSTG dataset [57]. Note that VidSTG [57] has bounding box and tracking
identity for all objects, but captions are not exhaustively provided. However,
DVOC-DS [58] uses VidSTG [57] for the Dense VOC task by removing the
captioning loss for missing captions during training and not evaluating the
missing captions during evaluation. We follow a similar setting. For the VIS
task, we evaluate on the OVIS dataset [36].

For OW-VIS, we use the standard evaluation metrics of open-world tracking
accuracy (OWTA) [2] for all, common and uncommon categories. For Dense VOC, we
use the captioned higher order tracking accuracy (CHOTA) [58], which depends
on the detection accuracy (DetA), association accuracy (AssA), and captioning
accuracy (CapA). We also report the frame-based METEOR score (APM). For VIS,
we use the standard evaluation metrics of average precision (AP, AP50, AP75)
and average recall (AR1, AR10) [54].

Figure 3: Example from the BURST validation data. The masks are superimposed
on the objects. The top-row shows examples of parachutes in the air and people
on the grass. The parachutes belong to the uncommon object category, _i.e_.,
parachutes were never seen during training. Our approach is able to detect and
retain the identity of the blue and the green parachutes as the green crosses
the blue parachute. The bottom row shows a person unboxing a leaf blower. The
carton of the leaf blower (gray mask), the leaf blower (maroon mask) and the
plastic wrapper (pink mask) are never seen during training. We are able to
consistently detect and track them along with the person (common object
category during training). Figure 4: An example from the VidSTG data. Our
approach is able to detect and track objects in the scene consistently and to
generate meaningful object-centric captions for each of the detected objects.

###  4.2 Main Results

Open-world video instance segmentation (BURST). Tab. 1 shows our results for
the OW-VIS task on the BURST dataset [2]. We report the open-world tracking
accuracy for all, common (comm.) and uncommon (unc.) categories. For the
uncommon classes, we achieve the state-of-the-art, improving upon the next
best method (Mask2Former [8]+DEVA [10]) by $\sim 6$ points on the BURST [2]
validation data and by $\sim 4$ points on the BURST test data. For the common
categories, our method ranks $2^{\mathrm{nd}}$ in the BURST validation data.
We use a SwinL [29] backbone, a clip-length of $T=1$, and DEVA [10] for the
temporal association of objects.

Dense video object captioning (VidSTG). Tab. 2 shows our results on the Dense
VOC task. We outperform DVOS-DS [58] on the captioning accuracy (CapA),
demonstrating that our captioning head with masked attention (Sec. 3.3) is
effective in generating object-centric captions. We improve upon DS-VOC on the
overall CHOTA metric, even though we slightly underperform on DetA and AssA.
Note that DVOS-DS is an offline method: the entire object trajectories are
used for generating the captions. Hence DVOS-DS cannot process videos with
more than $200$ frames. This is in contrast to our online method, where we
sequentially process short video clips of length $T=2$. This enables to
process very long videos. DVOS-DS uses a ViT [14] backbone, whereas we use
SwinL [29], which leads to a difference in DetA scores. We use a clip-length
$T=2$ for this experiment. Note that $T=2$ enables the object queries to be
spatio-temporally rich. This helps in generating better object-centric
captions. For tracking, we use CAROQ [13] to propagate object queries across
frames.

Video instance segmentation (OVIS). Tab. 3 shows our results for the closed-
world video instance segmentation task on the OVIS [36] dataset. In the
closed-world setting, we disable the open-world object queries. We notice that
the contrastive loss $\mathcal{L}_{\mathrm{cont}}$ (discussed in Sec. 3.4)
improves the closed-world results. We use a clip-length of $T=2$ in this
setting and CAROQ [13] to combine results from video clips.

###  4.3 Ablation Studies

Open-world object queries. Tab. 5 (first and second rows) shows how the open-
world object queries $q_{\mathrm{ow}}$, described in Sec. 3.2, help in
discovering new objects. In Tab. 5, ‘w/o $q_{\mathrm{ow}}$’ refers to the
setting without open-world object queries, _i.e_., we only use the closed-
world object queries to find the common (comm.) and uncommon (unc.) objects.
We observe that the performance drops by $2.6$ for uncommon categories and by
$0.3$ points for common categories in ‘w/o $q_{\mathrm{ow}}$’ as compared to
‘Ours’, where both the open- and closed-world object queries are used.

Masked attention. Tab. 5 shows that masked attention in the object-to-text
transformer of the captioning head, described in Sec. 3.3, helps in object-
centric captioning using the VidSTG [57] data.

The second row ‘w/o m.a.’ of Tab. 5 refers to the setting without masked
attention, _i.e_., the entire image-feature is used to calculate the cross-
attention in the object-to-text-transformer. The object-centric context is
only accumulated by concatenating the $i^{\mathrm{th}}$ object query with the
learnt text embeddings, as discussed in Sec. 3.3 and shown in Fig. 2. We
observe that the captioning accuracy CapA drops by $23$ points, indicating
that concatenating the object query with the text embeddings is not sufficient
for an object-centric focus.

The third row in Tab. 5, ‘bb. cap.’ (bounding box captioning), pursues the
opposite setting. Here, the images are cropped based on the object bounding
box predictions in the detection head. The cropped images are directly used
for captioning, ensuring that both the self and cross attention blocks in the
object-to-text transformer operate on object-centric features. Note, that we
don’t use masked attention in this setting. We see a drop in CapA of $5$
points in this setting. Although the cropping helps in retaining the object-
centric information, the overall context from the entire image is missing.

The fourth row in Tab. 5, ‘en. bb. cap.’ (enlarged bounding box captioning),
shows a similar setting as the third row, but the bounding boxes are first
enlarged by $10\%$ to provide more overall image context. The enlarged
bounding boxes are then used to crop the images for captioning. We observe a
drop in CapA of $3$ points in this setting, indicating that enlarging the
bounding boxes helps but is not sufficient to provide overall context.

The first row in Tab. 5, ‘Ours’ (our approach), where we retain the overall
context using self attention in the object-to-text transformer and focus on
the object-centric features using masked cross attention, performs the best
among these settings.

Contrastive loss. Tab. 5 (first and last rows) shows how the contrastive loss
$\mathcal{L}_{\mathrm{cont}}$, described in Sec. 3.4, helps in detecting both
the common (comm.) and uncommon (unc.) categories of objects. The performance
drops by $\sim 2$ points for both the common and uncommon categories for the
setting ‘w/o $\mathcal{L}_{\mathrm{cont}}$’, _i.e_., when the contrastive loss
is not used. The contrastive loss helps in removing highly overlapping false
positives in the closed-world setting and in discovering new objects in the
open-world setting. Tab. 3 further shows that the contrastive loss helps in
detecting objects in the closed-world setting.

###  4.4 Qualitative Results

We provide qualitative results generated by OW-VISCap in this section. Fig. 1
shows an example from the BURST [2] dataset. OW-VISCap is able to
simultaneously detect, track and caption objects in the given video frames.
The objects belong to both the open- and closed-world. Note that the BURST [2]
dataset doesn’t have object-centric captions available for training or
evaluation, hence our captioning head was not trained on BURST [2]. We train
the captioning head on the Dense VOC task (whose results are shown in Tab. 2).
We find this captioning-head to be effective in generating meaningful object-
centric captions even for objects never seen during training. Fig. 3 shows two
examples from the BURST validation data. We are able to consistently detect,
segment, and track the previously seen and unseen objects. Fig. 4 shows an
example from the VidSTG [57] data. Our approach is able to detect and track
objects in the scene consistently, and to generate meaningful object-centric
captions for each of the detected objects.

##  5 Conclusion

In this work, we introduce OW-VISCap to _jointly detect, segment, track, and
caption previously seen or unseen objects in videos_. We introduce open-world
object queries to encourage discovery of previously unseen objects without the
need of additional user-inputs. Instead of assigning a fixed label to detected
objects, we generate rich object-centric captions for each object by using
masked attention in an object-to-text transformer. Further, we introduce an
inter-query contrastive loss to ensure that the object queries differ from one
another. Our approach mostly matches or surpasses the state-of-the-art on the
diverse tasks of open-world video instance segmentation on the BURST dataset,
dense video object captioning on the VidSTG dataset, and closed-world video
instance segmentation on the OVIS dataset. Through an ablation study, we
demonstrate the effectiveness of each of our proposed components.

Acknowledgements: This work is supported in party by Agriculture and Food
Research Initiative (AFRI) grant no. 2020-67021-32799/project accession no.
1024178 from the USDA National Institute of Food and Agriculture: NSF/USDA
National AI Institute: AIFARMS. We also thank the Illinois Center for Digital
Agriculture for seed funding for this project. Work is also supported in part
by NSF under Grants 2008387, 2045586, 2106825, MRI 1725729.

## References

  * [1] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: ICCV (2021)
  * [2] Athar, A., Luiten, J., Voigtlaender, P., Khurana, T., Dave, A., Leibe, B., Ramanan, D.: Burst: A benchmark for unifying object recognition, segmentation and tracking in video. In: WACV (2023)
  * [3] Athar, A., Mahadevan, S., Osep, A., Leal-Taixé, L., Leibe, B.: Stem-seg: Spatio-temporal embeddings for instance segmentation in videos. In: ECCV (2020)
  * [4] Bertasius, G., Torresani, L.: Classifying, segmenting, and tracking object instances in video with mask propagation. In: ICCV (2020)
  * [5] Braso, G., Leal-Taixe, L.: Learning a neural solver for multiple object tracking. In: CVPR (June 2020)
  * [6] Caelles, A., Meinhardt, T., Brasó, G., Leal-Taixé, L.: Devis: Making deformable transformers work for video instance segmentation. arXiv preprint arXiv:2207.11103 (2022)
  * [7] Cheng, B., Choudhuri, A., Misra, I., Kirillov, A., Girdhar, R., Schwing, A.G.: Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764 (2021)
  * [8] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: CVPR (2022)
  * [9] Cheng, B., Schwing, A.G., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. In: NeurIPS (2021)
  * [10] Cheng, H.K., Oh, S.W., Price, B., Schwing, A., Lee, J.Y.: Tracking anything with decoupled video segmentation. In: ICCV (2023)
  * [11] Cheng, H.K., Tai, Y.W., Tang, C.K.: Rethinking space-time networks with improved memory coverage for efficient video object segmentation. In: NeurIPS (2021)
  * [12] Choudhuri, A., Chowdhary, G., Schwing, A.G.: Assignment-space-based multi-object tracking and segmentation. In: ICCV (2021)
  * [13] Choudhuri, A., Chowdhary, G., Schwing, A.G.: Context-aware relative object queries to unify video instance and panoptic segmentation. In: CVPR (2023)
  * [14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [15] Heo, M., Hwang, S., Oh, S.W., Lee, J.Y., Kim, S.J.: Vita: Video instance segmentation via object token association. arXiv preprint arXiv:2206.04403 (2022)
  * [16] Hornakova, A., Henschel, R., Rosenhahn, B., Swoboda, P.: Lifted disjoint paths with application in multiple object tracking. In: ICML (2020)
  * [17] Hu, Y.T., Huang, J.B., Schwing, A.G.: Videomatch: Matching based video object segmentation. In: ECCV (2018)
  * [18] Huang, D.A., Yu, Z., Anandkumar, A.: Minvis: A minimal video instance segmentation framework without video-based training. In: NeurIPS (2022)
  * [19] Huang, X., Xu, J., Tai, Y.W., Tang, C.K.: Fast video object segmentation with temporal aggregation network and dynamic template matching. In: CVPR (2020)
  * [20] Huang, X., Xu, J., Tai, Y.W., Tang, C.K.: Fast video object segmentation with temporal aggregation network and dynamic template matching. In: CVPR (2020)
  * [21] Jain, J., Li, J., Chiu, M.T., Hassani, A., Orlov, N., Shi, H.: Oneformer: One transformer to rule universal image segmentation. In: CVPR (2023)
  * [22] Ke, L., Ding, H., Danelljan, M., Tai, Y.W., Tang, C.K., Yu, F.: Video mask transfiner for high-quality video instance segmentation. In: ECCV (2022)
  * [23] Kim, D., Xie, J., Wang, H., Qiao, S., Yu, Q., Kim, H.S., Adam, H., Kweon, I.S., Chen, L.C.: Tubeformer-deeplab: Video mask transformer. In: ICCV (2022)
  * [24] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [25] Koner, R., Hannan, T., Shit, S., Sharifzadeh, S., Schubert, M., Seidl, T., Tresp, V.: Instanceformer: An online video instance segmentation framework. arXiv preprint arXiv:2208.10547 (2022)
  * [26] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [27] Liu, Y., Zulfikar, I.E., Luiten, J., Dave, A., Ramanan, D., Leibe, B., Ošep, A., Leal-Taixé, L.: Opening up open world tracking. In: CVPR (2022)
  * [28] Liu, Y., Zulfikar, I.E., Luiten, J., Dave, A., Ramanan, D., Leibe, B., Ošep, A., Leal-Taixé, L.: Opening up open world tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19045–19055 (2022)
  * [29] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV (2021)
  * [30] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [31] Lüddecke, T., Ecker, A.: Image segmentation using text and image prompts. In: CVPR (2022)
  * [32] Luiten, J., Fischer, T., Leibe, B.: Track to reconstruct and reconstruct to track. RAL (2020)
  * [33] Meinhardt, T., Kirillov, A., Leal-Taixe, L., Feichtenhofer, C.: Trackformer: Multi-object tracking with transformers. In: CVPR (2022)
  * [34] Munkres, J.: Algorithms for the assignment and transportation problems. J-SIAM (1957)
  * [35] Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Video object segmentation using space-time memory networks. In: ICCV (2019)
  * [36] Qi, J., Gao, Y., Hu, Y., Wang, X., Liu, X., Bai, X., Belongie, S., Yuille, A., Torr, P.H., Bai, S.: Occluded video instance segmentation. arXiv preprint arXiv:2102.01558 (2021)
  * [37] Qi, L., Kuen, J., Wang, Y., Gu, J., Zhao, H., Torr, P., Lin, Z., Jia, J.: Open world entity segmentation. PAMI (2022)
  * [38] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
  * [39] Thawakar, O., Narayan, S., Cholakkal, H., Anwer, R.M., Khan, S., Laaksonen, J., Shah, M., Khan, F.S.: Video instance segmentation in an open-world. arXiv preprint arXiv:2304.01200 (2023)
  * [40] Voigtlaender, P., Chai, Y., Schroff, F., Adam, H., Leibe, B., Chen, L.C.: Feelvos: Fast end-to-end embedding learning for video object segmentation. In: CVPR (2019)
  * [41] Voigtlaender, P., Changpinyo, S., Pont-Tuset, J., Soricut, R., Ferrari, V.: Connecting vision and language with video localized narratives. In: CVPR (2023)
  * [42] Voigtlaender, P., Krause, M., Os̆ep, A., Luiten, J., Sekar, B.B.G., Geiger, A., Leibe, B.: MOTS: Multi-object tracking and segmentation. In: CVPR (2019)
  * [43] Wang, Q., Zhang, L., Bertinetto, L., Hu, W., Torr, P.H.: Fast online object tracking and segmentation: A unifying approach. In: CVPR (2019)
  * [44] Wang, W., Feiszli, M., Wang, H., Tran, D.: Unidentified video objects: A benchmark for dense, open-world segmentation. In: ICCV (2021)
  * [45] Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., Xia, H.: End-to-end video instance segmentation with transformers. In: CVPR (2021)
  * [46] Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., Xia, H.: End-to-end video instance segmentation with transformers. In: CVPR (2021)
  * [47] Wu, J., Wang, J., Yang, Z., Gan, Z., Liu, Z., Yuan, J., Wang, L.: Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280 (2022)
  * [48] Wu, J., Jiang, Y., Sun, P., Yuan, Z., Luo, P.: Language as queries for referring video object segmentation. In: CVPR (2022)
  * [49] Wu, J., Jiang, Y., Zhang, W., Bai, X., Bai, S.: Seqformer: a frustratingly simple model for video instance segmentation. arXiv preprint arXiv:2112.08275 (2021)
  * [50] Wu, J., Liu, Q., Jiang, Y., Bai, S., Yuille, A., Bai, X.: In defense of online models for video instance segmentation. In: ECCV (2022)
  * [51] Xu, N., Yang, L., Fan, Y., Yang, J., Yue, D., Liang, Y., Price, B., Cohen, S., Huang, T.: Youtube-vos: Sequence-to-sequence video object segmentation. In: ECCV (2018)
  * [52] Xu, Z., Zhang, W., Tan, X., Yang, W., Huang, H., Wen, S., Ding, E., Huang, L.: Segment as points for efficient online multi-object tracking and segmentation. In: ECCV (2020)
  * [53] Yan, B., Jiang, Y., Sun, P., Wang, D., Yuan, Z., Luo, P., Lu, H.: Towards grand unification of object tracking. In: ECCV (2022)
  * [54] Yang, L., Fan, Y., Xu, N.: Video instance segmentation. In: ICCV (2019)
  * [55] Yang, S., Fang, Y., Wang, X., Li, Y., Fang, C., Shan, Y., Feng, B., Liu, W.: Crossover learning for fast online video instance segmentation. In: ICCV (2021)
  * [56] Zhang, L., Lin, Z., Zhang, J., Lu, H., He, Y.: Fast video object segmentation via dynamic targeting network. In: ICCV (2019)
  * [57] Zhang, Z., Zhao, Z., Zhao, Y., Wang, Q., Liu, H., Gao, L.: Where does it exist: Spatio-temporal video grounding for multi-form sentences. In: CVPR (2020)
  * [58] Zhou, X., Arnab, A., Sun, C., Schmid, C.: Dense video object captioning from disjoint supervision. arXiv preprint arXiv:2306.11729 (2023)
  * [59] Zou, X., Dou, Z.Y., Yang, J., Gan, Z., Li, L., Li, C., Dai, X., Behl, H., Wang, J., Yuan, L., et al.: Generalized decoding for pixel, image, and language. In: CVPR (2023)

## Supplementary Material — OW-VISCap: Open-World Video Instance Segmentation
and Captioning

Figure S1: Results on the BURST dataset. We successfully segment and track
closed-world objects (e.g., persons), and open-world objects (e.g., rackets),
throughout the video.

This is the supplementary material of the OW-VISCap paper. We develop OW-
VISCap, an approach for open-world video instance segmentation and captioning.
We test our approach on the task of open-world video instance segmentation
(OW-VIS), dense video object captioning (Dense VOC) and closed-world video
instance segmentation (VIS) in Sec. 4.

In this supplementary material, we provide additional analysis (Sec. 0.A) to
support the contributions we made in Sec. 3 of the main paper. We then discuss
the implementation details (Sec. 0.B) and limitations (Sec. 0.C) of our
approach.

Fig. S1 shows additional results on the BURST [2] dataset. We successfully
segment and track the closed-world objects: persons, and the open-world
objects: rackets, throughout the video. Fig. S2 shows results on the VidSTG
[57] dataset. The detected objects are tracked throughout the video and our
method generates meaningful captions for each object.

Figure S2: Results on the VidSTG dataset. Our approach detects, tracks and
generates meaningful object-centric captions for each object throughout the
video.

##  Appendix 0.A Additional Analysis

In this section we provide additional analysis on the different components
discussed in Sec. 3 of the main paper.

###  0.A.1 Open-World Embeddings as Object Proposals

We introduced open-world embeddings $e_{\mathrm{ow}}$ in Sec. 3.2 of the main
paper. These embeddings are modulated in the object transformer (Fig. 2) to
generate open-world object queries. We obtain the open-world embeddings by
encoding a grid of equally spaced points across the feature dimensions through
a prompt encoder. This encourages object discovery throughout the video-frame.
The open-world embeddings act as initial abstract object proposals.

In Fig. S3, we show that the open-world embeddings are strong object
proposals, even before they are modulated by the object transformer by being
combined with video frame features. The person, the spoon, the plate, and some
food on the plate are discovered by the open-world embeddings. Their
segmentation masks are obtained by the dot product between the open-world
embeddings and the video frame features. Further, we see a strong spatial
correlation between the grid of points and the segmentation masks generated by
the corresponding open-world embeddings. This suggests that encoding a grid of
points throughout the feature dimensions encourages object discovery
throughout the video frames.

###  0.A.2 Masked Attention for Object-Centric Captioning

In Tab. 5, we quantitatively show that masked-attention (Sec. 3.3) helps in
generating accurate object-centric captions for individual objects. Fig. S4
shows the high quality of the object-centric captions generated. The black
caption (‘a family sitting on a couch with a child’) is obtained when no mask
is provided in the object-to-text transformer. The entire image features are
seen during the cross attention operation in each layer of the object-to-text
transformer. The caption fails to capture the object-centric details.

The colored captions are generated with masked-attention for the individual
objects. The colored captions on the left clearly highlight the effectiveness
of masked attention to generate object-centric captions. For example, the
three persons (highlighted in cyan, grayish blue, and green) have distinct
captions, each one describing the individual identities of the corresponding
person. The school bag (light blue) is also described correctly. We want to
note that sometimes the method fails to generate meaningful object-centric
captions for small objects (captions on the right). We discuss this more in
Sec. 0.C.

Figure S3: Open-world embeddings act as strong object proposals. Further, a
strong spatial correlation exists between the grid of points and the
segmentation masks generated by the corresponding open-world embeddings. This
is highlighted by the color of the points in the grid, and the color of the
segmentation masks. Figure S4: Masked attention helps generate object-centric
captions (colored captions on the left). Providing no mask during cross
attention fails to capture object centric details (black caption). However,
masked-attention sometimes fails to generate meaningful object-centric
captions for small objects (colored captions on the right). Figure S5: The
repetitative predictions without contrastive loss (top-right) are highlighted
with red and cyan boxes. Contrastive loss (bottom-right) helps in suppressing
these repetitions. Figure S6: A failure mode, where the object identities
aren’t retained after prolonged occlusion. After a train crosses the screen
for a prolonged period of time ($\sim 30$ frames), the person initially
identified as blue is identified as green, and the bag is later identified as
light green instead of dark blue. The last caption for the bag is also
erroneous.

###  0.A.3 Contrastive Loss to Suppress Overlapping Predictions

In Tab. 3 and Tab. 5 of the main paper, we demonstrate the effectiveness of
using the contrastive loss $\mathcal{L}_{\mathrm{cont}}$ (discussed in Sec.
3.4) for object detection. This loss encourages that object queries differ
from each other, among others by suppressing highly overlapping predictions.
We highlight this in Fig. S5. The left image shows a frame from the OVIS [36]
dataset. The top-right and bottom-right images show a few predictions from our
network trained without (top) and trained with (bottom) the contrastive loss.
The repetitive predictions for the top-right image are highlighted with red
and cyan boxes. The contrastive loss helps in removing these repetitions.

##  Appendix 0.B Implementation Details

In this section, we provide the implementation details of OW-VISCap. We first
describe the architecture of different components discussed in Sec. 3. We then
discuss our choice of hyper-parameters for all experiments discussed in Sec.
4. We also discuss the resources and the licenses of the code-bases and
datasets used in the paper.

###  0.B.1 Architecture

Prompt encoder. Our prompt encoder discussed in Sec. 3.2 is a lightweight
network, following the architecture of the prompt encoder used in SAM [24]. We
initialize our prompt encoder from SAM [24] and fine-tune it on the BURST [2]
dataset.

Captioning head. Our captioning head discussed in Sec. 3.3 consists of an
object-to-text transformer and a frozen large language model (LLM). The
object-to-text transformer decoder has $11$ transformer layers. Each layer has
a self attention, masked cross attention (discussed in Sec. 3.3) operations,
followed by a feed forward network. The object-to-text transformer and the
text embeddings $e_{\mathrm{text}}$ are initialized from BLIP-2 [26]. We use a
frozen OPT-2.7B model as the LLM. We fine-tune the object-to-text transformer
and the text embeddings on the VidSTG [57] dataset.

###  0.B.2 Hyper-Parameters

We now discuss the hyper-parameters used in this work. For experiments on the
BURST [2] dataset, we encode a grid of $7\times 7$ point across the width and
height of the image features to obtain the open-world embeddings
$e_{\mathrm{ow}}$ discussed in Sec. 3.2. Hence the total number of open-world
object queries $N_{\mathrm{obj,ow}}$ is $49$. We also experimented with a grid
of $4\times 4$ and $10\times 10$, but didn’t see a significant change in
performance. For experiments on the VidSTG [57] dataset, the number of text
embeddings $e_{\mathrm{text}}$ is $32$. In all experiments, the maximum number
of closed world objects ($N_{\mathrm{obj,cw}}$) in a given video for a
ResNet-50 backbone is $100$, and for a Swin-L backbone is $200$. We use a
feature dimension $C$ (Sec. 3.1) of $256$ in all models.

We trained the models with an initial learning rate of $0.0001$ and ADAMW [30]
optimizer with a weight decay of $0.05$. We use a batch size of $8$. The
networks were first initialized with weights from Mask2Former [8] trained on
the COCO image instance segmentation dataset. We then fine-tune the models on
the respective BURST [2], VidSTG [57], and OVIS [36] datasets for $10,000$,
$16,000$ and $8,000$ iterations respectively.

###  0.B.3 Resources

We used $8$ NVIDIA A100 GPUs to run the experiments presented in this paper.
Each experiment took roughly $10$ GPU hours of training on the A100 GPUs for
the BURST experiments, $16$ GPU hours for the VidSTG experiments, and $8$ GPU
hours for the VIS dataset experiments.

###  0.B.4 Licenses

Our code is built on Mask2Former [8] which is majorly licensed under the MIT
license, with some portions under the Apache-2.0 License. We also build on SAM
[24], which is released under the Apache 2.0 License and BLIP-2 [26] which is
released under the MIT license. The OVIS [36] dataset is released under the
Attribution-NonCommercial-ShareAlike (CC BY-NC-SA) License. The VidSTG [57]
and BURST [2] datasets are released under the MIT license.

##  Appendix 0.C Limitations

In this section, we show some failure modes of our proposed approach and
discuss limitations. Fig. S3 shows that our approach sometimes fails to detect
some open-world objects that a human may find to be of interest. For example,
the grinder on the left, the window at the top-right, _etc_., are not detected
by the network. The colored captions on the right side of Fig. S4 show that
our approach sometimes fails to generate meaningful object-centric captions
for small objects. For the purple object (cushion on a sofa), the caption
(‘the the the …’) is not meaningful since it fails to form a complete sentence
or capture the identity of the object it represents. For the red object (other
cushions on a sofa), the caption (‘a family sits on the couch’) is not object-
centric since it fails to provide a description specific to the object. Fig.
S6 further highlights a failure mode. After a train crosses the scene for a
prolonged period of time ($\sim 30$ frames), object identities may be lost.

These issues can be addressed by stronger strategies for open-world object
discovery, stronger caption-generators, and integrating better object
trackers, which we leave for future work.