# A revised asteroid polarization-albedo relationship using WISE/NEOWISE data

Joseph R. Masiero11affiliation: Jet Propulsion Laboratory/California Institute
of Technology, 4800 Oak Grove Dr., MS 321-520, Pasadena, CA 91109, USA,
<EMAIL_ADDRESS>, A. K. Mainzer11affiliation: Jet Propulsion
Laboratory/California Institute of Technology, 4800 Oak Grove Dr., MS 321-520,
Pasadena, CA 91109, USA<EMAIL_ADDRESS>, T. Grav22affiliation:
Planetary Science Institute, 1700 East Fort Lowell, Suite 106, Tucson, AZ
85719-2395 , J. M. Bauer11affiliation: Jet Propulsion Laboratory/California
Institute of Technology, 4800 Oak Grove Dr., MS 321-520, Pasadena, CA 91109,
USA<EMAIL_ADDRESS>33affiliation: Infrared Processing and
Analysis Center, California Institute of Technology, Pasadena, CA 91125 USA ,
E. L. Wright44affiliation: UCLA Astronomy, PO Box 91547, Los Angeles, CA
90095-1547 USA , R. S. McMillan55affiliation: Lunar and Planetary Laboratory,
University of Arizona, 1629 East University Blvd, Kuiper Space Science Bldg.
#92, Tucson, AZ 85721-0092 USA , D. J. Tholen66affiliation: Institute for
Astronomy, University of Hawaii, Honolulu, HI 96822 USA , A. W.
Blain77affiliation: Department of Physics and Astronomy, University of
Leicester, University Road, Leicester, LE1 7RH, United Kingdom

###### Abstract

We present a reanalysis of the relationship between asteroid albedo and
polarization properties using the albedos derived from the Wide-field Infrared
Survey Explorer. We find that the function that best describes this relation
is a three-dimensional linear fit in the space of $\log$ (albedo)-$\log$
(polarization slope)-$\log$ (minimum polarization). When projected to two
dimensions the parameters of the fit are consistent with those found in
previous work. We also define $p^{\star}$ as the quantity of maximal
polarization variation when compared with albedo and present the best fitting
albedo-$p^{\star}$ relation. Some asteroid taxonomic types stand out in this
three-dimensional space, notably the E, B, and M Tholen types, while others
cluster in clumps coincident with the S- and C-complex bodies. We note that
both low albedo and small ($D<30~{}$km) asteroids are under-represented in the
polarimetric sample, and we encourage future polarimetric surveys to focus on
these bodies.

##  1 Introduction

As light scatters off the surface of atmosphereless bodies, it is instilled
with a small linear polarization. The degree of linear polarization of the
scattered light measured by the observer is a function of the phase angle of
observation and the composition and structure of the surface, in particular
the interrelated parameters of albedo, index of refraction, and space between
scattering elements (e.g. Muinonen, 1989; Shkuratov et al., 1994). Early work
quantified the relation between phase angle (the angle between the direction
to the sun and the observer as seen from the target, $\alpha$) and
polarization (Dollfus & Zellner, 1979) and this effect can be used in parallel
with the magnitude-phase effect to probe the scattering physics of
atmosphereless surfaces (Muinonen et al., 2002, 2009).

As expected from classical scattering models, the light reflected from a
surface is polarized perpendicular to the scattering plane for large phase
angles, which is referred to as a positive polarization. For small phase
angles, however, light acquires a polarization in the scattering plane due to
an increase in the dominance of second-order scattering. This case is referred
to as negative polarization, as it is perpendicular to the positive case and
thus carries a negative sign when the polarization coordinate system is
rotated to account for the viewing geometry. The angle where the phase curve
transitions from positive to negative is referred to as the inversion angle
($\alpha_{0}$). By definition the value of the polarization must go to zero at
$\alpha=0^{\circ}$, though some work has suggested that surfaces may have a
secondary trough at very small angles related to the optical opposition effect
(Rosenbush et al., 1997). Cellino et al. (2005a) find no evidence for a
polarimetric opposition effect in their sample, though high albedo objects are
not represented there.

From the studies of the scattering properties of the lunar surface, a
relationship was found between albedo and the parameters used to describe the
polarimetric-phase effect of the lunar regolith (Bowell et al., 1973) that was
then extended to asteroids (Zellner et al., 1974; Cellino et al., 1999), of
the form:

$\displaystyle\log p_{V}=C_{1}\log h+C_{2}$ (1) $\displaystyle\log
p_{V}=C_{3}\log P_{min}+C_{4}$ (2)

where $p_{V}$ is the geometric albedo, $h$ is the linear slope of the phase
curve at the inversion angle, and $P_{min}$ is the value of the largest
negative polarization (i.e. the depth of the negative trough), usually
expressed as an absolute value. We show an illustration of these parameters
and two typical polarization-phase curves in Figure 1. We note that the
polarization shown in this figure is the $P_{r}$ value that has been rotated
to account for viewing geometry, such that $P_{r}>0$ is the amplitude of
polarization perpendicular to the scattering plane and $P_{r}<0$ is the
amplitude parallel to the scattering plane. A polarization component $\pm
45^{\circ}$ from the scattering plane is typically not observed at any phase
angle for asteroids, and thus is ignored in this diagram.

Cellino et al. (1999) present the most recent best-fitting values for the
constants in the above equations: $C_{1}=-1.118\pm 0.071$, $C_{2}=-1.779\pm
0.062$, $C_{3}=-1.357\pm 0.140$, and $C_{4}=-0.858\pm 0.030$. In this work, we
revise the best-fitting values for these constants in light of new albedo data
from the Wide-field Infrared Survey Explorer (WISE, Wright et al., 2010) and
the planetary science extension NEOWISE (Mainzer et al., 2011a). The aim of
this work is two-fold: firstly, while WISE provides us with albedos for a
large fraction of the known asteroids, calibration of this relationship will
allow it to be applied to objects that were not observed by WISE; secondly,
the behavior of the polarization of asteroid surfaces helps us determine the
surface mineralogy and this relationship represents a critical component in
this determination. Through application of both thermal infrared and
polarimetric data we can gain a broader understanding of the behavior of
asteroids across the Solar system.

Figure 1: Illustration of the typical polarization-phase behavior for two
different types of asteroids, with the inversion angle ($\alpha_{0}$), the
minimum polarization ($P_{min}$), and the slope of polarization at the
inversion angle ($h$) labeled for the dashed blue curve. Example curves for a
generic S-type asteroid (solid red line) and a generic C-type (dashed blue
line) are shown.

##  2 Data

We draw our list of polarimetric properties for asteroids from a range of
sources. The dominant contributor is the Astronomical Polarimetric Database
presented in the Planetary Data System (PDS) (Lupishko & Vasilyev, 2008) which
was a compilation of the polarimetric properties of individual asteroids in
the literature up to the data of publication. We also incorporate values for
$h$, $P_{min}$, and/or $\alpha_{0}$ for asteroids given by: Cellino et al.
(1999, 2005a, 2005b); Fornasier et al. (2006); Gil-Hutton (2007a); Gil-Hutton
et al. (2007b, 2008); Masiero & Cellino (2009); Belskaya et al. (2010). We
note that as these data are drawn from a range of different instruments,
uncertainties in the absolute calibration may result in a larger scatter than
is actually present. A comprehensive survey of polarimetric properties of a
large number of objects conducted with a single instrument would reduce this
possible source of error, and so is strongly encouraged.

Determination of $h$, $P_{min}$, and $\alpha_{0}$ all require polarimetric
measurements spanning a range of phase angles. The inversion angle can
typically be determined to a reasonable level of accuracy with a few bounding
measurements at $\alpha\sim 20^{\circ}$. Polarimetric slope is more difficult
to determine, especially for objects located farther from the sun that are
rarely observable at phase angles much beyond the inversion angle (e.g.
objects that do not come within $\sim 2.9~{}$AU of the sun can never be
observed at phase angles $\alpha>20^{\circ}$). Careful timing of observations
can ensure adequate phase coverage that will allow for an accurate
determination of the slope. The depth of minimum polarization is often the
most difficult parameter to determine for some objects, as it requires
observing at small phase angles that are not frequently available for
asteroids in the inner Main Belt, Hungaria, Mars Crosser, and NEO populations.
Additionally, determining this value requires evenly spaced observations over
the full branch of negative polarization, rather than just a few bounding
measurements as required for both $\alpha_{0}$ and $h$. As such, relative
errors on $P_{min}$ tend to be larger than measured for the other polarimetric
parameters. Where errors on polarimetric parameters were not given by the
source, we assume values based on the errors on the published data in those
sources.

The albedos we use for this work are drawn from the values derived for Main
Belt asteroids (MBAs) published in Masiero et al. (2011). For objects in the
NEO or Mars Crossing populations, we draw albedos from the appropriate lists
(Mainzer et al., 2011e, f) which were derived using a method identical to that
used for the MBAs. All of the objects with both defined polarimetric phase
curves as well as WISE-determined albedos had low identifying numbers,
implying that they were some of the first objects discovered, and thus likely
to preferentially sample the largest minor bodies of the solar system. These
large asteroids were more likely to have been seen in multiple bands by WISE
which allows for fitting of the beaming parameter. Mainzer et al. (2011b) show
that in cases such as this the error on albedo as an absolute measurement is
$\sim 20\%$ of the measured albedo value, however internal comparisons are
better than this limit.

A primary concern in any analysis of albedos derived from infrared-determined
diameters is the quality of the optical measurements used. We draw our $H$
magnitudes from the Minor Planet Center’s orbital element catalog
(MPCORB111http://www.minorplanetcenter.net/iau/MPCORB.html), as discussed in
Masiero et al. (2011). While other studies have found an offset between
measured magnitudes and those predicted from the $H$ absolute magnitude value,
with objects being fainter than expected (e.g. Jurić et al., 2002; Parker et
al., 2008), Mainzer et al. (2011b) find that, in general, no offset
corrections to $H$ are required for the most recent releases of the
magnitudes. An exception to this result has been found for some objects with
unusually high albedos in Masiero et al. (2011) and Mainzer et al. (2011e);
many of these objects are coincident in orbital-element-space with the
Hungarias and the Vesta family. Harris et al. (1989) found that the commonly
assumed value of $G=0.15$ is inappropriate for these types of high-albedo
objects, and a value of $G\sim 0.4$ may be more appropriate. Revising $G$ to
this value would result in an offset of up to $\sim 0.3~{}$mag in the $H$
magnitude depending on the initial $H$ fit, however this correction is not
required for most objects. In the past, photometric measurements for many
asteroids that contributed to the $H$ magnitudes in the MPCORB catalog were
acquired with unfiltered CCDs. New, filtered observations and refined handling
of previous photometry have largely mitigated the effect of unfiltered
measurements on $H$ values. (T. Spahr, 2012, private communication).

Mainzer et al. (2011c) present a comparison of the WISE albedos to the IRAS
albedos and find a good match for most objects, though some scatter is seen,
especially at the smallest sizes where the IRAS signal-to-noise ratios were
poor compared to WISE. This albedo error assumes moderate-to-low light curve
amplitudes and well characterized $H$ and $G$ values. This error will result
in an uncertainty in the offsets of the linear fits (i.e. $C_{2}$ and $C_{4}$
in Equations 1 and 2) though the slopes should be unaffected. We include this
error in our fits below. We note that recently Muinonen et al. (2010) have
introduced a three-parameter photometric system ($H$,$G_{1}$,$G_{2}$) to
better characterize the behavior of the photometric phase effect which may
reduce some of these errors, but we note that this system requires accurate
photometry over a large phase window, which is not available for many
asteroids.

We show the polarimetric and albedo data used for this work in Table
LABEL:tab.data (ellipses indicate an unmeasured polarimetric property). When
these two data sets are combined, we have $65$ objects with measured albedos
and polarimetric slopes, and $112$ with albedos and minimum polarization
values. This result is a improvement compared to the data set presented by
Cellino et al. (1999), who performed a similar analysis using IRAS albedos of
$37$ objects for the slope-albedo fit and $16$ for the minimum polarization-
albedo fit. We note that due to the brightness requirements of most
polarimeters and the polarimetric survey strategies employed, these lists are
dominated by the largest known asteroids. Approximately half our sample have
sizes over $100~{}$km, and three-quarters are larger than $50~{}$km. Thus
while the largest asteroids are well sampled, there is a distinct lack of
small bodies in these lists. We also note that despite the fact that low-
albedo objects dominate the Main Belt population (Masiero et al., 2011), they
are under-represented in the polarimetric surveys (see below). As WISE is
sensitive to thermal infrared light, the detection probability for asteroids
is effectively unbiased with respect to the albedos of the objects observed
(Mainzer et al., 2011e), and thus the distribution of albedos seen with WISE
is a more accurate representation of the true population than is the
distribution seen for optically-selected samples. Extending polarimetric
coverage to both smaller sizes and low albedo objects through a large-scale
campaign is critical to extending and generalizing the trends seen here and in
previous work.

Table 1: Compiled asteroid albedos and polarimetric properties Asteroid | $p_{V}$ | $\alpha_{0}$ | $h$ | $P_{min}$
---|---|---|---|---
2 | $0.142\pm 0.018$ | $18.1\pm 0.1$ | $0.228\pm 0.003$ | $1.38\pm 0.05$
5 | $0.245\pm 0.051$ | $19.1\pm 0.1$ | $0.096\pm 0.050$ | $0.70\pm 0.05$
6 | $0.269\pm 0.049$ | $20.8\pm 0.2$ | $0.091\pm 0.050$ | $0.80\pm 0.05$
8 | $0.261\pm 0.048$ | $20.0\pm 0.1$ | $0.104\pm 0.003$ | $0.68\pm 0.05$
9 | $0.134\pm 0.016$ | $21.8\pm 0.1$ | $0.102\pm 0.003$ | $0.74\pm 0.05$
10 | $0.058\pm 0.005$ | $...$ | $...$ | $1.50\pm 0.05$
11 | $0.158\pm 0.036$ | $18.9\pm 0.2$ | $0.124\pm 0.003$ | $0.73\pm 0.05$
12 | $0.140\pm 0.014$ | $20.8\pm 0.2$ | $0.121\pm 0.003$ | $0.73\pm 0.01$
13 | $0.069\pm 0.022$ | $21.7\pm 0.5$ | $0.257\pm 0.003$ | $2.10\pm 0.05$
14 | $0.221\pm 0.022$ | $20.5\pm 0.2$ | $0.105\pm 0.003$ | $0.82\pm 0.10$
15 | $0.206\pm 0.055$ | $20.6\pm 0.2$ | $0.087\pm 0.005$ | $0.72\pm 0.02$
17 | $0.160\pm 0.009$ | $...$ | $0.131\pm 0.003$ | $0.74\pm 0.05$
18 | $0.221\pm 0.082$ | $21.6\pm 0.1$ | $0.101\pm 0.003$ | $0.87\pm 0.05$
19 | $0.050\pm 0.020$ | $21.7\pm 0.2$ | $0.305\pm 0.003$ | $1.72\pm 0.05$
22 | $0.169\pm 0.061$ | $...$ | $...$ | $0.83\pm 0.04$
24 | $0.064\pm 0.016$ | $...$ | $0.191\pm 0.003$ | $1.63\pm 0.10$
27 | $0.201\pm 0.058$ | $...$ | $0.099\pm 0.003$ | $0.70\pm 0.05$
29 | $0.157\pm 0.035$ | $22.0\pm 0.2$ | $0.098\pm 0.003$ | $0.88\pm 0.10$
30 | $0.171\pm 0.034$ | $19.8\pm 0.5$ | $0.104\pm 0.003$ | $0.78\pm 0.05$
31 | $0.045\pm 0.044$ | $...$ | $...$ | $1.32\pm 0.10$
32 | $0.230\pm 0.065$ | $...$ | $...$ | $0.63\pm 0.05$
39 | $0.245\pm 0.056$ | $21.0\pm 0.1$ | $0.090\pm 0.003$ | $0.79\pm 0.05$
40 | $0.195\pm 0.019$ | $20.8\pm 0.2$ | $0.100\pm 0.003$ | $0.85\pm 0.05$
46 | $0.052\pm 0.011$ | $...$ | $...$ | $1.54\pm 0.10$
47 | $0.067\pm 0.009$ | $...$ | $0.204\pm 0.003$ | $1.44\pm 0.05$
51 | $0.100\pm 0.026$ | $...$ | $0.292\pm 0.050$ | $1.86\pm 0.05$
54 | $0.049\pm 0.008$ | $22.2\pm 0.5$ | $0.357\pm 0.050$ | $1.95\pm 0.05$
56 | $0.050\pm 0.006$ | $19.7\pm 0.2$ | $0.318\pm 0.003$ | $1.47\pm 0.05$
57 | $0.182\pm 0.047$ | $...$ | $...$ | $0.71\pm 0.10$
58 | $0.059\pm 0.005$ | $...$ | $...$ | $1.70\pm 0.10$
63 | $0.159\pm 0.028$ | $19.8\pm 0.1$ | $0.102\pm 0.003$ | $0.70\pm 0.05$
64 | $0.676\pm 0.223$ | $18.2\pm 0.2$ | $0.036\pm 0.001$ | $0.32\pm 0.05$
68 | $0.207\pm 0.025$ | $...$ | $...$ | $0.68\pm 0.05$
70 | $0.040\pm 0.009$ | $...$ | $...$ | $1.83\pm 0.05$
71 | $0.248\pm 0.035$ | $...$ | $0.061\pm 0.005$ | $0.61\pm 0.02$
73 | $0.186\pm 0.018$ | $...$ | $...$ | $0.76\pm 0.06$
75 | $0.098\pm 0.014$ | $20.2\pm 0.1$ | $0.103\pm 0.017$ | $...$
77 | $0.153\pm 0.046$ | $...$ | $...$ | $1.25\pm 0.14$
80 | $0.182\pm 0.026$ | $...$ | $...$ | $0.75\pm 0.05$
83 | $0.086\pm 0.021$ | $...$ | $...$ | $1.47\pm 0.10$
84 | $0.053\pm 0.017$ | $20.3\pm 0.5$ | $0.306\pm 0.050$ | $1.49\pm 0.05$
85 | $0.063\pm 0.025$ | $...$ | $...$ | $1.36\pm 0.10$
89 | $0.185\pm 0.034$ | $...$ | $0.119\pm 0.050$ | $0.90\pm 0.05$
95 | $0.056\pm 0.009$ | $...$ | $...$ | $1.78\pm 0.05$
97 | $0.206\pm 0.046$ | $22.1\pm 0.1$ | $0.174\pm 0.018$ | $...$
113 | $0.223\pm 0.031$ | $...$ | $0.081\pm 0.005$ | $...$
114 | $0.088\pm 0.010$ | $...$ | $...$ | $1.24\pm 0.10$
115 | $0.654\pm 0.124$ | $...$ | $...$ | $0.71\pm 0.05$
118 | $0.139\pm 0.031$ | $...$ | $...$ | $0.80\pm 0.12$
121 | $0.077\pm 0.010$ | $...$ | $...$ | $1.72\pm 0.05$
125 | $0.115\pm 0.027$ | $...$ | $0.145\pm 0.033$ | $0.83\pm 0.02$
129 | $0.157\pm 0.026$ | $...$ | $...$ | $0.90\pm 0.05$
131 | $0.164\pm 0.011$ | $...$ | $0.208\pm 0.059$ | $...$
132 | $0.120\pm 0.008$ | $...$ | $0.146\pm 0.006$ | $1.13\pm 0.03$
135 | $0.152\pm 0.050$ | $...$ | $...$ | $1.06\pm 0.10$
138 | $0.161\pm 0.028$ | $...$ | $0.103\pm 0.020$ | $...$
139 | $0.045\pm 0.023$ | $...$ | $0.262\pm 0.050$ | $1.31\pm 0.05$
141 | $0.049\pm 0.010$ | $20.6\pm 0.5$ | $0.330\pm 0.050$ | $1.78\pm 0.05$
145 | $0.043\pm 0.004$ | $...$ | $...$ | $1.86\pm 0.05$
153 | $0.046\pm 0.008$ | $...$ | $...$ | $1.05\pm 0.05$
182 | $0.210\pm 0.059$ | $...$ | $...$ | $0.64\pm 0.05$
184 | $0.107\pm 0.019$ | $...$ | $...$ | $0.93\pm 0.06$
188 | $0.157\pm 0.055$ | $...$ | $0.140\pm 0.015$ | $...$
189 | $0.199\pm 0.024$ | $...$ | $...$ | $1.26\pm 0.10$
192 | $0.288\pm 0.040$ | $19.8\pm 0.1$ | $0.084\pm 0.003$ | $0.75\pm 0.05$
197 | $0.239\pm 0.026$ | $...$ | $...$ | $0.79\pm 0.08$
201 | $0.097\pm 0.006$ | $...$ | $...$ | $1.00\pm 0.05$
204 | $0.163\pm 0.044$ | $...$ | $...$ | $0.83\pm 0.12$
216 | $0.111\pm 0.034$ | $...$ | $...$ | $1.27\pm 0.05$
217 | $0.044\pm 0.005$ | $...$ | $...$ | $0.82\pm 0.05$
230 | $0.171\pm 0.076$ | $20.6\pm 0.2$ | $0.122\pm 0.003$ | $0.94\pm 0.05$
234 | $0.151\pm 0.034$ | $27.0\pm 2.0$ | $...$ | $1.60\pm 0.20$
250 | $0.112\pm 0.021$ | $...$ | $...$ | $0.88\pm 0.08$
259 | $0.042\pm 0.005$ | $...$ | $...$ | $1.25\pm 0.05$
270 | $0.254\pm 0.043$ | $...$ | $...$ | $0.65\pm 0.05$
305 | $0.182\pm 0.028$ | $...$ | $...$ | $0.64\pm 0.10$
306 | $0.174\pm 0.060$ | $...$ | $...$ | $0.66\pm 0.10$
324 | $0.063\pm 0.012$ | $20.0\pm 0.1$ | $0.278\pm 0.003$ | $1.46\pm 0.05$
334 | $0.051\pm 0.016$ | $...$ | $...$ | $1.32\pm 0.05$
338 | $0.165\pm 0.028$ | $...$ | $...$ | $0.98\pm 0.10$
345 | $0.059\pm 0.012$ | $...$ | $...$ | $1.55\pm 0.05$
347 | $0.213\pm 0.041$ | $22.6\pm 0.1$ | $0.113\pm 0.003$ | $0.78\pm 0.03$
349 | $0.153\pm 0.018$ | $...$ | $...$ | $0.39\pm 0.05$
351 | $0.171\pm 0.046$ | $...$ | $...$ | $0.74\pm 0.09$
354 | $0.173\pm 0.032$ | $...$ | $...$ | $0.51\pm 0.10$
356 | $0.053\pm 0.015$ | $...$ | $...$ | $1.50\pm 0.10$
377 | $0.056\pm 0.025$ | $19.8\pm 0.2$ | $0.206\pm 0.005$ | $1.76\pm 0.04$
384 | $0.190\pm 0.040$ | $...$ | $...$ | $0.94\pm 0.35$
396 | $0.139\pm 0.025$ | $...$ | $...$ | $1.34\pm 0.09$
409 | $0.050\pm 0.010$ | $19.9\pm 0.2$ | $0.191\pm 0.005$ | $...$
410 | $0.043\pm 0.007$ | $...$ | $0.313\pm 0.050$ | $1.94\pm 0.05$
415 | $0.086\pm 0.009$ | $...$ | $...$ | $1.28\pm 0.10$
423 | $0.066\pm 0.005$ | $...$ | $...$ | $1.40\pm 0.05$
441 | $0.139\pm 0.026$ | $...$ | $...$ | $1.41\pm 0.12$
451 | $0.069\pm 0.006$ | $...$ | $...$ | $1.62\pm 0.05$
466 | $0.086\pm 0.009$ | $...$ | $...$ | $1.60\pm 0.10$
511 | $0.073\pm 0.006$ | $19.4\pm 0.1$ | $0.277\pm 0.003$ | $1.69\pm 0.05$
532 | $0.202\pm 0.039$ | $...$ | $0.122\pm 0.003$ | $0.78\pm 0.05$
550 | $0.137\pm 0.024$ | $...$ | $0.157\pm 0.005$ | $...$
558 | $0.117\pm 0.010$ | $...$ | $...$ | $0.75\pm 0.06$
584 | $0.244\pm 0.060$ | $19.1\pm 0.1$ | $0.108\pm 0.003$ | $0.64\pm 0.05$
600 | $0.177\pm 0.036$ | $...$ | $...$ | $0.43\pm 0.20$
602 | $0.052\pm 0.007$ | $...$ | $...$ | $1.76\pm 0.10$
624 | $0.077\pm 0.020$ | $...$ | $...$ | $1.30\pm 0.05$
625 | $0.197\pm 0.058$ | $...$ | $0.070\pm 0.003$ | $...$
654 | $0.043\pm 0.011$ | $20.5\pm 0.5$ | $0.280\pm 0.050$ | $1.46\pm 0.10$
662 | $0.193\pm 0.028$ | $...$ | $...$ | $1.32\pm 0.33$
674 | $0.206\pm 0.033$ | $...$ | $...$ | $0.81\pm 0.10$
704 | $0.076\pm 0.010$ | $15.7\pm 0.1$ | $0.305\pm 0.003$ | $1.45\pm 0.10$
737 | $0.136\pm 0.043$ | $...$ | $...$ | $0.84\pm 0.05$
787 | $0.120\pm 0.022$ | $...$ | $0.087\pm 0.003$ | $...$
796 | $0.205\pm 0.041$ | $...$ | $0.124\pm 0.011$ | $0.98\pm 0.02$
849 | $0.115\pm 0.016$ | $...$ | $...$ | $0.95\pm 0.05$
857 | $0.225\pm 0.026$ | $...$ | $...$ | $0.75\pm 0.16$
863 | $0.112\pm 0.016$ | $18.1\pm 0.2$ | $0.052\pm 0.005$ | $0.40\pm 0.10$
887 | $0.230\pm 0.018$ | $...$ | $0.101\pm 0.050$ | $0.76\pm 0.05$
925 | $0.253\pm 0.053$ | $19.6\pm 0.2$ | $0.065\pm 0.005$ | $...$
1036 | $0.212\pm 0.026$ | $20.6\pm 0.2$ | $0.112\pm 0.003$ | $0.84\pm 0.02$
1052 | $0.273\pm 0.074$ | $...$ | $...$ | $0.67\pm 0.05$
1058 | $0.242\pm 0.024$ | $...$ | $...$ | $0.69\pm 0.10$
1105 | $0.102\pm 0.017$ | $...$ | $...$ | $1.20\pm 0.16$
1355 | $0.466\pm 0.082$ | $18.2\pm 0.1$ | $0.083\pm 0.020$ | $...$
1627 | $0.153\pm 0.046$ | $...$ | $0.131\pm 0.003$ | $...$
1672 | $0.094\pm 0.016$ | $...$ | $0.131\pm 0.003$ | $...$
1685 | $0.292\pm 0.127$ | $...$ | $0.099\pm 0.003$ | $...$
2131 | $0.198\pm 0.034$ | $...$ | $...$ | $0.86\pm 0.15$
2577 | $0.377\pm 0.062$ | $20.9\pm 0.1$ | $0.124\pm 0.044$ | $...$
3169 | $0.423\pm 0.067$ | $19.6\pm 0.1$ | $0.276\pm 0.018$ | $...$
6249 | $0.878\pm 0.140$ | $22.4\pm 0.1$ | $0.164\pm 0.035$ | $...$
6911 | $0.454\pm 0.083$ | $...$ | $...$ | $0.83\pm 0.16$
Table 1: (continued)

##  3 Revised Polarimetric-Albedo Relationship

In Figures 2-4 we compare the measured WISE albedo to the slope of the
polarization beyond the inversion angle, the depth of the negative branch of
polarization, and the inversion angle, respectively, for all objects with
recorded values for these parameters. We distinguish objects in the Hungaria
region and in the NEO population as red squares and cyan triangles,
respectively. While the NEOs appear consistent with the MBAs, the Hungaria
objects deviate from the general trend significantly. As discussed in Masiero
et al. (2011) the albedos for these objects are suspect: large deviations in
the magnitude-phase slope parameter from the assumed $G=0.15$ used for most
asteroids can result in incorrect $H$ values, and thus poorly constrained
albedos (Harris et al., 1989). Alternatively, large-amplitude long-period
light curves may also corrupt the $H$ values calculated from optical
photometry. We are currently working on a program to better constrain the
photometric parameters and albedos of these objects, but for the following
discussion we will disregard the Hungaria asteroids.

Figure 2: Albedo vs. slope of the polarization phase curve beyond the
inversion angle. Objects located in the Hungaria region are noted as red
squares, objects in the NEO population as cyan triangles. The green dashed
line shows the best fit found for Equation 1 with our data. Figure 3: The same
as Figure 2, but for albedo vs. depth of the minimum polarization branch. The
green dashed line shows the best fit found for Equation 2 with our data.
Figure 4: The same as Figure 2, but for albedo vs. inversion angle.

We see no overall trend between albedo and inversion angle in our data. The
object with the anomalously high inversion angle is (234) Barbara, the
principal member of the “Barbarian” group of objects with strange polarimetric
properties (Cellino et al., 2006). The object with an inversion angle well
below the general trend is (704) Interamnia: some F-class objects like
Interamnia have previously been shown to display unusually small inversion
angles (Belskaya et al., 2005).

We see the expected general trends when looking at slope and Pmin, with low
albedo objects showing steeper slopes and deeper troughs. We note, however,
that the lowest albedo objects, those with $p_{V}<0.04$, have almost no
representation in the polarimetric sample despite representing over $10\%$ of
the total population of MBAs observed by WISE, even before correcting the
population for objects without optical followup (and thus without measured
albedos). We therefore cannot comment on the reliability of the polarization-
phase relation at the lowest albedos. A campaign of polarimetric observations
of low albedo asteroids is critical to test these relations at their low
albedo extreme.

We find that the optimal description of the relation between albedo and
polarimetric parameters is a linear fit in the three dimensional space of
$\log p_{V}$-$\log h$-$\log P_{min}$. We use only those objects where both
polarimetric parameters are measured to an accuracy of $20\%$ or better,
leaving us with $41$ objects in our high-confidence sample. We perform
orthogonal distance regression on the three dimensional data, using the
associated errors on each measurement to determine the best fitting linear
parameters as well as each parameter’s error. We then reduce the best-fit
parameters back to two-dimensional projections, which result in the following
constant parameters for the relationships in Equations 1 and 2:

$C_{1}=-1.207\pm 0.067$ $C_{2}=-1.892\pm 0.141$ $C_{3}=-1.579\pm 0.084$
$C_{4}=-0.880\pm 0.106$

These projected fits are shown as green dashed lines in Figures 2 and 3. With
the exception of $C_{3}$, these parameters are all within one-sigma of the
values found by Cellino et al. (1999), and all are within 1.5 sigma. As the
WISE albedos for the largest asteroids have been shown to be generally
consistent with the IRAS values (Mainzer et al., 2011c) and all of the objects
used here are in the size range sampled by IRAS, this agreement was not
unexpected. Of the objects in our high-confidence polarimetric sample only $6$
were not observed by IRAS, however the WISE albedos are all derived from a
minimum of $5$ observations (and an average of $>10$) spread over time and
thus are less sensitive to rotation effects. As the WISE data cover MBAs down
to a few kilometers and NEOs to much smaller sizes, future polarimetric
surveys focusing on smaller asteroids will allow this relationship to be
tested over a more extensive size range.

We can also project our fit of the polarimetric properties onto an axis of
maximal variation. We define $p^{\star}$ as the quantity of maximum
polarimetric variation and find a best-fitting transform of:

$p^{\star}=(0.79\pm 0.02)\log h+(0.61\pm 0.03)\log P_{min}$

Using our data we find a best-fitting relation between $p^{\star}$ and albedo
of:

$\log p_{V}=(-1.58\pm 0.09)+(-1.04\pm 0.04)p^{\star}$

We show $p^{\star}$ compared to albedo for all of the high-confidence objects
with both measured $h$ and $P_{min}$, along with this fit, in Figure 5. We
label (64) Angelina as “E”, (2) Pallas as “B”, and (132) Aethra as “M”
following their Tholen taxonomic classifications (Neese, 2010). The clusters
of objects with Tholen S and C taxonomic classifications are labeled as such,
and include other objects within those taxonomic complexes (e.g. F- and
D-types are included in the C-complex) . We indicate (71) Niobe with “n” in
this plot; though it has a Tholen class of S it is distinct enough from the
general cluster to warrant mention. Additional polarimetric and spectroscopic
followup of this object will help determine why its properties differ from the
general S complex. We focus on Tholen classifications here as this taxonomic
system shows the greatest distinction in albedo between the different types
(Mainzer et al., 2011d).

Figure 5: The maximal variation of the polarimetric properties $p^{\star}$ as
defined in the text compared to the measured albedo. The best-fitting
relationship projected from the three-dimensional fit is shown as the dashed
line. Text labels denote specific asteroids or groupings as discussed in the
text.

In addition to the lack of the lowest albedo objects in the polarimetric
sample, we observe an over-representation of high albedo objects compared to
the distribution for all similarly sized MBAs. Figure 6 shows the distribution
of albedos of all objects in the sample with high-confidence polarimetric
properties used to derive the linear three dimensional fit (the smallest of
which is $D\sim 35~{}$km), as well as all MBAs larger than $30~{}$km in
diameter that were observed by WISE. The difference in these two distributions
can be traced to the optical selection bias in the acquisition and measurement
of the polarimetric properties of asteroids: very high signal-to-noise levels
are needed to reach the polarimetric sensitivities that allow for accurate
measurement of $P_{min}$ and $h$. Thus, even though at a given phase angle low
albedo objects will show larger degrees of polarization, the reduction in
photons received from these sources make these measurements less precise.
Focusing future polarimetric surveys on low albedo asteroids will help make
this sample more representative of the true distribution of MBAs.

Figure 6: Normalized albedo distribution of the high-confidence polarimetric
sample (solid) and all MBAs larger than $30~{}$km that were detected during
the fully cryogenic portion of the NEOWISE survey (dotted).

##  4 Conclusions

Using the newly available albedos from the WISE space telescope thermal
infrared survey data, we have fit the relationships between albedo, the slope
of the polarimetric-phase curve beyond the inversion angle, and the maximum
depth of the negative polarization trough. We restrict ourselves to objects
with well characterized polarimetric properties (i.e. relative errors
$<20\%$). Due to the selection of the polarimetrically observed objects this
results in our sample consisting of only objects larger than $D>30~{}$km, with
nearly three-quarters having $D>50~{}$km.

We find that the function that best describes the albedo and polarimetry is a
three dimensional linear fit in $\log p_{V}$-$\log h$-$\log P_{min}$ space.
Orthogonal distance regression allows us to find the best fitting parameters
while accounting for measurement error on all parameters. When the best fit
line is projected to two dimensions, we find the resultant fit parameters are
all within $1.5$ sigma of those found by Cellino et al. (1999). We also define
a new polarimetric quantity $p^{\star}$ that describes the maximum variation
in polarimetric properties when compared with albedo.

We observe distinct separation of some taxonomic classes in $p^{\star}$ space.
In particular E-type, B-type, and some M-type asteroids are far removed from
the clumps that trace the more generic S- and C-complex objects. Asteroid (71)
Niobe also holds a distinct location in this space despite its S-type
classification under the Tholen system, and warrants further study. We note
that the principal component (PC) analysis of Niobe from the Eight Color
Asteroid Survey indicates that it is on the edge of the S-complex (Tholen,
1984) and it has a PC4 in the bottom two percent of all asteroids in that
survey (one of only 3 S-type or probable S-type objects with PC4 that low
Zellner et al., 2009).

Finally, despite the prevalence of low albedo asteroids seen throughout the
Main Belt (Masiero et al., 2011) we find that they are under-represented in
the polarimetric sample. Notably, roughly $10\%$ of MBAs have albedos
$p_{V}<0.04$, but there are no objects in our polarimetric sample with albedos
this low. We recommend that future surveys focus on measuring polarization-
phase curves for low albedo asteroids to properly sample this population.

## Acknowledgments

The authors wish to thank referee Alberto Cellino for his helpful review of
this paper. J.R.M. was supported by an appointment to the NASA Postdoctoral
Program at JPL, administered by Oak Ridge Associated Universities through a
contract with NASA. This publication makes use of data products from the Wide-
field Infrared Survey Explorer, which is a joint project of the University of
California, Los Angeles, and the Jet Propulsion Laboratory/California
Institute of Technology, funded by the National Aeronautics and Space
Administration. This publication also makes use of data products from NEOWISE,
which is a project of the Jet Propulsion Laboratory/California Institute of
Technology, funded by the Planetary Science Division of the National
Aeronautics and Space Administration. This research has made use of the
NASA/IPAC Infrared Science Archive, which is operated by the Jet Propulsion
Laboratory, California Institute of Technology, under contract with the
National Aeronautics and Space Administration.

## References

  * Belskaya et al. (2005) Belskaya, I.N., Shkuratov, Yu.G., Efimov, Yu.S., et al., 2005, Icarus, 178, 213.
  * Belskaya et al. (2010) Belskaya, I.N., Fornasier, S., Krugly, Yu.N., et al., 2010, A&A, 515, 29.
  * Bowell et al. (1973) Bowell, E., Dollfus, A., Zellner, B. & Geake, J.E., 1973, Proc. of the Lunar Science Conference, 4, 3167.
  * Cellino et al. (1999) Cellino, A., Gil Hutton, R., Tedesco, E.F., di Martino, M. & Brunini, A., 1999, Icarus, 138, 129.
  * Cellino et al. (2005a) Cellino, A., Gil Hutton, R., di Martino, M., Bendjoya, Ph., Belskaya, I.N. & Tedesco, E.F., 2005a, Icarus, 179, 304.
  * Cellino et al. (2005b) Cellino, A., Yoshida, F., Anderlucci, E., et al., 2005b, Icarus, 179, 297.
  * Cellino et al. (2006) Cellino, A., Belskaya, I.N., Bendjoya, Ph., et al., 2006, Icarus, 180, 565.
  * Dollfus & Zellner (1979) Dollfus, A. & Zellner, B., 1979, Asteroids (Univ of Arizona Press), 170.
  * Fornasier et al. (2006) Fornasier, S., Belskaya, I.N., Shkuratov, Yu.G., et al., 2006, A&A, 455, 371.
  * Gil-Hutton (2007a) Gil-Hutton, R., 2007a, A&A, 464, 1127.
  * Gil-Hutton et al. (2007b) Gil-Hutton, R., Lazzaro, D. & Benavidez, P., 2007b, A&A, 468, 1109.
  * Gil-Hutton et al. (2008) Gil-Hutton, R., Mesa, V., Cellino, A. et al., 2008, A&A, 482, 309.
  * Harris et al. (1989) Harris, A.W., Young, J.W., Contreiras, L. et al., 1989, Icarus, 81, 365.
  * Jurić et al. (2002) Jurić, M., Ivezić, Z̆., Lupton, R.H., et al., 2002, AJ, 124, 1776.
  * Lupishko & Vasilyev (2008) Lupishko, D.F. and Vasilyev, S.V., Eds., Asteroid Polarimetric Database V6.0. EAR-A-3-RDR-APD-POLARIMETRY-V6.0. NASA Planetary Data System, 2008.
  * Mainzer et al. (2011a) Mainzer, A.K., Bauer, J.M., Grav, T., Masiero, J., Cutri, R.M., Dailey, J., Eisenhardt, P., McMillan, R.M. et al., 2011a, ApJ, 731, 53.
  * Mainzer et al. (2011b) Mainzer, A.K., Grav, T., Masiero, J., et al., 2011b, ApJ, 736, 100.
  * Mainzer et al. (2011c) Mainzer, A., Grav, T., Masiero, J., et al., 2011c, ApJL, 737, 9.
  * Mainzer et al. (2011d) Mainzer, A.K., Grav, T., Masiero, J., et al., 2011d, ApJ, 741, 90.
  * Mainzer et al. (2011e) Mainzer, A.K., Grav, T., Bauer, J., et al., 2011e, ApJ, 743, 156.
  * Mainzer et al. (2011f) Mainzer, A.K., Grav, T., Bauer, J., Masiero, J., McMillan, R.S., Cutri, R.M. et al., 2011f, in prep.
  * Masiero & Cellino (2009) Masiero, J. & Cellino, A., 2009, Icarus, 199, 333.
  * Masiero et al. (2011) Masiero, J., Mainzer, A.K., Grav, T. et al., 2011, ApJ, 741, 68.
  * Muinonen (1989) Muinonen, K., 1989, Proc. URSI International Symp. on Electromagnetic Theory, 428.
  * Muinonen et al. (2002) Muinonen, K., Piironen, J., Shkuratov, Y., Ovcharenko, A. & Clark, B., 2002, Asteroids III, ed. Bottke, Cellino, Paolicchi & Binzel (Univ of Arizona Press), 123.
  * Muinonen et al. (2009) Muinonen, K., Penttilä, A., Cellino, A., et al., 2009, M&PS, 44, 1937.
  * Muinonen et al. (2010) Muinonen, K., Belskaya, I.N., Cellino, A., et al., 2010, Icarus, 209, 542.
  * Neese (2010) Neese, C., Ed., Asteroid Taxonomy V6.0. EAR-A-5-DDR-TAXONOMY-V6.0. NASA Planetary Data System, 2010.
  * Parker et al. (2008) Parker, A., Ivezić, Z̆., Jurić, M., Lupton, R., Sekora, M.D. & Kowalski, A., 2008, Icarus, 198, 138.
  * Rosenbush et al. (1997) Rosenbush, V.K., Avramchuk, V.V., Rosenbush, A.E. & Mishchenko, M.I., 1997, ApJ, 487, 402.
  * Shkuratov et al. (1994) Shkuratov, Yu.G., Muinonen, K., Bowell, E., et al., 1994, E&MP, 65, 201.
  * Tedesco et al. (2002) Tedesco, E.F., Noah, P.V., Noah, M. & Price, S.D., 2002, AJ, 123, 1056.
  * Tholen (1984) Tholen, D.J., 1984, Ph.D. Thesis, Arizona Univ., Tucson.
  * Wright et al. (2010) Wright, E.L., Eisenhardt, P., Mainzer, A.K., Ressler, M.E., Cutri, R.M., Jarrett, T., Kirkpatrick, J.D., Padgett, D., et al., 2010, AJ, 140, 1868.
  * Zellner et al. (1974) Zellner, B., Gehrels, T. & Gradie, J., 1974, AJ, 79, 1100.
  * Zellner et al. (2009) Zellner, B., Tholen, D.J. & Tedesco E.F., 2009, Eight Color Asteroid Survey. EAR-A-2CP-3-RDR-ECAS-V4.0. NASA Planetary Data System, 2009.


heightadjust=all, floatrowsep=columnsep

# End-to-End Video Text Spotting with Transformer

Weijia Wu1 Debing Zhang2 Ying Fu3 Chunhua Shen1 Hong Zhou1†
Yuanqiang Cai4† Ping Luo5 Weijia Wu1 Yuanqiang Cai2 Chunhua Shen1 Debing
Zhang3 Ying Fu4 Hong Zhou1† Ping Luo5
1Zhejiang University
2Beijing University of Posts and Telecommunications
3Kuaishou Technology
4Beijing Institute of Technology
5The University of Hong Kong

###### Abstract

Recent video text spotting methods usually require the three-staged pipeline,
_i.e.,_ detecting text in individual images, recognizing localized text,
tracking text streams with post-processing to generate final results. The
previous methods typically follow the tracking-by-match paradigm and develop
sophisticated pipelines, which is an not effective solution. In this paper,
rooted in Transformer sequence modeling, we propose a simple, yet effective
end-to-end trainable video text DEtection, Tracking, and Recognition framework
(TransDeTR), which views the VTS task as a direct long-range temporal modeling
problem. TransDeTR mainly includes two advantages: 1) Different from the
explicit match paradigm in the adjacent frame, the proposed TransDeTR tracks
and recognizes each text implicitly by the different query termed ‘text query’
over long-range temporal sequence (more than 7 frames). 2) TransDeTR is the
first end-to-end trainable video text spotting framework, which simultaneously
addresses the three sub-tasks (_e.g.,_ text detection, tracking, recognition).
Extensive experiments on four video text datasets (_e.g.,_ ICDAR2013 Video,
ICDAR2015 Video) are conducted to demonstrate that TransDeTR achieves the
state-of-the-art performance with up to $11.0\%$ improvements on detection,
tracking, and spotting tasks. The code can be found at $\tt
github.com/weijiawu/TransDETR$.

##  1 Introduction

Text spotting is one of the fundamental tasks in computer vision. In recent
years, image-based text spotting has made extraordinary progress with a lot of
great algorithms [56, 1, 20, 41, 17, 45, 46], but much less effort was spent
on video text spotting, which is the base of video-and-language tasks such as
video retrieval [5, 14], video caption [33] and driving navigation systems
[28]. Yin et al. [49] firstly surveys and provides a detailed introduction for
video text spotting task (VTS), which requires algorithms to simultaneously
detect, track and recognize each text instance in a video sequence. Compared
with text spotting in static image [8], video text spotting is more
challenging. The quality of the video frames is generally worse than that of
static images, due to motion blur, out of focus, and artifacts issues. More
importantly, the video text spotting task requires text spatio-temporal
information, _i.e.,_ the same text tracking trajectory(Tracking ID), which
cannot be provided by image-based text spotting methods.

Figure 1: Comparisons of different video text spotting pipeline. (a) Wang et
al. [43] tracks and recognizes text by transcription matching; (b-c) Cheng et
al. [3, 4] tracks and recognizes text with metric learning; (d) Wu et al. [47]
adopts IoU-based match to address the task; (e) Without complicated matching,
hand-crafted components, an simpler, end-to-end pipeline with transformer is
proposed in this work.

Existing video text spotting methods typically develop sophisticated pipelines
to tackle the task and view the video text spotting tasks as the text
instances match problem in adjacent frames. As shown in Figure 1 (a)-(e), the
previous methods [47, 4, 43] usually use multiple models and hand-crafted
strategies for the video text spotting task, which firstly tackles each frame
in a video sequence, and then associates the similar text instances in the
adjacent frames by various matching strategies(_i.e.,_ use their IoU,
transcription, and feature). For example, IoU-based methods [47, 31] relies on
image text detection models [41, 56] and complex human-designed matching
rules, where two text instances are associated while their IoU is higher than
the given threshold. And one separate recognition model is used to recognize
the text content. Similarly, transcription-based [25, 43] and Re-ID-based
methods [3, 7] compute the text recognition transcription and feature
similarity from different text instances in adjacent frames, to join pairs
with the same text instance. Besides, these methods only deal with temporal
information in the adjacent frames, they are not favorable to long temporal
text tracking and recognition, because it is difficult to ensure the match
correct of the same text instance in each adjacent frame. Thus, a simple,
effective end-to-end VTS framework with long-range modeling is highly
desirable.

Here, we take a deeper look at the video text spotting tasks. The core of the
task is that how to better obtain a steady tracking trajectory and recognition
results for the same text instance in consecutive video frames. Recently, many
transformer-based works [38, 2, 57, 51] have shown the potential of the
Transformer structure [35] for modeling long-range dependencies and learning
temporal information across multiple frames. Therefore, we attempt to explore
a simple, but efficient video text spotting paradigm based on transformer.

In this paper, motivated by the query concept from MOTR [51], we firstly
propose an end-to-end trainable video text DEtection, Tracking and Recognition
framework with Transformer (TransDeTR), which views the video text spotting
task as a direct end-to-end long-range temporal variation problem, not the
previous match problem. We propose a new concept text query and one temporal
tracking loss to model video text spotting task over long-range sequence. Each
text query is responsible to predict the entire track trajectory and the
content of a text. Since each text query follows the same text object all the
time once the query is matched by one object, the paradigm can naturally
remove the matching associations in adjacent frame and some hand-crafted
components, _e.g.,_ NMS. During inference phrase, without hand-crafted match
strategies, TransDeTR use the classification scores of each text prediction to
determine when the text appears and disappears. As shown in Figure. 1,
TransDeTR is a simple yet effective end-to-end framework, which includes three
parts, _e.g.,_ backbone, transformer encoder-decoder, and the recognition head
with rotated RoI. Encoder-decoder naturally models the tracking box with text
queries, and the recognition head with rotated RoI enables the end-to-end
training. Given a video clip that consists of multiple frames as input, the
TransDeTR outputs the sequence of detection, tracking, and recognition results
for each text instance in the video directly.

Our main contributions are summarized as follows:

  * •

We firstly propose an end-to-end trainable video text spotting framework with
Transformer, named TransDeTR, which simultaneously solve text detection,
tracking, and recognition tasks seamlessly and naturally in one unified
framework. Without hand-crafted components(_e.g.,_ the match of objects in
adjacent frames, NMS), the framework is significantly different from existing
approaches, considerably simplifying the overall pipeline.

  * •

Different from the tracking-by-match paradigm over the adjacent frame,
TransDeTR views the VTS task as a long-range ( more than 7 frames) temporal
variation problem, which tracks and recognizes the same text in sequence with
one text query.

  * •

The proposed TransDeTR achieves the state-of-the-art performance on four
datasets for multi-stage tasks (_i.e.,_ text detection, tracking and
spotting). Especially, TransDeTR achieves $72.8\%$ ${\rm ID_{F1}}$ for video
text spotting task on ICDAR2015 [8], with $11.3\%$ improvements than the
previous SOTA methods.

##  2 Related work

###  2.1 Text Spotting in Image

Various methods [13, 21, 17] based on deep learning have been proposed and
have improved the performance considerably for image-level text spotting. Li
et al. [13] proposed the first end-to-end trainable scene text spotting
method, which successfully uses an RoI Pooling [29] to joint detection and
recognition features. Its improved version [39] further significantly improves
the performance. Liao et al. [21] propose a Mask TextSpotter which subtly
refines Mask R-CNN and uses character-level supervision to detect and
recognize characters simultaneously. Liu et al. [16] adopt an anchor-free
mechanism to improve both the training and inference speed, and proposed an
RoIRotate to enable the end-to-end text spotting training. PAN++ [42] proposed
an end-to-end text spotting framework with masked roi, which reformulates a
text line as a text kernel surrounded by peripheral pixels. However, these
image-based methods can not obtain temporal information in the video, which is
essential for some downstream tasks such as video understanding.

###  2.2 Text Spotting in Video

Compared to text spotting algorithms in the image, there are rare video text
spotting methods. Yin et al. [49] provide a detailed survey, summarizes and
compares the existing text detection, tracking, and recognition methods in a
video before 2016. As shown in Fig. 1 (a), Wang et al. [43] links texts in the
current frame and several previous frames to obtain the tracking results by
hand-crafted post-processing, such as IoU-based associations. As shown in Fig.
1 (b), Cheng et al. [3] propose a video text spotting framework by only
recognizing the localized text one-time instead of frame-wisely recognition,
which needs at least three-step with many hyper-parameters, including single-
frame detection, aggregation, quality score. As shown in Fig. 1 (c), Free [4]
is a improved version for faster and robuster video text spotting base on
Cheng et al. [3], which achieve better and faster performance. Nguyen et al.
[26] performs character detection and recognition via scanning-window
templates trained with mixture models. Rong et al. [31] presents a hand-
crafted framework of scene text recognition in multiple frames based on
feature representation of scene text character (STC) for character prediction
and conditional random field (CRF) model for word configuration. As shown in
Fig. 1 (d), Wu et al. [47] applies object features from the previous frame as
a tracking query for the current frame, then uses IoU match to obtain tracking
and recognition results. The above methods include multiple models and hand-
crafted components, which are complex pipelines and need multiple steps to
generate the tracking trajectory and recognition results. More detailed
comparison are presented in Fig. 1. Besides, the above methods only can deal
with temporal information in adjacent frames by text instance match, but it is
difficult to ensure the match correct in each frame.

###  2.3 Transformers in Vision

Transformer was first proposed in [35] as a new paradigm for the machine
translation task. Since the 2020 year, Transformers start to show promises in
solving computer vision tasks. DETR [2, 57] propose a novel object detection
paradigm based on Transformers, which largely simplifies the traditional
detection pipeline, and achieves on excellent performances compared with CNN-
based detectors [29]. Besides, there is a popularity of using transformer
architecture in other vision tasks, such as segmentation [54], object tracking
[34, 51, 53] and even backbone construction [18]. Lately, some works [34, 51]
show using a transformer in processing sequential visual data also make
remarkable shots, especially for temporal tasks. MOTR [51] introduce a novel
concept of track query and the contiguous query passing mechanism for
multiple-object Tracking and long-range temporal relation. TransTrack [34]
track object by applying object features from the previous frame as a query of
the current frame. All the above methods prove the good capacity of
transformer in modeling temporal relation, but for video text spotting or
multi-orient text tracking, to the best of our knowledge, there are still no
transformer-based solutions. Therefore, we propose a simple, but effective
TransDeTR method, which shows convincingly high performance on the popular
benchmark.

Figure 2: The overall architecture of TransDeTR. It contains three main
components: 1). a backbone(_e.g.,_ ResNet, PVT [40]) is used to extract
feature of video sequences; 2) a Transformer encoder models the relations of
features, and a weight-shared decoder models each text move trajectory by text
query. For the initial $F_{1}$ frame, an empty query set (yellow box) is
injected into the decoder to localize the initial objects and generate the
initial text query for the next frame ($F_{2}$). For the $F_{t}$ frames, the
specific text set from the previous frame is concatenated with an empty query
set to generate the text query for the current frame; 3) an recognition head
with Rotated RoI is designed to recognize text.

##  3 Our Method: TransDeTR

The proposed method tackles the video text spotting task by modeling it as a
direct sequence prediction problem, where each text query is responsible to
predict an entire text instance trajectory and the corresponding text content.
To achieve this goal, two ingredients are essential: (1) The simple pipeline:
without multiple models and hand-crafted strategies(_e.g.,_ NMS, Track Post-
Processing), TransDeTR tracks and recognizes text instances during continuous
video frames directly. The whole pipeline includes three parts: backbone,
transformer-based encoder and decoder, and recognition head with Rotated RoI.
(2) Temporal Tracking Loss for Text Query: for each frame, there are two kinds
of text tracking losses, _i.e.,_ tracked object with the same query(text
query) from the previous frames and new appeared text object with empty query.
Temporal Tracking Loss is a sequence-aware tracking loss for modeling long-
range temporal trajectory.

###  3.1 TransDeTR architecture

The overall TransDeTR architecture is depicted in Figure 2, which contains
three main components: A backbone (_e.g.,_ ResNet, PVT [40]) is used to
extract feature representation, transformer-based encoder network and decoder
network models the relations of pixel-level features and detects new-born text
instance and trace already existed instance, the attention-based recognition
head with Rotatedt RoI is utilized to obtain the final recognition results.

Backbone. The backbone extracts the original pixel-level feature sequence of
the input video clip. Assume that the initial video clip with $n$ consecutive
frames is denoted by $x_{\rm t}\in{\\{x_{1},x_{2},...,x_{n}\\}}$. A
transformer backbone [40] is used to generate the corresponding lower-
resolution activation map $f_{\rm t}\in{\\{f_{1},f_{2},...,f_{n}\\}}$.

Transformer Encoder. We adopt deformable transformer encoder [57] to model the
similarities among all the pixel level features for the extracted features
$f_{\rm t}$. The corresponding feature output for each frame can be denoted by
$D(f_{\rm t})$, where $t$ corresponds to $t$-th frame.

Transformer Decoder. The decoder follows the standard architecture of the
deformable DETR [57], which decodes the top pixel features that can represent
the text instances of each frame. The difference with the deformable DETR
decoder is that our $N$ queries are composed of two parts: empty queries
$Q_{\emptyset}$ and text queries $Q_{text}$ from the previous frame, as shown
in Figure 2. For the first frame $x_{1}$, the corresponding feature $D(f_{\rm
1})$ together with the fixed-length empty queries $Q(\emptyset)$ are input to
the transformer decoder to generate the initialized text object and text
queries $Q_{1}$. And then, we concatenate text queries $Q_{1}$ and empty
queries $Q(\emptyset)$ to generate a new query set for the $x_{2}$ frame.
Similarly, for any $x_{t}$ ($t>1$) frame, the corresponding $N$ text instance
queries are composed of text queries $Q_{t-1}$ from the previous frame and
empty queries $Q(\emptyset)$.

Rotated RoI Align. To enable end-to-end training, similar to RRPN [23], RRPN++
[22], we adopt the Rotated Region-of-Interest (Rotated RoI) to extract the
features of each text from the output feature map of upsampling. The whole
process can be divided into three steps. 1) We firstly upsample and
concatenate low-level feature maps and high-level semantic feature maps
$f_{\rm t}\in{\\{f_{1},f_{2},...,f_{4}\\}}$ from the shared backbone to obtain
the feature maps $F$, whose size is same as that of the input image. 2)
Translation matrix T and rotation matrix R parameters for affine
transformation are computed via predicted or ground truth coordinates of text
proposals. 3) Affine transformations are applied to the feature map $F$ for
each region respectively, and horizontal feature maps of text regions are
obtained. For the last two step, given one feature map and the text region-of-
interest representation ($c_{x},c_{y},c_{h},c_{w},\theta$).
$c_{x},c_{y},c_{h},c_{w},\theta$ denotes the center coordinate, height, width,
and orientation angle of the Rotated RoI, respectively. It is easy to produce
the final Rotated RoI feature using the affine transformation:

$\displaystyle\left[\begin{array}[]{c}x^{\prime}\\\ y^{\prime}\\\
1\end{array}\right]$
$\displaystyle=\mathbf{R}\mathbf{T}\left[\begin{array}[]{c}x\\\ y\\\
1\end{array}\right]$ (1)
$\displaystyle=\left[\begin{array}[]{ccc}\cos\alpha&\sin\alpha&0\\\
-\sin\alpha&\cos\alpha&0\\\
0&0&1\end{array}\right]\left[\begin{array}[]{lll}1&0&-c_{x}\\\ 0&1&-c_{y}\\\
0&0&1\end{array}\right]\left[\begin{array}[]{c}x\\\ y\\\ 1\end{array}\right]$

where $(x,y)$ is a point in the input feature map, and
$(x^{\prime},y^{\prime})$ refers to the corresponding points in the rotated
feature map. The final horizontal feature of each text is extracted from the
final Rotated RoI feature, and resized to the fixed size ($h\times w$).

Recognition Head. Benefiting from the shared backbone feature and Rotated RoI,
we design an attention-based recognition head for end-to-end training.
Following the PAN++ [42], our head is also a seq2seq model composed of a
starter and a decoder. The starter produces SOS’s feature vector with a linear
transformation (embedding layer), which is the input of the decoder at the
initial time step. The decoder predicts symbol $y_{rec}$ with two LSTM layers
and one multi-head attention layer at each time step, and the output symbol of
the previous step is fed into later LSTM until predicting the end of the
string (EOS) token. The attention-based recognition head is a basic and simple
recognition network, which enables the model to learn a character-level
language model representing output class dependencies.

###  3.2 Temporal Tacking Loss for Text Query

Multi-orient Box Matching for Single Frame. To fit the rotated text instances,
similar to TransVTSpotter [47], we use the multi-orient box bipartite matching
for detecting and tracking rotated scene text, where the core idea is an angle
prediction of a rotated bounding box. Similar to DETR [2], the proposed method
infers a fixed-size set of $N$ predictions, then produces an optimal bipartite
matching between predicted and ground-truth objects, finally optimizes object-
specific (rotated bounding box) losses. For $t$-th frame, let us denote the
ground truth set of objects by $y^{t}$, and
$\hat{y}=\\{\hat{y}_{i}^{t}\\}_{i=1}^{N}$ the set of $N$ predictions. To find
a bipartite matching between these two sets, we search for a permutation of
$N$ elements $\sigma\in\mathfrak{S}_{N}$ with the lowest cost:

$\hat{\sigma}^{t}=\operatorname*{arg\,min}_{\sigma\in\mathfrak{S}_{N}}\sum_{i}^{N}{\cal
L}_{\rm match}(y_{i}^{t},\hat{y}^{t}_{\sigma(i)}),$ (2)

where ${\cal L}_{\rm match}(y_{i}^{t},\hat{y}^{t}_{\sigma(i)})$ is a pair-wise
_matching cost_ (Hungarian algorithm [12]) between ground-truth $y_{i}$ and a
prediction with index $\sigma(i)$ in $t$-th frame. Each element $i$ of the
ground-truth set can be seen as a $y_{i}=(c_{i},b_{i},a_{i})$ where $c_{i}$ is
the target class label, $b_{i}\in[0,1]^{4}$ is a vector that defines ground-
truth box center coordinates and its height and width relative to the image
size, and $a_{i}$ is rotation angle between the longest edge of arbitrary-
oriented box and horizontal line (x-axis). For the prediction with index
$\sigma(i)$ we define probability of class $c_{i}$ as
$\hat{p}_{\sigma(i)}(c_{i})$, the predicted box as $\hat{b}_{\sigma(i)}$, and
the predicted angle as $\hat{a}_{\sigma(i)}$. Thus we define ${\cal L}_{\rm
match}(y_{i}^{t},\hat{y}^{t}_{\sigma(i)})$ as:

$\displaystyle{\cal L}_{\rm match}(y_{i}^{t},\hat{y}^{t}_{\sigma(i)})$
$\displaystyle=-\mathds{1}_{\\{c_{i}\neq\emptyset\\}}\hat{p}^{t}_{\sigma(i)}(c_{i})$
(3) $\displaystyle+\mathds{1}_{\\{c_{i}\neq\emptyset\\}}{\cal L}_{\rm
box}(b^{t}_{i},\hat{b}^{t}_{\sigma(i)})$
$\displaystyle+\mathds{1}_{\\{c_{i}\neq\emptyset\\}}{\cal L}_{\rm
angle}(a^{t}_{i},\hat{a}^{t}_{\sigma(i)})\,.$

Finally, we could compute the detection loss function with all pairs matched
at the $t$-th frame:

$\displaystyle{\cal L}_{\rm Hungarian}(y^{t},\hat{y}^{t})$
$\displaystyle=\sum_{i=1}^{N}\Bigl{[}-\log\hat{p}^{t}_{\hat{\sigma}(i)}(c_{i})$
(4) $\displaystyle+\mathds{1}_{\\{c_{i}\neq\emptyset\\}}{\cal L}_{\rm
box}(b^{t}_{i},\hat{b}^{t}_{\hat{\sigma}}(i))$
$\displaystyle+\mathds{1}_{\\{c_{i}\neq\emptyset\\}}{\cal L}_{\rm
angle}(a^{t}_{i},\hat{a}^{t}_{\sigma(i)})\Bigl{]}\,,$

where ${\cal L}_{\rm box}(\cdot)$ is a linear combination of the $\ell_{1}$
loss and the generalized IoU loss [30, 2]. Specifically, the linear
combination is defined by $\lambda_{\rm iou}{\cal L}_{\rm
iou}(b_{i},\hat{b}_{\sigma(i)})+\lambda_{\rm
L1}||b_{i}-\hat{b}_{\sigma(i)}||_{1}$ where $\lambda_{\rm iou},\lambda_{\rm
L1}\in\mathbb{R}$ are hyperparameters. And ${\cal L}_{\rm angle}(\cdot)$
refers to a cosine angle embedding loss $1-cos(\hat{a}_{\sigma(i)}-a_{i})$,
enabling the network to fit rotated text instances.

Temporal Tracking Loss over Multiple Frames. Multi-oriented box matching solve
rotated scene text detection by the rotation angle prediction, how to track
each text instance with the same text query accurately becomes another vital
problem. Motivated by DETR [2, 57] and MOTR [51], we propose a concept text
query to model video text spotting over long-range sequence. If the match in
Equation. 2 is successful, the matched text query $Q_{text}$ is responsible to
track the corresponding text instance until it disappears.

To achieve this goal, the fixed-size set of $N$ queries needs to contain two
parts: empty queries $Q(\emptyset)$ for detecting the new-born text instances
and text queries $Q_{text}$ for tracking already tracked text instances. For
new-born text instances, an optimal bipartite matching in Equation. 2 is
produced between prediction from empty queries $Q(\emptyset)$ and ground truth
of new-born text instances. For already tracked text instances, without the
optimal bipartite matching, the already matched text queries $Q_{text}$ are
inherited from the previous frame. Finally, the matched empty queries
$Q(\emptyset)$ and the inherited text queries $Q_{text}$ are concatenated
together and input to the Transformer decoder to update the representation in
the current frame. For $t$-th frame, the fixed-size of $N$ queries can be
formulated as:

$\begin{split}Q^{t}_{text}=\left\\{\begin{array}[]{lcl}Q^{t}_{\sigma(i)}(\emptyset)&&t=1\\\
Q^{t}_{\sigma(i)}(\emptyset)\cup Q^{t-1}_{text}&&t\geq
2\end{array}\right.\end{split}$ (5)

where $\sigma(i)$ refers to the lowest matching cost index from Hungarian
algorithm in Equation. 2. For the initial frame($t=1$), since no already
matched text queries $Q_{text}$ from the previous frame, empty query set with
index $\sigma(i)$ is used to localize the initial objects and generate the
initial text query set for the next frame($t=2$). For $t$-th frame($t>=2$),
the query set contain two parts. Empty query set with index $\sigma(i)$ from
the pair-wise matching is used to detect new-born text instances. Text query
set $Q_{text}$, as the already matched in the previous frame, track already
tracked text instances.

Different from the previous methods [4, 43] training with single frame without
temporal information, TransDeTR takes a video clip (around 8 frames) as input
to learn long-range temporal variances. The overall loss in the whole video
clip sequence($T$ frames in total) could be computed:

$\begin{split}\mathcal{L}_{track}(y,\hat{y})=\frac{\sum\limits_{t=1}^{T}({\cal
L}_{\rm
Hungarian}(y^{t},\hat{y}^{t}))+\mathcal{L}(y^{t},\hat{y}^{t}_{Q^{t-1}_{text}}))}{\sum\limits_{t=1}^{T}(G^{t})}\,,\end{split}$
(6)

where $G^{t}$ denotes the total number of the ground-truths at $T_{i}$ frame.
Similar to the ${\cal L}_{\rm Hungarian}(y^{t},\hat{y}^{t})$ in Equation. 4,
the $\mathcal{L}(y^{t},\hat{y}^{t}_{Q^{t-1}_{text}})$ is the detection loss of
a single frame. The difference is it corresponding to the already matched text
queries $Q^{t-1}_{text}$ and $\hat{\sigma}^{t-1}$ from the previous frame.

###  3.3 Loss Function

The proposed pipeline mainly contains two losses, _i.e.,_ temporal tracking
loss and recognition loss, which are responsible for three different tasks,
_i.e.,_ text detection, tracking, and recognition tasks. The whole loss
function can be formulated as Equation. 7:

$\mathcal{L}=\sigma_{1}\mathcal{L}_{track}+\sigma_{2}\mathcal{L}_{rec}~{},$
(7)

where $\sigma_{1}$ and $\sigma_{2}$ are two weight parameters. The loss
function $\mathcal{L}_{rec}$ of text recognition refers to Cross-Entropy loss
between ground truth and predicted words. $\mathcal{L}_{track}$ is the
temporal tracking loss composed of two parts, and each part includes three su-
losses, _e.g.,_ class loss, rotation angle loss ${\cal L}_{\rm angle}(\cdot)$,
and generalized IoU loss ${\cal L}_{\rm box}(\cdot)$, as shown in Equ. 6.

###  3.4 Inference

During inference, many hand-crafted components and hyper-parameter, _e.g.,_
explicit track associations, NMS are no longer required. The fixed-size set of
$N$ queries are used to predict a set of text moving trajectories and the
corresponding content. And we use classification scores of prediction
$\hat{a}_{\sigma(i)}$ to determine when a text instance appears and
disappears. A text instance appears while classification scores are higher
than one threshold $\tau$. In turn, we remove the text instance while its
classification score is lower than the threshold $\tau$. In all experiments,
we set the threshold $\tau$ to $0.5$.

Table 1: Multi-Oriented Box Matching(MOBM) for detection and tracking task on
ICDAR2015video [8]. Cosine loss brings more gains than L1 loss.

Method | Precision | Recall | F-measure |  ${\rm ID_{F1}}$$\uparrow$ |  ${\rm MOTA}$$\uparrow$ |  ${\rm MOTP}$$\uparrow$
---|---|---|---|---|---|---
ours,w/o MOBM | 67.7 | 56.1 | 61.5 | 41.1 | 18.4 | 68.7
ours,w/ MOBM($L1$) | 77.4 | 70.5 | 73.8 | 63.3 | 45.6 | 76.1
ours,w/ MOBM($Cosine$) | 80.6 | 70.2 | 75.0 | 65.5 | 47.7 | 74.1

Table 2: Effect of Temporal Tracking Loss(Text Query-based) for Text Tracking Task. Deformable-DETR [57] is used as the base detector for text tracking on ICDAR2015 (video) [8]. The related settings of IoU and Transcription-based follows the work [43]. Method |  ${\rm ID_{F1}}$$\uparrow$ |  ${\rm MOTA}$$\uparrow$ |  ${\rm MOTP}$$\uparrow$
---|---|---|---
IoU-based | 56.3 | 43.1 | 75.8
Transcription-based | 56.9 | 43.5 | 75.9
Temporal Tracking Loss for Text Query | 65.5 | 47.7 | 74.1
Table 3: The input sequence length for text tracking task on ICDAR2015 (video) [8]. Long-range temporal information brings huge gains. Length |  ${\rm ID_{F1}}$$\uparrow$ |  ${\rm MOTA}$$\uparrow$ |  ${\rm MOTP}$$\uparrow$ |  ${\rm M\mbox{-}Tracked}$$\uparrow$ |  ${\rm M\mbox{-}Lost}$$\downarrow$
---|---|---|---|---|---
3 | 61.2 | 44.8 | 74.1 | 38.9 | 34.5
4 | 63.4 | 45.5 | 74.1 | 40.1 | 33.2
5 | 65.2 | 47.3 | 74.0 | 42.0 | 32.5
6 | 65.5 | 47.7 | 74.1 | 42.0 | 32.1

(a) Effect of decoder layer number and the shorter side of input image.
‘Shorter Side’ denotes the length of image shorter side.

(b) Comparison with Free [4] in FPs, FNs and ID switch for text tracking task
on ICDAR2013 [9]. ‘$\times 10$’ denotes $10$ times.

##  4 Experiments

###  4.1 Datasets

ICDAR2013 Video [9] is proposed in the ICDAR2013 Robust Reading Competition,
which contains 13 videos for training and 15 videos for testing. These videos
are harvested from indoors and outdoors scenarios, and each text is labeled as
a quadrangle with 4 vertexes in word-level.

ICDAR2015 Video [8] is the expanded version of ICDAR2013 (video), which
consists of a training set of 25 videos (13,450 frames) and a test set of 24
video (14,374 frames). Similar to ICDAR2013 (video), text instances in this
dataset are labeled at the word level. Quadrilateral bounding boxes and
transcriptions are provided.

Minetto [24] contains 5 videos in outdoor scenes. The resolution is fixed as
640×480. Each text is labeled in the form of axis-aligned bounding box. As all
videos are for testing, we use the model trained on ICDAR2015 Video to
evaluate this dataset directly.

YouTube Video Text (YVT) [26] dataset is harvested from YouTube, contains $30$
videos, where $15$ videos for training and $15$ videos for testing. Different
from the above datasets, it contains web videos except for scene videos.

###  4.2 Implementation Details

All the experiments are conducted on PyTorch with Tesla V100 GPUs. We use the
MOTR [51] as our basic network. For all experiments in our paper, the model is
firstly pretrained on the COCO-Text [36], and then finetune on other video
datasets. COCO-Text is the largest scene text detection dataset with 63,686
images, which reuses the images from MS-COCO dataset [15]. For static images
from COCO-Text, we apply the random shift [55] to generate video clips with
pseudo tracks. AdamW [19] as the optimizer for total 20 epochs with the
initial learning rate of 2e-4\. The learning rate decays to 2e-5 at 10 epochs.
The batch size is set to be 1, where each batch contains 8 frames. The data
augmentation includes random horizontal, scale augmentation, resizing the
images whose shorter side is by 480-800 pixels while the longer side is by at
most 1333 pixels. Following the setting in MOTR [51], we randomly sample key
frames from video sequences with an interval to solve the problem of different
frame rates, where the max interval is set to 5 in our experiments. All
inference speed performances are tested with a batch size of 1 on a V100 GPU
and a 2.20GHz CPU in a single thread. In the metric of the paper, Mostly
Tracked ($M\mbox{-}Tracked$) refers to the number of text instance tracked for
at least 80 percent of lifespan, Mostly Lost ($M\mbox{-}Lost$) denotes the
number of objects tracked less than 20 percent of lifespan in tracking.

Inconsistency from Evaluation Metric. The inconsistency between the reported
performance and the original paper lies in the update of the evaluation
metric. From 2020, Robust Reading Competition Website
111https://rrc.cvc.uab.es/?ch=3&com=evaluation&task=1 has updated the
evaluation method, and the old metric and ground truth is not available. All
works have already used the new metric after 2020. As for ICDAR2015 Video, we
test it via the online evaluation on Robust Reading Competition Website. Other
datasets are evaluated by the offline metric, and we have provided the metric
in our open source code.

###  4.3 Ablation Study

Here, we conduct five groups of ablation experiments to study the core factors
of TransDeTR.

Multi-Oriented Box Matching. To evaluate the importance of the multi-oriented
box matching cost, we train several models turning it ${\cal L}_{\rm
angle}(a^{t}_{i},\hat{a}^{t}_{\sigma(i)})$ on and off. As shown in Table 1,
for detection and tracking tasks on ICDAR2015 [8], we train a model without
multi-oriented box matching and angle prediction, a model with
$L1\mbox{-}based$ angle loss, and a model with $Cosine\mbox{-}based$ angle
loss. With multi-oriented box matching (cosine loss), the model achieves
$13.5\%$ improvements of F-measure than the counterpart without the matching
for the text detection task, $24.4\%$ ${\rm ID_{F1}}$ improvements for text
tracking task on ICDAR2015 video.

Temporal Tracking Loss. Query-based temporal text loss plays a vital role in
the proposed video text tracker. As shown in Table 2, we evaluate the
effectiveness by comparing with other association tracking methods, _e.g.,_
IoU-based association tracker and transcription-based association tracker.
Following the work [43], IoU-based and transcription-based association
trackers link and match text objects in adjacent frames by IoU and edit
distance of text instances. The query-based method with temporal text loss
shows a significant improvement than the other methods, especially for ${\rm
ID_{F1}}$, which achieves up to $8.6\%$ improvement. The previous tracing-by-
match paradigm requires successful text match in each adjacent frame, which is
difficult to achieve. Matching failure in any adjacent frames would lead to
the ID switching. The query-based model temporal text instances relationship
during multi-frame, instead of matching text instance in the adjacent frames.
The novel paradigm enhances the tracking stably by alleviating the problem of
ID switching and object missing.

Video Sequence Length. The input video sequences of different lengths make a
different impact on temporal relation modeling. As shown in Table 3 (c), we
experiment with various video sequence lengths to explore the effect of the
sequence length setting. The longer input video sequences, the better
performance, the video sequence with six frames shows $4.3\%$ improvement than
the video of three frames. With the increasing of video sequence length, the
model presents a better performance, and achieves the best with $65.5\%$ ${\rm
ID_{F1}}$ and $47.7\%$ ${\rm MOTA}$. Therefore, we argue that the longer video
sequence with more frames contributes to better temporal relation modeling.

Decoder Layer and Shorter Side of Input Image. We evaluate the importance of
decoder layer number and the image shorter side in Figure 3.4. With the
increasing of decoder layer number, the performance presents a gradual
improvement, achieving the highest accuracy with $6$ layers. Similarly, better
performance is presented with a shorter side of image increasing. And the
performance drops dramatically at least $5\%$ while the shorter side is less
than $500$ pixels, where much low-level feature information lost. In all
experiments, we set the decoder layer number and shorter side of the image to
$6$ and $800$ for the trade-off between accuracy and speech, respectively.

Efficiency for fixing ID switching, FP and FN. To further present the
efficiency of TransDeTR for text tracking task, we summarize and compare the
number of false-positive, false-negative, and ID switches on ICDAR2013 [9] in
Figure 3.4. Compared with Free [4], our TransDeTR suppresses obviously false
positive, false negative, and ID switch by long-sequence modeling. Especially
for false negatives, the improved performance is quite remarkable, with a
1,239 false negatives decrease. We argue that the improvements mainly come
from these undetected texts in the middle frames. The previous paradigms rely
on the image detection model, which is difficult to obtain a steady detection
performance for all text instances during long-sequence video frames. Our
TransDeTR searches these texts with low confidence by query-based long-
sequence modeling, which off huge dependence on detection model.

Table 4: Video text detection performance on ICDAR2013(video) [9]. TransDeTR presents better performance and faster inference speed. Method | Video Text Detection/% | FPS
---|---|---
Precision | Recall | F-measure
Epshtein et al. [6] | 39.8 | 32.5 | 35.9 | -
Zhao et al. [52] | 47.0 | 46.3 | 46.7 | -
Yin et al. [48] | 48.6 | 54.7 | 51.6 | -
Khare et al. [10] | 57.9 | 55.9 | 51.7 | -
Wang et al. [37] | 58.3 | 51.7 | 54.5 | -
Shivakumara et al. [32] | 61.0 | 57.0 | 59.0 | -
Wu et al. [44] | 63.0 | 68.0 | 65.0 | -
Yu et al. [50] | 82.4 | 56.4 | 66.9 | -
Wei et al. [7] | 75.5 | 64.1 | 69.3 | -
Free [4] | 79.7 | 68.4 | 73.6 | 8.8
TransDeTR (ours) | 80.6 | 70.2 | 75.0 | 15.4
Figure 3: Tracking and spotting Visualization for different methods. Compared with Free [4] (top row), our TransDeTR (bottom row) shows better performance for solving false positive and ID switch problems. Table 5: Text tracking performance on YVT(video) [26], ICDAR2013(video) [9] and Minetto [24]. ‘M-Tracked’ and ‘M-Lost’ denote ‘Mostly Tracked’ and ‘Mostly Lost’, respectively. ∗ denotes our implement with the official code and weight. Dataset | Method | Video Text Tracking/% | FPS
---|---|---|---
${\rm ID_{F1}}$$\uparrow$ |  ${\rm MOTA}$$\uparrow$ |  ${\rm MOTP}$$\uparrow$ |  ${\rm M\mbox{-}Tracked}$$\uparrow$ |  ${\rm M\mbox{-}Lost}$$\downarrow$
ICDAR2015[8] |  USTB_TexVideo [8] | 25.9 | 7.4 | 70.8 | 7.4 | 66.1 | -
StradVision-1 [8] | 25.9 | 7.9 | 70.2 | 6.5 | 70.8 | -
USTB(II-2) [8] | 21.9 | 12.3 | 71.8 | 4.8 | 72.3 | -
AJOU [11] | 36.1 | 16.4 | 72.7 | 14.1 | 62.0 | -
Free [4] | 57.9 | 43.2 | 76.7 | 36.6 | 44.4 | 8.8
TransVTSpotter [47] | 57.3 | 44.1 | 75.8 | 34.3 | 33.7 | 9.0
TransDeTR | 65.5 | 47.7 | 74.1 | 42.0 | 32.1 | 16.7
Minetto[24] |  Zuo et al. [58] | - | 56.4 | 73.1 | - | - | -
Pei et al. [27] | - | 73.1 | 57.7 | - | - | -
AGD&AGD[50] | - | 75.6 | 74.7 | - | - | -
Yu et al.[50] | - | 81.3 | 75.7 | - | - | -
TransVTSpotter[47] | 74.7 | 84.1 | 77.6 | - | - | 9.0
TransDeTR | 76.6 | 86.5 | 75.5 | 94.3 | 0 | 18.2
ICDAR2013[9] |  YORO∗ [3] | 62.5 | 47.3 | 73.7 | 33.1 | 45.3 | 14.3
TransDeTR | 67.2 | 54.7 | 76.6 | 43.5 | 33.2 | 15.4
YVT[26] |  Free[4] | - | 54.0 | 78.0 | - | - | -
TransDeTR | 73.4 | 55.7 | 76.2 | 57.2 | 33.8 | 17.5

###  4.4 Comparison with State-of-the-arts

We compare TransDeTR against state-of-the-art methods in terms of three tasks,
_i.e.,_ video text detection, video text tracking and video text spotting.

Video text detection. Following previous works [4], ICDAR2013 video as a
popular public dataset is used for use to verify the effectiveness of
TransDeTR. Table 4 presents the video text detection performance on ICDAR2013
(video) [9]. TransDeTR achieves the best result among methods without any
bells and whistles. Compare with Free [4] using a powerful base detector EAST
[56], our TransDeTR still achieves $1.4\%$ F-measure improvement without hand-
crafted components such as NMS. Therefore, we consider that the query-based
temporal modeling from our TransDeTR can help the network to learn a better
feature representation for video text detection. Note: we do not evaluate the
performance on ICDAR2015 video due to the test set annotation of that is not
available.

Figure 4: Visualization of TransDeTR for video text spotting results on ICDAR2015video [8]. Table 6: Video text spotting performance on ICDAR2015(video). ‘aug’ denotes data augmentation for random cropping. Method | Backbone | End to End Video Text Spotting/% | FPS
---|---|---|---
${\rm ID_{F1}}$$\uparrow$ |  ${\rm MOTA}$$\uparrow$ |  ${\rm MOTP}$$\uparrow$ |  ${\rm M\mbox{-}Tracked}$$\uparrow$ |  ${\rm M\mbox{-}Lost}$$\downarrow$
Separate Framework |  |  |  |  |  |  |
USTB_TexVideo(II-2) [8] |  | 21.3 | 13.2 | 66.6 | 6.6 | 67.7 | -
USTB_TexVideo [8] |  | 28.2 | 15.6 | 68.5 | 9.5 | 60.7 | -
StradVision-1 [8] |  | 32.0 | 9.0 | 70.2 | 8.9 | 59.5 | -
Free [4] | ResNet50 | 61.9 | 53.0 | 74.9 | 45.5 | 35.9 | 8.8
TransVTSpotter[47] | ResNet50 | 61.5 | 53.2 | 74.9 | - | - | 9.0
End-to-End framework |  |  |  |  |  |  |
TransDeTR | ResNet50 | 70.4 | 58.4 | 75.2 | 32.0 | 20.8 | 16.7
TransDeTR (aug) | ResNet50 | 72.8 | 60.9 | 74.6 | 33.6 | 20.8 | 16.7

Video text tracking. Since the tracking task is the core task in this work, we
adopt four common datasets(_i.e.,_ ICDAR2015 [8], YVT (video) [26], ICDAR2013
(video) [9] and Minetto [24]) to evaluate the task. Table 5 presents
performance comparison on ICDAR2015 (video) [8]. Without convolutions and
hand-crafted components such as NMS, our TransDeTR achieves a great
improvement of $7.6\%$ ${\rm ID_{F1}}$ than the current SOTA method [4]. This
is reasonable, ${\rm M\mbox{-}Tracked}$ of TransDeTR is obvious improvement
with $5.4\%$, which denotes TransDeTR can track text instance more robustly
and accurately. For the improvement, Fig. 3 presents more direct visualization
with less false negatives. Besides, compare with the previous works(_e.g.,_
Free [4]) using multi-stage matching and refinement, our TransDeTR is a simple
end-to-end framework. As for ICDAR2013 video dataset, compared with YORO [3],
our TransDeTR present better performance with $4.7\%$ ${\rm ID_{F1}}$ and
$7.4\%$ ${\rm MOTA}$ improvements, while its inference speed is faster. For
Minetto and YVT datasets, our TransDeTR also shows a powerful performance with
$2.4\%$ and $1.7\%$ MOTA improvements than the previous methods, respectively.
The better performance on YVT verify the robustness and generalization of our
TransDeTR, while the model is trained on ICDAR2015 video and test directly on
its test set. Besides, to further present the effectiveness of TransDeTR, we
visualize the tracking and spotting results of Free [4](SOTA method) and
TransDeTR in Figure 3. Compared with Free [4], our TransDeTR shows a more
stable and accurate tracking trajectory by alleviating the problem of ID
switching and object missing (red circle). And more statistics and analysis
can be found in Fig. 3.4 and ablation study.

Video text spotting. Video text spotting task, as the final aim, requires one
to simultaneously track and recognize text instances accurately in a video
sequence. Table 6 provides the performance comparison of the task on
ICDAR2015(video) [8]. TransDeTR inherits the advantage from query-based
tracking, achieves $11.3\%$ (${\rm ID_{F1}}$) with $7.7\%$(${\rm MOTA}$)
improvements than the current SOTA method, while the inference speed is faster
($16.7$ FPS $v.s$ $9.0$ FPS). Meanwhile, TransDeTR drastically reduces the
‘Mostly Lost’ by alleviating up to $15.0\%$ false negatives. Besides,
comparing the SOTA method Free [4] with a complicated pipeline(_e.g.,_ NMS,
Aggregation, Match and Score), our TransDeTR is a simple and clean framework
without hand-crafted components. And more surprising is our TransDeTR is a
real end-to-end trainable framework, compared previous works with multiple
models and hand-crafted strategies. To further present the effectiveness of
the proposed method, we list more high-quality visualization results in
Figure. 4.

##  5 Potential Negative Societal Impact and Limitations

Potential Negative Societal Impact Similar to the previous video text spotting
methods [4], the proposed TransDeTR have a potential negative risk. Some
important private information in video can be easily collected automatically
by the method engaged in fraud, marketing, or other illegal activities.

Limitations The proposed method shows great performance on video text
detection, tracking, and spotting tasks. But the speech with around $10$ FPS
is a limitation for its application. Compared with the existing real-time
image-based detector, the speech of our TransDeTR still needs further
optimization. And we will continue to explore a real-time transformer-based
video text spotting method in future work.

##  6 Conclusion

In this paper, we firstly present an end-to-end trainable video text
DEtection, Tracking and Recognition framework with Transformers (TransDeTR),
which views the VTS task as a direct long-range temporal variation problem.
Without explicit heuristics, TransDeTR models each temporal arbitrary-oriented
boxes tracking trajectory and text content implicitly by different ‘text
query’. With temporal tracking loss over multiple frames and recognition head,
the overall pipeline is significantly different and simpler than existing
approaches. Without convolutions and hand-crafted components such as dense
anchors and NMS, our TransDeTR achieves $65.5\%$ ${\rm ID_{F1}}$ for tracking
task and $72.8\%$ ${\rm ID_{F1}}$ for video text spotting task on ICDAR2015
(video), with $8.2\%$ and $11.3\%$ improvements than the previous SOTA method,
respectively. Meantime, the inference time is faster than previous methods. We
hope that the long-range modeling approach can be applied to more video-and-
language tasks in the future.

## References

  * [1] Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee.  Character region awareness for text detection.  In CVPR, pages 9365–9374, 2019.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, pages 213–229, 2020.
  * [3] Zhanzhan Cheng, Jing Lu, Yi Niu, Shiliang Pu, Fei Wu, and Shuigeng Zhou.  You only recognize once: Towards fast video text spotting.  In ACM MM, pages 855–863, 2019.
  * [4] Zhanzhan Cheng, Jing Lu, Baorui Zou, Liang Qiao, Yunlu Xu, Shiliang Pu, Yi Niu, Fei Wu, and Shuigeng Zhou.  Free: A fast and robust end-to-end video text spotter.  TIP, 30:822–837, 2020.
  * [5] Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang.  Dual encoding for video retrieval by text.  TPAMI, 2021.
  * [6] Boris Epshtein, Eyal Ofek, and Yonatan Wexler.  Detecting text in natural scenes with stroke width transform.  In CVPR, pages 2963–2970, 2010.
  * [7] Wei Feng, Fei Yin, Xu-Yao Zhang, and Cheng-Lin Liu.  Semantic-aware video text detection.  In CVPR, pages 1695–1705, 2021.
  * [8] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, and Shijian Lu.  Icdar 2015 competition on robust reading.  In ICDAR, pages 1156–1160, 2015.
  * [9] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, and Sergi Robles Mestre.  Icdar 2013 robust reading competition.  In ICDAR, pages 1484–1493, 2013.
  * [10] Vijeta Khare, Palaiahnakote Shivakumara, Raveendran Paramesran, and Michael Blumenstein.  Arbitrarily-oriented multi-lingual text detection in video.  MTA, 76(15):16625–16655, 2017.
  * [11] Hyung Il Koo and Duck Hoon Kim.  Scene text detection via connected component clustering and nontext filtering.  TIP, 22(6):2296–2305, 2013.
  * [12] Harold W. Kuhn.  The hungarian method for the assignment problem.  Naval Research Logistics Quarterly, 2(1):83–97, 1955.
  * [13] Hui Li, Peng Wang, and Chunhua Shen.  Towards end-to-end text spotting with convolutional recurrent neural networks.  In ICCV, pages 5238–5246, 2017.
  * [14] Pandeng Li, Yan Li, Hongtao Xie, and Lei Zhang.  Neighborhood-adaptive structure augmented metric learning.  In AAAI, 2022.
  * [15] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, pages 740–755. Springer, 2014.
  * [16] Xuebo Liu, Ding Liang, Shi Yan, Dagui Chen, Yu Qiao, and Junjie Yan.  Fots: Fast oriented text spotting with a unified network.  In CVPR, pages 5676–5685, 2018.
  * [17] Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang.  Abcnet: Real-time scene text spotting with adaptive bezier-curve network.  In CVPR, pages 9809–9818, 2020.
  * [18] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [19] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [20] Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai.  Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes.  In ECCV, pages 67–83, 2018.
  * [21] Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai.  Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes.  In ECCV, pages 67–83, 2018.
  * [22] Jianqi Ma.  Rrpn++: Guidance towards more accurate scene text detection.  arXiv preprint arXiv:2009.13118, 2020.
  * [23] Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue.  Arbitrary-oriented scene text detection via rotation proposals.  IEEE Transactions on Multimedia, 20(11):3111–3122, 2018.
  * [24] Rodrigo Minetto, Nicolas Thome, Matthieu Cord, Neucimar J Leite, and Jorge Stolfi.  Snoopertrack: Text detection and tracking for outdoor videos.  In ICIP, pages 505–508, 2011.
  * [25] Takeshi Mita and Osamu Hori.  Improvement of video text recognition by character selection.  In ICDAR, pages 1089–1093, 2001.
  * [26] Phuc Xuan Nguyen, Kai Wang, and Serge Belongie.  Video text detection and recognition: Dataset and benchmark.  In WACV, pages 776–783, 2014.
  * [27] Wei-Yi Pei, Chun Yang, Li-Yu Meng, Jie-Bo Hou, Shu Tian, and Xu-Cheng Yin.  Scene video text tracking with graph matching.  IEEE Access, 6:19419–19426, 2018.
  * [28] Sangeeth Reddy, Minesh Mathew, Lluis Gomez, Marçal Rusinol, Dimosthenis Karatzas, and CV Jawahar.  Roadtext-1k: Text detection & recognition dataset for driving videos.  In ICRA, pages 11074–11080, 2020.
  * [29] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  NeurIPS, 28:91–99, 2015.
  * [30] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In CVPR, pages 658–666, 2019.
  * [31] Xuejian Rong, Chucai Yi, Xiaodong Yang, and Yingli Tian.  Scene text recognition in multiple frames based on text tracking.  In ICME, pages 1–6, 2014.
  * [32] Palaiahnakote Shivakumara, Liang Wu, Tong Lu, Chew Lim Tan, Michael Blumenstein, and Basavaraj S Anami.  Fractals based multi-oriented text detection system for recognition in mobile video images.  PR, 68:158–174, 2017.
  * [33] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov.  Unsupervised learning of video representations using lstms.  In ICML, pages 843–852, 2015.
  * [34] Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo.  Transtrack: Multiple-object tracking with transformer.  arXiv preprint arXiv:2012.15460, 2020.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, pages 5998–6008, 2017.
  * [36] Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie.  Coco-text: Dataset and benchmark for text detection and recognition in natural images.  arXiv preprint arXiv:1601.07140, 2016.
  * [37] Lan Wang, Yang Wang, Susu Shan, and Feng Su.  Scene text detection and tracking in video with background cues.  In ICMR, pages 160–168, 2018.
  * [38] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li.  Transformer meets tracker: Exploiting temporal context for robust visual tracking.  In CVPR, pages 1571–1580, 2021.
  * [39] Peng Wang, Hui Li, and Chunhua Shen.  Towards end-to-end text spotting in natural scenes.  TPAMI, 2021.
  * [40] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [41] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao.  Shape robust text detection with progressive scale expansion network.  In CVPR, pages 9336–9345, 2019.
  * [42] Wenhai Wang, Enze Xie, Xiang Li, Xuebo Liu, Ding Liang, Yang Zhibo, Tong Lu, and Chunhua Shen.  Pan++: Towards efficient and accurate end-to-end spotting of arbitrarily-shaped text.  TPAMI, 2021.
  * [43] Xiaobing Wang, Yingying Jiang, Shuli Yang, Xiangyu Zhu, Wei Li, Pei Fu, Hua Wang, and Zhenbo Luo.  End-to-end scene text recognition in videos based on multi frame tracking.  In ICDAR, volume 1, pages 1255–1260, 2017.
  * [44] Liang Wu, Palaiahnakote Shivakumara, Tong Lu, and Chew Lim Tan.  A new technique for multi-oriented scene text line detection and tracking in video.  TMM, 17(8):1137–1152, 2015.
  * [45] Weijia Wu, Ning Lu, Enze Xie, Yuxing Wang, Wenwen Yu, Cheng Yang, and Hong Zhou.  Synthetic-to-real unsupervised domain adaptation for scene text detection in the wild.  In ACCV, 2020.
  * [46] Weijia Wu, Jici Xing, and Hong Zhou.  Textcohesion: Detecting text for arbitrary shapes.  arXiv preprint arXiv:1904.12640, 2019.
  * [47] Weijia Wu, Debing Zhang, Yuanqiang Cai, Sibo Wang, Jiahong Li, Zhuang Li, Yejun Tang, and Hong Zhou.  A bilingual, openworld video text dataset and end-to-end video text spotter with transformer.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [48] Xu-Cheng Yin, Xuwang Yin, Kaizhu Huang, and Hong-Wei Hao.  Robust text detection in natural scene images.  TPAMI, 36(5):970–983, 2013.
  * [49] Xu-Cheng Yin, Ze-Yu Zuo, Shu Tian, and Cheng-Lin Liu.  Text detection, tracking and recognition in video: a comprehensive survey.  TIP, 25(6):2752–2773, 2016.
  * [50] Hongyuan Yu, Yan Huang, Lihong Pi, Chengquan Zhang, Xuan Li, and Liang Wang.  End-to-end video text detection with online tracking.  PR, 113, 2021.
  * [51] Fangao Zeng, Bin Dong, Tiancai Wang, Cheng Chen, Xiangyu Zhang, and Yichen Wei.  Motr: End-to-end multiple-object tracking with transformer.  arXiv preprint arXiv:2105.03247, 2021.
  * [52] Xu Zhao, Kai-Hsiang Lin, Yun Fu, Yuxiao Hu, Yuncai Liu, and Thomas S Huang.  Text from corners: a novel approach to detect text and caption in videos.  TIP, 20(3):790–799, 2010.
  * [53] Zelin Zhao, Ze Wu, Yueqing Zhuang, Boxun Li, and Jiaya Jia.  Tracking objects as pixel-wise distributions, 2022.
  * [54] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, and Fu.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In CVPR, pages 6881–6890, 2021.
  * [55] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking objects as points.  In ECCV, pages 474–490, 2020.
  * [56] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang.  East: an efficient and accurate scene text detector.  In CVPR, pages 5551–5560, 2017.
  * [57] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.
  * [58] Ze-Yu Zuo, Shu Tian, Wei-yi Pei, and Xu-Cheng Yin.  Multi-strategy tracking based text detection in scene videos.  In ICDAR, pages 66–70, 2015.