11institutetext: Niels Bohr International Academy, Niels Bohr Institute,
Blegdamsvej 17, DK-2100 Copenhagen Ø, Denmark
11email<EMAIL_ADDRESS>22institutetext: Departament d’Astronomia i
Astrofísica, Universitat de València, Dr Moliner 50, 46100, Burjassot
(València), Spain

# Gravitational Wave Memory Imprints on the CMB from Populations
of Massive Black Hole Mergers

Lorenz Zwick 11 David O’Neill 11 Kai Hendriks 11 Philip Kirkeberg 11 Miquel
Miravet-Tenés 22

(Received April 10, 2024; accepted ???)

###### Abstract

Aims. To showcase and characterise the rich phenomenology of temperature
fluctuation patterns that are imprinted on the CMB by the gravitational wave
memory (GWM) of massive black hole (BH) mergers.

Methods. We analyse both individual binaries as well as populations of
binaries, distributed in local cosmological boxes at a given redshift.

Results. The magnitude of the temperature fluctuations scales primarily as a
function of binary total mass and pattern angular scale, and accumulates as a
random-walk process when populations of mergers are considered. Fluctuations
of order $\sim 10^{-10}$ K are easily reached across scales of $\sim
1^{\prime}$ to $\sim 1^{\circ}$ for realistic volumetric merger rates of 10-3
Mpc-3 Gyr-1, as appropriate for massive galaxies at $z=1$. We determine
numerically that GWM temperature fluctuations result in a universal power
spectrum with a scaling of $P(k)\propto k^{2.7}$.

Conclusions. While not detectable given the limitations of current all-sky CMB
surveys, our work explicitly shows how every black hole merger in the Universe
left us its unique faint signature.

###### Key Words.:

Black hole physics – gravitational wave memory – cosmic background radiation

##  1 Introduction

Gravitational wave memory (GWM) is a phenomenon in general relativity, in
which the passing of a GW leaves a permanent, transverse displacement in a
ring of test masses (Zel’dovich & Polnarev, 1974; Christodoulou, 1991; Thorne,
1992; Blanchet & Damour, 1992; Talbot et al., 2018). The observational
consequences of GWM are plentiful, ranging from improved parameter estimation
for individual compact object mergers (Braginsky & Thorne, 1987; Favata, 2009,
2010; Lasky et al., 2016; Tiwari et al., 2021), the existence of astrophysical
GWM backgrounds in various frequency bands (Zhao & Cao, 2022), manifestations
in cosmology (Tolish & Wald, 2016; Bieri et al., 2017; Jokela et al., 2022),
and finally its influence on the radiation from the cosmic microwave
background (CMB).

The latter is perhaps the most intriguing, as CMB observations constitute the
cornerstone of our current understanding of the early Universe (Aghanim et
al., 2020). GWM results in both the red-shifting of CMB radiation as well as
the mixing of power from different modes. Both of these aspects have been
beautifully studied in Madison (2020) and Madison (2021) for the first time.
However, the aforementioned works do not attempt to make an explicit
connection with astrophysical populations of sources, opting instead to treat
the problem with more generality. Additionally, these studies focus solely on
the deflection to CMB photon paths as an observable consequence of GWM. Very
recently, the former point has been addressed in Boybeyi et al. (2024),
showing how realistic populations of mergers produce an accumulation of GWM
that affects the astrometry of far-away point sources.

In this Letter, we focus instead on a different observable: the temperature
fluctuations directly imprinted on the CMB by the GWM of massive black hole
(MBH) mergers. We find that the accumulation of GWM caused by multiple mergers
grows such fluctuations in a random-walk-like fashion (see also Boybeyi et
al., 2024), leading to a rich and potentially extremely informative
phenomenology of patterns that lives a few orders of magnitude below the
observational limitations of current all-sky CMB surveys.

##  2 GWM from binary mergers sourcing CMB temperature fluctuations

GWs propagate from their source radially as transverse-traceless waves. Here
we assume the source to be a compact object binary with total mass $M$ and
inclination $\iota$. We expect GWM to be present once information about the
binary merger reaches a given space-time location, at a luminosity distance
$D$. At lowest order in the post-Newtonian expansion, the strain tensor for
linear GWM at merger is given by (Blanchet, 2014):

$\displaystyle\begin{split}h_{ij}^{\rm{mem}}&=-\frac{5}{28}\frac{GM}{c^{2}D}\sqrt{\frac{16\pi}{30}}\\\
&\quad\times\left(\begin{array}[]{ccc}0&0&0\\\
0&-(1+\cos\iota^{2})&-2\cos\iota\\\
0&-2\cos\iota&(1+\cos\iota^{2})\end{array}\right),\end{split}$ (1)

where $G$ is Newton’s gravitational constant, $c$ is the speed of light, and
the GW had been propagating in the $x$ direction. Note that here we assumed
equal mass binary components, and will do so for the remainder of this work.
Photons travelling through space may experience permanent distortions and
deflections caused by the GWM from one such merger event. Here we focus solely
on the resulting change in photon wavelength. While GWM also causes path
deflections (Madison, 2020, see also Kopeikin et al. (1999); Book & Flanagan
(2011) for the analogous effect from general GWs), the latter will be
dominated by gravitational lensing from the entire dark matter halo, by a
factor $\sim M_{\rm Halo}/M$. We leave the convolution of these two effects
for future work.

We devise a strategy to calculate the imprint of GWM on 2D patches of the
CMB111See https://www.youtube.com/watch?v=s2d9ROsMoF0 for a visualisation of
our setup.. We model MBH mergers as events, distributed in a local box placed
at a particular redshift. Each event $E$ is characterised by the following
labels:

$\displaystyle E:=\left[t_{\rm m},x,y,z,M,\iota,\gamma\right],$ (2)

where $t_{\rm m}$ is the local time of the merger, $x$, $y$ and $z$ its
spatial coordinates, while $\gamma$ is an additional orientation angle of the
MBH binary with respect to the observer reference frame. Each event is given a
total mass $M$, and is sampled from a uniform distribution in time,
inclination and orientation. The spatial coordinates of the events are instead
sampled from a 3D distribution with a given power spectrum of the form:

$\displaystyle P_{\rm 3D}(k)\propto k^{-n},$ (3)

with $k$ being the Fourier mode. Large values of $n$ describe highly spatially
clustered events. For the majority of our analysis, we draw events from an
uniform distribution, i.e. in the following we set $n=0$ unless otherwise
stated. Note that here we do not consider any possible clustering in time.

For our purposes it suffices to assume that GWM acts as a step function, which
propagates information about the merger event outwards in a shell. We model it
with a Heaviside function ${\Theta_{\rm H}}$:

$\displaystyle h_{ij}(\mathbf{x},t)=h_{ij}^{\rm{mem}}\times\Theta_{\rm
H}\left(t-t_{\rm m}-\frac{|\mathbf{x}_{\mathrm{m}}-\mathbf{x}|}{c}\right),$
(4)

where $\mathbf{x}_{\mathrm{m}}$ is the location of the merger. CMB photons
that intersect the shell are therefore instantaneously affected by GWM. Their
wavelength is either stretched or squeezed by the transverse-traceless metric
strain projected along the photon propagation vector. The resulting change in
wavelength is therefore maximised when photons intersect the expanding GWM
shell at right angles. Across CMB patches, this effect manifests as a
correlated wavelength perturbation pattern.

Hereafter, we will use Wien’s displacement law to relate a small change in
photon wavelength to a corresponding change in bulk blackbody temperature:

$\displaystyle\frac{\Delta\lambda_{\rm ph}}{\lambda_{\rm
ph}}\approx-\frac{\Delta T}{T},$ (5)

where $\lambda_{\rm ph}$ is the photon’s wavelength at peak spectral radiance
and $T$ is the mean CMB temperature. To produce 2D temperature fluctuation
maps, we can apply the GWM of single or multiple events to a surface of
photons, being careful to project each GWM strain correctly.

##  3 Temperature fluctuations from individual Major Mergers

Figure 1: Temperature fluctuation patterns for an individual merger event with
different inclinations $\iota=0,\pi/2$ and orientation $\gamma=0,\pi/4$. The
amplitude and physical scale of the fluctuations can be found by using Eq. 6.
Note that the average temperature fluctuation always vanishes, by merit of the
tracelessness of the GWM tensor. The grey dashed line illustrates the size of
the GWM shell at the time the photon surface meets the location of the source.
Figure 2: Three realisations of the CMB temperature fluctuations caused by the
GWM of a population of mergers. The merger events have a mass of $M=10^{9}$ M⊙
and are drawn from a uniform distribution in space-time and inclination and
orientation angles. As the number of mergers is increased according to a
specified volumetric merger rate $\mathcal{R}$, both the amplitude and the
standard deviation of the temperature fluctuations grows following a random
walk scaling. The colorbars are adjusted to account for this behaviour. With
sufficient sources within a box ($\sim 10$ or more), the temperature
fluctuations tend to follow universal power spectrum with a scaling $\propto
k^{-2.7}$, where $k$ are the modes.

In Fig. 1 we showcase the full GMW imprint of an individual merger event on
the CMB temperature, where we vary both inclination and orientation angles. We
find that the shape of the patterns is determined by the angles $\iota$ and
$\gamma$, while the amplitude and scale are determined by the mass $M$, the
merger time $t_{\rm m}$, the location of the merger along the line of sight
and the inclination (to a lesser degree). As expected, the average temperature
across the 2D image is unaffected. The patterns roughly scale with the size of
the GWM shell (grey dashed line) at the time the photon surface matches with
the location of the source.

We find the following scaling for the maximum amplitude of temperature
fluctuations caused by an individual event:

$\displaystyle\frac{\Delta T_{\mathrm{max}}}{T(z_{\rm m})}=$ $\displaystyle
2.6\times
10^{-13}\left(\frac{10^{8}\mathrm{pc}}{R}\right)\left(\frac{M}{10^{9}M_{\odot}}\right)$
$\displaystyle\times\left(1+0.48\cos{2\iota}\right)$ (6)

where $R>0$ is the radius of the GWM shell and $T(z_{\rm m})$ denotes the mean
CMB temperature at the redshift of the merger event. Here we scale our results
with values typical for SMBH binaries, as they are the loudest astrophysical
source of GWs, and therefore GWM. The maximum temperature perturbation scales
inversely with $R$, meaning that photons that intersect the GWM shell close to
the merger experience the largest temperature perturbations.

In order to turn a physical scale to an angular scale we need to specify a
redshift. Here we scale our results with a redshift $z=1$, which roughly
corresponds to the peak of the massive galaxy volumetric merger rate (Fakhouri
et al., 2010; O’Leary et al., 2021; Conselice et al., 2022). We find the
following scaling:

$\displaystyle\frac{\Delta T_{\mathrm{max}}}{T(z_{\rm m})}\approx 5.35\times
10^{-11}\left(\frac{1^{\prime}}{\beta}\right)\left(\frac{M}{10^{9}M_{\odot}}\right)\left(\frac{\mathcal{D}_{\theta}(1)}{\mathcal{D}_{\theta}(z_{\rm
m})}\right),$ (7)

where $\beta$ is the angular scale of the GWM shell, and we now neglect the
inclination dependance. Here $\mathcal{D}_{\theta}$ is the angular diameter
distance of a given cosmology, which peaks at roughly $z\sim 1$ in
$\Lambda$-CDM (Peebles, 1993). For our purposes, redshifts of $\sim 1$ are
therefore the most unfavourable, as by this effect alone temperature
fluctuations at a given angular scale will have a factor $\sim 2$ higher
amplitudes if produced at $z=7$ rather than $z=1$. Note that here we chose to
scale the results with one minute of arc, as the typical resolution of state-
of-the-art all-sky CMB missions is $\sim 5^{\prime}$ (The COrE Collaboration
et al., 2011; Planck Collaboration et al., 2020b).

In reality, the temperature patterns produced by the GWM of mergers are not
static. Different isochrone surfaces of CMB photons continuously traverse any
local box that contains MBH mergers, causing the projected patterns of
temperature fluctuations t vary in time, roughly in proportion to their
physical scale (as $\sim R/c$). We showcase several movies of time-evolving
fluctuations, the footage of which can be found on
https://www.youtube.com/watch?v=2iwVHoZ6VF4&list=PLllD9CwiToPKY7hjYHaJDGrB5KBjRPyi4&index=1.
For a single event, the GWM shell propagates radially with the passing of
time, affecting photons at greater distances from the source. The result is to
enlarge the physical scale of the pattern as well as decreasing its amplitude.
We can express the typical temperature fluctuation, as seen by a local
observer, as a function of the elapsed time $\tau$ after the merger has
occurred:

$\displaystyle\frac{\Delta T_{\mathrm{max}}}{T(z_{\rm m})}=$ $\displaystyle
8.4\times
10^{-13}\left(\frac{10^{8}\mathrm{yr}}{\tau}\right)\left(\frac{M}{10^{9}M_{\odot}}\right).$
(8)

##  4 Temperature Fluctuations from populations of Mergers

Fig. 2 shows three realisations of the temperature fluctuation patterns in a
local box of size $\sim(100\,{\rm{Mpc}})^{3}$, populated with $10$, $100$ and
$1000$ sources, respectively. Here the events all have the same mass of
$10^{9}$ $M_{\odot}$, but are otherwise drawn from uniform distributions.
While the resulting temperature fluctuations seem to increase in complexity
and resolution, they are composed of superpositions (ignoring non-linearities
of the order $h_{ij}^{2}$) of the individual patterns showcased in Fig. 1,
appearing at different scales and with different orientations and
inclinations.

As the mean of the fluctuations is always vanishing, we expect both the
maximum and the standard deviation of temperature fluctuations within a given
box to roughly follow a random walk process (see also Boybeyi et al., 2024).
Both of the aforementioned quantities should scale as $\sqrt{N}$, where $N$ is
the total amount of merger events in the box. Here we explicitly confirm the
hypothesis by setting up local boxes of various sizes. We populate the boxes
with events corresponding to a given volumetric merger rate $\mathcal{R}$. A
specific realisation of one such test is visualised in Fig. 3, showing both
the maximum and standard deviations of the temperature fluctuations averaged
over a total of $\sim 3\times 10^{4}$ realisations with varying number of
events. These particular realisations were performed in a box of length 100
Mpc, roughly corresponding to a few degrees of angular scale at $z=1$. The
masses are set to $10^{9}$ M⊙ and the remaining parameters are drawn from
uniform distributions. Throughout our tests, we find that both the maximum and
the standard deviation of temperature fluctuations in a given CMB patch scale
with the square root of the average local volumetric merger rate in the
corresponding box. The maximum follows:

$\displaystyle\frac{\Delta T_{\rm{max}}}{T(z_{\rm box})}=1.1\times
10^{-11}\left(\frac{\mathcal{R}(M)}{10^{-4}\,\rm{Mpc}^{-3}\,\rm{yr}^{-1}}\right)^{1/2}\left(\frac{M}{10^{9}\,\rm{M}_{\odot}}\right),$
(9)

though always presents a significant scatter, varying by a factor of
approximately $2$ to $5$ at $1\sigma$, depending on the total number of
events. Note that here $T(z_{\rm box})$ denotes the average local CMB
temperature, a quantity that is well defined only for sufficiently small
boxes. The standard deviation follows:

$\displaystyle\frac{\Delta T_{\rm{std}}}{T(z_{\rm box})}=1.6\times
10^{-12}\left(\frac{\mathcal{R}(M)}{10^{-4}\,\rm{Mpc}^{-3}\,\rm{yr}^{-1}}\right)^{1/2}\left(\frac{M}{10^{9}\,\rm{M}_{\odot}}\right),$
(10)

with significantly less scatter. Volumetric merger rates of order $10^{-4}$
Mpc-3 Gyr-1 to $10^{-3}$ Mpc-3 Gyr-1 are what is typically observed for the
major mergers of massive galaxies with stellar mass above $10^{11}$ M⊙ at
$z\sim 1$ (Bell et al., 2006; O’Leary et al., 2021). According to
observations, such galaxies typically host black holes in the range $10^{7}$
M⊙ to several $10^{9}$ M⊙ depending on galaxy morphology (Reines & Volonteri,
2015; Zhu et al., 2021), with an occupation fraction that approaches unity for
more massive galaxies (Volonteri et al., 2016). We scaled Eqs. 9 and 10 with
the volumetric merger rates and BH masses appropriate to such galaxies,
assuming that the BH merger rate would be roughly of the same order. As
discussed before, the temperature fluctuations vary in time. We showcase an
animation for the case of 20 randomly distributed sources on
https://www.youtube.com/watch?v=jli4n9j6BDk&list=PLllD9CwiToPKY7hjYHaJDGrB5KBjRPyi4&index=3.
We observe the superposition of patterns from individual sources, each of
which expands and fades during its evolution according to timescales of order
$R_{i}/c$, where $R_{i}$ are the scales of each individual pattern.

We find that the GWM effect of a population of mergers gives rise to a
characteristic power spectrum in the resulting CMB temperature fluctuation.
Fig. 4 shows the power spectrum from 100 merger population realisations of
boxes of local size $(30\,\rm{Mpc})^{3}$ containing $10$, $100$ and $1000$
mergers. Due to the $\sqrt{N}$ proportionality of the maximum temperature
fluctuation as described above, the amplitude of the power spectrum is
similarly increasing with the density of mergers. The larger scatter present
for lower merger density is caused by small-number statistics, where with too
few sources the initial merger distribution can be very inhomogenous when
drawn randomly. We fit a power law $P(k)\propto k^{n}$, to the curves
displayed in Fig. 4, disregarding the lowest ($k<3$) and highest modes
($k>40$) to avoid boundary and resolution effects of the temperature map. Here
$k$ is in units of the image resolution, and can be easily transformed to
physical scales given the scale of the CMB patch. We find the best fit scaling
to be $n=-3.04\pm 0.67$, $n=-2.83\pm 0.35$, $n=-2.70\pm 0.13$ for patterns
caused by 10, 100 and 1000 sources respectively. We find that the scaling of
the power spectrum of temperature fluctuations converges to a universal value
of:

$\displaystyle P^{\rm{GWM}}_{\rm{2D}}(k)\propto k^{-2.7},$ (11)

within all CMB patches that contain a sufficient number of merger events.

Figure 3: We show the maximum and the standard deviation of temperature
fluctuations for CMB photons that passed through a (100 Mpc)3 cosmological
box. The box is populated with $10^{9}$ M⊙ mergers according to a given
volumetric merger rate, which are otherwise sampled uniformly in space, time,
inclination and orientation. The results are averaged over a total of $\sim
3\times 10^{4}$ realisations (pale lines), an both the mean (solid lines) and
the standard deviation (shaded areas enclosed by black lines) of the
realisations are shown. Figure 4: Realisations of the power spectrum of CMB
temperature fluctuations caused by the GWM of $10$, $100$ and $1000$ mergers
wihtin a local $(30\,\rm{Mpc})^{3}$ box (pale lines). We find that the power
spectra follow a universal scaling $\propto k^{-2.7}$, where $k$ are the modes
of the image (here in units of image resolution). The mean (thick lines) and
the 2$\sigma$ contours (shaded areas) are also displayed.

##  5 Towards Detecting GWM imprints

The temperature fluctuations patterns analysed in this work showcase how every
single BH merger leaves its signature on CMB radiation. Though hidden below a
myriad of other signals (see e.g. Planck Collaboration et al., 2020a;
Raghunathan, 2022; Bianchini & Millea, 2023), the entire merger history of BHs
is marked on the oldest image of our Universe. Nevertheless, while
contemplating such aesthetically pleasing thoughts, the astrophysicist is
called to ask the following question: are such patterns actually detectable?
Here we offer a few insights, in the understanding that detecting such
fluctuations may well be exceedingly challenging: Though real, they are small
in amplitude and have to be de-convolved from lensing, path deflections and
other sources of noise. We welcome input from the broader community to explore
some non trivial detection techniques (see e.g. Kusiak et al., 2023, for an
attempt to reduce contamination from astrophysical noise sources).

Holmberg 15 is a nearby ultra-massive BH with a mass of $\sim 4\times 10^{10}$
M⊙, located at a redshift of $z\sim 0.05$ (Mehrgan et al., 2019). Is it
possible to exclude that it experienced a major merger in its recent past?
Suppose a CMB microwave interferometer sensitive to wavelengths
$\lambda_{\rm{sens}}$ of $\sim$ 2 mm and temperature fluctuations of $\Delta
T_{\rm sens}$ (see Dickinson et al., 2004, as an example). The latter would
require the following baseline $B$ in order to resolve the GWM pattern around
a presumed major merger with total mass $M$ at $z=0.05$:

$\displaystyle B\approx 0.15\,\rm{km}\times\left(\frac{\Delta T_{\rm
sens}}{10^{-5}\,\rm{K}}\right)\left(\frac{4\times
10^{10}\,\rm{M}_{\odot}}{M}\right)\left(\frac{2\,\rm{mm}}{\lambda_{\rm{sens}}}\right),$
(12)

where we used Eq. 7 and $T_{\rm CMB}=2.73$ K. Such baselines are clearly
realistic, and at the corresponding resolution of several $1^{\prime\prime}$
the GWM pattern would appear at scales of tens of kpc, approximately $10^{4}$
years after merger. Improving the temperature fluctuation sensitivity beyond
the typical values of current all-sky CMB surveys would allow to measure the
patterns at larger scales, and probe whether a merger occurred further back in
time. It is worth noting here that the GWM fluctuations will most likely be
self-lensed, increasing their amplitude.

Our estimates for the random walk like accumulation of GWM apply to single
mass populations of sources in local boxes. A complete assessment requires
modelling the merger rates of all MBHs across cosmic time (see Boybeyi et al.,
2024, for a possible approach). At higher redshifts in particular, however,
the mass spectrum and merger rates of MBHs are purely speculative. The latter
may well be much higher than what is adopted in this paper (see some estimates
on the merger rate of primordial black holes for an extreme example Ali-
Haïmoud et al., 2017; Huang et al., 2024). We have shown that the power
spectrum of temperature fluctuations is universal as soon as sufficient
mergers are present in a given patch of CMB. For a volumetric merger rate of
$10^{-3}$ Mpc-3 Gyr-1, this is realised for a box with angular scale of
approximately $1^{\circ}$, of which there are 41253 independent realisations
on the full sky. Knowing the scaling a priori, is it possible to coherently
add up power from all such patches in a way that suppresses other
contributions? Additionally, among the 41253 there are between 2 and 3 patches
that are 4 sigma statistical outliers. Here outlier could refer to several of
the quantities relevant to the accumulation of GWM, e.g. the merger rate,
typical MBH masses, or simply to the realisation statistics of the random
walk. There is always some probability that in some region of the sky the
effects we discussed may be unusually large, though quantifying this would
require a fully cosmological treatment over the entire sky. In the same vein,
we note that our analysis should be convolved with an evolving cosmological
metric, which will most likely increase the magnitude of the effect by an
appreciable amount.

## Acknowledgements

L.Z., K.H. and D.ON. acknowledge support from ERC Starting Grant No.
121817–BlackHoleMergs. M.M.T. acknowledges support from the Ministerio de
Ciencia, Innovación y Universidades del Gobierno de España through the “Ayuda
para la Formación de Profesorado Universitario” (FPU) No. FPU19/01750 and the
”Ayuda FPU Complementaria de Movilidad para Estancias Breves en Centros
Extranjeros” No. EST23/00420. D.ON gratefully acknowledges support from Villum
Fonden Grant No. 29466 and the Danish Independent Research Fund through Sapere
Aude Starting Grant No. 121587. P.K. acknowledges support from the Stefan
Rozental og Hanna Kobylinski Rozentals Fond. The authors acknowledge Daniel J.
D’Orazio, Johan Samsing and Martin Pessah for helpful discussions. The authors
are grateful for the existence and convenient location of Søernes Ølbar.

## References

  * Aghanim et al. (2020) Aghanim, N., Akrami, Y., Ashdown, M., et al. 2020, Astronomy & Astrophysics, 641, A6
  * Ali-Haïmoud et al. (2017) Ali-Haïmoud, Y., Kovetz, E. D., & Kamionkowski, M. 2017, Phys. Rev. D, 96, 123523
  * Bell et al. (2006) Bell, E. F., Phleps, S., Somerville, R. S., et al. 2006, ApJ, 652, 270
  * Bianchini & Millea (2023) Bianchini, F. & Millea, M. 2023, Phys. Rev. D, 107, 043521
  * Bieri et al. (2017) Bieri, L., Garfinkle, D., & Yunes, N. 2017, Classical and Quantum Gravity, 34, 215002
  * Blanchet (2014) Blanchet, L. 2014, Living Rev. Relat., 17, 2
  * Blanchet & Damour (1992) Blanchet, L. & Damour, T. 1992, Phys. Rev. D, 46, 4304
  * Book & Flanagan (2011) Book, L. G. & Flanagan, E. E. 2011, Phys. Rev. D, 83, 024024
  * Boybeyi et al. (2024) Boybeyi, T., Mandic, V., & Papageorgiou, A. 2024, arXiv e-prints, arXiv:2403.07614
  * Braginsky & Thorne (1987) Braginsky, V. B. & Thorne, K. S. 1987, Nature, 327, 123
  * Christodoulou (1991) Christodoulou, D. 1991, Phys. Rev. Lett., 67, 1486
  * Conselice et al. (2022) Conselice, C. J., Mundy, C. J., Ferreira, L., & Duncan, K. 2022, ApJ, 940, 168
  * Dickinson et al. (2004) Dickinson, C., Battye, R. A., Carreira, P., et al. 2004, MNRAS, 353, 732
  * Fakhouri et al. (2010) Fakhouri, O., Ma, C.-P., & Boylan-Kolchin, M. 2010, MNRAS, 406, 2267
  * Favata (2009) Favata, M. 2009, The Astrophysical Journal, 696, L159–L162
  * Favata (2010) Favata, M. 2010, Classical and Quantum Gravity, 27, 084036
  * Huang et al. (2024) Huang, H.-L., Jiang, J.-Q., & Piao, Y.-S. 2024, Phys. Rev. D, 109, 063515
  * Jokela et al. (2022) Jokela, N., Kajantie, K., & Sarkkinen, M. 2022, Phys. Rev. D, 106, 064022
  * Kopeikin et al. (1999) Kopeikin, S. M., Schäfer, G., Gwinn, C. R., & Eubanks, T. M. 1999, Phys. Rev. D, 59, 084023
  * Kusiak et al. (2023) Kusiak, A., Surrao, K. M., & Hill, J. C. 2023, Phys. Rev. D, 108, 123501
  * Lasky et al. (2016) Lasky, P. D., Thrane, E., Levin, Y., Blackman, J., & Chen, Y. 2016, Phys. Rev. Lett., 117, 061102
  * Madison (2020) Madison, D. R. 2020, Phys. Rev. Lett., 125, 041101
  * Madison (2021) Madison, D. R. 2021, Phys. Rev. D, 103, 083515
  * Mehrgan et al. (2019) Mehrgan, K., Thomas, J., Saglia, R., et al. 2019, ApJ, 887, 195
  * O’Leary et al. (2021) O’Leary, J. A., Moster, B. P., Naab, T., & Somerville, R. S. 2021, MNRAS, 501, 3215
  * Peebles (1993) Peebles, P. J. E. 1993, Principles of Physical Cosmology
  * Planck Collaboration et al. (2020a) Planck Collaboration, Aghanim, N., Akrami, Y., et al. 2020a, A&A, 641, A1
  * Planck Collaboration et al. (2020b) Planck Collaboration, Aghanim, N., Akrami, Y., et al. 2020b, A&A, 641, A3
  * Raghunathan (2022) Raghunathan, S. 2022, ApJ, 928, 16
  * Reines & Volonteri (2015) Reines, A. E. & Volonteri, M. 2015, ApJ, 813, 82
  * Talbot et al. (2018) Talbot, C., Thrane, E., Lasky, P. D., & Lin, F. 2018, Phys. Rev. D, 98, 064031
  * The COrE Collaboration et al. (2011) The COrE Collaboration, Armitage-Caplan, C., Avillez, M., et al. 2011, arXiv e-prints, arXiv:1102.2181
  * Thorne (1992) Thorne, K. S. 1992, Phys. Rev. D, 45, 520
  * Tiwari et al. (2021) Tiwari, S., Ebersold, M., & Hamilton, E. Z. 2021, Phys. Rev. D, 104, 123024
  * Tolish & Wald (2016) Tolish, A. & Wald, R. M. 2016, Phys. Rev. D, 94, 044009
  * Volonteri et al. (2016) Volonteri, M., Dubois, Y., Pichon, C., & Devriendt, J. 2016, MNRAS, 460, 2979
  * Zel’dovich & Polnarev (1974) Zel’dovich, Y. B. & Polnarev, A. G. 1974, Sov. Ast., 18, 17
  * Zhao & Cao (2022) Zhao, Z.-C. & Cao, Z. 2022, Science China Physics, Mechanics, and Astronomy, 65, 119511
  * Zhu et al. (2021) Zhu, P., Ho, L. C., & Gao, H. 2021, ApJ, 907, 6


# ODM: A Text-Image Further Alignment Pre-training Approach for Scene Text
Detection and Spotting

Chen Duan
<EMAIL_ADDRESS>First Author and Second Author contribute equally to
this work. Pei Fu11footnotemark: 1
<EMAIL_ADDRESS>Shan Guo
<EMAIL_ADDRESS>Qianyi Jiang
<EMAIL_ADDRESS>Xiaoming Wei
<EMAIL_ADDRESS>
###### Abstract

In recent years, text-image joint pre-training techniques have shown promising
results in various tasks. However, in Optical Character Recognition (OCR)
tasks, aligning text instances with their corresponding text regions in images
poses a challenge, as it requires effective alignment between text and OCR-
Text (referring to the text in images as OCR-Text to distinguish from the text
in natural language) rather than a holistic understanding of the overall image
content. In this paper, we propose a new pre-training method called OCR-Text
Destylization Modeling (ODM) that transfers diverse styles of text found in
images to a uniform style based on the text prompt. With ODM, we achieve
better alignment between text and OCR-Text and enable pre-trained models to
adapt to the complex and diverse styles of scene text detection and spotting
tasks. Additionally, we have designed a new labeling generation method
specifically for ODM and combined it with our proposed Text-Controller module
to address the challenge of annotation costs in OCR tasks, allowing a larger
amount of unlabeled data to participate in pre-training. Extensive experiments
on multiple public datasets demonstrate that our method significantly improves
performance and outperforms current pre-training methods in scene text
detection and spotting tasks. Code is available at ODM.

##  1 Introduction

Optical Character Recognition (OCR) has garnered significant attention in the
field of computer vision for its remarkable performance in automated data
entry, document analysis, instant translation, and more. Most existing methods
for obtaining OCR results involve a two-stage process, including a text
detection model and a text recognition model, or directly utilizing an end-to-
end text spotting model. Furthermore, many of these methods initialize the
model through pre-training on the ImageNet dataset [37].

Figure 1: Comparisons of different pre-training strategies. (a) Obtain the
pre-trained model through mask image modeling, taking only image embeddings as
inputs. (b) Obtain the pre-trained model through mask language modeling, which
simultaneously takes both OCR-Text and image as inputs. (c) Our approach
obtains the pre-trained model through OCR-Text destylization modeling.

Pre-training techniques have recently gained significant attention for their
outstanding performance across a wide range of computer vision tasks. Two
commonly used pre-training methods in computer vision are: (1) Masked Image
Modeling (MIM) pre-training [1, 49, 10, 3], which focuses on learning visual
contextualized representations solely from images. This method is typically
applied to vision-dominated tasks, such as image classification. (2) Masked
Language Modeling (MLM) [6, 51, 52], which utilizes both text and image as
inputs and extracts semantic information from both modalities. However, when
applying these pre-training methods in the OCR field, two specific issues
arise: (1) With the MIM-based method, there are instances where the text in
the image is completely obscured by the masked patch, primarily due to the
relatively small proportion of the text. This has the potential to hinder the
pre-trained model’s ability to effectively learn textual feature information.
(2) The MLM-based method, although achieving weakly supervised training by
masking the text input, does not explicitly exploit text location information
during training. This can lead to ineffective alignment between text and image
features, as well as inadequate handling of image information. These
challenges highlight the need for innovative approaches that specifically
address the unique requirements of OCR.

In this paper, we propose a novel pre-training technique called OCR-Text
Destylization Model (ODM) to address the challenges of text-image alignment in
OCR tasks. As illustrated in Fig. 1, unlike existing MIM and MLM methods, ODM
introduces a new pixel-level image reconstruction modeling based on text
prompts. Since OCR tasks primarily focus on the text within the image while
considering other pixels irrelevant, ODM aims to reconstruct a binary image
that removes the text style and enforces alignment between the text and OCR-
Text. This is achieved by utilizing a pixel-level reconstruction approach
instead of the traditional three-channel reconstruction. To further enhance
the model’s understanding of the text, we propose a Text-Controller module.
This module guides the image encoder to identify and interpret the OCR-Text,
facilitating the alignment between the text and OCR-Text. Additionally, we
have designed a novel method for generating ODM labels, effectively addressing
the issue of inadequate pixel-level labels in the dataset. By leveraging font
files, text, and location labels, we generate binary images with a unified
font style, as illustrated in Fig. 2, which showcases some examples of OCR-
Text destylization images. These advancements in ODM and the Text-Controller
module, along with the novel label generation method, contribute to improved
text-image alignment in OCR tasks.

(a)

(b)

(c)

(d)

Figure 2: The upper row and lower row represent the original images and their
corresponding destylized labels, respectively. (a), (b), (c), and (d) are
taken from the ICDAR15 [15], CTW1500 [26], TotalText [5], and LSVT [41]
datasets, respectively.

In summary, the main contributions are three-fold:

(1) We propose a simple yet effective pre-training method called ODM, which
focuses on learning features specifically for OCR-Text. By using pixel-level
labels with a uniform style, we successfully destylize OCR-Text, improving
text comprehension. This crucial feature information enables the pre-trained
model to adapt well to various scenarios in text detection and spotting tasks.

(2) We introduce a novel Text-Controller module that helps regulate the
model’s output, enhancing its understanding of OCR-Text. With this module, our
method does not require a perfect match between the input image and the text
pair. As a result, we can utilize weakly annotated data (i.e., using other OCR
recognition engines to obtain the text and location in the image and filter it
based on recognition confidence, text size, etc.), which can greatly reduce
the annotation cost.

(3) Experimental results on public datasets demonstrate that ODM delivers
outstanding performance and surpasses existing pre-training techniques across
a range of scene text detection and spotting datasets.

##  2 Related Work

Figure 3: The overall architecture of ODM. The text is encoded by the Text-
Controller to get the encoded text features, and the image is encoded by the
image encoder to get the encoded image features. The text features and image
features interact through cross-attention, and finally output destylization
binary image.

Scene Text Detection. Scene text detectors based on deep learning can
primarily be categorized into regression based [38, 18, 57, 32, 11, 42] and
segmentation based [50, 46, 47, 22, 58, 23, 55] methods. Regression-based
methods perceive scene text as an object and directly regress the bounding box
of the text instance. Segmentation-based methods treat the text detection task
as a semantic segmentation problem. They obtain a segmentation map by directly
segmenting the text instance and subsequently group the segments into a box
through post-processing.

Scene Text Spotting. Scene text spotting represents the unification of
detection and recognition processes within a singular framework. Numerous
studies have improved performance by simultaneously learning detectors and
recognizers. In [19, 20, 2, 25, 12, 7], end-to-end implementation is achieved
by training the detection and recognition separately. The Mask TextSpotter
[30, 31, 21] series performs character segmentation during the recognition
process. The ABCNet [27, 28] series obtains detection coordinates through
control points of the Bezier curve. SwinTextSpotter [13] and ESTextSpotter
[14] use separate detection-recognition heads corresponding to different
annotation formats.

Furthermore, some methods directly decode coordinates and recognition results
directly by predicting sequence, utilizing the structure of transformer
encoder and decoder. Both SPTS [34] and SPTS V2 [29] obtain coordinates by
predicting the central point of the text instance, employing an auto-
regressive approach to predict the central point and word transcription
tokens. UNITS [16] can handle various types of detection formats through
prompts, and they can extract text beyond the number of trained text
instances. DeepSolo [54] inspired by DETR, enables the decoder to
simultaneously perform text detection and recognition.

Vision-Language Pre-training. Modern pre-training methods generally involve
MLM [6] or MIM [10, 3, 49], or a combination of both. The MLM task randomly
masks a set of text tokens from the input and reconstructs them based on the
context around the masked tokens. MIM, on the other hand, randomly masks a
percentage of image patches and predicts the RGB values of raw pixels. STKM
[45] learns text knowledge from datasets with image-level text annotations,
the acquired text knowledge can subsequently be transferred to various text
detectors. Inspired by CLIP [35], VLPT [40] adopts fine-grained cross-modality
interaction to align unimodal embeddings for learning better representations
of backbone via carefully designed pre-training tasks. oCLIP [52] proposes an
MLM-based vision-language pre-training method, which has achieved excellent
performance in text detection and text spotting tasks. StrucTexTv2 [56]
implements pre-training through MIM, where it randomly masks some text word
regions in the input images and feeds them into the encoder.

While these methods have shown the effectiveness of pre-training in enhancing
OCR performance, they either do not utilize higher-level textual semantic
information as input or do not explicitly utilize the positional information
of OCR-Text, making it difficult to achieve effective alignment between text
and OCR-Text. Inspired by MaskFeat [48], which achieves pre-training by
reconstructing the HOG features of masked image regions, and considering that
glyph has been employed in some OCR tasks [33, 4, 53, 43] with proven
efficacy. We propose utilizing the Text Controller module to reconstruct the
corresponding destylized glyph of the text. This approach enables visible
alignment between text and OCR-Text. As shown in Fig. 4, our method can better
attend to OCR-Text, highlighting its superiority in learning visual text
representations for scene text image tasks.

##  3 Methodology

(a) Origin

(b) No change

(c) Drop-Text

(d) Noise-Text

Figure 4: Illustration of the proposed Text-Controller Module: The attention
heatmap (from the cross-attention layer) of the text branch under different
input scenarios is depicted. (a)The original image. (b) The text input
consists of three instances: “Rootin”, “Ridge”, and “Toymakers”. (c) The
“Toymakers” instance is discarded. (d) A non-existent instance “Sjehf” is
added.

We introduce ODM, a pre-training technique that effectively aligns text and
OCR-Text, leveraging the intrinsic characteristics of OCR-Text and explicitly
learning the location information. The overall pipeline of our method is
depicted in Fig. 3. Our network consists of two input branches: the image
encoder, which employs ResNet50 [9] to extract features from input images, and
the text encoder, designed to extract textual features. By applying cross-
attention between the extracted textual and visual features, we generate image
features that align with textual features. These aligned features are then
processed by a decoder to generate destylization binary images.

###  3.1 The Text-Controller Module

To address the challenge of the image encoder-decoder architecture in
comprehending the significance of characters, we introduce the Text-Controller
module to regulate the feature extraction of the image and align the OCR-Text
features with textual features in the hidden space.

Drop-Text. General text encoders, such as CLIP [35], are designed to process
text descriptions as input, with the primary objective of establishing
alignment between text and image features. In the Text-Controller Module,
control over the decoder is executed through prompts. Specifically, when a
portion of the OCR-Text is input, the model is expected to only reconstruct
that part of the binary image, treating the remaining OCR-Text as the
background. During the training phase, the input OCR-Text is selected
randomly, with a ratio varying from 0% to 100%. This strategy encourages the
model to focus more on aligning the text with the corresponding OCR-Text.

Noise-Text. Noise, such as inaccurate labels, can potentially impact the
model’s training effectiveness. In contrast, we have introduced a concept
termed Noise-Text, which leverages noise to augment the model’s performance.
This approach entails adding noise to the text encoder’s input, introducing
non-existent OCR-Text by altering its Token encoding value. The integration of
these disruptive elements empowers the model to align the features of the text
and OCR-Text more effectively, even in more intricate and challenging
scenarios.

In Fig. 4, we demonstrate the performance of cross-attention on images when
employing different strategies in Text-Controller Module. Our approach
achieves alignment between the text and corresponding OCR-Text while remaining
unaffected by noisy text.

###  3.2 OCR-Text Destylization

To ensure that the image encoder can learn the fundamental features of OCR-
Text, we have designed a simple decoder to reconstruct the destylized glyph of
the text. This decoder comprises only basic FPN [24] layer upsampling and 1*1
convolution.

Fig. 5 demonstrates the predicted results on some real images using our
proposed method. The training dataset only consists of SynthText [8] and does
not include these real images. From the results, it can be observed that our
proposed method effectively achieves OCR-Text destylization.

Figure 5: The upper row and lower row represent the original images and their
corresponding predicted results, respectively.

###  3.3 Loss Function

The ODM produces a binary image, conceptualizing supervised training as a
pixel-level segmentation task. As a result, we optimize the model using a
binary cross-entropy loss function for training:

${\mathcal{L}_{seg}}=\frac{1}{N}\sum_{i=1}^{N}-\left[{y}_{i}\log({p}_{i})+(1-{y}_{i})\log(1-{p}_{i})\right]$
(1)

where $N$ represents the number of pixels, $p_{i}$ represents the predicted
result, and $y_{i}$ represents the Ground Truth.

On the other hand, the training output of ODM is to generate a destylization
binary image, rather than a text segmentation map that maintains proportional
consistency with the original image. Consequently, relying exclusively on
pixel-level cross-entropy loss poses a challenge in effectively guiding the
model to learn the destylization of characters. To optimize at the feature
level, we incorporate the OCR LPIPS loss function proposed by OCR-VQGAN [36].
Specifically, we utilize a well-trained detector (Unet-VGG16[39]) to input
both the Ground Truth and the model’s predicted binary images. Subsequently,
the $\mathcal{L}$1 Loss is computed to foster the learning of a rich latent
space and the destylized character glyph images, which is defined as follows:

${\mathcal{L}_{ocr}}=\sum_{l}\frac{1}{H_{l}W_{l}}\left\|{VGG}_{l}(\hat{y})-{VGG}_{l}(y)\right\|_{1}$
(2)

where $H_{l}$ and $W_{l}$ respectively represent the height and width of the
output feature map in the l-th layer, $VGG$ represents a well-trained
detector, $\hat{y}$ represents the predicted value, and $y$ represents the
Ground Truth.

At the same time, drawing upon the batch-level contrastive loss proposed by
CLIP, we expedite the mapping of text and images into the same semantic space
by maximizing the similarity among positive samples and minimizing that among
negative samples, which is defined as follows:

$\begin{split}{\mathcal{L}_{bc}}&=CE(\frac{exp(I,{T}_{+})}{\textstyle\sum_{i=1}^{B}exp(I,{T}_{i})},y(I))\\\
&\quad+CE(\frac{exp(T,{I}_{+})}{\textstyle\sum_{i=1}^{B}exp(T,{I}_{i})},y(T))\end{split}$
(3)

where $CE$ represents the cross-entropy loss, $I$ represents image features,
$T$ represents text features, $I_{+}$ represents the image feature
corresponding to the current text feature, $T_{+}$ represents the text feature
corresponding to the current image feature, and $y(*)$ represents the Ground
Truth.

The loss function $\mathcal{L}$total is the sum of segmentation loss
$\mathcal{L}$seg, OCR LPIPS loss $\mathcal{L}$ocr, and batch contrastive loss
$\mathcal{L}$bc, formulated as follows:

$\mathcal{L}_{total}=\alpha\mathcal{L}_{seg}+\beta\mathcal{L}_{ocr}+\gamma\mathcal{L}_{bc}$
(4)

where the weights $\alpha$, $\beta$, and $\gamma$ are empirically set to 1, 1,
and 0.5, respectively.

###  3.4 Label Generation

One of the challenges in supervised training for ODM is the creation of data
labels, particularly the acquisition of fine-grained pixel-level labels, which
is both challenging and costly. To address this, we propose a method for
generating pixel-level labels.

For four-point annotations, we calculate the dimensions of the quadrilateral
and estimate the size and position of each character based on the number of
characters. We then populate these characters using a font file, such as
“NotoSans-Regular”. For multi-point annotations, we employ the image synthesis
approach from ABCNet [27] and utilize the Bezier curves provided by the labels
to compute the curvature of the text. Similar to four-point annotations, we
determine the size and position of each character from the multi-point
annotations and calculate the slope using the top-left point of each character
and the subsequent point. Once we obtain the slope, we adjust the orientation
of the characters accordingly. Some examples of the generated labels are shown
in Fig. 2.

During the process of acquiring pixel-level labels, there may be discrepancies
in the spacing between the original OCR-Text and the pixel-level labels. As a
result, there might be instances where individual characters in our generated
labels do not align precisely with the position of OCR-Text characters in the
original image. However, this discrepancy does not impact our task since our
approach involves transforming the original image into a destylization binary
image rather than performing pixel-by-pixel text segmentation.

##  4 Experiment

###  4.1 Datasets

In our experiments, we employ a substantial quantity of publicly accessible
datasets, executing pre-training on the SynthText [8] dataset and fine-tuning
on text detection and text spotting tasks utilizing datasets like ICDAR15
[15], CTW1500 [26], and TotalText [5].

SynthText is a synthetic dataset for text detection and recognition,
consisting of more than 800K synthesized images. Each image is annotated with
text regions and their corresponding textual content. The synthetic texts in
the dataset are generated by integrating real texts into natural images. The
positions, orientations, sizes, and appearances of the synthetic texts are
randomly varied to simulate text in real-world scenarios.

ICDAR2015 consists of 1500 images, allocates 1000 images for training, and the
remaining 500 for testing. The ICDAR2015 contains different types of text,
including horizontal text and multi-oriented text.

CTW1500 is a dataset for curved text instances, consisting of 1500 images,
comprising 1500 images, of which 1000 are allocated for training and the
remaining 500 for testing.

TotalText is a dataset for curved text instances, consisting of 1555 images,
with 1255 designated for training and the remaining 300 for testing.

Table 1: The performance of different models on ICDAR15, CTW1500, and
TotalText for scene text detection is presented in the table. “PD” and “Syn”
refer to the pre-training dataset and SynthText dataset, respectively. “+ODM”
refers to our pre-trained model on the SynthText dataset, which is adopted for
fine-tuning. $\dagger$ represents the scores obtained after reproducing the
model, and $\varDelta$ represents the improvement.

Method | PD | ICDAR 15 | CTW1500 | TotalText
---|---|---|---|---
P | R | H | P | R | H | P | R | H
FCENet[58]$\dagger$ | - | 82.43 | 88.34 | 85.28 | 85.7 | 80.1 | 83.1 | 78.1 | 84.85 | 81.34
FCENet + ODM | Syn | 91.38 | 82.19 | 86.54 | 84.92 | 82.33 | 83.60 | 84.05 | 80.67 | 82.32
$\varDelta$ |  |  |  | \+ 1.26 |  |  | \+ 0.5 |  |  | \+ 0.98
DBNet++[23]$\dagger$ | - | 91.61 | 81.46 | 86.24 | 79.06 | 79.50 | 79.28 | 81.84 | 80.22 | 81.02
DBNet++ + ODM | Syn | 91.87 | 85.94 | 88.81 | 81.99 | 81.98 | 81.97 | 88.01 | 82.25 | 85.03
$\varDelta$ |  |  |  | \+ 2.57 |  |  | \+ 2.69 |  |  | \+ 4.01

Table 2: Comparison with existing scene text pre-training techniques on
PSENet. “PD” and “Syn” refer to the pre-training dataset and SynthText
dataset, respectively. “+ODM” refers to our pre-trained model on the SynthText
dataset, which is adopted for fine-tuning. $\dagger$ represents the scores
obtained after reproducing the model, and $\varDelta$ represents the
improvement.

Method | PD | ICDAR 15 | CTW1500 | TotalText
---|---|---|---|---
P | R | H | P | R | H | P | R | H
PSENet[46]$\dagger$ | - | 83.96 | 76.36 | 79.98 | 80.6 | 75.6 | 78.0 | 75.1 | 81.8 | 78.3
PSENet[46] | Syn | 86.2 | 79.4 | 82.7 | 81.8 | 77.8 | 79.7 | 87.8 | 79.0 | 82.6
PSENet + STKM[45] | Syn | 87.78 | 84.06 | 85.88 | 85.08 | 78.23 | 81.51 | 85.08 | 78.23 | 81.51
PSENet + oCLIP[52] | Syn | 88.95 | 80.98 | 84.78 | 86.3 | 79.6 | 82.8 | 90.7 | 80.8 | 85.5
PSENet + oCLIP[52] | Web | - | - | - | 87.5 | 79.9 | 83.5 | 92.2 | 82.4 | 87.0
PSENet + ODM | Syn | 88.43 | 85.41 | 86.90 | 85.86 | 85.38 | 85.62 | 88.56 | 83.37 | 85.94
$\varDelta$ |  |  |  | \+ 6.92 |  |  | \+ 7.62 |  |  | \+ 7.64

###  4.2 Implementation Details

Training. In our proposed method, we employ ResNet50 as the image encoder,
with a 6-layer transformer [44] serving as the text encoder. The
reconstruction decoder is constructed from FPN and 1x1 convolutions. During
the training phase, we configure the image size to 512x512, set the text
length for the Text-Controller to 25, and cap the maximum number of text
instances at 32. The learning rate is established at 1e-4. The network
training is conducted on 8 A100 GPUs, with each card handling a batch size of
64, resulting in a total batch size of 512. The entire training process spans
100 epochs.

Fine-tuning. We executed evaluations on a variety of OCR tasks, encompassing
text detection methods such as DB++ [23]111https://github.com/open-
mmlab/mmocr/tree/main/configs/textdet/dbnetpp, FCENet
[58]222https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/fcenet,
and PSENet [46]333https://github.com/open-
mmlab/mmocr/tree/main/configs/textdet/psenet, as well as text spotting
techniques including ABCNet [27]444https://github.com/aim-
uofa/AdelaiDet/blob/master/configs/BAText, DeepSolo
[54]555https://github.com/ViTAE-Transformer/DeepSolo, and SPTS
[34]666https://github.com/shannanyinxiang/SPTS. In the comparative
experiments, the only variable altered is the backbone weights, with all other
settings maintained consistently.

Evaluation protocol. We use intersection over union (IoU) to determine whether
the model correctly detects the region of text, and we calculate precision
(P), recall (R), and Hmean (H) for comparison following ICDAR2015 [15].

###  4.3 Comparison with Detection Methods

To evaluate the effectiveness of our proposed method on text detection tasks,
we conducted comparable experiments with DB++, PSENet, and FCENet. Our model
underwent initial pre-training on the SynthText dataset and was subsequently
fine-tuned on different datasets. The results of these experiments are shown
in Tab. 1 and Tab. 2. The networks utilizing our pre-trained backbone
demonstrated improvements compared to their original counterparts, with PSENet
showing particularly notable improvement. This suggests that our pre-trained
model weights can effectively focus on text regions within the scene,
showcasing the significant potential of our pre-trained weights.

Table 3: The performance of different models on ICDAR15 for scene text
spotting is presented in the table. “PD” and “Syn” refer to the pre-training
dataset and SynthText dataset, respectively. “+ODM” refers to our pre-trained
model on the SynthText dataset, which is adopted for fine-tuning. $\dagger$
represents the scores obtained after reproducing the model, and $\varDelta$
represents the improvement. ‘D’ and ‘E’ represent Detection and End-to-End,
respectively. ‘S’, ‘W’, and ‘G’ donate using strong, weak, and generic
lexicons, respectively.

Method | PD | D-H | E-S | E-W | E-G
---|---|---|---|---|---
ABCNet[27]$\dagger$ | - | 85.22 | 78.01 | 73.86 | 68.09
ABCNet + ODM | Syn | 87.56 | 80.87 | 75.75 | 70.01
$\varDelta$ |  | \+ 2.34 | \+ 2.86 | \+ 1.89 | \+ 1.92
DeepSolo[54]$\dagger$ | - | 86.44 | 83.50 | 79.36 | 74.19
DeeoSolo + ODM | Syn | 88.08 | 85.83 | 80.67 | 75.2
$\varDelta$ |  | \+ 1.64 | \+ 2.3 | \+ 1.31 | \+ 1.01
SPTS[34]$\dagger$ | - | - | 77.5 | 70.2 | 65.8
SPTS + ODM | Syn | - | 78.8 | 72.1 | 67.8
$\varDelta$ |  |  | \+ 1.3 | \+ 1.9 | \+ 2.0

Table 4: The performance of different models on CTW1500 for scene text
spotting is presented in the table. “PD” and “Syn” refer to the pre-training
dataset and SynthText dataset, respectively. “+ODM” refers to our pre-trained
model on the SynthText dataset, which is adopted for fine-tuning. $\dagger$
represents the scores obtained after reproducing the model, and $\varDelta$
represents the improvement. ‘D’ and ‘E’ represent Detection and End-to-End,
respectively. ‘N’ and ‘F’ represent None and Full, respectively.

Method | PD | D-H | E-N | E-F
---|---|---|---|---
ABCNet[27]$\dagger$ | - | 83.69 | 64.17 | 78.66
ABCNet + ODM | Syn | 85.40 | 65.06 | 79.79
$\varDelta$ |  | \+ 1.71 | \+ 0.89 | \+ 1.13
DeepSolo[54]$\dagger$ | - | 86.31 | 76.35 | 84.31
DeeoSolo + ODM | Syn | 86.58 | 78.07 | 85.65
$\varDelta$ |  | \+ 0.27 | \+ 1.72 | \+ 1.34
SPTS[34]$\dagger$ | - | - | 74.2 | 82.4
SPTS + ODM | Syn | - | 78.2 | 84.2
$\varDelta$ |  |  | \+ 4.0 | \+ 1.8

###  4.4 Comparison with Spotting Methods

To evaluate the effectiveness of our proposed method on text spotting tasks,
we conducted comparable experiments with ABCNet, DeepSolo, and SPTS, covering
a wide range of task scenarios. Our model was initially pre-trained on the
SynthText dataset and then fine-tuned on different datasets. The results of
these experiments are presented in Tab. 3 and Tab. 4. The experimental results
demonstrate that utilizing our pre-trained weights for fine-tuning
significantly improves the performance of the models across various datasets.
This indicates that the pre-trained weights acquired through our proposed pre-
training method accurately locate the positions of OCR-Text and extract image
features that capture the semantic information of the text instance.

###  4.5 Weakly Supervised Pre-training

To evaluate the effectiveness of our proposed method in aligning text with
images in weakly labeled data, we assess its efficacy. We first employ PPOCRv3
[17] to generate pseudo labels (i.e., text and position in the image) on the
400,000 weakly annotated images from the LSVT [41] dataset. We then generate
the corresponding pixel-level destylized annotations using our proposed label
generation method. To ensure the quality of pre-training, we only select
labels with inference confidence exceeding 0.9 and a text size larger than 32
pixels. We pre-train our model with these generated labels and subsequently
fine-tune different scene text detectors using 30,000 fully annotated images
from the LSVT dataset. The results presented in Tab. 5 demonstrate that our
proposed method, when exclusively utilizing the pseudo labels for pre-
training, achieves approximately a 1.5% improvement in the Hmean score after
fine-tuning on both DBNet++ and PSENet models. This demonstrates that our
method can perform consistently even under weakly supervised scenarios and
partially addresses the issue of a large amount of unlabeled data not being
able to participate in pre-training.

Table 5: The performance of different models on LSVT for scene text detection
is presented in the table. “PD” and “LSVT” refer to the pre-training dataset
and LSVT dataset, respectively. “+ODM” refers to our pre-trained model with
400,000 pseudo-label images in the LSVT dataset, which is adopted for fine-
tuning. $\dagger$ represents the scores obtained after reproducing the model,
and $\varDelta$ represents the improvement.

Method | PD | P | R | H
---|---|---|---|---
PSENet[46]$\dagger$ | - | 64.61 | 72.29 | 68.23
PSENet + ODM | LSVT | 66.33 | 73.60 | 69.78
$\varDelta$ |  |  |  | \+ 1.55
DBNet++[23]$\dagger$ | - | 71.21 | 79.28 | 75.02
DBNet++ + ODM | LSVT | 75.81 | 77.16 | 76.48
$\varDelta$ |  |  |  | \+ 1.46

###  4.6 Comparison with Pre-training Methods

We compare our proposed method with existing scene text pre-training
strategies, including STKM and oCLIP. To investigate the effectiveness of
different pre-training objectives, we conducted ablative experiments with
PSENet on three datasets: ICDAR15, CTW1500, and TotalText. As shown in Tab. 2,
when pre-training on the same set of data, our proposed method outperforms the
existing pre-training techniques. Furthermore, when fine-tuning on the CTW1500
dataset, our proposed method, which was pre-trained on SynthText alone, even
surpasses the performance of oCLIP, which was pre-trained on 40 million web
images.

Table 6: Proportion ablation study of the proposed Text-Controller module on
CTW1500. “PD” and “Syn” refer to the pre-training dataset and SynthText
dataset, respectively. “TP” refers to the selected text proportion. “+ODM”
refers to our pre-trained model on the SynthText dataset, which is adopted for
fine-tuning. $\dagger$ represents the scores obtained after reproducing the
model.

Method | PD | TP | P | R | H
---|---|---|---|---|---
PSENet[46] | - | - | 80.6 | 75.6 | 78.0
PSENet[46] | Syn | - | 81.8 | 77.8 | 79.7
PSENet + ODM | Syn | 0% | 82.41 | 84.19 | 83.29
PSENet + ODM | Syn | 30% | 84.83 | 82.18 | 83.48
PSENet + ODM | Syn | 50% | 83.76 | 84.78 | 84.27
PSENet + ODM | Syn | 70% | 85.32 | 82.37 | 83.82

Table 7: Ablation study of our proposed components on TotalText. We fine-tune
PSENet by using the pre-trained models with different modules. “TE”, “DT”,
“NT”, and “OL” refer to Text Encoder, Drop-Text, Noise-Text, and OCR Loss,
respectively.

Method | TE | DT | NT | OL | P | R | H
---|---|---|---|---|---|---|---
PSENet[46] |  |  |  |  | 75.10 | 81.80 | 78.30
PSENet + ODM |  |  |  |  | 85.08 | 78.55 | 81.68
PSENet + ODM | ✓ |  |  |  | 86.66 | 82.75 | 84.66
PSENet + ODM | ✓ | ✓ |  |  | 88.06 | 82.97 | 85.44
PSENet + ODM | ✓ |  | ✓ |  | 88.10 | 82.57 | 85.24
PSENet + ODM | ✓ | ✓ | ✓ |  | 88.17 | 83.21 | 85.62
PSENet + ODM | ✓ | ✓ | ✓ | ✓ | 88.56 | 83.37 | 85.94

###  4.7 Ablation Experiments

Proportion Ablation We conducted experiments to assess the influence of
selected proportions in our Drop-Text and Noise-Text strategies. Four groups
of experiments were performed with selected proportions of 0%, 30%, 50%, and
70% respectively. The selected proportions for these strategies were the only
variables adjusted, while all other configurations remained constant. We pre-
trained the models on the SynthText dataset with varying proportions and
transferred the pre-trained weights to fine-tune PSENet on the CTW1500
dataset. The results are presented in Tab. 6. The performance of the model was
effectively enhanced by facilitating text-image alignment through the Drop-
Text and Noise-Text strategies. The impact of varying proportions on
performance was substantial. A small proportion of text instances resulted in
minimal performance improvement, possibly due to the insufficient changes in a
small number of words to achieve effective text-image alignment. This minimal
improvement can be attributed to the errors introduced during training due to
the lack of significant changes in the text instances. On the other hand,
using a large proportion of text instances also had a limited effect on
performance. This can be attributed to the fact that when a substantial number
of words are either eliminated or added as noise, the model captures fewer
valid features during training, leading to difficulties in convergence and
potentially biased learning. Hence, it is important to select an appropriate
proportion that ensures the model captures an adequate number of valid
features. This helps the model align text with OCR-Text in more complex
scenarios and enhances the robustness of the model.

Module Ablation: In our research, we investigated the contributions of several
proposed modules. We trained the models with different combinations of these
modules on the SynthText dataset and subsequently fine-tuned the pre-trained
weights using PSENet on the TotalText dataset, as shown in Tab. 7. The
empirical results indicated that for the task of OCR-Text Desytlization, the
text feature furnished by the Text-Controller module aids the model in better
understanding and locating OCR-Text. Additionally, our proposed Drop-Text and
Noise-Text strategies effectively bolster the model’s performance by
intensifying the alignment between text and OCR-Text.

##  5 Conclusion

This paper introduces ODM, a novel pre-trained method that aims to transform
diverse styles of text found in images into a uniform style based on the text
prompt. By leveraging the pre-trained model generated through ODM, it can
seamlessly integrate into existing detection and spotting networks, resulting
in significant performance improvements. To address the challenge of
annotation costs in OCR tasks, we propose a new labeling generation method
designed specifically for ODM. Additionally, we introduce the Text-Controller
module, which helps regulate the model output and improves its understanding
of OCR-Text. By combining these approaches, we enable a larger amount of
unlabeled data to be used in the pre-training process, effectively reducing
annotation costs. Our extensive ablation and comparative experiments
demonstrate the effectiveness and robustness of our model. The results
highlight the potential of ODM in OCR pre-training and its valuable
contributions to the advancement of scene text detection and spotting tasks.
Looking ahead, we plan to explore the potential of this method in other
domains, such as document analysis, handwriting recognition, and other complex
scene text scenarios.

## References

  * Bao et al. [2021] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ , 2021.
  * Borisyuk et al. [2018] Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar.  Rosetta: Large scale system for text detection and recognition in images.  In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 71–79, 2018.
  * Chen et al. [2022] Jun Chen, Ming Hu, Boyang Li, and Mohamed Elhoseiny.  Efficient self-supervised vision pretraining with local masked reconstruction.  _arXiv preprint arXiv:2206.00790_ , 2022.
  * Chen et al. [2024] Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei.  Textdiffuser: Diffusion models as text painters.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Ch’ng et al. [2020] Chee-Kheng Ch’ng, Chee Seng Chan, and Cheng-Lin Liu.  Total-text: toward orientation robustness in scene text detection.  _International Journal on Document Analysis and Recognition (IJDAR)_ , 23(1):31–52, 2020.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Feng et al. [2019] Wei Feng, Wenhao He, Fei Yin, Xu-Yao Zhang, and Cheng-Lin Liu.  Textdragon: An end-to-end framework for arbitrary shaped text spotting.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9076–9085, 2019.
  * Gupta et al. [2016] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2315–2324, 2016.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2021a] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Mae: Masked autoencoders are scalable vision learners.  _arXiv preprint arXiv:2111.06377_ , 2021a.
  * He et al. [2021b] Minghang He, Minghui Liao, Zhibo Yang, Humen Zhong, Jun Tang, Wenqing Cheng, Cong Yao, Yongpan Wang, and Xiang Bai.  Most: A multi-oriented scene text detector with localization refinement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8813–8822, 2021b.
  * He et al. [2018] Tong He, Zhi Tian, Weilin Huang, Chunhua Shen, Yu Qiao, and Changming Sun.  An end-to-end textspotter with explicit alignment and attention.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5020–5029, 2018.
  * Huang et al. [2022] Mingxin Huang, Yuliang Liu, Zhenghao Peng, Chongyu Liu, Dahua Lin, Shenggao Zhu, Nicholas Yuan, Kai Ding, and Lianwen Jin.  Swintextspotter: Scene text spotting via better synergy between text detection and text recognition.  In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4593–4603, 2022.
  * Huang et al. [2023] Mingxin Huang, Jiaxin Zhang, Dezhi Peng, Hao Lu, Can Huang, Yuliang Liu, Xiang Bai, and Lianwen Jin.  Estextspotter: Towards better scene text spotting with explicit synergy in transformer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 19495–19505, 2023.
  * Karatzas et al. [2015] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al.  Icdar 2015 competition on robust reading.  In _2015 13th international conference on document analysis and recognition (ICDAR)_ , pages 1156–1160. IEEE, 2015.
  * Kil et al. [2023] Taeho Kil, Seonghyeon Kim, Sukmin Seo, Yoonsik Kim, and Daehee Kim.  Towards unified scene text spotting based on sequence generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15223–15232, 2023.
  * Li et al. [2022] Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, et al.  Pp-ocrv3: More attempts for the improvement of ultra lightweight ocr system.  _arXiv preprint arXiv:2206.03001_ , 2022.
  * Liao et al. [2017a] Minghui Liao, Baoguang Shi, Xiang Bai, Xinggang Wang, and Wenyu Liu.  Textboxes: A fast text detector with a single deep neural network.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2017a.
  * Liao et al. [2017b] Minghui Liao, Baoguang Shi, Xiang Bai, Xinggang Wang, and Wenyu Liu.  Textboxes: A fast text detector with a single deep neural network.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2017b.
  * Liao et al. [2018] Minghui Liao, Baoguang Shi, and Xiang Bai.  Textboxes++: A single-shot oriented scene text detector.  _IEEE transactions on image processing_ , 27(8):3676–3690, 2018.
  * Liao et al. [2020a] Minghui Liao, Guan Pang, Jing Huang, Tal Hassner, and Xiang Bai.  Mask textspotter v3: Segmentation proposal network for robust scene text spotting.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_ , pages 706–722. Springer, 2020a.
  * Liao et al. [2020b] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai.  Real-time scene text detection with differentiable binarization.  In _Proceedings of the AAAI conference on artificial intelligence_ , pages 11474–11481, 2020b.
  * Liao et al. [2022] Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai.  Real-time scene text detection with differentiable binarization and adaptive scale fusion.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(1):919–931, 2022.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017.
  * Liu et al. [2018] Xuebo Liu, Ding Liang, Shi Yan, Dagui Chen, Yu Qiao, and Junjie Yan.  Fots: Fast oriented text spotting with a unified network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5676–5685, 2018.
  * Liu et al. [2019] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang.  Curved scene text detection via transverse and longitudinal sequence connection.  _Pattern Recognition_ , 90:337–345, 2019.
  * Liu et al. [2020] Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang.  Abcnet: Real-time scene text spotting with adaptive bezier-curve network.  In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9809–9818, 2020.
  * Liu et al. [2021] Yuliang Liu, Chunhua Shen, Lianwen Jin, Tong He, Peng Chen, Chongyu Liu, and Hao Chen.  Abcnet v2: Adaptive bezier-curve network for real-time end-to-end text spotting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(11):8048–8064, 2021.
  * Liu et al. [2023] Yuliang Liu, Jiaxin Zhang, Dezhi Peng, Mingxin Huang, Xinyu Wang, Jingqun Tang, Can Huang, Dahua Lin, Chunhua Shen, Xiang Bai, et al.  Spts v2: single-point scene text spotting.  _arXiv preprint arXiv:2301.01635_ , 2023.
  * Lyu et al. [2018] Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai.  Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 67–83, 2018.
  * Lyu et al. [2021] Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai.  Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(2):532–548, 2021.
  * Ma et al. [2018] Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue.  Arbitrary-oriented scene text detection via rotation proposals.  _IEEE transactions on multimedia_ , 20(11):3111–3122, 2018.
  * Ma et al. [2023] Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang, Di Niu, Haonan Lu, and Xiaodong Lin.  Glyphdraw: Learning to draw chinese characters in image synthesis models coherently.  _arXiv preprint arXiv:2303.17870_ , 2023.
  * Peng et al. [2022] Dezhi Peng, Xinyu Wang, Yuliang Liu, Jiaxin Zhang, Mingxin Huang, Songxuan Lai, Jing Li, Shenggao Zhu, Dahua Lin, Chunhua Shen, et al.  Spts: single-point text spotting.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 4272–4281, 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rodriguez et al. [2023] Juan A Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez.  Ocr-vqgan: Taming text-within-image generation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3689–3698, 2023.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115:211–252, 2015.
  * Shi et al. [2017] Baoguang Shi, Xiang Bai, and Serge Belongie.  Detecting oriented text in natural images by linking segments.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2550–2558, 2017.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  2014\.
  * Song et al. [2022] Sibo Song, Jianqiang Wan, Zhibo Yang, Jun Tang, Wenqing Cheng, Xiang Bai, and Cong Yao.  Vision-language pre-training for boosting scene text detectors.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15681–15691, 2022.
  * Sun et al. [2019] Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al.  Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1557–1562. IEEE, 2019.
  * Tang et al. [2022] Jingqun Tang, Wenqing Zhang, Hongye Liu, MingKun Yang, Bo Jiang, Guanglong Hu, and Xiang Bai.  Few could be better than all: Feature sampling and grouping for scene text detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4563–4572, 2022.
  * Tuo et al. [2023] Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng, and Xuansong Xie.  Anytext: Multilingual visual text generation and editing.  _arXiv preprint arXiv:2311.03054_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Wan et al. [2021] Qi Wan, Haoqin Ji, and Linlin Shen.  Self-attention based text knowledge mining for text detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5983–5992, 2021.
  * Wang et al. [2019a] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao.  Shape robust text detection with progressive scale expansion network.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9336–9345, 2019a.
  * Wang et al. [2019b] Wenhai Wang, Enze Xie, Xiaoge Song, Yuhang Zang, Wenjia Wang, Tong Lu, Gang Yu, and Chunhua Shen.  Efficient and accurate arbitrary-shaped text detection with pixel aggregation network.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8440–8449, 2019b.
  * Wei et al. [2022] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14668–14678, 2022.
  * Xie et al. [2022] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9653–9663, 2022.
  * Xu et al. [2019] Yongchao Xu, Yukang Wang, Wei Zhou, Yongpan Wang, Zhibo Yang, and Xiang Bai.  Textfield: Learning a deep direction field for irregular scene text detection.  _IEEE Transactions on Image Processing_ , 28(11):5566–5579, 2019.
  * Xu et al. [2020] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al.  Layoutlmv2: Multi-modal pre-training for visually-rich document understanding.  _arXiv preprint arXiv:2012.14740_ , 2020.
  * Xue et al. [2022] Chuhui Xue, Wenqing Zhang, Yu Hao, Shijian Lu, Philip HS Torr, and Song Bai.  Language matters: A weakly supervised vision-language pre-training approach for scene text detection and spotting.  In _European Conference on Computer Vision_ , pages 284–302. Springer, 2022.
  * Yang et al. [2024] Mingkun Yang, Biao Yang, Minghui Liao, Yingying Zhu, and Xiang Bai.  Class-aware mask-guided feature refinement for scene text recognition.  _Pattern Recognition_ , 149:110244, 2024.
  * Ye et al. [2023] Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, and Dacheng Tao.  Deepsolo: Let transformer decoder with explicit points solo for text spotting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19348–19357, 2023.
  * Yu et al. [2023a] Wenwen Yu, Yuliang Liu, Wei Hua, Deqiang Jiang, Bo Ren, and Xiang Bai.  Turning a clip model into a scene text detector.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6978–6988, 2023a.
  * Yu et al. [2023b] Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang, Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang.  Structextv2: Masked visual-textual prediction for document image pre-training.  _arXiv preprint arXiv:2303.00289_ , 2023b.
  * Zhou et al. [2017] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang.  East: an efficient and accurate scene text detector.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 5551–5560, 2017.
  * Zhu et al. [2021] Yiqin Zhu, Jianyong Chen, Lingyu Liang, Zhanghui Kuang, Lianwen Jin, and Wayne Zhang.  Fourier contour embedding for arbitrary-shaped text detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3123–3131, 2021.