# Non-Linear Convergence of Solar-like Stars Chromospheres using Millimeter,
Sub-millimeter, and Infrared Observations

F. Tapia-Vázquez Instituto de Radioastronomía y Astrofísica, Universidad
Nacional Autónoma de México,
P.O. Box 3-72, 58090, Morelia, Michoacán, México.  V. De la Luz  Escuela
Nacional de Estudios Superiores Unidad Morelia, Universidad Nacional Autónoma
de México, Morelia, 58190, México

(Received -; Revised -; Accepted -)

###### Abstract

In this work, we present a new methodology to fit the observed and synthetic
spectrum of solar-like stars at millimeter, submillimeter and infrared
wavelengths through semiempirical models of the solar chromosphere. We use the
Levenberg-Marquardt algorithm as a Non-Linear method, PakalMPI as the
semiempirical model of the solar chromosphere, and recent observations from
the Atacama Large Millimeter/submillimeter Array (ALMA) of Alpha Centauri A as
a test case. Our results show that we can use solar chromospheric
semiempirical models as an input model to reproduce the observed spectrum of
solar-like stars. The new profiles show similarities to the solar chromosphere
as a minimum of temperature (without the restriction from CO emission) and a
plateau in the high chromosphere. Our method provides a new fast numerical
tool to estimate the physical conditions of solar-like stars.

stars: chromospheres — methods: numerical — radiative transfer — Sun:
chromosphere — Sun: radio radiation — Sun: infrared

††journal: ApJS

##  1 Introduction

The particular characteristics of the emission at millimeter, submillimeter,
and infrared wavelengths in the solar chromosphere allow us to estimate their
temperature and density using indirect methodologies like semiempirical models
(Vernazza et al., 1981; Fontenla et al., 1993; Avrett, & Loeser, 2008). These
models are an important tool for a wide set of studies e.g. solar and stellar
chromospheres (Loukitcheva et al., 2004; Linsky, 2017), temperature minimum
(Liseau et al., 2013; De la Luz et al., 2014), solar flares (Machado et al.,
1980; Trottet et al., 2015) and Sun component features (Fontenla et al.,
2006).

In terms of solar atmospheric temperatures, the semiempirical models predict
values close to photospheric temperatures that decreases until a minimum, then
increase dramatically until the coronae (Vernazza et al., 1981; Avrett, &
Loeser, 2008) in a thin layer of around $2200$ km. This layer is known as the
solar chromosphere. The chromosphere remains observable by different spectral
ranges that include Ultraviolet (UV) in the continuum and line emission, in
the visible (mainly H$\alpha$) and in the millimeter, submillimeter and
infrared wavelengths. This last wavelength range becomes more important as
improvements in the sensitivity and space resolution in modern radio
telescopes (Nakajima et al., 1995; Kudaka et al., 2015; Wedemeyer et al.,
2016).

The process to compute an atmospheric model begin testing the initial
conditions of a semiempirical model (Vernazza et al., 1976; Carlsson, 1992;
Dere et al., 1997; De la Luz et al., 2010). The process includes three steps:
i) small modifications in an initial temperature profile, ii) compute the
density and pressure required to guarantee hydrostatic, hydrodynamic or
magnetohydrodynamic equilibrium, and iii) compute the synthetic spectrum to
compare with observations. Differences between synthetic and observed spectrum
are solved making changes in the atmospheric temperature profile at different
altitudes until synthetic and observed spectrum converges. The number of
iterations to converge both observations and the synthetic spectrum restrict
the precision of the computation.

Recent observations at sub-mm wavelengths of Solar-like stars allow computing
for the first time the stellar atmosphere at chromosphere altitudes using as
start model the solar chromospheric by millimeter-infrared wavelength range
observations (Liseau et al., 2016). However, the differences between the solar
chromosphere and the structure of the atmosphere of Solar-like stars requires
thousands of iterations to fit observations and the synthetic spectrum.

In this work, we present a new non-linear model to converge automatically
observed and synthetic spectrum of solar-like chromospheres at millimeter-
infrared wavelength. We used the Levenberg-Marquardt (LM) method to fitting
observations and synthetic spectrum using PakalMPI as parametric function (De
la Luz et al., 2011). The LM algorithm use three parameters: a set of points
in the electromagnetic spectrum as the independent variable ($\nu$) in the LM
scope, PAKALMPI as function model ($f(\nu,T_{r})$), and the temperature model
($T_{r}$) as a set of discrete points used to computed

$T_{b}=f(\nu,T_{r}),$ (1)

where $T_{b}$ is the synthetic spectrum. In our case, the observed spectrum
that corresponds with the frequencies is the dependent data ($T^{o}_{b}$). Our
model does not take into account limb brightening contribution. We are
computing the brightness temperature in the center of the stellar disk.

In Section 2 we present the region of the spectrum under study (independent
variable). In Section 3 present our parametric semiempirical model PakalMPI
(function model). In Section 4 the set of observations of Solar-like stars
used as test cases (dependent data). In Section 5 we describe the general
approximation and restriction of the LM method used in this work. In Section 6
we show the results of the case test. Finally in Section 7 we present our
discussion and conclusions.

##  2 The independent variable: The millimeter,submillimeter, and infrared
wavelengths

The independent variable in the LM scope represents the set of data that are
evaluated in the function model. In this work, the independent variable
($\nu$) is a set of frequencies of the electromagnetic spectrum at millimeter,
submillimeter, and infrared wavelengths

$\nu=\\{\nu_{1},\nu_{2},...,\nu_{k}\\},$ (2)

where $k$ is the number of points in the observed electromagnetic spectrum and
$\nu_{i}$ are central frequencies of radio observations constrained to cover
the 10 bands by ALMA (from $35$GHz to $950$GHz) and Spitzer ($12.5$THz):

$35\mbox{GHz}\leq\nu_{i}\leq 12.5\mbox{THz}.$

This range of frequencies allow us to use the tomographic properties of the
emission to estimate the radial temperature and density profile evolved in the
synthetic electromagnetic spectrum (De la Luz, 2016).

##  3 The parametric function model: PakalMPI

LM requires a function model that receives the independent variable ($\nu$)
and a set of parameters ($T_{r}$) that fit the model (Equation 1). In our
case, we modify PakalMPI model (De la Luz et al., 2010, 2011) to work as
parametric function. PakalMPI model uses Bremsstrahlung (Dulk, 1985; Güdel,
2002), Inverse Bremsstrahlung (Golovinskii, & Zon, 1980), and H-
(Zheleznyakov, 1996) mechanisms as source of opacity. However, Bremsstrahlung
and H- are the most important mechanism of thermal emission in the
chromosphere (De la Luz et al., 2011).

The set of parameters are temperature points that represent the radial
temperature profile under study

$T_{r}=\\{T_{1},T_{2},..,T_{n}\\},$

where $n$ is the number of altitudes in the radial temperature profile and
$T_{i}$ are the ordered radial temperatures against distance over the
photosphere from a semiempirical model. We used C7 model (Avrett, & Loeser,
2008) as initial condition (radial temperature and density) because is a
consistent model of the solar chromosphere especially on the temperature
minimum (De la Luz et al., 2014). The C7 model is a one-dimensional and time
independent model of the average quiet-Sun. This model replaced the earlier
model C of Vernazza et al. (1981) and take into account observations of the
Extreme Ultraviolet (EUV) spectrum and two measurements in the submillimeter
region to get a good average of the temperature minimum. However, this model
does not reproduce correctly the observations in the submillimeter, millimeter
and radio range (De la Luz, 2016). De la Luz et al. (2014) showed that by
modifying the temperature profile it is possible compute a better fit in this
region of the spectrum. The new model obtained was compatible with the theory
presented by Avrett, & Loeser (2008).

The temperature profile is passed as arguments together with the frequency
ranges to PakalMPI. Then, PakalMPI compute in three steps the synthetic
spectrum: i) compute a new radial density profile iteratively using the radial
temperature profile to satisfy the hydrostatic equilibrium, ii) compute the
ionization stages of a set of atoms that represents the chemical composition
of the atmosphere, and iii) solves the radiative transfer equation in the set
of requested frequencies. The two last steps can be computed in parallel using
the Message Passing Interface (MPI) methodology. Detailed information about
the steps can be found in De la Luz et al. (2010).

PakalMPI output includes the synthetic spectrum, the radial profiles of all
the ions, temperature, density, and pressure of the atmosphere in equilibrium.
This output is used in further steps of our methodology.

##  4 The dependent data: Observations of Solar-like stars at millimeter,
submillimeter, and infrared wavelengths

In this work, the dependent variable is represented by the observed spectrum

$T_{b}=\\{T_{b}^{\nu_{1}},T_{b}^{\nu_{2}},...,T_{b}^{\nu_{k}}\\},$ (3)

where $T_{b}^{\nu_{k}}$ is a observed point in the electromagnetic spectrum.

The development of new infrastructure (Atacama Large Millimeter/submillimeter
Array (ALMA), Large Millimeter Telescope (LMT), etc.) allow us to study main
sequence stars in mm - sub-mm regime. The emission at these wavelengths is
originated in the chromosphere mainly to free-free emission (Dulk, 1985;
Loukitcheva et al., 2004; Wedemeyer et al., 2016), and neutral interaction
(Zheleznyakov, 1996) but in general, it is not well-constrained (Cranmer et
al., 2013). The first binary stellar system that has been observed and studied
in this regime has been Alpha Centauri ($\alpha$ Cen) located at $1.3384\pm
0.0011$pc (Kervella et al., 2017). The system $\alpha$ Cen is composed of two
stars: $\alpha$ Cen A (G2 V) and $\alpha$ Cen B (K1 V). In both, the minimum
temperature was observed by Liseau et al. (2013) in the far-infrared. Liseau
et al. (2016) showed that an atmospheric structure similar to the Sun can be
adapted to solar-like stars.

In order to test our model, we have chosen $\alpha$ Cen A since it is often
considered a solar twin (Cayrel de Strobel, 1996) and has observations from
the far infrared to the millimeter (Liseau et al., 2016).

##  5 Levenberg-Marquardt approach: The Kinich-Pakal model

There are several implementations of the Levenberg-Marquadt algorithm
(Marquardt, 1963; Press et al., 1986; Pujol, 2007). In this work we use least
square fit (LsqFit) package developed by Julia Optim team (Mogensen et al.,,
2018).

The model that we present in this work is called Kinich-Pakal (KP). The aim of
KP is to compute the best model of stellar atmosphere to fit the synthetic
spectrum to observations at millimeter, submillimeter, and infrared
wavelengths.

In Figure 1 we show the pseudo code of KP. We start reading the initial
conditions of a single model that includes: radial temperature, Hydrogen
density, and pressure. KP read the radial profiles and the observations and
start the computations iterative using PakalMPI as a function model.

In each iteration, KP computes the hydrostatic equilibrium of the input
atmosphere and compute their synthetic spectrum. The computations of the
synthetic spectrum is performed in parallel for each frequency. If the
differences between the observed and the synthetic spectrum are greater than
an epsilon stop flag, then the computed atmosphere becomes the initial
condition for the next iteration and the process continues to step forward.

KP has several flags that constrain the Levenberg-Marquardt method. The
constraints include: set the global limits of the solution, limit the solution
in a defined range, and the absolute error tolerance against the set of
observations. The performance of KP depends of the flag constraints and is
studied in the following section.

##  6 Test Case: $\alpha$ Centauri A

$\alpha$ Cen A and B was the first stellar system resolved in submillimeter by
ALMA (Liseau et al., 2015). Table 1 show the properties of the observations of
$\alpha$ Cen A during the Cycle $2$ of ALMA between July $2014$ and May
$2015$. In Table 2 we show additional observations of $\alpha$ Cen A. In the
following subsections, we describe the set of test used by KP.

###  6.1 Infrastructure for the computations

For the following tests, we run KP in a single node of the cluster of the
Center of Supercomputing of Space Weather of the National Laboratory of Space
Weather in Mexico (LANCE). The node of the cluster have $40$ threads with
Intel Xeon E52670 v2 a $2.5$GHz, $128$GB of RAM (DDR3-$1866$) and $4TB$ for
storage. The operating system is Linux $4.2.0$ SMP for $64$bits.

###  6.2 Test of auto convergence

Before starting these simulations with the real observed spectrum, we test the
model using a synthetic spectrum produced by a solar atmosphere in hydrostatic
equilibrium.

We use as input atmospheric model a set of radial profiles with variations of
a pre-calculated atmosphere in equilibrium over the photosphere (z). We use
the C7 model with variation of temperature of $-1000$K, $-100$K, $-10$K,
$+10$K, $+100$K, and $+1000$K as shown in Figure 2a. We replace the observed
spectrum used in this test with a synthetic spectrum computed with PakalMPI
for the C7 model (Test Spectrum). The idea of replace the observed spectrum is
to keep under control the input and the output of the model.

KP computed successfully these models in around $3$ hour using a single node
with $28$ threads (one for each frequency). Table 3 show the frequencies and
the computed synthetic spectrum used in this test.

In order to determinate the region where the emission is generated, we include
in Figure 2c the normalized Contribution Function (CF) defined by

$\mbox{CF}=j_{\nu}\exp{(-\tau_{\nu}),}$ (4)

where $j_{\nu}=\kappa_{\nu}B(T)$ and

$\tau_{\nu}=\int\kappa_{\nu}dz,$ (5)

is the optical depth. We found that our model computes the emission between
$140$km and $2120$km for $12.5$THz and $35$GHz respectively. The gray regions
in Figure 2a-c shows the outer boundary of the model where the convergence
method can not be applied directly. On the other hand, the synthetic spectrum
computed with these models fit the Test Spectrum (Figure 2d). The only result
that not converge completely to the initial condition is the $+1000$K model
(segmented black line in Figure 2b). The auto convergence test show that our
methodology has an upper boundary for the initial condition of $+1000$K.

In the following tests, we use the real spectrum of $\alpha$ Cen A with
different restrictions in KP to know the stability of the model.

###  6.3 Test without Restrictions (KP2019A)

In the second test, we use real observations of $\alpha$ Cen A using C7 as an
input model without restrictions (i.e. the temperature model can take any
value, even negative). KP makes $12730$ iterations in $49.5$ hours in order to
obtain the model KP2019A. The results are shown in Figure 3a-b as dotted line.
KP fixes the observed spectrum (Figure 3a) but the atmospheric model presents
changes in the radial temperature that are very complicated to explain
physically. The model shows the following characteristics: high temperature at
around $1500$km over the photosphere $T_{r}>11500$K, negative temperatures at
$1950$km, and abrupt changes of temperature that reaches values between
$3\times 10^{5}$K and $-2\times 10^{5}$K that are physically impossible.
Analyzing in detail the ray path of the solution of the radiative transfer
equation, we found that negative temperatures are due Bremsstrahlung opacity.
This opacity has the following form:

$\kappa_{\nu}=C\frac{n_{e}n_{i}}{T^{3/2}}\nu^{2},$

where $C$ is a constant, $n_{e}$ is the electron density, $n_{i}$ the ion
density, $T$ the temperature, and $\nu$ the frequency. For temperatures less
than $3000$K the ion density of all the species that we study drops almost to
zero ($n_{i}\approx 0$) due there is not enough energy to ionize the medium,
especially Hydrogen. That’s means that the atmosphere becomes optically thin.
In the other way, when the temperatures are higher than $2\times 10^{4}$K all
the medium is practically ionized, then the opacity can be rewritten as

$\kappa_{\nu}\approx C\frac{n_{e}^{2}}{T^{3/2}}\nu^{2},$

the $n_{e}$ reaches a limit, however, the temperature can increase and as a
consequence, the opacity again drops to zero. In both cases: low and high
temperatures the atmosphere becomes optically thin. These results show that
exist solutions computed by KP in the radiative transfer equations without
physical interpretations. In the following test, we constrain the minimum
temperature in the solution to try to solve this feature of the model.

###  6.4 Test Constraining Positive Temperatures (KP2019B)

In this test, we constrain the lower temperature limit in $3000$K. KP produced
the KP2019B model after computed $11300$ models in around $44$ hours. In the
Figure 3a-b we show the results as dashed-point line. We observe that the
computed spectrum not exactly fix the observations of $\alpha$ Cen A (Figure
3a) like the last test. The radial temperature profile again shows
temperatures without physical interpretations (Figure 3b). After $1600$km, the
radial temperature increase suddenly with temperatures around $10^{5}$K.

###  6.5 Test with Bounded Temperature (KP2019C)

We bounded the temperature taking both: the initial model as reference and the
results of the Subsection 6.2. We use C7 model and a boundary of $\pm 1000$K
as constraining. KP computed $7968$ models in around $31$ hours to produce our
model called KP2019C. In Figure 3a-b we show the results as dashed line.
Figure 3a shows that the spectrum computed by KP is close to fit the observed
spectrum. However, we can observe that for low frequencies the model not fix
exactly the observed spectrum. The temperature model (Figure 3b) shows similar
C7 model shape but with three main differences: i) at photospheric altitudes
themKP2019C show a higher temperature than C7 model, ii) the temperature
minimum of KP2019C is closer to the photosphere and shows a lower temperature
of around $3200$K than C7 model, and iii) the temperature in the high
chromosphere of KP2019C (after $1500$km) is higher than C7 by $1000$K. In the
following test, we analyze the impact of the spatial resolution of the radial
temperature profile.

###  6.6 Testing the Spatial Resolution (KP2019D)

The C7 model starts with $140$ layers divided into three parts: photosphere
($20$ layers), chromosphere ($81$ layers), and corona ($39$ layers) but in
this test, we interpolate the model to increase the density 4 times in the
chromospheric region up to $324$ layers. KP2019D model is composed of $383$
layers.

We used the same constraints as in the last subsection. To calculate the final
spectrum, KP computes $23585$ models in around $156$ hours. The cyan solid
lines in Figure 3a-b shows the results of KP2019D model. In figure 3a, we plot
the spectrum obtained by the model showed in Figure 3b. We observe the same
similarities of KP2019C but with changes in temperatures with more detailed
spatial resolution. For example, the peak of temperature of KP2019C at around
$1000$km now is a set of spikes.

###  6.7 Smoothing Model (KP2019E)

We smooth the radial temperature profile from KP2019D using the filter of
Savitzky-Golay with a nine degrees polynomial (Savitzky & Golay, 1964) to know
the effect of these spikes in the spectrum. This last profile was equilibrated
again with PakalMPI to guarantee the hydrostatic equilibrium. The results are
plotted in red colors in the same Figure 3a-b. We called this model KP2019E.

In the model KP2019E, we eliminate the spikes and compute their synthetic
spectrum using PakalMPI. In Figure 3a we observe the results of eliminating
the spikes in the temperature model. At low frequencies, the spectrum has a
better fix. At frequencies lower than $300$GHz the spectrum not fix exactly
the observations but are inside of the observations error limits for all the
points of the spectrum, except at $700$GHz that correspond to the Band 9 of
ALMA. This observation is the responsible of the increase of temperature in
the radial profile between $450$km and $1100$km over the photosphere. The
model KP2019E shows a continue profile without spikes. The temperature minimum
is lower ($\sim 3000$K) and closer to the photosphere ($\sim 500$km) than C7
model. The radial temperature of KP2019E after $1500$km is higher than C7
model.

In Figure 3 we show the results of computing the CF (panel c) and optical
depth (panel d) for the KP2019E model at $35$GHz, $97.5$GHz, $233$GHz,
$679$GHz, and $12491.5$GHz. We found that for the case of $12491$GHz the CF
have a maximum at altitudes closed to $250$km over the photosphere. For the
other cases, the CF shows three peaks at $679$GHz, two peaks at $233$GHz, and
one peak at $97.5$GHz and $35$GHz. These peaks in the CF shows similarities
with the reported solar CF (Selhorst et al., 2019) that show two peaks (around
$230$GHz) and one peak (around $100$GHz).

##  7 Discussion and Conclusions

In this work, we present KP a non-linear method to compute a stellar
atmosphere to fix observed and synthetic spectrum trough a semiempirical model
of the chromosphere at millimeter, submillimeter and infrared wavelengths. KP
uses the Levenberg-Marquardt algorithm to minimize the differences between
synthetic and observed spectrum running PakalMPI to equilibrate
hydrostatically the atmosphere and compute their synthetic spectrum. KP uses
MPI libraries, C, and Julia Languages to solve the model in parallel where it
is possible to divide the computations.

We test the autoconvergence of the method using a set of variations of C7
model and found that variations greater than $+1000$K over the profile impacts
in the final radial temperature model. This numerical behavior shows that the
initial conditions should be lower than $+1000$K over the final solution. We
can not compute directly stars with spectral class far to solar-like
classification. Computations with small variations in the radial temperature
profile to reproduce their emission is mandatory, specially for stars colder
than the Sun. We used the radial density of C7 model as an initial condition
for all the simulations. However, Figure 2c show that the CF for low
frequencies is truncated due to low density at this altitude. In order to fix
the CF a wider chromosphere than C7 model of density is required.

In this context, we use spectral observations of $\alpha$ Cen A at millimeter,
submillimeter, and infrared wavelengths and C7 semiempirical model as input
model with the following constraints: without restrictions, limit the lower
temperature, and bounding temperature profile. We found that the solution of
KP is acceptable only in the regimen of ”bounding temperature” as shows in the
model KP2019D. A detailed simulation using greater spatial resolution shows
that the shape of the profile preserves but the profile shows several spikes
that are numerical artifacts in the radial temperature profile. These spikes
are related to the increase in the brightness temperature in some points of
the observed spectrum. Specifically at Band 3 ($97.5$GHz) and 9 ($679$GHz) of
ALMA.

We eliminate the spikes using the filter of Savitzky-Golay with a polynomial
of nine degree. We compute the hydrostatic equilibrium of this last profile
and called to this model KP2019E. The synthetic spectrum obtained from KP2019E
fixed all the observed points inside of their observational errors for
$\alpha$ Cen A except band 3 and 9 of ALMA. The spectrum computed from KP2019E
suggest that both bands presents an over flux in agreement with the analysis
of Liseau (2019).

The CF for frequencies at $35$GHz, $97.5$GHz, $233$GHz, $679$GHz, and
$12491.5$GHz (ALMA and Spitzer coverage, Figure 3c) shows similarities (one
peak of CF at $100$GHz and two peaks at $230$GHz) in comparison with the solar
models (Selhorst et al., 2019). In the KP2019E model, the maximum peaks for
the aforementioned frequencies occurs at a height of $2140$km, $1625$km,
$1150$km, $510$km, and $275$km that corresponds with the distance over the
stellar photosphere where $\tau_{\nu}\approx 1$ (Figure 3d).

The results obtained by the methodology here presented show that KP is capable
of finding the physical conditions of stellar atmospheres of Solar-Like stars
using their observed spectra at millimeter, submillimeter, and infrared
wavelengths. We show, that KP is stable and a fast method that can compute
thousands of atmospheres in a few hours.

The impact of the radial density as initial condition in the computation of
the radial temperature profile as well as the estimation of a wider solar
chromosphere using our methodology is considered as future work. KP can be
found under request to the authors.

The authors thank the referee for helpful and valuable comments on this paper.
This work was supported by the Ciencia Básica (254497) CONACyT Fellowship.

## References

  * Avrett, & Loeser (2008) Avrett, E. H., & Loeser, R. 2008, ApJS, 175, 229
  * Carlsson (1992) Carlsson, M. 1992, Cool Stars, Stellar Systems, and the Sun, 499
  * Cayrel de Strobel (1996) Cayrel de Strobel, G. 1996, A&A Rev., 7, 243
  * Cranmer et al. (2013) Cranmer, S. R., Wilner, D.J. and MacGregor, M.A., 2013, ApJ, 772(2), p.149
  * De la Luz et al. (2010) De la Luz, V., Lara, A., Mendoza-Torres, J. E., et al. 2010, ApJS, 188, 437
  * De la Luz et al. (2011) De la Luz, V., Lara, A., & Raulin, J.-P. 2011, ApJ, 737, 1
  * De la Luz et al. (2014) De la Luz, V., Chavez, M., Bertone, E. et al., 2014, Sol Phys, 289: 2879
  * De la Luz (2016) De la Luz, V. 2016, ApJ, 825, 138
  * Dere et al. (1997) Dere, K. P., Landi, E., Mason, H. E., et al. 1997, A&AS, 125, 149
  * Dulk (1985) Dulk, G. A. 1985, ARA&A, 23, 169
  * Fontenla et al. (1993) Fontenla, J. M., Avrett, E. H., & Loeser, R. 1993, ApJ, 406, 319
  * Fontenla et al. (2006) Fontenla, J. M., Avrett, E., Thuillier, G., et al. 2006, ApJ, 639, 441
  * Golovinskii, & Zon (1980) Golovinskii, P. A., & Zon, B. A. 1980, Zhurnal Tekhnicheskoi Fiziki, 50, 1847
  * Güdel (2002) Güdel, M. 2002, ARA&A, 40, 217
  * Kervella et al. (2017) Kervella, P., Bigot, L., Gallenne, A., et al. 2017, A&A, 597, A137
  * Kudaka et al. (2015) Kudaka, A. S., Cassiano, M. M., Marcon, R., et al. 2015, Sol. Phys., 290, 2373
  * Linsky (2017) Linsky, J. L. 2017, ARA&A, 55, 159
  * Liseau et al. (2013) Liseau, R., Montesinos, B., Olofsson, G., et al. 2013, A&A, 549, L7
  * Liseau et al. (2015) Liseau, R., Vlemmings, W., Bayo, A., et al. 2015, A&A, 573, L4
  * Liseau et al. (2016) Liseau, R., De la Luz, V., O’Gorman, E., et al. 2016, A&A, 594, A109
  * Liseau (2019) Liseau, R. 2019, arXiv e-prints, arXiv:1904.03043
  * Loukitcheva et al. (2004) Loukitcheva, M., Solanki, S. K., Carlsson, M. et al., 2004, A&A, 419, 747
  * Machado et al. (1980) Machado, M. E., Avrett, E. H., Vernazza, J. E., et al. 1980, ApJ, 242, 336
  * Marquardt (1963) Marquardt, D. W. 1963, Journal of the Society for Industrial and Applied Mathematics, 11, 431
  * Mogensen et al., (2018) Mogensen, P. K., & Riseth, A. N. 2018, Journal of Open Source Software, 3, 615
  * Nakajima et al. (1995) Nakajima, H., Nishio, M., Enome, S., et al. 1995, Journal of Astrophysics and Astronomy Supplement, 16, 437
  * Pujol (2007) Pujol, J. 2007, Geophysics, 72, W1
  * Press et al. (1986) Press, W. H., Flannery, B. P., & Teukolsky, S. A. 1986, Cambridge: University Press
  * Savitzky & Golay (1964) Savitzky, A., & Golay, M. J. E. 1964, Analytical Chemistry, 36, 1627
  * Selhorst et al. (2019) Selhorst, C. L., Simões, P. J. A., Brajša, R., et al. 2019, ApJ, 871, 45
  * Trottet et al. (2015) Trottet, G., Raulin, J.-P., Mackinnon, A., et al. 2015, Sol. Phys., 290, 2809
  * Vernazza et al. (1976) Vernazza, J. E., Avrett, E. H., & Loeser, R. 1976, ApJS, 30, 1
  * Vernazza et al. (1981) Vernazza, J. E., Avrett, E. H., & Loeser, R. 1981, ApJS, 45, 635
  * Wedemeyer-Böhm & Wöger (2007) Wedemeyer-Böhm, S. and Wöger, F., 2007, Proceedings of the IAU, 3(S247), p.66-73.
  * Wedemeyer et al. (2016) Wedemeyer, S., Bastian, T., Brajša, R., et al. 2016, Space Sci. Rev., 200, 1
  * White et al. (2017) White, S. M., Iwai, K., Phillips, N. M., et al. 2017, Sol. Phys., 292, 88
  * Zheleznyakov (1996) Zheleznyakov, V. V. 1996, Astrophysics and Space Science Library

Figure 1: Diagram of operation for KP. KP uses a parametric function (PAKALMPI), a set of dependent data (observations) and independent variables (frequency) as inputs. LsqFit function starts the Levenberg-Marquart algorithm. PAKALMPI compute the hydrostatic equilibrium and solve the radiative transfer equation. LM receives the computed spectrum from PAKALMPI and compares it with observations. If the error is higher than predefined $\delta$ then LM modifies the temperature profile and repeats the process. If the error is less than $\delta$ then KP produces a new semiempirical model.  Figure 2: Convergence analysis using C7 model for a set of initial conditions of radial temperature profile with $-1000$K, $-100$K, $-10$K, $+10$K, $+100$K, and $+1000$K. (a) Set of models used as initial conditions for the chromosphere using as reference the C7 model. (b) Radial temperature profile of each computed model in hydrostatic equilibrium. (c) Contribution functions of the C7 model for frequencies close to those observed by ALMA and Spitzer. (d) Relative error of the synthetic spectrum for each model taking as reference the C7 model. All the models recovery the C7 synthetic spectrum but $+1000$K model not recovery the radial temperature profile. The models $+1000$K, $-1000$K, and $+100$K shows negative values for radial temperature at altitudes higher than $2130$km over the photosphere where the model can not be applied (gray regions panel b-c).  Figure 3: $\alpha$ Cen A model obtained from KP. (a) The dotted line shows the model KP2019A without restrictions, dashed-point line presents the KP2019B model with positive temperatures restriction, dashed line is KP2019C model with bounded temperature, cyan solid line indicate the synthetic spectrum from the raw temperature profile (KP2019D model), the red solid line is the synthetic spectrum after smooth (KP2019E model), and the solid black line is the synthetic spectrum of C7 solar model. The circle markers are observations of $\alpha$ Cen A from Liseau et al. (2016, 2013), and the triangle markers are the sun observations from ALMA White et al. (2017). (b) Radial temperature profiles over the photosphere. We show that the radial temperature for $\alpha$ Cen A between $290$ and $1560$km is lower than the solar model. In the outer boundary, the temperature remains higher than the solar model. (c) Normalized Contribution Function (CF) of the KP2019E model of $\alpha$ Cen A for the same frequencies. For the case of $12491$GHz, we found that the CF has a maximum close to $250$km over the photosphere. For the other cases, the CF shows three peaks for $679$GHz and two peaks for $233$GHz and one peak for $97.5$GHz and $35$GHz. (d) Optical depth for $35$GHz, $97.5$GHz, $233$GHz, $679$GHz, and $12491.5$GHz for KP2019E model of $\alpha$ Cen A.  Wavelength | Frequency | Facility | Flux | Brightness Temperature
---|---|---|---|---
$[$mm] |  $[$GHz] |  |  $[$mJy] |  $[$K]
3.075 | 97.5 | ALMA Band 3 | 3.33$\pm$0.01 | 8618$\pm$31
2.068 | 145 | ALMA Band 4 | 6.33$\pm$0.08 | 7316$\pm$88
1.287 | 233 | ALMA Band 6 | 13.58$\pm$0.08 | 6087$\pm$36
0.873 | 343.5 | ALMA Band 7 | 26.06$\pm$0.19 | 5351$\pm$49
0.740 | 405 | ALMA Band 8 | 35.32$\pm$0.21 | 5242$\pm$55
0.442 | 679 | ALMA Band 9 | 107.20$\pm$1.50 | 5678$\pm$79
Table 1: ALMA Brightness temperatures for $\alpha$ Cen A from Liseau et al. (2016). Wavelength | Frequency | Facility | Flux | Brightness Temperature
---|---|---|---|---
$[$mm] |  $[$GHz] |  | [$\rm mJy$] | [K]
0.870 | 345 | LABOCA | 28.00$\pm$7 | 5738$\pm$1432
0.500 | 600 | Herschel-SPIRE | 80$\pm$30 | 5421$\pm$2018
0.350 | 857 | Herschel-SPIRE | 145.00$\pm$28 | 4822$\pm$927
0.250 | 1199 | Herschel-SPIRE | 240.00$\pm$50 | 4084$\pm$845
0.160 | 1874 | Herschel-PACS | 560.00$\pm$60 | 3920$\pm$394
0.100 | 2998 | Herschel-PACS | 1410.00$\pm$50 | 3909$\pm$135
0.070 | 4283 | Herschel-PACS | 3350$\pm$28 | 4540$\pm$37
0.024 | 12491.5 | Spitzer-MIPS | 28530$\pm$580 | 47364$\pm$91
Table 2:  Brightness temperatures at infrared wavelengths for $\alpha$ Cen A from Liseau et al. (2013). Frequency | Brightness Temperature | Frequency | Brightness Temperature
---|---|---|---
$[$GHz] | [K] |  $[$GHz] | [K]
35 | 7298 | 800 | 6600
40 | 7150 | 900 | 6591
50 | 6975 | 1000 | 6577
60 | 6878 | 2000 | 5918
70 | 6820 | 3000 | 5268
80 | 6781 | 4000 | 4982
90 | 6754 | 5000 | 4866
100 | 6734 | 6000 | 4823
200 | 6662 | 7000 | 4831
300 | 6641 | 8000 | 4838
400 | 6630 | 9000 | 4844
500 | 6621 | 10000 | 4854
600 | 6615 | 11000 | 4866
700 | 6608 | 12500 | 4886
Table 3: Synthetic spectrum between 35 GHz and 12.5 THz by PAKALMPI using the
C7 model.


heightadjust=all, floatrowsep=columnsep

# SelfText Beyond Polygon: Unconstrained Text Detection with Box Supervision
and Dynamic Self-Training

Weijia Wu1∗ Enze Xie2∗ Ruimao Zhang2 Wenhai Wang3 Guan Pang4 Zhen Li5
Hong Zhou1† Ping Luo2

###### Abstract

Although a polygon is a more accurate representation than an upright bounding
box for text detection, the annotations of polygons are extremely expensive
and challenging. Unlike existing works that employ fully-supervised training
with polygon annotations, we propose a novel text detection system termed
SelfText Beyond Polygon (SBP) with Bounding Box Supervision (BBS) and Dynamic
Self Training (DST), where training a polygon-based text detector with only a
limited set of upright bounding box annotations.

For BBS, we firstly utilize the synthetic data with character-level
annotations to train a Skeleton Attention Segmentation Network (SASN). Then
the box-level annotations are adopted to guide the generation of high-quality
polygon-liked pseudo labels, which can be used to train any detectors. In this
way, our method achieves the same performance as text detectors trained with
polygon annotations ( _i.e.,_ both are 85.0% F-score for PSENet on ICDAR2015
). For DST, through dynamically removing the false alarms, it is able to
leverage limited labeled data as well as massive unlabeled data to further
outperform the expensive baseline. We hope SBP can provide a new perspective
for text detection to save huge labeling costs. Code is available at: $\tt
github.com/weijiawu/SBP$.

††1 Zhejiang University, China.††2 The University of Hong Kong, Hong Kong
SAR.††3 Nanjing University, China.††4 Facebook AI, USA.††5 Chinese University
of Hong Kong, Shenzhen.††† Corresponding author.
(zhouh@mail.bme.zju.edu.cn)††∗ Authors contributed equally.
<EMAIL_ADDRESS>
##  1 Introduction

Scene text detection [28, 16, 40, 21, 22], which aims to locate texts in the
wild, has achieved much attention in recent years because of its numerous
applications, _e.g.,_ instant translation, image retrieval, scene parsing.
Since text instances in the real-world scenario are usually with various
sizes, directions and shapes, most scene text detection methods [30, 21, 31]
utilize polygon annotation to train robust text detectors. Although the
polygon is more accurate than the upright bounding box(bounding box equals
upright bounding box with two points in this paper), the labeling cost of
polygons is extremely high, limiting its large-scale extension in real-world
applications.

Figure 1: The comparisons of data cost and the performance of the proposed
method. (a) Annotating more points are more expensive (the price of 100 text
instances). (b) Data collection is much cheaper than annotation. (c) The
proposed SBP achieves competitive performance while saving huge labeling costs
for PSENet [30] on Total-Text [4]. Note that the data cost information is
obtained from Amazon Mechanical Turk (MTurk), and we make a rough cost
evaluation of Total-Text.

To address the above-mentioned issue, in this paper, we propose a novel system
termed SelfText Beyond Polygon (SBP) for unconstrained scene text detection
with extremely low data collection and labeling costs, _i.e.,_ with limited
bounding box annotated data and a larger amount of unlabeled data. Our
motivations are inspired by the following observations. First, although
bounding box annotation is limited by its performance, there is still a cheap
way of labeling to roughly distinguish the foreground and background
locations. As illustrated in Fig. 1 (a), according to the Amazon Mechanical
Turk (MTurk), the upright bounding box annotations are about 4$\times$ cheaper
than polygon annotations [4, 38]. Thus, how to effectively utilize the
bounding box annotations to boost the detection accuracy becomes critical in
such a situation. Second, even if the upright bounding box annotation is
adopted, the cost is still very high when the data volume breakthroughs the
million range. Fortunately, the data collection cost is usually much cheaper
than labeling with the amount of the data increases, just as shown in Fig. 1
(b). Thus how to progressively mining the valuable information hidden in the
massive unlabeled data to further improve the accuracy of the detector is
another problem that will be addressed in this paper.

In practice, we deal with the above problems by proposing two novel schemes
named Bounding Box Supervision (BBS) and Dynamic Self-Training (DST) to
efficiently use a limited set of bounding-box annotated data and massive
unlabeled data, respectively. For the former one, we firstly apply the
synthetic data whose character-level annotations can be automatically
generated for free, to pre-train a newly proposed Skeleton Attention
Segmentation Network (SASN). By exploiting the text data with various
geometric information (_e.g.,_ the straight and the curved data ) in the
training phase, the SASN could effectively capture the changes of the text
appearance in different scenes. Then bounding box annotations are used to crop
the text region in the real images which are fed into SASN to generate high-
quality polygon-liked pseudo labels. By splicing all of these local pseudo
labels, the global one would be obtained and applied to train any text
detectors. For the later one, we propose a Dynamic Self-Training (DST) scheme,
which adopts the filtering and multi-scale inference strategies to reduce the
effect of false positive and false negative in pseudo labels for the unlabeled
data. In addition, the dynamic mixed training strategy is also used to limit
the use of the pseudo labels in the late stage of training. As shown in Fig. 1
(c), by using BBS and DST, PSENet [30] achieves $4\times$ cheaper data cost
compare with that of using polygon annotation, and obtaining $33.5\%$ F-score
improvements compare with that of directly using bounding box annotation on
Total-Text [4]. The main contributions are four folds:

(1) We introduce a Bounding Box Supervision strategy, which utilizes free
synthetic data and box-level annotations to achieve high quality polygon-liked
real annotations (_i.e.,_ pseudo labels).

(2) We introduce a Skeleton Attention Segmentation Network for capturing the
structure of the texts with arbitrary shapes and restraining noise
interference.

(3) We propose a stable self training scheme termed Dynamic Self-Training to
utilize large scale unlabeled data and limited labeled data for robust
detectors training.

(4) The proposed SelfText Beyond Polygon is simple but surprisingly effective
and practical, which saves 70%+(rough estimation) cost and achieves almost the
same performance compared with fully-supervised methods.

##  2 Related Work

###  2.1 Supervised Text Detection

Scene text detection has achieved remarkable progress in the deep learning
era. Previous methods [16, 39] focused on horizontal text detection. CTPN [28]
adopted Faster RCNN [26] and modified RPN to detect texts. Then many famous
methods [23, 17] aim at solving multi-oriented text detection. EAST [40] used
FCN [19] to predict text score map, distance map and angle map in an anchor-
free manner. Recent works focus on curved text detection [1, 35]. TextSnake
[21] modeled curved text instance as a series of orderd disks and a text
center line. SPCNet [33] and Mask Text Spotter [22] are based on Mask RCNN,
which translate curved text detection as a instance segmentation problem.
PSENet [30] and PAN [31] treat text instance as kernels with different scales,
and reconstrust the whole text instance in the post-processing.

Figure 2: Illustration of the whole SelfText Beyond Polygon. (a) The pipeline
of the Bounding Box Supervision, which training with synthetic data and
inference on real data to generated pseudo label, details as shown in Fig. 3.
(b) The pipeline of Dynamic Self-Training, which utilizes limited data and
massive unlabeled data to train detectors for saving cost.

###  2.2 Weak Supervised Object Detection

To alleviate the expensive data cost problem, there have been various weakly
supervised object detection (WSOD) works [29] based on image-level
annotations. Bilen _et al._ [2] proposed an architecture called WSDDN to
select and classify region proposals simultaneously. ContextLocNet [12]
further improves WSDDN by taking contextual region into consideration. Beyond
that, some works tried to utilize the temporal information in videos. Kwak _et
al._ [15] discovered object appearance representation across videos and
tracked the object in temporal space for supervision. Wang _et al._ [32]
performed unsupervised tracking on videos and clustered similar features.
Recently, Yang _et al._ [36] and Kim _et al._ [14] proposed activity-driven
methods, which exploits action classes as contextual information to localize
objects. Different from the above methods, we develop a novel pipeline for
weakly supervised text detection, which generating polygon-liked pseudo label
with bounding-box annotation (2 points) instead of polygon annotation (4-20
points) to train networks.

###  2.3 Self-training with noisy labels

Self-training, where a teacher model is used to generate labels for the
unlabeled set the student can train on, is a widely-used methods in semi-
supervised learning. Xie _et al._ [34] use strong data augmentation in self-
training but only for image classification. Ilija _et al._ [25] proposed a
data distillation method to improve instance detection level tasks and vote on
unlabeled data to get pseudo-label. Recently, noise labels have been shown to
have positive effects in self-training for object detection [41], which
motivates our work. In this work, we introduce a novel dynamic self-training
approach to minimizing the negative effects of false negatives and false
positives in pseudo labels.

##  3 Methods

As shown in Fig. 2, we describe the details of the proposed SelfText Beyond
Polygon (SBP), which has two key components termed the Bounding Box
Supervision (BBS) and the Dynamic Self-Training (DST).

Figure 3: The whole pipeline of Bounding Box Supervision. (a) The architecture
of Skeleton Attention Segmentation Network, which is composed of two streams:
regular stream and skeleton stream. (b) The details of Skeleton Attention
Module, which refine the input feature map by weighting with attention map.
(c) The detailed structure of the Decoder. How to generate polygon-liked
pseudo label includes two steps: (1) Blue arrows show training SASN with
synthetic data based on character annotations. (2) Red arrows illustrate
generating pseudo labels on real data based on bounding box annotations.

###  3.1 Bounding Box Supervision

Following the segmentation with bounding boxes [6], we propose a weakly
supervised method, termed Bounding Box Supervision (BBS), which can achieve
competitive accuracy with low-cost bounding box annotations for text
detection. As shown in Fig. 2 (a), BBS utilizes a novel proposed Skeleton
Attention Segmentation Network (SASN), which is shown in Fig. 3, to generate
polygon pseudo labels based on the given bounding box annotations. The process
composes three steps. (1) In the first step ( see blue arrows in Fig. 2 ), we
train SASN with free synthetic data based on character annotation. (2) In the
second step ( see red arrows in Fig. 2 ), the box-level annotations are
utilized to crop the real image, which are fed into the SASN for generating
polygon-liked pseudo labels. (3) By splicing all of the local pseudo labels,
the global one is obtained. In this way, bounding box annotations can be
converted to high-quality polygon pseudo labels. The detectors ( _e.g.,_
PSENet [30] ) trained on these pseudo labels can achieve almost the same
performance as those trained on polygon annotations.

####  3.1.1 Skeleton Attention Segmentation Network

Network Architecture. As presented in Fig. 3 (a), we use ResNet50 [11] as the
backbone network for the SASN, and extract three levels of features ( _i.e.,_
$C1,C2,C3$ ) from different downsampled scales ( _i.e.,_ $1/4,1/8,1/16$ ).
After that, the skeleton stream fuses $C1$ and $C3$ to predict the skeleton
attention map. At the same time, the regular stream segment the text by
applying the Skeleton Attention Module. Although text segmentation in a
bounding box is easier than ain the whole image, there still exists a bit of
noise. For example two texts in one bounding box, which is illustrated in the
Fig. 3 (a) with green circle. To restrain the noise interference and segment
the body text precisely, we carefully design the Skeleton Attention Module,
which generates the skeleton map to drive the attention mechanism.

Skeleton Attention. Fig. 3 (b) illustrates the details of the Skeleton
Attention Module (SA). Given an input sample
$(x_{i},y_{i})\in\\{(x_{1},y_{1}),(x_{2},y_{2}),...(x_{n},y_{n})\\}$, where
$x_{i}$ and $y_{i}$ denote the $i$-th image and its labels. We use $x_{i}^{k}$
to denote the $k$-th pixel of the $i$-th training image, with $y_{i}^{k}$= $0$
for the background and $y_{i}^{k}$ = $1$ for the text pixel. We also define
the text skeleton ground-truth as a soft label. For the $k$-th pixel in the
text region of $i$-th image, we first calculate the shortest distance
$d_{i}^{k}$ between the $k$-th pixel to its nearest background pixel, and then
the value $p_{i}^{k}$ is defined as the soft skeleton label of $k$-th pixel by
normalizing $d_{i}^{k}$ to $[0,1]$:

$\small\vspace{-0.2cm}p_{i}^{k}=\sum_{k}\frac{d_{i}^{k}}{d_{i}^{*}}\,,$ (1)

where $d_{i}^{*}$ is the maximum value of $\\{d_{i}^{k}\\}$ in the $i$-th
image.

Intuitively, the pixels close to the skeleton of the text instance should have
a greater value than the boundary pixels ( see text skeleton label in Fig. 3
(a) ). Since the soft label is a decimal representing the degree of distance,
it is incompatible with the commonly binary cross-entropy loss. Besides, L1
and L2 loss are not sensitive to the distance distribution among $[0,1]$ [26].
Therefore, to handle the soft label, we modify the cross-entropy loss into a
“soft” form. For a pixel $x_{i}^{k}$, $p_{i}^{k}$ denotes the value of the
$k$-th pixel in the ground truth skeleton map. $\mathcal{F}$ indicates neural
networks. Thus the text skeleton loss in Fig. 3 (a) is defined as follows:

$\small\mathcal{L}_{ske}=-\sum_{k}log(1-\left|p_{i}^{k}-\mathcal{F}(x_{i}^{k})\right|)\,.$
(2)

In practice, the predicted skeleton map is downsampled to obtain multi-scale
maps (_i.e.,_ 1/4, 1/8, 1/16), which are transferred to the regular stream
(the thick yellow arrow in Fig. 3 (a) ). And then, the extracted feature map
(_e.g.,_ $C1$) and the corresponding scale skeleton map are as the input of
the Skeleton Attention Module. The two feature maps before and after the
skeleton attention are concatenated and fed into a channel attention block. At
last, it outputs the refined feature map. Note that the proposed Skeleton
Attention Module is shared for the high-level and the low-level feature maps (
_i.e.,_ $C1,C2,C3$ ).

####  3.1.2 Suitable Synthetic Datasets

Existing real-world text datasets can be divided two types: straight (_e.g.,_
ICDAR2015 [13]) and curved (_e.g.,_ Total-Text [4]), depending on the shape of
the text instance. However, the curved synthetic text data is scarce.
SynthText [9], the most widely-used synthetic datasets, does not contain
curved text, which leads to a serious domain gap between synthetic data and
real data ( _e.g.,_ Total-Text ) in curved text distribution (
$0.2\%~{}v.s.~{}58\%$ ), as shown in Fig. 4. Therefore, we also adopt Curved
SynthText [20] to align the data distribution with curved dataset. In this
work, we use SynthText [9] to train SASN for straight text line and use Curved
SynthText [20] to match curved text line.

Figure 4: The shape distribution difference between synthetic data and real
data causes serious domain gap.

###  3.2 Dynamic Self Training for Text Detection

Inspired by previous works [34, 41], we propose a novel self-training
strategy, termed Dynamic Self Training, to reduce label cost by exploiting
large-scale unlabeled data.

####  3.2.1 Overview of Dynamic Self-Training

Fig. 2 (b) gives an overview of the proposed dynamic self-training scheme. In
such case, we have a large set of unlabeled images
$\\{\widetilde{x}_{i}\\}_{i=1}^{m}$ and limited labeled images
$\\{(x_{j},y_{j})\\}_{j=1}^{n}$. Here $y_{i}$ could indicate the polygon-like
label generated from BBS or the manually labeled ground truth. We firstly use
the labeled images to train an initial detector $\theta$. Then we apply the
detector $\theta$ to generate multi-scale prediction results
$\\{\widetilde{y}_{i}\\}_{i=1}^{m}$ as the foreground maps of unlabeled images
$\\{\widetilde{x}_{i}\\}_{i=1}^{m}$. When $\\{\widetilde{y}_{i}\\}_{i=1}^{m}$
is obtained, we can further calculate the background maps
$\\{\widetilde{z}_{i}\\}_{i=1}^{m}$ by filtering false negatives, as shown in
Fig. 5. Specifically, false negatives is filtered by thresholding the distance
score $\widetilde{t}_{i}^{k}$ of each negative sample pixel, which is
calculated by a distance transform function $D(\cdot)$ [3] and edge detection
$E(\cdot)$ ( _i.e.,_ Canny [8] ). By leveraging the information from
$\\{\widetilde{y}_{i}\\}_{i=1}^{m}$ and $\\{\widetilde{z}_{i}\\}_{i=1}^{m}$,
we could generate the high-quality pseudo label for unlabeled data. Please
refer to Sec. 3.2.2 for more details. After that, we conduct the dynamic mixed
retraining of the model by using both
$\\{(\widetilde{x}_{i},\widetilde{y}_{i},\widetilde{z}_{i})\\}_{i=1}^{m}$ and
$\\{(x_{j},y_{j})\\}_{j=1}^{n}$, and adaptively adjust the number of the
unlabeled data in a mini-batch. Sec. 3.2.3 gives the detailed training
process.

Algorithm 1 summarizes the scheme of DST. $E(\cdot)$ represents gradient edge
detector [8], which is utilized to filter text region with regular change of
gradient. $D(\cdot)$ represents a distance transform function [3]. $M(\cdot)$
and $\phi$ refer to the multi-scale inference and the scale set, respectively.
The shortest distance $\widetilde{t}_{i}^{k}$ between $k$-th pixel and
gradient edge ( _i.e.,_ the white line in Fig. 5 (b) ) can be calculated by
the distance transform function. The pixel with bigger $\widetilde{t}_{i}^{k}$
is usually farther from gradient edge, leading to the lower probability to be
the foreground.

Figure 5: Background Filtering Process. (a) The pseudo labels contains False
Negatives. (b) Generated edge map by edge detector ( _i.e.,_ Canny [8] ). (c)
Generated distance map by using Distance Transform. (d) The final refined
negative samples ( _i.e.,_ white regions ).

Input: Labeled images ${(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})}$

unlabeled images ${\widetilde{x}_{1},\widetilde{x}_{2},...,\widetilde{x}_{m}}$

multi-scale $s_{1},s_{2},...,s_{t}$

Output: trained parameters $\theta$

training the model $\theta$ on
${(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})}$ (just as in normal
supervised learning)

while  _epoch_ do

multi-scale inference $\widetilde{y}_{i}$ $\leftarrow$
$M(f(\widetilde{x}_{i}|\phi,\theta))$

compute edge map $e_{i}\leftarrow E(\widetilde{x}_{i})$

compute distance map $\widetilde{t}_{i}^{k}\leftarrow D(1-e_{i})$

filtering negative samples $\widetilde{z}_{i}$

if  _$\widetilde{t}_{i}^{k} >\sigma$_ then

$\widetilde{z}_{i}\leftarrow 1$;

else

$\widetilde{z}_{i}\leftarrow 0$;

end if

dynamic mixed retraining model $\theta$ on the union of
${(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})}$ and
${(\widetilde{x}_{1},\widetilde{y}_{1},\widetilde{z}_{1}),(\widetilde{x}_{2},\widetilde{y}_{2},\widetilde{z}_{2}),...,(\widetilde{x}_{m},\widetilde{y}_{m},\widetilde{z}_{m}})$

end while

Algorithm 1 Dynamic Self-Training Method.

####  3.2.2 High-quality Pseudo Label Generation

Multi-Scale Inference. As shown in Algorithm 1, we use multi-scale inference
to exploit hard positive samples as many as possible. When generating pseudo
labels $\\{\widetilde{y}_{i}\\}_{i=1}^{m}$, the unlabeled image is resized to
a pre-defined set of scales $\phi$ = $\\{\varepsilon-3\psi$,
$\varepsilon-2\psi$, $\varepsilon-\psi$, $\varepsilon$, $\varepsilon+\psi$,
$\varepsilon+2\psi$, $\varepsilon+3\psi\\}$ to generate multi-scale
predictions, where $\varepsilon$ denotes the shorter side of an image and
$\psi$ is set as 32 in practice. And then, the Locality-Aware NMS [40] is used
to ensemble the multi-scale inference results of each unlabeled image. Such
aggregating scheme often obtains a prediction that is always superior to any
of the model’s predictions under a single scale, especially for most text
detectors without FPN [18].

Filtering Negative Samples. To obtain the accurate background maps
$\\{\widetilde{z}_{i}\\}_{i=1}^{m}$, we need to identify and discard the false
negative regions. In consideration of the particularity of text, we use the
edge detection and distance transform to filter pixels and select the ones far
away from the gradients of image as the final background pixels. As shown in
Fig. 5, for an input image $\widetilde{x}_{i}$, we firstly obtain the
corresponding edge map by edge detector. And then, the distance transform (
_i.e.,_ $D(.)$ in Algorithm 1 ) is used to calculated the distance
$\widetilde{t}_{i}^{k}$ between $k$-th pixel and $b$-th pixel, where $b$-th
pixel is the nearest one of $k$-th pixel on gradient edge ( _i.e.,_ the white
parts in Fig. 5 (b) ). Similar to Eqn. 1, the $\widetilde{t}_{i}^{k}$ is
normalized to $[0,1]$, and Fig. 5 (c) presents the visualization of the
distance map. Finally, the background maps $\\{\widetilde{z}_{i}\\}$ are
calculated by thresholding the score map $\\{\widetilde{t}_{i}^{k}\\}$ with a
lower threshold $\sigma$, which is set as $0.1$ in all of the experiments.

####  3.2.3 Dynamic Mixed Training

The generated pseudo labels usually contain false positive (FP) and false
negative (FN), which causes unsteady convergence of network. To tackle this
problem and making full use of unlabeled data, we propose the dynamic mixed
training, which optimizes models with more unlabeled data ( _e.g.,_ $28$
unlabeled images and $6$ labeled images ) in the early stage of training, and
utilize more unlabeled data in the later stage. In the training stage, a batch
size data with the volume $\tau$ is composed of $\alpha$ labeled images and
$\beta$ unlabeled images, where $\beta+\alpha=\tau$. Here $\alpha$ can be
calculated with the formula, $\alpha=\frac{\rm current\ epoch}{\rm max\
epoch}*\tau$. The loss function in each iteration can be written as:

$\displaystyle\mathcal{L}_{dmt}=$
$\displaystyle\frac{1}{\alpha}\sum_{j=1}^{\alpha}\mathcal{L}_{1}(~{}y_{j},~{}\mathcal{G}(~{}x_{j},~{}\theta~{})~{})$
(3) $\displaystyle+$
$\displaystyle\frac{1}{\beta}\sum_{i=1}^{\beta}\mathcal{L}_{2}(~{}(~{}\widetilde{y}_{i},~{}\widetilde{z}_{i}~{}),\mathcal{G}(~{}\widetilde{x}_{i},~{}\theta~{})~{}),$

where $\mathcal{G}$ denotes the text detector. $\mathcal{L}_{1}(\cdot)$ and
$\mathcal{L}_{2}(\cdot)$ denotes corresponding loss of detector on labeled
data and pseudo labeled data, respectively.

##  4 Experiments

###  4.1 Datasets and Experimental Settings

Pure Synthetic Datasets. _SynthText_ [9] consist of 800k synthetic images
generated by adding variants of multi-oriented text with random fonts, size,
and color. _Curved SynthText_
111https://github.com/PkuDavidGuan/CurvedSynthText [20] generates 80m curved
texts with character level annotation by revising the text rendering module of
the SynthText engine.

Real Datasets. _ICDAR2015_ [13] includes 1,000 training and 500 testing images
with quadrilateral annotation. _Total-Text_[5] is a English curved text
dataset contains 1,555 images, which includes 3 different text orientations:
horizontal, multioriented, and curved. _MSRA-TD500_ [37] consists of 500
training and 200 testing images for detecting multi-lingual long texts of
arbitrary orientation. _ICDAR2017-MLT_ [24] consists of 18000 images with
texts in 9 languages for multi-lingual text detection.

Synthetic Data | Size | Evaluation on Total-Text/%
---|---|---
Precision | Recall | F-score
SynthText | 50k | 78.3 | 73.5 | 75.9
SynthText | 500k | 80.3 | 73.5 | 76.8
SynthText | 1m | 79.7 | 74.0 | 76.7
Curved SynthText | 50k | 80.8 | 75.0 | 77.8
Curved SynthText | 500k | 81.7 | 75.6 | 78.5
Curved SynthText | 1m | 81.3 | 75.1 | 78.1
Table 1: Synthetic Data for BBS: suitable annotation method bring a large gain. The Skeleton Attention and PSENet [30] are used in the experiment. Method | Synthetic Data | Evaluation on Total-Text/%
---|---|---
Precision | Recall | F-score
Baseline | SynthText(500k) | 77.2 | 73.0 | 75.0
Baseline+SA | SynthText(500k) | 80.3 | 73.5 | 76.8
Baseline | Curved SynthText(500k) | 80.5 | 73.2 | 76.7
Baseline+SA | Curved SynthText(500k) | 81.7 | 75.6 | 78.5
Table 2: Skeleton Attention: SA bring a large gain regardless of dataset. The ‘SA’ denotes the ‘Skeleton Attention’. PSENet [30] is adopted as the detector. Method | Evaluation on ICDAR2015/%
---|---
Precision | Recall | F-score
Baseline | 59.2 | 59.3 | 59.2
Baseline,ST | 70.1 | 67.3 | 68.6
Baseline,ST,Iter | 70.8 | 68.1 | 69.2
Baseline,ST,Iter,DM | 71.2 | 68.5 | 69.8
Baseline,ST,Iter,DM,DS | 72.5 | 70.1 | 71.3
Baseline,ST,Iter,DM,DS,Fil | 73.7 | 69.9 | 71.7
Table 3: Ablation study of DST: we use PSENet [30] as the base detector, and randomly split the ICDAR2015 training set into 100 labeled and 900 unlabeled images. ‘Baseline’ and ‘ST’ denote ‘training with only labeled data’ and ‘self-training on unlabeled data’, respectively. ‘DM’, ‘DS’, ‘Fil’, ‘Iter’ denote ‘dynamic mixed training’, ‘multi-scale inference’, ‘filtering’ and ‘iterative self-training’, respectively. Method | Annotation | Pre | ICDAR2015/% | MSRA-TD500/% | ICDAR2017-MLT/% | Total-Text/%
---|---|---|---|---|---|---
P | R | F | P | R | F | P | R | F | P | R | F
Strong Supervision |  |  |  |  |  |  |  |  |  |  |  |  |  |
CTPN[28] | GT | - | 74.2 | 51.6 | 60.9 | - | - | - | - | - | - | - | - | -
SegLink[27] | GT | ✓ | 73.1 | 76.8 | 75.0 | 86.0 | 70.0 | 77.0 | - | - | - | 30.3 | 23.8 | 26.7
EAST[40] | GT | - | 80.5 | 72.8 | 76.4 | 81.7 | 61.6 | 70.2 | - | - | - | - | - | -
PixelLink[7] | GT | - | 82.9 | 81.7 | 82.3 | 83.0 | 73.2 | 77.8 | - | - | - | - | - | -
TextSnake[21] | GT | ✓ | 84.9 | 80.4 | 82.6 | 83.2 | 73.9 | 78.3 | - | - | - | 82.7 | 74.5 | 78.4
PSENet[30] | GT | - | 81.5 | 79.7 | 80.6 | - | - | - | 73.7 | 68.2 | 70.8 | 81.8 | 75.1 | 78.3
PSENet[30] | GT | ✓ | 86.9 | 84.5 | 85.7 | - | - | - | - | - | - | 84.0 | 78.0 | 80.9
EAST † | GT | - | 76.9 | 77.1 | 77.0 | 71.8 | 69.1 | 70.4 | 68.1 | 63.2 | 65.6 | - | - | -
EAST† | GT | ✓ | 82.0 | 82.4 | 82.2 | 77.9 | 76.5 | 77.2 | 70.3 | 62.8 | 66.4 | - | - | -
PSENet† | GT | - | 81.6 | 79.5 | 80.5 | 80.6 | 77.7 | 79.1 | 73.1 | 67.3 | 70.1 | 80.4 | 76.5 | 78.4
PSENet† | GT | ✓ | 86.4 | 83.5 | 85.0 | 84.1 | 85.0 | 84.5 | 72.5 | 69.1 | 70.8 | 83.4 | 78.1 | 80.7
Box Supervision |  |  |  |  |  |  |  |  |  |  |  |  |  |
EAST † | b-GT | - | 65.8 | 63.8 | 64.8 | 40.5 | 31.1 | 35.2 | 66.3 | 59.6 | 62.8 | - | - | -
EAST†+BBS | PL | - | 77.8 | 78.2 | 78.0 | 71.3 | 70.2 | 70.7 | 67.3 | 64.1 | 65.7 | - | - | -
EAST† | b-GT | ✓ | 70.8 | 72.0 | 71.4 | 48.3 | 42.4 | 45.2 | 67.2 | 60.1 | 63.5 | - | - | -
EAST†+BBS | PL | ✓ | 81.3 | 82.2 | 81.8 | 77.4 | 75.5 | 76.4 | 67.6 | 64.9 | 66.3 | - | - | -
PSENet† | b-GT | - | 70.2 | 69.1 | 69.6 | 47.2 | 36.9 | 41.4 | 67.2 | 61.4 | 64.2 | 46.5 | 43.6 | 45.0
PSENet†+BBS | PL | - | 82.9 | 77.6 | 80.2 | 80.3 | 77.5 | 78.9 | 72.4 | 69.3 | 70.8 | 81.7 | 75.6 | 78.5
PSENet† | b-GT | ✓ | 72.7 | 74.3 | 73.5 | 47.5 | 39.5 | 43.1 | 66.4 | 63.1 | 64.7 | 51.9 | 47.5 | 49.6
PSENet†+BBS | PL | ✓ | 86.8 | 83.3 | 85.0 | 84.4 | 84.7 | 84.5 | 73.8 | 67.7 | 70.6 | 82.5 | 77.6 | 80.1
Table 4: The results of BBS on ICDAR2015, MSRA-TD500, ICDAR2017-MLT, Total-
Text. $\dagger$ refers to our testing performance. The ‘GT’, ‘b-GT’, ‘PL’
refer to the ’Ground Truth’, ‘Bounding-box annotation of ground truth’,
‘Pseudo Label generated by the SASN training on SynthText [10] or curved
SynthText [20]’. ‘P’, ‘R’, ‘F’ and “Pre” refer to ‘Precision’, ‘Recall’,
‘F-score’ and ‘pretraining on external data’, respectively. In green and in
bold are highlighted for comparison.

Implementation Details. _Bounding Box Supervision._ All of the experiments use
the same strategy: (1) training SASN with synthetic data based on character
annotation. (2) generating the pixel level pseudo label on real data based on
bounding box annotation with SASN. (3) training the detectors (_i.e.,_ EAST
and PSENet) with the pseudo label. The stochastic gradient descent(SGD)
optimizer is adopted with a momentum of 0.9 and a weight decay of 0.0005. The
batch size is set to 8 per GPU. The learning rate is initialized to 0.02 and
decayed with the power of 0.9 for 16 epochs. During training and inference,
the crop images are resized to a resolution of 128$\times$128\. _Dynamic Self-
Training._ All of the experiments use the same training strategy: (1) we
firstly choose a part of training data as labeled data to train the detectors.
(2) generating pseudo label in the rest of the data with the proposed
components. (3) fine-tune the detectors on whole data. To evaluate the
effectiveness of the DST, we randomly split the training set into labeled data
and unlabeled data, and using one ratio control the labeled data size in the
total training set. PSENet [30] and EAST [40] are adopted as the base
detectors because of the popularity. While involving PSENet [30] and EAST [40]
experiments, all settings are following the original paper.

###  4.2 Ablation Study

In this part, we conduct three groups ablation experiments to analyze BBS and
DST. More ablation (_e.g.,_ Box supervision _v.s_. Strong supervision, Labeled
data size for DST) experiments are in supplementary materials.

Synthetic data for BBS. To understand the impact of synthetic data with
different text shape distribution, we compare the performance of curved
Synthtext [20] and Synthtext [10]. As shown in Tab. 1, using curved Synthext
obtains better performance, $2\%$ absolute improvement over that using
Synthext, because of the serious domain gap of text shape between Total-Text
(curved text) and Synthtext (straight text). The performance tends to
stabilize while synthetic data size exceeds $500k$, so we randomly select
$500k$ synthetic data to train SASN in all experiments.

Skeleton Attention for BBS. Skeleton Attention is the key to the SASN. Tab. 2
gives the ablation study about the attention on Total-Text [4]. No matter
which synthetic data, Skeleton Attention can obtain up to $2\%$ improvement
compares to the baseline without the attention. In addition, Fig. 6 gives the
visualization of the attention map, which enables the network more interested
in the text skeleton.

Components for DST. Dynamic mixed training, multi-scale inference and
filtering are the keys for DST. We make an assumption that these components
independent of each other, which improve the performance of model from
different levels by minimizing the effects of false negatives and false
positives. Tab. 3 also indirectly proves the conclusion. By combining with
dynamic mixed training, multi-scale inference and filtering, the F-score
achieved a $69.8\%$, $71.3\%$ and $71.7\%$ , respectively, making $0.6\%$,
$0.5\%$ and $0.4\%$ absolute improvements over the baseline.

###  4.3 Experiments on Scene Text Detection

In this section, we firstly present the experiments of BBS and DST,
respectively. And then, the combined experiments are given in supplementary
materials.

####  4.3.1 Bounding Box Supervision

Quadrilateral-type datasets. Tab. 4 lists the experimental results of various
methods on ICDAR2015, ICDAR2017 and MSRA-TD500 datasets. For _ICDAR2015_ ,
PSENet [30] using pseudo label achieves almost the same performance ($80.2\%$
and $85.0\%$ using the pre-trained model on SynthText) with that using ground
truth, which prove high-quality of pseudo label generated by BBS. On the
contrary, directly training the detector with bounding box annotation has an
unsatisfactory F-score ($69.6\%$ and $73.5\%$), which exists more than $10\%$
gap compare with that using polygon annotation. EAST [40] has a similar case
with PSENet. Moreover, the F-score ($78.0\%$) of using the pseudo label has a
$1.0\%$ improvement over that ($77.0\%$) of using ground truth. For _MSRA-
TD500_ , annotations are provided at the line level, including the spaces
between words in the box. Therefore, bounding box annotation on MSRA-TD500
usually contains large plenty of background, which causes poor performance
($35.2\%$ for EAST and $41.4\%$ for PSENet). In this case, the pseudo label
from BBS still brings almost the same performance ($70.7\%$ for EAST and
$78.9\%$ for PSENet) with that ($70.4\%$ for EAST and $79.1\%$ for PSENet) of
polygon annotation. In addition, after using the pre-trained model on
SynthText, PSENet and EAST all obtain almost the same improvement no matter
using which labels (pseudo label or ground truth). For _ICDAR2017_ , the
performance is similar to ICDAR15, pseudo label has a high-quality performance
compare with that using ground truth. The difference is that the bounding box
also brings relatively good performance ($62.8\%$ for EAST and $64.2\%$ for
PSENet). The main reason is that texts in ICDAR17 have a small tilting angle
and size.

Polygon-type dataset. Tab. 4 lists the experimental results on Total-Text [4].
The annotation on Total-Text is complex and polygonal in shape. The great
performance ($78.5\%$) of using pseudo label further proves the significance
of our work, and Fig. 7 gives some visualization of ground truth and pseudo
label. Similar to MSRA-TD500, bounding box annotation on Total-Text also
contains plenty of backgrounds, which causes poor performance ($45.0\%$ and
$49.6\%$). And using the pseudo label generated by BBS still can achieve a
great performance ($78.5\%$ and $80.1\%$) with $33.5\%$ and $30.5\%$
improvements.

Method | Ratio | Pre | ICDAR2015/%
---|---|---|---
P | R | F
Strong Supervision |  |  |  |  |
EAST † | 100% | - | 76.9 | 77.1 | 77.0
EAST† | 100% | ✓ | 82.0 | 82.4 | 82.2
PSENet† | 100% | - | 81.6 | 79.5 | 80.5
PSENet† | 100% | ✓ | 86.4 | 83.5 | 85.0
Self-Training |  |  |  |  |
EAST † | 50% | - | 77.1 | 75.7 | 76.4
EAST†+DST | 50% | - | 77.6 | 77.5 |  77.5 (+1.1)
EAST † | 50% | ✓ | 79.1 | 79.2 | 79.1
EAST†+DST | 50% | ✓ | 82.6 | 79.7 |  81.1 (+2.0)
PSENet† | 50% | - | 76.6 | 73.6 | 75.1
PSENet†+DST | 50% | - | 79.8 | 76.8 |  78.3 (+3.2)
PSENet† | 50% | ✓ | 78.9 | 76.6 | 77.7
PSENet†+DST | 50% | ✓ | 84.0 | 82.9 |  83.5 (+5.8)
Table 5: The results of DST on ICDAR2015. ‘Ratio’, ‘P’, ‘R’, and ‘F’ refer to ‘the proportion of using labeled data’, ‘Precision’, ‘Recall’ and ‘F-score’, respectively. In blue are the gaps of at least (+1.1) point. Method | Ratio | Pre | ICDAR2017-MLT/%
---|---|---|---
P | R | F
Strong Supervision |  |  |  |  |
EAST † | 100% | - | 68.1 | 63.2 | 65.6
PSENet† | 100% | - | 73.1 | 67.3 | 70.1
Self-Training |  |  |  |  |
EAST † | 50% | - | 70.4 | 60.2 | 64.5
EAST †+DST | 50% | - | 69.2 | 63.5 |  66.2 (+1.7)
PSENet† | 50% | - | 69.1 | 68.4 | 68.7
PSENet†+DST | 50% | - | 70.6 | 69.8 |  70.2 (+1.5)
Table 6: The results of DST on ICDAR2017-MLT. ‘Ratio’, ‘P’, ‘R’, and ‘F’ refer
to ‘the proportion of using labeled data’, ‘Precision’, ‘Recall’ and
‘F-score’, respectively. In blue are the gaps of at least (+1.5) point.

####  4.3.2 Dynamic Self Training

We present the experimental results on ICDARs in this subsection, and more
experiments(_i.e.,_ Total-Text) are in the supplementary material. For
_ICDAR2015_ , Tab. 5 lists the experimental results of the self-training, and
the ‘Ratio’ denotes only using part of training set as the labeled data. By
using DST, EAST using half labeled data and half unlabeled data obtains the
great performance ($77.5\%~{}v.s.~{}77.0\%$ and $81.1\%~{}v.s.~{}82.2\%$),
which is almost the same as that of strong supervision. After using the pre-
trained model on SynthText, the performance can be further improved with 2.0%
gap. Similar to EAST, PSENet also has achieved a competitive performance
($78.3\%~{}v.s.~{}80.5\%$ and $83.5\%~{}v.s.~{}85.0\%$) with only half labeled
training set. For _ICDAR2017_ , Tab. 6 lists the experimental results of the
self-training. Similar to ICDAR15, our method all achieve competitive
performance ($66.2\%~{}v.s.~{}65.6.0\%$ and $70.2\%~{}v.s.~{}70.1\%$) no
matter which detectors are used.

Figure 6: The visualization of BBS on Skeleton Attention. The Skeleton
Attention enables the network to focus on the skeleton of body text instance
for segmenting precisely. Figure 7: The visualization of bounding-box
annotation and the pseudo labels generated by BBS on Total-Text [4].

##  5 Conclusion

In this paper, we present a simple but surprisingly effective and practical
method termed SelfText Beyond Polygon (SBP), which includes two components:
Bounding Box Supervision (BBS) and Dynamic Self-Training (DST). The BBS with
only bounding box annotation achieve almost the same performance as using
expensive polygon annotation. The DST training with limited labeled data and
unlabeled data to further reduce the data cost. The experiments showed that
our method achieves almost the same performance as that of strong supervision
while saving $70\%$ \+ data cost, which can provide a new perspective for
weakly supervised text detection and save much money for the industry.

##  6 Appendix

###  6.1 More Analysis for Dynamic Mixed Training

As shown in Fig. 8, fully supervision (green line) usually has a stable
convergence effect ($\pm 0.2\%$), normal self-training (red line) has a
performance fluctuation ($\pm 2.0\%$), because of the adverse effects of false
negatives (FNs) and false positives (FPs). Fortunately, Dynamic Mixed Training
could achieve competitive convergence performance ($\pm 0.8\%$) by dynamically
adjusting the number of unlabeled data in a mini-batch, as shown in the blue
line. The basic idea is simple but effective, the network training with few
unlabeled data for minimizing the adverse effects of FNs and FPs in the last
training stage, and using massive unlabeled data for mining the valuable
information in the early training stage. We argue the self-training requiring
such an adjusting strategy to help stable convergence.

Figure 8: Dynamic Mixed Training. Dynamic Self-Training achieves much stabler convergence compare with common self-training. The EAST [40] is used as the base detector, and the training set of ICDAR2015 is divided into half labeled data and half unlabeled data. ‘Self-Training’ denotes traditional self training without dynamic mixed training. Figure 9: The pseudo labels without bounding box annotation contain large amounts of false negatives and false positives, whose performance is not up to using bounding box (_i.e.,_ 59.4% _v.s._ 78.0% for EAST [40] on ICDAR2015). Method | Annotation | Evaluation on Total-Text/%
---|---|---
Precision | Recall | F-score
PSENet | GT | 80.4 | 76.5 | 78.4
PSENet | b-GT | 46.5 | 43.6 | 45.0
PSENet | PL(ST) | 80.3 | 73.5 |  76.8 (+31.8)
PSENet | PL(Curved ST) | 81.7 | 75.6 |  78.5 (+33.5)
Evaluation on MSRA-TD500/%
PSENet | GT | 80.6 | 77.7 | 79.1
PSENet | b-GT | 47.2 | 36.9 | 41.4
PSENet | PL(ST) | 80.3 | 77.5 |  78.9 (+37.5)
EAST | GT | 71.8 | 69.1 | 70.4
EAST∗ | PL(ST) | 66.6 | 53.5 | 59.4
EAST | b-GT | 40.5 | 31.1 | 35.2
EAST | PL(ST) | 71.3 | 70.2 |  70.7 (+35.5)
Table 7: Box supervision _v.s_. Strong supervision. ‘GT’, ‘b-GT’, ‘PL(ST)’, ‘PL(Curved ST)’ denote the ‘Ground Truth’, ‘Bounding box annotation’, ‘Pseudo label generated by BBS with SyntheText’, ‘Pseudo label generated by BBS with Curved SyntheText’, respectively. ∗ denotes ‘Pseudo label generated by the detector without bounding box annotation’. In blue are the gaps of at least (+31.8) point. Method | Ratio | Evaluation on ICDAR2015/%
---|---|---
Precision | Recall | F-score
Baseline | 10% | 59.2 | 59.3 | 59.2
Baseline+DST | 10% | 73.7 | 69.3 |  71.7 (+10.1)
Baseline | 30% | 71.8 | 69.3 | 70.5
Baseline+DST | 30% | 76.6 | 76.0 |  76.3 (+5.5)
Baseline | 50% | 76.6 | 73.6 | 75.1
Baseline+DST | 50% | 79.8 | 76.8 |  78.3 (+3.2)
Baseline | 70% | 77.4 | 76.4 | 76.9
Baseline+DST | 70% | 79.1 | 77.4 |  78.2 (+1.3)
Baseline | 90% | 80.3 | 78.9 | 79.6
Baseline+DST | 90% | 80.6 | 79.8 |  80.2 (+0.6)
Baseline | 100% | 81.6 | 79.5 | 80.5
Table 8: Labeled data size for DST. We use PSENet [30] as the base detector.
‘Ratio’ denotes $\frac{labeled\;data\;size}{total\;data\;size}$, where the
total data size is 1,000 training images for ICDAR2015 [13]. In blue are the
gaps of at least (+0.6) point. Figure 10: The visualization of bounding-box
annotation, polygon-based ground truth and polygon-based pseudo labels
generated by BBS on two datasets: Total-Text [4] and MSRA-TD500[37].

###  6.2 Ablation Study

Box supervision _v.s_. Strong supervision. Compare with strong supervision,
Bounding Box Supervision (BBS) not only can save significant costs, but also
achieves almost the same performance as that of strong supervision. In Tab. 7,
we adopt two detectors(_i.e.,_ PSENet [30] and EAST [40]) and two datasets
(_i.e.,_ Total-Text [4] and (MSRA-TD500 [37]) to prove the conclusion. On
Total-Text, PSENet using the pseudo label generated by BBS achieve a
satisfactory F-score ($78.5\%$), which is almost the same as using the ground
truth ($78.4\%$), with $33.5\%$ improvement over that of using the bounding
box ($45.0.5\%$). On MSRA-TD500, compared with that using ground truth, EAST
and PSENet all achieve the competitive F-scores ($70.7\%$ _v.s._ $70.4\%$ and
$78.9\%$ _v.s._ $79.1\%$) with the pseudo label from BBS, which have great
improvements ($35.5\%$ and $37.5\%$, respectively). The competitive
performance of BBS proves the practicability efficiency of the pseudo label,
Fig. 10 gives some visualization of the pseudo label.

Besides, bounding box annotation as the priori location is necessary for BBS.
Fig. 9 gives a visualization comparison between based on bounding box
annotation and without the box annotation, the pseudo label without box priori
location contains large amounts of false negatives and false positives (the
red circle), which causes unsatisfactory performances (_i.e.,_ 59.4% _v.s._
78.0% for EAST [40] on ICDAR2015 [13]), as shown in Tab. 7.

Labeled data size for DST. The target of Dynamic Self-Training is using
limited labeled data and unlabeled data to train detectors. However, how much
labeled data is enough for training a competitive model. Tab. 8 give a
correlation between labeled data proportion and performance on ICDAR2015 [13].
F-score is increasing with labeled data proportion increase and achieves a
competitive performance ($78.3\%$) with $50\%$ labeled data. Besides, DST
brings a significant improvement up to $10.1\%$ while using only $10.0\%$
labeled training set.

## References

  * [1] Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee.  Character region awareness for text detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9365–9374, 2019.
  * [2] Hakan Bilen and Andrea Vedaldi.  Weakly supervised deep detection networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2846–2854, 2016.
  * [3] Gary Bradski and Adrian Kaehler.  Learning opencv: Computer vision with the opencv library.  ” O’Reilly Media, Inc.”, 2008.
  * [4] Chee Kheng Ch’ng and Chee Seng Chan.  Total-text: A comprehensive dataset for scene text detection and recognition.  In Proc. Int. Conf. Document Analysis Recogn., pages 935–942, 2017\.
  * [5] Chee Kheng Ch’ng and Chee Seng Chan.  Total-text: A comprehensive dataset for scene text detection and recognition.  In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 935–942. IEEE, 2017.
  * [6] Jifeng Dai, Kaiming He, and Jian Sun.  Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation.  In Proceedings of the IEEE international conference on computer vision, pages 1635–1643, 2015.
  * [7] Dan Deng, Haifeng Liu, Xuelong Li, and Deng Cai.  Pixellink: Detecting scene text via instance segmentation.  In Proc. AAAI Conf. Artificial Intell., pages 6773–6780, 2018\.
  * [8] Lijun Ding and Ardeshir Goshtasby.  On the canny edge detector.  Pattern Recognition, 34(3):721–725, 2001.
  * [9] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2315–2324, 2016.
  * [10] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2315–2324, 2016\.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] Vadim Kantorov, Maxime Oquab, Minsu Cho, and Ivan Laptev.  Contextlocnet: Context-aware deep network models for weakly supervised localization.  In European Conference on Computer Vision, pages 350–365. Springer, 2016.
  * [13] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al.  Icdar 2015 competition on robust reading.  In Proc. Int. Conf. Document Analysis Recogn., pages 1156–1160, 2015.
  * [14] Daesik Kim, Gyujeong Lee, Jisoo Jeong, and Nojun Kwak.  Tell me what they’re holding: Weakly-supervised object detection with transferable knowledge from human-object interaction.  In AAAI, pages 11246–11253, 2020.
  * [15] Suha Kwak, Minsu Cho, Ivan Laptev, Jean Ponce, and Cordelia Schmid.  Unsupervised object discovery and tracking in video collections.  In Proceedings of the IEEE international conference on computer vision, pages 3173–3181, 2015.
  * [16] Minghui Liao, Baoguang Shi, Xiang Bai, Xinggang Wang, and Wenyu Liu.  Textboxes: A fast text detector with a single deep neural network.  In Proc. AAAI Conf. Artificial Intell., pages 4161–4167, 2017\.
  * [17] Minghui Liao, Zhen Zhu, Baoguang Shi, Gui-song Xia, and Xiang Bai.  Rotation-sensitive regression for oriented scene text detection.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 5909–5918, 2018\.
  * [18] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [19] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 3431–3440, 2015\.
  * [20] Shangbang Long, Yushuo Guan, Bingxuan Wang, Kaigui Bian, and Cong Yao.  Rethinking irregular scene text recognition.  arXiv, pages arXiv–1908, 2019.
  * [21] Shangbang Long, Jiaqiang Ruan, Wenjie Zhang, Xin He, Wenhao Wu, and Cong Yao.  Textsnake: A flexible representation for detecting text of arbitrary shapes.  Proc. Eur. Conf. Comp. Vis., pages 20–36, 2018.
  * [22] Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai.  Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes.  In Proc. Eur. Conf. Comp. Vis., pages 67–83, 2018.
  * [23] Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue.  Arbitrary-oriented scene text detection via rotation proposals.  IEEE Trans. Multimedia, 20(11):3111–3122, 2018.
  * [24] Nibal Nayef, Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng, Dimosthenis Karatzas, Zhenbo Luo, Umapada Pal, Christophe Rigaud, Joseph Chazalon, et al.  Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification-rrc-mlt.  In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 1454–1459. IEEE, 2017.
  * [25] Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He.  Data distillation: Towards omni-supervised learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4119–4128, 2018.
  * [26] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Proc. Advances in Neural Inf. Process. Syst., pages 91–99, 2015\.
  * [27] Baoguang Shi, Xiang Bai, and Serge Belongie.  Detecting oriented text in natural images by linking segments.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2550–2558, 2017\.
  * [28] Zhi Tian, Weilin Huang, Tong He, Pan He, and Yu Qiao.  Detecting text in natural image with connectionist text proposal network.  In Proc. Eur. Conf. Comp. Vis., pages 56–72, 2016.
  * [29] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan.  Learning to detect salient objects with image-level supervision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 136–145, 2017.
  * [30] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao.  Shape robust text detection with progressive scale expansion network.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 9336–9345, 2019\.
  * [31] Wenhai Wang, Enze Xie, Xiaoge Song, Yuhang Zang, Wenjia Wang, Tong Lu, Gang Yu, and Chunhua Shen.  Efficient and accurate arbitrary-shaped text detection with pixel aggregation network.  In Proc. IEEE Int. Conf. Comp. Vis., pages 8440–8449, 2019.
  * [32] Xiaolong Wang and Abhinav Gupta.  Unsupervised learning of visual representations using videos.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2015.
  * [33] Enze Xie, Yuhang Zang, Shuai Shao, Gang Yu, Cong Yao, and Guangyao Li.  Scene text detection with supervised pyramid context network.  In Proc. AAAI Conf. Artificial Intell., volume 33, pages 9038–9045, 2019.
  * [34] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687–10698, 2020.
  * [35] Yongchao Xu, Yukang Wang, Wei Zhou, Yongpan Wang, Zhibo Yang, and Xiang Bai.  Textfield: Learning a deep direction field for irregular scene text detection.  IEEE Transactions on Image Processing, 28(11):5566–5579, 2019.
  * [36] Zhenheng Yang, Dhruv Mahajan, Deepti Ghadiyaram, Ram Nevatia, and Vignesh Ramanathan.  Activity driven weakly supervised object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2917–2926, 2019.
  * [37] Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu.  Detecting texts of arbitrary orientations in natural images.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 1083–1090, 2012\.
  * [38] Liu Yuliang, Jin Lianwen, Zhang Shuaitao, and Zhang Sheng.  Detecting curve text in the wild: New dataset and new solution.  arXiv preprint arXiv:1712.02170, 2017.
  * [39] Zheng Zhang, Chengquan Zhang, Wei Shen, Cong Yao, Wenyu Liu, and Xiang Bai.  Multi-oriented text detection with fully convolutional networks.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 4159–4167, 2016\.
  * [40] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang.  East: an efficient and accurate scene text detector.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 5551–5560, 2017\.
  * [41] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le.  Rethinking pre-training and self-training.  arXiv preprint arXiv:2006.06882, 2020.