# Identification of a potential ultra-low Q value electron capture decay
branch in 75Se via a precise Penning trap measurement of the mass of 75As

M. Horana Gamage  corresponding author<EMAIL_ADDRESS>Department of
Physics, Central Michigan University, Mount Pleasant, Michigan, 48859, USA  R.
Bhandari  Department of Physics, Central Michigan University, Mount Pleasant,
Michigan, 48859, USA  G. Bollen  Facility for Rare Isotope Beams, East
Lansing, Michigan, 48824, USA  Department of Physics and Astronomy, Michigan
State University, East Lansing, Michigan 48824, USA  N. D. Gamage  Facility
for Rare Isotope Beams, East Lansing, Michigan, 48824, USA  A. Hamaker
National Superconducting Cyclotron Laboratory, East Lansing, Michigan, 48824,
USA  Department of Physics and Astronomy, Michigan State University, East
Lansing, Michigan 48824, USA  D. Puentes  National Superconducting Cyclotron
Laboratory, East Lansing, Michigan, 48824, USA  Department of Physics and
Astronomy, Michigan State University, East Lansing, Michigan 48824, USA  M.
Redshaw  Department of Physics, Central Michigan University, Mount Pleasant,
Michigan, 48859, USA  National Superconducting Cyclotron Laboratory, East
Lansing, Michigan, 48824, USA  R. Ringle  Facility for Rare Isotope Beams,
East Lansing, Michigan, 48824, USA  S. Schwarz  Facility for Rare Isotope
Beams, East Lansing, Michigan, 48824, USA  C. S. Sumithrarachchi  Facility for
Rare Isotope Beams, East Lansing, Michigan, 48824, USA  I. Yandow  National
Superconducting Cyclotron Laboratory, East Lansing, Michigan, 48824, USA
Department of Physics and Astronomy, Michigan State University, East Lansing,
Michigan 48824, USA

(August 27, 2024)

###### Abstract

Background

Low energy $\beta$ and electron capture (EC) decays are important systems in
neutrino mass determination experiments. An isotope with an ultra-low Q value
$\beta$-decay to an excited state in the daughter with $Q_{es}$ $<$ 1 keV
could provide a promising alternative candidate for future experiments. 75Se
EC and 75Ge $\beta$-decay represent such candidates, but a more precise
determination of the mass of the common daughter, 75As, is required to
evaluate whether their potential decay branches are energetically allowed and
ultra-low.

Purpose

Perform a precise atomic mass measurement of 75As and combine the result with
the precisely known atomic masses of 75Se and 75Ge, along with nuclear energy
level data for 75As to evaluate potential ultra-low Q value decay branches in
the EC decay of 75Se and the $\beta$-decay of 75Ge.

Method

The LEBIT Penning trap mass spectrometer at the Facility for Rare Isotope
Beams was used to perform a high-precision measurement of the atomic mass of
75As via cyclotron frequency ratio measurements of 75As+ to a 12C${}_{6}^{+}$
reference ion.

Results

The 75As mass excess was determined to be ME(75As) = –73 035.98(43) keV, from
which the ground-state to ground-state Q values for 75Se EC and 75Ge
$\beta$-decay were determined to be 866.50(44) keV and 1179.01(44) keV,
respectively. These results were compared to energies of excited states in
75As at 865.4(5) keV and 1172.0(6) keV to determine Q values of 1.1(7) keV and
7.0(7) keV for the potential ultra-low EC and $\beta$-decay branches of 75Se
and 75Ge, respectively.

Conclusion

The candidate ultra-low Q value EC and $\beta$-decay branches of 75Se and 75Ge
to excited states in 75As, identified by Gamage, et al. [N.D. Gamage,
Hyperfine Interact. 240, 43 (2019)] are seen to be energetically allowed. The
75Se EC decay to the 865.4 keV excited state in 75As is potentially ultra-low
with $Q_{es}$ $\approx$ 1 keV. However, a more precise determination of the
865.4(5) keV level in 75As is required to draw a more definite conclusion
about the energy of this potential decay channel.

##  I Introduction

The ongoing development of sensitive, high-precision experimental techniques
and sophisticated theoretical calculations has continued to make nuclear
$\beta$-decay an important tool in modern physics, for example, in tests of
the Standard Model (SM) and searches for physics beyond the SM Hardy and
Towner (2015); Severijns  _et al._ (2006); Burkey  _et al._ (2022), and in
investigations of the nature and absolute mass of the neutrino Avignone  _et
al._ (2008); Friedrich  _et al._ (2021); Aker (2019); Monreal and Formaggio
(2009); Gastaldo (2017); Faverzani  _et al._ (2016).

Extreme cases of $\beta$-decay e.g. highly-forbidden decays and very low
energy decays, pose significant experimental challenges in the detection of
these rare processes, and theoretical challenges in describing them. Ultra-low
(UL) Q value $\beta$-decays, in which the parent nucleus decays to an excited
state in the daughter with $Q_{es}$ $<$ 1 keV, provide a particularly
interesting case, since they have both experimental and theoretical
implications. An isotope with an ultra-low Q value $\beta$-decay could
potentially be used as a new candidate in direct neutrino mass determination
experiments Cattadori  _et al._ (2004, 2007); Suhonen (2014); Kopp and Merle
(2010). They can also provide a powerful tool to test the role of atomic
interference effects in nuclear $\beta$-decay Mustonen and Suhonen (2010a);
Suhonen and Mustonen (2010), which become more significant for very low energy
decays.

To date, only one ultra-low Q value $\beta$-decay has been observed, that of
115In to the 3/2+ first excited state in 115Sn Cattadori  _et al._ (2004). The
confirmation that this decay branch is energetically allowed was made by
precise Penning trap measurements of the 115In – 115Sn mass difference at
Florida State University Mount  _et al._ (2009) and with JYLFTRAP at the
University of Jyväskylä Wieslander  _et al._ (2009), combined with the
precisely known energy of the 3/2+ level in 115Sn Blachot (2005);
Zheltonozhsky  _et al._ (2018). Using the results of Ref. Mount  _et al._
(2009) and Zheltonozhsky  _et al._ (2018), the resulting $Q_{\textrm{UL}}$ =
0.147(10) keV, making this the lowest known Q value $\beta$-decay. Theoretical
calculations of the partial half-life for 115In decay to 115Sn(3/2+) showed a
significant discrepancy compared to experimental results Mustonen and Suhonen
(2010a); Suhonen and Mustonen (2010). To identify potential candidates for
future direct neutrino mass determination experiments, and to further
theoretical developments, additional ultra-low Q value decays are required to
be identified and to have their Q values and partial half-lives measured.

A large number of potential ultra-low Q value decays do in fact exist—see for
example Refs. Mustonen and Suhonen (2010b); Suhonen (2014); Kopp and Merle
(2010); Haaranen and Suhonen (2013); Mustonen and Suhonen (2011), and Ref.
Gamage  _et al._ (2019) for a recent comprehensive evaluation. The
identification of these ultra-low Q value decay candidates is made from a
comparison of the ground-state to ground-state (gs-gs) Q value, $Q_{gs}$,
obtained from atomic mass data for parent and daughter isotopes Wang  _et al._
(2021), and nuclear energy level data for the daughter nnd . In order to
determine if a decay to an excited state in the daughter is energetically
allowed and ultra-low, both $Q_{gs}$, and the energy of the final state in the
daughter, $E^{*}$, must be known to $\ll$ 1 keV. While nuclear energy levels
are often known to precisions of $\sim$10 – 100 eV, atomic masses are usually
not, except in cases where they have been measured precisely by dedicated
Penning trap experiments.

In the last few years, there has been growing interest in performing precise
Penning trap measurements to determine $Q_{es}$ values for potential ultra-low
Q value decay candidates that have been identified. Studies have been
performed on 131Cs Karthein  _et al._ (2019) at ISOLTRAP; 135Cs de Roubin  _et
al._ (2020), 72As Ge  _et al._ (2021a), and 159Dy Ge  _et al._ (2021b) at
JYFLTRAP; 112,113Ag and 115Cd with the Canadian Penning Trap Gamage  _et al._
(2022), and 89Sr and 139Ba at LEBIT Sandler  _et al._ (2019). These
measurements showed that 135Cs could have an ultra-low Q value $\beta$-decay
branch, and that in 159Dy EC an experimentally confirmed Mys (1969) decay to
the 11/2+ state in the daughter has a $Q_{es}$ = 1.2(2) keV. The other
candidates were ruled out.

In this paper we report on precise determinations of the $Q_{gs}$ values for
75Ge $\beta$-decay and 75Se EC decay via a precise measurement of the mass of
the common daughter, 75As. These candidates were first identified in Ref.
Gamage  _et al._ (2019), along with 89Sr and 139Ba Sandler  _et al._ (2019),
as cases where the mass of the stable daughter isotope was known less
precisely than that of the unstable parents, making them ideal cases for study
with the offline ion sources available at Penning trap facilities such as
LEBIT.

Fig. 1 shows the energy levels and decay schemes for the $A=75$ Ge-As-Se
triplet. The potential ultra-low Q value decays are indicated by dashed blue
arrows. For 75Ge, the potential ultra-low Q value $\beta$-decay is to the
1172.0(6) keV level in 75As, identified as a negative parity state with spin
between 1/2 and 7/2 nnd . For 75Se, there are two potential ultra-low Q value
EC decay branches: to the 865.4(5) keV level, identified as having $J^{\pi}$ =
3/2- or 5/2-, and to the 860.4(4) keV 1/2+ level. The $Q_{gs}$ values listed
in Fig. 1 are calculated using data from the most recent atomic mass
evaluation, AME2020 Wang  _et al._ (2021), and are limited by the 0.88
keV/$c^{2}$ uncertainty in the mass of 75As. The uncertainties in the masses
of 75Ge and 75Se are 0.052 keV/$c^{2}$ and 0.073 keV/$c^{2}$, respectively,
and contribute negligibly to the uncertainty in the Q value. The resulting
potential ultra-low Q value decay energies to the respective excited states in
75As are $Q^{\beta}_{es}$(75Ge) = 5.2(1.1) keV and
$Q^{\textrm{EC}}_{es}$(75Se) = –0.7(1.0) keV and 4.3(1.0) keV. Hence, a new
Penning trap measurement of the 75As mass is called for to check the accuracy
of the Q values and eliminate uncertainty due to current uncertainty in the
75As mass.

Figure 1: (color online) Decay scheme for 75Se EC and 75Ge $\beta$-decay. The
main decay branches are indicated by solid black arrows with corresponding
branching ratios. The potential ultra-low Q value decay branches are indicated
by dashed blue arrows. Spin assignments in braces indicate the range of
possible values for that level. The ground-state to ground-state Q values are
obtained using data from the AME2020 Wang  _et al._ (2021). Energy levels and
Q values are given in units of keV.

##  II Experimental Description

The Low Energy Beam and Ion Trap (LEBIT) Ringle  _et al._ (2009, 2013),
located at the Facility for Rare Isotope Beams (FRIB) was used to perform
precise cyclotron frequency ratio measurements of singly-charged 75As+ and
12C${}_{6}^{+}$ carbon cluster ions with a similar mass-to-charge ratio. The
75As+ ions were delivered to the LEBIT facility from a batch mode ion source
(BMIS) where they were produced as a by-product during normal BMIS operation.
The 12C${}_{6}^{+}$ ions were produced with a laser ablation source (LAS) Izzo
_et al._ (2016) in which a Sigradur® sig  glassy carbon target was installed.
In either case, ions first enter the LEBIT cooler and buncher Schwarz  _et
al._ (2016), in which they are thermalized by a helium buffer gas, confined in
a linear rf Paul trap, and then released as low emittance, $\sim$100 ns length
pulses. Ions are then directed toward the Penning trap, which has a hyperbolic
structure and is housed in a 9.4 T magnetic field.

In the Penning trap, ions undergo three normal modes of motion: reduced-
cylotron, magnetron, and axial, with frequencies $f_{+}$, $f_{-}$, and
$f_{z}$, respectively. The three normal mode frequencies can be combined to
determine the free-space cyclotron frequency, $f_{c}$. Pertinent to the work
described here is the relationship

$f_{c}=f_{+}+f_{-}=\frac{qB}{2\pi m},$ (1)

where $q/m$ is the charge-to-mass ratio of the ion, and $B$ is the magnetic
field strength. At LEBIT, the cyclotron frequency of ions in the Penning trap
is determined using the time-of-flight ion cyclotron resonance (TOF-ICR)
technique Gräff  _et al._ (1980); König  _et al._ (1995). Ions are parked on
an initial magnetron orbit by a Lorentz steerer Ringle  _et al._ (2007) and
are then driven with a quadrupolar radio frequency (rf) pulse at a frequency
$f_{\textrm{rf}}$ close to the free-space cyclotron frequency of the ion.
Depending on the frequency difference $\left|f_{\textrm{rf}}-f_{c}\right|$,
magnetron motion can be partly or fully converted to cyclotron motion. The
ions are then released from the trap and impinge on a micro-channel plate
(MCP) detector that is used to determine the time-of-flight from the trap to
the MCP, as well as the number of ions that were in the trap. An ion’s time-
of-flight to the MCP is minimized upon full conversion of magnetron to
cyclotron motion. Hence, $f_{\textrm{rf}}$ is varied around $f_{c}$ and a
characteristic TOF resonance curve can be mapped out.

In this work, we used the Ramsey TOF-ICR technique Bollen  _et al._ (1992);
George  _et al._ (2007a), in which two time separated $\pi$/2 pulses are used
instead of a single $\pi$-pulse to couple the magnetron and cyclotron modes.
This technique results in an interference-type pattern in the TOF resonance,
and has been shown to enable a determination of the cyclotron frequency to at
least 3 times higher precision in the same measurement time compared to the
traditional TOF-ICR technique George  _et al._ (2007b). An example Ramsey TOF-
ICR resonance for 75As+ from this work is shown in Fig. 2.

Figure 2: (Color online) A 500 ms cyclotron frequency resonance curve for
75As+. The solid red line is a fit to the theoretical line shape Kretzschmar
(2007).

A single resonance like the one shown in Fig. 2 took $\sim$30 mins, consisted
of $\sim$2500 – 5000 ions, and enabled a determination of the 75As+ or
12C${}_{6}^{+}$ frequency to $\sim$15 mHz. In order to determine the cyclotron
frequency ratio of 12C${}_{6}^{+}$ to 75As+ and to account for temporal
magnetic field drifts, we alternated between measurements with the two ions,
performing 65 measurement pairs in total.

Deviations from an ideal Penning trap i.e. a magnetic field that is not
perfectly uniform, and an electric field that is not purely quadrupolar,
combined with the finite normal mode amplitudes of ions in the trap, result in
shifts to the measured frequencies. The preparation of ions of different $m/q$
in the Penning trap can result in them having different normal mode amplitudes
and hence different systematic frequency shifts. While these shifts largely
cancel in the cyclotron frequency ratio, when comparing ions of different
nominal $m/q$ a residual shift to the cyclotron frequency ratio can remain. In
this work, our cyclotron frequency ratio involved 12C${}_{6}^{+}$ and 75As+,
which differ by three atomic mass units. To evaluate the potential shift to
the ratio, we took additional cyclotron frequency ratio data for
12C${}_{6}^{+}$ vs 85,87Rb+ and 12C${}_{7}^{+}$. The uncertainty in the mass
of the carbon cluster ions and the rubidium ions (the masses of which have
been determined previously with the Florida State University Penning trap
Mount  _et al._ (2010)) are negligible compared to the statistical
uncertainties in our measurements.

##  III Data and Analysis

To evaluate the cyclotron frequency ratio from our alternating measurements of
12C${}_{6}^{+}$ and 75As+, pairs of 12C${}_{6}^{+}$ cyclotron frequency
measurements were first linearly interpolated to obtain the corresponding
$f_{c}^{\textrm{int}}(^{12}$C${}_{6}^{+})$ value at the time of the
interleaved $f_{c}(^{75}$As${}^{+})$ measurement. This procedure eliminated
potential systematic shifts to the cyclotron frequency ratio due to linear
magnetic field drifts. A series of cyclotron frequency ratios, corresponding
to inverse mass ratios of the two ions were found from

$R=\frac{f_{c}(^{75}\textrm{As}^{+})}{f_{c}^{\textrm{int}}(^{12}\textrm{C}_{6}^{+})}=\frac{m(^{12}\textrm{C}_{6}^{+})}{m(^{75}\textrm{As}^{+})}.$
(2)

The individual measurements of $R$ were then used to find the weighted
average, $\bar{R}$, the associated statistical uncertainty, and the Birge
Ratio Birge (1932), as shown in Fig. 3. For the 12C${}_{6}^{+}$ vs 75As+ data
set, consisting of 65 individual ratio measurements, the Birge Ratio was found
to be 1.07(6) indicating that the statistical uncertainties in the individual
measurements are a reasonable estimate of the statistical spread of the data.
Nevertheless, we conservatively inflated the statistical uncertainty in the
weighted average ratio by the Birge Ratio.

Figure 3: (Color online) Individual cyclotron frequency ratio measurements of
$f_{c}$(75As+)/$f_{c}^{\textrm{int}}$(12C${}_{6}^{+}$) (black squares). The
red band shows the uncertainty in the average ratio, without systematic
corrections and associated uncertainties applied. The blue line and band show
the ratio obtained using the 75As mass from AME2020 Wang  _et al._ (2021) and
the associated uncertainty.

Our auxiliary measurements of 12C${}_{6}^{+}$ vs 85,87Rb+ and 12C${}_{7}^{+}$
did show a systematic shift in the ratio when comparing ions of different
nominal $m/q$ of –0.57(18) $\times$ 10-9 per u/e. This result was used to
apply a correction of 1.7(5) $\times$ 10-9 to the 12C${}_{6}^{+}$ vs 75As+
ratio. We also performed a Z-class analysis to evaluate potential shifts to
the cyclotron frequency from the Coulomb interaction when there is more than
one ion in the trap Kellerbauer  _et al._ (2003). This analysis showed a
linear trend, but with a slope of 0.52(51) $\times$ 10-9 per ion in the trap,
consistent with no shift. In our data analysis, we limited the number of ions
in the trap to between one and five. Assuming an average of three ions in the
trap, a correction of –1.6(1.5) $\times$ 10-9 should be applied to the data.
We applied half this value as a correction with an uncertainty equal to the
applied correction.

In the course of the analysis we uncovered an additional systematic effect:
when ions arrive at the MCP, they create a time-of-flight (TOF) distribution,
the width of which depends on the energy difference between ions that are
resonant with the rf quadrupole drive in the trap (i.e. magnetron motion is
fully converted to cyclotron motion) and those that are not (i.e. no
conversion). There is also some spread due to the variation in the initial
conditions of ions in the trap. In addition, there can be background noise or
ion events in the time window, and so a gate is placed around the TOF
distribution of the ions of interest. For 75As+ ions, a systematic variation
in the cyclotron frequency was observed as a function of location of the high-
end cut-off time of the gate i.e. on where the cut-off was made for ions with
a longer TOF, which corresponds to non-resonant ions. This systematic effect
was not observed for 12C${}_{6}^{+}$ or 12C${}_{7}^{+}$ ions produced with the
LAS, or 85,87Rb+ ions produced with a surface ionization source. Hence, we
concluded that there could be contaminant ions in the beam arriving from the
BMIS that were not identified, and which could cause systematic shifts to the
75As+ ions’ frequency via ion-ion interactions.

To account for this effect, we systematically varied the high-end cut-off time
of the gate, and recorded the cyclotron frequency and $\chi^{2}$ of the fit
averaged over all 65 measurements. We found that the $\chi^{2}$ varied
quadratically around its minimum—when more ion events from the longer TOF tail
of the distribution were included, the fit was statistically worse, and,
likewise, when the cut was made further into the TOF distribution, more data
were discarded and the fit was again worse. We used the gate-setting
corresponding to the minimum $\chi^{2}$ (obtained from a fit of $\chi^{2}$ vs
gate width) in our final analysis. The cyclotron frequency we obtained as a
function of gate width also varied around a minimum value. The minimum
$f_{c}$, however, did not occur at the minimum $\chi^{2}$. The minimum $f_{c}$
was 0.007 Hz lower, and the maximum $f_{c}$ 0.015 Hz higher than our nominal
value at $\chi^{2}$-min. These correspond to shifts in the
75As+/12C${}_{6}^{+}$ ratio of 3.7 and 7.7 $\times$ 10-9, respectively. We
used the average of these two values, 5.7 $\times$ 10-9, to conservatively
estimate the additional uncertainty to include in our final ratio to account
for this systematic effect. This corresponds to an uncertainty in the mass of
75As of about 0.4 keV/$c^{2}$. A summary of our ratio measurement and the
systematic corrections and uncertainties applied are shown in Table 1.

Table 1:  Error budget for 75As+ vs 12C${}_{6}^{+}$ cyclotron frequency ratio measurement. $\sigma_{\mathrm{st}}$($\times$BR) is the statistical uncertainty (inflated by the Birge Ratio, BR = 1.07). $\Delta R_{\mathrm{ion}}$, $\Delta R_{\mathrm{\delta M}}$, and $\Delta R_{\mathrm{gate}}$ are the systematic corrections (and corresponding uncertainties in parentheses) to account for frequency shifts due to there being more than one ion in the trap, the effect of comparing ions of dissimilar $m/q$, and the effect associated with how the gate on the TOF distribution was set (see text for details). $\bar{R}^{{}^{\prime}}$ is the corrected ratio with statistical, systematic,and total uncertainties shown respectively in parentheses. Source | Contribution to
---|---
Ratio ($\times$ 10-9) |  $M$(75As) (keV/$c^{2}$)
$\sigma_{\mathrm{st}}$ | 1.5 | 0.11
$\sigma_{\mathrm{st}}\times$BR | 1.6 | 0.12
$\Delta R_{\mathrm{ion}}$ | –0.8(8) | 0.06(6)
$\Delta R_{\mathrm{\delta M}}$ | 1.7(5) | –0.12(4)
$\Delta R_{\mathrm{gate}}$ | 0.0(5.7) | 0.0(0.4)
Final ratio |  $\bar{R}^{{}^{\prime}}$ = $0.961\ 004\ 378\ 4(16)(58)(60)$

After accounting for systematic effects, the corrected average ratio,
$\bar{R}^{{}^{\prime}}$, was used to determine the atomic mass of 75As in
atomic mass units using the mass and binding energy of the carbon cluster
reference molecule and accounting for the mass of the missing electrons,
$m_{e}$:

$M(^{75}\textrm{As})=\left[6M(^{12}\textrm{C})-m_{e}+BE(^{12}\textrm{C}_{6})\right]\bar{R}^{{}^{\prime}-1}+m_{e}.$
(3)

The first ionization energies of 75As and 12C6 are both $\approx$10 eV, so the
difference is $<$1 eV and is negligible. The 12C6 molecular binding energy, on
the other hand, is $-31.7$ eV for the most stable cyclic structure Kosimov
_et al._ (2008), and therefore is included in Eqn. (3). The mass excess was
then calculated from

$ME(^{75}\textrm{As})=\left[M(^{75}\textrm{As})-75\right]\times F,$ (4)

where the conversion factor $F=931\,494.102\,42(28)$ (keV/$c^{2}$)/u from Ref.
Tiesinga  _et al._ (2021) was used.

##  IV Results and Discussion

Table 2:  Mass excess for 75As obtained in the work along with the result from the AME2020 Wang  _et al._ (2021) and the difference $\Delta$ME = MELEBIT – MEAME Isotope | Ref. | This work | AME2020 |  $\Delta$ME
---|---|---|---|---
(keV/$c^{2}$) | (keV/$c^{2}$) | (keV/$c^{2}$)
75As |  12C6 | $-73\ 035.98(43)$ | $-73\ 034.20(88)$ | –1.78(98)

Our final mass excess value for 75As is given in Table 2, and compared with
the value from the AME2020 Wang  _et al._ (2021). Our result is 1.78(98)
keV/$c^{2}$ lower than the AME2020 value and a factor of 2 more precise. This
result indicates that 75As is more strongly bound than suggested by the
AME2020 value, thus increasing the $Q_{gs}$ values of 75Se and 75Ge, which can
be found via

$Q^{\textrm{EC}}_{gs}\left({}^{75}\textrm{Se}\right)=\left[ME\left({}^{75}\textrm{Se}\right)-ME\left({}^{75}\textrm{As}\right)\right]c^{2},$
(5)

and

$Q^{\beta}_{gs}\left({}^{75}\textrm{Ge}\right)=\left[ME\left({}^{75}\textrm{Ge}\right)-ME\left({}^{75}\textrm{As}\right)\right]c^{2}.$
(6)

Our calculated $Q^{\textrm{EC}}_{gs}$ and $Q^{\beta}_{gs}$ values for 75Se and
75Ge, obtained using our new value for $M(^{75}$As) and the mass of the 75Se
and 75Ge parent atoms from AME2020 Wang  _et al._ (2021), are given in Table
III.

The Q value for the decay to an excited state in the daughter nuclide with
energy $E^{*}$ can then be found via

$Q_{es}^{\beta/\textrm{EC}}=Q_{gs}^{\beta/\textrm{EC}}-E^{*}.$ (7)

For the case of 75Se EC decay, two potential excited states in 75As with
energies close to $Q^{\textrm{EC}}_{gs}(^{75}$Se) were identified: the
860.0(4) keV 1/2+ level, and the 865.4(5) keV 3/2+ or 5/2+ level. For 75Ge
$\beta$-decay, the 1172.0(6) keV $J^{-}$ state with $J$ = 1/2, 3/2, 5/2 or 7/2
was identified as a potential daughter state with a low energy decay Q value.
Our $Q_{gs}$ values are compared with the relevant $E^{*}$ values in Table
III. Our results indicate that all three decays listed are energetically
allowed at least at the 1.5$\sigma$ level. The 75Ge $\beta$-decay to the 1172
keV state in 75As is ruled out as not being ultra-low (i.e. $Q_{es}^{\beta}$
$>$ 1 keV) at the 9$\sigma$ level. Likewise, the EC decay of 75Se to the 860
keV level in 75As is ruled out as an ultra-low decay branch, at the 8$\sigma$
level. However, the EC decay of 75Se to the 865 keV level in 75As has
$Q_{es}^{\textrm{EC}}\approx$ 1 keV, making it a potential candidate for an
ultra-low Q value decay. The uncertainty in the $Q_{es}^{\textrm{EC}}$ value
listed in Table III is predominantly (60 % contribution) due to the 0.5 keV
uncertainty in the 865 keV level, with the remaining 40 % contribution due to
the 0.4 keV uncertainty in our $M(^{75}$As) value. Hence, a more precise
determination of both the $Q^{\textrm{EC}}_{gs}(^{75}$Se) value and the
$E^{*}$(865.4 keV) level energy are required to further evaluate this
potential ultra-low Q value EC decay.

In fact, a recent measurement reported on the arXiv by the JYFLTRAP group does
provide a more precise mass for 75As Ramalho  _et al._ (2022). Our result for
$ME$(75As) agrees with the JYLFTRAP result, $ME$(75As) = –73035.521(42)
keV/$c^{2}$, which is a factor of ten more precise, at the 1$\sigma$ level.
The JYFLTRAP results also indicates that 75Se EC decay to the 865.4 keV level
in 75As is energetically allowed with a potentially ultra-low Q value,
$Q_{es}\lesssim$ 1 keV, but that a more precise determination of the 865.4 keV
level is necessary.

Table 3:  Q values based on the mass measurement listed in Table 2 and Eqns. (5) and (6). The column $E^{*}$ gives the energy of the excited state of the daughter nucleus that the potential ultra-low Q value decay would go to. The result for the Q value of the decay branch to the excited state, $Q_{es}$, is calculated from Eqn. (7). Parent | Decay | $Q_{gs}$ |  $E$* | $Q_{es}$
---|---|---|---|---
(keV) | (keV) | (keV)
75Se | EC | 866.50(44) | 865.4(5) | 1.10(67)
75Se | EC | 866.50(44) | 860.4(4) | 6.10(60)
75Ge | $\beta$ | 1179.01(44) | 1172.0(6) | 7.01(74)

##  V Conclusion

Using Penning trap mass spectrometry, we have performed a precise measurement
of the mass of 75As. Our result reduces the uncertainty in the mass by about a
factor of two compared to the value adopted in the most recent atomic mass
evaluation, AME2020 Wang  _et al._ (2021) and shows a shift of –1.78
keV/$c^{2}$. Using this new mass and the more precisely known masses of 75Se
and 75Ge from AME2020, we were able to determine new ground-state to ground-
state Q values for the EC decay of 75Se and the $\beta$-decay of 75Ge. These Q
values were combined with excited state energies in 75As to determine the
$Q_{es}$ values of potential ultra-low energy decays of 75Se and 75Ge. Our
results indicate that the $\beta$-decay of 75Ge to the 1172.0(6) keV, negative
parity state with angular momentum between 1/2 and 7/2, is energetically
allowed, but with a Q value of 7.0(7) keV, ruling it out as a potential ultra-
low Q value decay. Likewise, the EC decays of 75Se to the 865.4(5) keV 3/2- or
5/2- level, and to the 860.0(4) keV 1/2+ level are also energetically allowed
with Q values of 1.1(7) keV and 6.1(6) keV, respectively. Our result is in
agreement with a recently reported measurement by the JYFLTRAP group. Both
results indicate that the EC decay of 75Se to the 865 keV level in 75As with a
$Q_{es}$ $\lesssim$ 1 keV is of interest as a potential ultra-low Q value
decay candidate and that, in order to further evaluate this potential decay
branch, a more precise determination of the energy of the 865 keV level is
required.

## Acknowledgments

This research was supported by Central Michigan University, Michigan State
University and the Facility for Rare Isotope Beams, and the National Science
Foundation under Contracts No. PHY-1565546, PHY-2111185, and No. PHY-2111302.

## References

  * Hardy and Towner (2015) J. C. Hardy and I. S. Towner, Phys. Rev. C 91, 025501 (2015).
  * Severijns  _et al._ (2006) N. Severijns, M. Beck, and O. Naviliat-Cuncic, Rev. Mod. Phys. 78, 991 (2006).
  * Burkey  _et al._ (2022) M. T. Burkey, G. Savard, A. T. Gallant, N. D. Scielzo, J. A. Clark, T. Y. Hirsh, L. Varriano, G. H. Sargsyan, K. D. Launey, M. Brodeur, D. P. Burdette, E. Heckmaier, K. Joerres, J. W. Klimes, K. Kolos, A. Laminack, K. G. Leach, A. F. Levand, B. Longfellow, B. Maaß, S. T. Marley, G. E. Morgan, P. Mueller, R. Orford, S. W. Padgett, A. Pérez Galván, J. R. Pierce, D. Ray, R. Segel, K. Siegl, K. S. Sharma, and B. S. Wang, Phys. Rev. Lett. 128, 202502 (2022).
  * Avignone  _et al._ (2008) F. T. Avignone, S. R. Elliott, and J. Engel, Rev. Mod. Phys. 80, 481 (2008).
  * Friedrich  _et al._ (2021) S. Friedrich, G. B. Kim, C. Bray, R. Cantor, J. Dilling, S. Fretwell, J. A. Hall, A. Lennarz, V. Lordi, P. Machule, D. McKeen, X. Mougeot, F. Ponce, C. Ruiz, A. Samanta, W. K. Warburton, and K. G. Leach, Phys. Rev. Lett. 126, 021803 (2021).
  * Aker (2019) K. C. Aker, M. et al. (KATRIN Collaboration), Phys. Rev. Lett. 123, 221802 (2019).
  * Monreal and Formaggio (2009) B. Monreal and J. A. Formaggio, Phys. Rev. D 80, 051301(R) (2009).
  * Gastaldo (2017) E. C. Gastaldo, L. et al. (ECHo Collaboration), The European Physical Journal Special Topics 226, 1623 (2017).
  * Faverzani  _et al._ (2016) M. Faverzani _et al._ (HOLMES Collaboration), Journal of Low Temperature Physics 184, 922 (2016).
  * Cattadori  _et al._ (2004) C. M. Cattadori, M. D. Deo, M. Laubenstein, L. Pandola, and V. I. Tretyak, Nucl. Phys. A 748, 333 (2004).
  * Cattadori  _et al._ (2007) C. M. Cattadori, M. De Deo, M. Laubenstein, L. Pandola, and V. I. Tretyak, Physics of Atomic Nuclei 70, 127 (2007).
  * Suhonen (2014) J. Suhonen, Physica Scripta 89, 054032 (2014).
  * Kopp and Merle (2010) J. Kopp and A. Merle, Phys. Rev. C 81, 045501 (2010).
  * Mustonen and Suhonen (2010a) M. T. Mustonen and J. Suhonen, J. Phys. G: Nucl. Part. Phys. 37, 064008 (2010a).
  * Suhonen and Mustonen (2010) J. Suhonen and M. Mustonen, Progress in Particle and Nuclear Physics 64, 235 (2010), neutrinos in Cosmology, in Astro, Particle and Nuclear Physics.
  * Mount  _et al._ (2009) B. Mount, M. Redshaw, and E. G. Myers, Phys. Rev. Lett. 103, 122502 (2009).
  * Wieslander  _et al._ (2009) J. S. E. Wieslander, J. Suhonen, T. Eronen, M. Hult, V.-V. Elomaa, A. Jokinen, G. Marissens, M. Misiaszek, M. T. Mustonen, S. Rahaman, C. Weber, and J. Äystö, Phys. Rev. Lett. 103, 122501 (2009).
  * Blachot (2005) J. Blachot, Nuclear Data Sheets 104, 967 (2005).
  * Zheltonozhsky  _et al._ (2018) V. A. Zheltonozhsky, A. M. Savrasov, N. V. Strilchuk, and V. I. Tretyak, Europhys. Lett. 121, 12001 (2018).
  * Mustonen and Suhonen (2010b) M. T. Mustonen and J. Suhonen, AIP Conf. Proc. 1304, 401 (2010b).
  * Haaranen and Suhonen (2013) M. Haaranen and J. Suhonen, The European Physical Journal A 49, 93 (2013).
  * Mustonen and Suhonen (2011) M. Mustonen and J. Suhonen, Physics Letters B 703, 370 (2011).
  * Gamage  _et al._ (2019) N. D. Gamage, R. Bhandari, M. H. Gamage, R. Sandler, and M. Redshaw, Hyp. Int. 240, 43 (2019).
  * Wang  _et al._ (2021) M. Wang, W. Huang, F. Kondev, G. Audi, and S. Naimi, Chinese Physics C 45, 030003 (2021).
  * (25) “National nuclear data center,” https://www.nndc.bnl.gov/.
  * Karthein  _et al._ (2019) J. Karthein, D. Atanasov, K. Blaum, S. Eliseev, P. Filianin, D. Lunney, V. Manea, M. Mougeot, D. Neidherr, Y. Novikov, L. Schweikhard, A. Welker, F. Wienholtz, and K. Zuber, Hyperfine Interactions 240, 61 (2019).
  * de Roubin  _et al._ (2020) A. de Roubin, J. Kostensalo, T. Eronen, L. Canete, R. P. de Groote, A. Jokinen, A. Kankainen, D. A. Nesterenko, I. D. Moore, S. Rinta-Antila, J. Suhonen, and M. Vilén, Phys. Rev. Lett. 124, 222503 (2020).
  * Ge  _et al._ (2021a) Z. Ge, T. Eronen, A. de Roubin, D. A. Nesterenko, M. Hukkanen, O. Beliuskina, R. de Groote, S. Geldhof, W. Gins, A. Kankainen, A. Koszorús, J. Kotila, J. Kostensalo, I. D. Moore, A. Raggio, S. Rinta-Antila, J. Suhonen, V. Virtanen, A. P. Weaver, A. Zadvornaya, and A. Jokinen, Phys. Rev. C 103, 065502 (2021a).
  * Ge  _et al._ (2021b) Z. Ge, T. Eronen, K. S. Tyrin, J. Kotila, J. Kostensalo, D. A. Nesterenko, O. Beliuskina, R. de Groote, A. de Roubin, S. Geldhof, W. Gins, M. Hukkanen, A. Jokinen, A. Kankainen, A. Koszorús, M. I. Krivoruchenko, S. Kujanpää, I. D. Moore, A. Raggio, S. Rinta-Antila, J. Suhonen, V. Virtanen, A. P. Weaver, and A. Zadvornaya, Phys. Rev. Lett. 127, 272301 (2021b).
  * Gamage  _et al._ (2022) N. D. Gamage, R. Sandler, F. Buchinger, J. A. Clark, D. Ray, R. Orford, W. S. Porter, M. Redshaw, G. Savard, K. S. Sharma, and A. A. Valverde, arXiv:2202.12874 (2022).
  * Sandler  _et al._ (2019) R. Sandler, G. Bollen, N. D. Gamage, A. Hamaker, C. Izzo, D. Puentes, M. Redshaw, R. Ringle, and I. Yandow, Phys. Rev. C 100, 024309 (2019).
  * Mys (1969) _Proceedings of the Conference on the Electron Capture and Higher Order Processes in Nuclear Decays (Debrecen, Hungary, 1968)_ , Vol. 1 (Eoetvoe, Budapest, 1969) pp. 102 - 7.
  * Ringle  _et al._ (2009) R. Ringle, G. Bollen, A. Prinke, J. Savory, P. Schury, S. Schwarz, and T. Sun, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 604, 536 (2009).
  * Ringle  _et al._ (2013) R. Ringle, S. Schwarz, and G. Bollen, Int. J. Mass Spec. 349, 87 (2013).
  * Izzo  _et al._ (2016) C. Izzo, G. Bollen, S. Bustabad, M. Eibach, K. Gulyuz, D. J. Morrissey, M. Redshaw, R. Ringle, R. Sandler, S. Schwarz, and A. A. Valverde, Nucl. Instrum. Methods Phys. Res., Sect. B 376, 60 (2016).
  * (36) “Htw sigradur,” http://www.htw-germany.com.
  * Schwarz  _et al._ (2016) S. Schwarz, G. Bollen, R. Ringle, J. Savory, and P. Schury, Nucl. Instrum. Methods Phys. Res. A 816, 131 (2016).
  * Gräff  _et al._ (1980) G. Gräff, H. Kalinowsky, and J. Traut, Zeit. Phy. A 297, 35 (1980).
  * König  _et al._ (1995) M. König, G. Bollen, H.-J. Kluge, T. Otto, and J. Szerypo, Int. J. Mass. Spectrom. Ion Process 142, 95 (1995).
  * Ringle  _et al._ (2007) R. Ringle, G. Bollen, A. Prinke, J. Savory, P. Schury, S. Schwarz, and T. Sun, International Journal of Mass Spectrometry 263, 38 (2007).
  * Bollen  _et al._ (1992) G. Bollen, H.-J. Kluge, T. Otto, G. Savard, and H. Stolzenberg, Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms 70, 490 (1992).
  * George  _et al._ (2007a) S. George, K. Blaum, F. Herfurth, A. Herlert, M. Kretzschmar, S. Nagy, S. Schwarz, L. Schweikhard, and C. Yazidjian, International Journal of Mass Spectrometry 264, 110 (2007a).
  * George  _et al._ (2007b) S. George, S. Baruah, B. Blank, K. Blaum, M. Breitenfeldt, U. Hager, F. Herfurth, A. Herlert, A. Kellerbauer, H.-J. Kluge, M. Kretzschmar, D. Lunney, R. Savreux, S. Schwarz, L. Schweikhard, and C. Yazidjian, Phys. Rev. Lett. 98, 162501 (2007b).
  * Kretzschmar (2007) M. Kretzschmar, International Journal of Mass Spectrometry 264, 122 (2007).
  * Mount  _et al._ (2010) B. J. Mount, M. Redshaw, and E. G. Myers, Phys. Rev. A 82, 042513 (2010).
  * Birge (1932) R. T. Birge, Phys. Rev. 40, 207 (1932).
  * Kellerbauer  _et al._ (2003) A. Kellerbauer, K. Blaum, G. Bollen, F. Herfurth, H. J. Kluge, M. Kuckein, E. Sauvan, C. Scheidenberger, and L. Schweikhard, The European Physical Journal D - Atomic, Molecular, Optical and Plasma Physics 22, 53 (2003).
  * Kosimov  _et al._ (2008) D. P. Kosimov, A. A. Dzhurakhalov, and F. M. Peeters, Phys. Rev. B 78, 235433 (2008).
  * Tiesinga  _et al._ (2021) E. Tiesinga, P. J. Mohr, D. B. Newell, and B. N. Taylor, Journal of Physical and Chemical Reference Data 50, 033105 (2021).
  * Ramalho  _et al._ (2022) M. Ramalho, Z. Ge, T. Eronen, D. A. Nesterenko, J. Jaatinen, A. Jokinen, A. Kankainen, J. Kostensalo, J. Kotila, M. I. Krivoruchenko, J. Suhonen, K. S. Tyrin, and V. Virtanen, arXiv:2203.00457 (2022).


22footnotetext: † Corresponding author.11institutetext: University of Science
and Technology of China, Hefei, China 22institutetext: iFLYTEK Research,
Hefei, China
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>

# NAMER: Non-Autoregressive Modeling for Handwritten Mathematical Expression
Recognition

Chenyu Liu 1University of Science and Technology of China, Hefei, China
12iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>Jia Pan 2iFLYTEK Research, Hefei,
China

<EMAIL_ADDRESS><EMAIL_ADDRESS>Jinshui Hu 2iFLYTEK Research,
Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>Baocai Yin 2iFLYTEK Research,
Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>Bing Yin 2iFLYTEK Research, Hefei,
China

<EMAIL_ADDRESS><EMAIL_ADDRESS>
Mingjun Chen 2iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>Cong Liu 2iFLYTEK Research, Hefei,
China

<EMAIL_ADDRESS><EMAIL_ADDRESS>Jun Du† 1University of Science and
Technology of China, Hefei, China 1 Qingfeng Liu 1University of Science and
Technology of China, Hefei, China 12iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>1University of Science and
Technology of China, Hefei, China 12iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>2iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>2iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>2iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>2iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>2iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>2iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>1University of Science and
Technology of China, Hefei, China 11University of Science and Technology of
China, Hefei, China 12iFLYTEK Research, Hefei, China

<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, Handwritten Mathematical Expression Recognition (HMER) has gained
considerable attention in pattern recognition for its diverse applications in
document understanding. Current methods typically approach HMER as an image-
to-sequence generation task within an autoregressive (AR) encoder-decoder
framework. However, these approaches suffer from several drawbacks: 1) a lack
of overall language context, limiting information utilization beyond the
current decoding step; 2) error accumulation during AR decoding; and 3) slow
decoding speed. To tackle these problems, this paper makes the first attempt
to build a novel bottom-up Non-AutoRegressive Modeling approach for HMER,
called NAMER. NAMER comprises a Visual Aware Tokenizer (VAT) and a Parallel
Graph Decoder (PGD). Initially, the VAT tokenizes visible symbols and local
relations at a coarse level. Subsequently, the PGD refines all tokens and
establishes connectivities in parallel, leveraging comprehensive visual and
linguistic contexts. Experiments on CROHME 2014/2016/2019 and HME100K datasets
demonstrate that NAMER not only outperforms the current state-of-the-art
(SOTA) methods on ExpRate by 1.93%/2.35%/1.49%/0.62%, but also achieves
significant speedups of 13.7$\times$ and 6.7$\times$ faster in decoding time
and overall FPS, proving the effectiveness and efficiency of NAMER.

###### Keywords:

Non-autoregressive modeling Handwritten mathematical expression recognition
OCR

##  1 Introduction

Handwritten Mathematical Expression Recognition is a fundamental task in the
OCR and pattern recognition field, playing a wide-ranging role in document
understanding, teaching and education, as well as office automation. With the
developments of deep learning [23, 16, 18, 37], sequence modeling [1, 8, 36],
and text recognition [33, 34, 9] over the past decade, more and more HMER
algorithms [47, 49, 53, 4, 46, 27, 52] have been proposed with large
performance improvements.

(a) Existing AR modeling of HMER.

(b) A brief illustration of NAMER.

Figure 1: Comparison between existing AR based HMER methods and the proposed
NAMER. All processes in NAMER follow a NAR manner.

Different from text recognition and speech recognition, there are many
complicated symbol relations in mathematical expressions, making HMER more
challenging. For example, a mathematical expression “\frac{x^{y}}{z_{1}}” has
three relations: fraction, exponent, and subscript. To tackle this,
researchers have proposed to convert HMER to an image-to-markup [11]
generation task. Hence, lots of encoder-decoder based sequence modeling
methods [1, 8, 36] have been improved to deal with HMER successfully.
Specifically, given an image, an encoder is firstly used to extract visual
features, and an attention based decoder is then designed to recognize LaTeX
strings token by token, following an AR manner. See Fig. 1(a) for a brief
illustration. Moreover, researchers have persistently strived to advance HMER,
focusing on refining both structured modeling and contextual inference. Tree
based [49, 54, 40, 10] and syntax-aware [46] targets are proposed for image-
to-sequence prediction with robuster structural recognitions, while symbol
counting [27], bidirectional modeling [53, 4] and coverage modeling [52] are
designed to enhance the contextual inference abilities of HMER.

Despite great performance improvements on HMER, there are still some
weaknesses of current methods. Firstly, current models lack the ability to
flexibly utilize the overall visual or linguistic context features. Although
the bidirectional and coverage modeling designs have already improved the
utilization of context, current models are still prone to misclassify
ambiguous symbols that human can easily recognize them correctly by using
vision-language context informations. Secondly, because of the mechanism that
current models need to recognize tokens in an autoregressive manner, inference
speeds of these models are slow.

To jointly improve the utilization of overall vision-language infos and
recognition efficiency in HMER, we propose a new Non-Autoregressive Modeling
framework for HMER, called NAMER. Our motivation is from the recognition
ability and process of humans: (1) humans don’t recognize and understand HMEs
in a predefined LaTeX string or symbol tree order, instead, humans usually
start from recognizing symbols in an approximate left-to-right order and
determining relations between these symbols; (2) humans have the strong
ability to recognize from arbitrary local symbols to global results in a
bottom-up manner, and when facing ambiguous symbols, the overall bottom
information of local visual features and local recognition results are
dynamically used. Inspired by this, we aim to build a HMER system which have
two key characteristics: one is the mechanism of recognizing or tokenizing
local symbols in parallel, the other is recovering the connectivities and
relations between these tokens in a non-autoregressive (NAR) manner. To
achieve this, NAMER adopts a two-stage, bottom-up approach while keeping end-
to-end training. As illustrated in Fig. 1(b), the first stage predicts all
visible symbols and local relation tokens, including some imaginary
relationship symbols like “^”, along with their approximate locations in
parallel, providing coarse symbol-instance level results. The second stage
refines these predicted coarse tokens while simultaneously predicting the
connectivity probabilities among them in parallel. Both stages follow a NAR
manner, and the resultant output constructs a Directed Acyclic Graph (DAG)
that can easily be converted to the LaTeX format or Symbol Layout Trees[49].
Consequently, NAMER is a NAR recognition system that doesn’t require decoding
tokens sequentially, resulting in significant speedups. Moreover, the two-
stage NAR design not only enhances flexibility in revising misclassified
tokens by leveraging global visual and linguistic context features more
effectively but also ensures robustness in reconstructing the structure of
HMEs from local to global.

To verify the effectiveness of NAMER, we conducted comprehensive experiments
on the CROHME 2014 [30], 2016 [31], 2019 [29] datasets and HME100K [46]
dataset. In short, NAMER significantly enhances decoding speed while
maintaining SOTA performance. Across the CROHME datasets, NAMER achieves
ExpRates of 60.51%, 60.24%, and 61.72% without augmentations on CROHME 2014,
2016, and 2019, respectively, surpassing SOTA methods (58.57%, 57.89%, 59.71%)
by an average of 1.93%. Regarding inference speed, NAMER demonstrates a
13.7$\times$ and 6.70x$\times$ improvement in decoding time and overall
inference time, respectively, compared to the previous SOTA method, CoMER
[52]. In summary, the main contributions of this paper are as follows:

  1. 1.

NAMER represents the first endeavor, to the best of our knowledge, in
constructing a non-autoregressive HMER system without extra annotations;

  2. 2.

The VAT, PGD modules, and Bipartite Matching training strategy are
meticulously designed to facilitate a bottom-up NAR decoding mechanism;

  3. 3.

NAMER demonstrates significant improvements in both decoding speed and memory
usage while achieving state-of-the-art performance.

##  2 Related Work

###  2.1 HMER Methods

Traditional HMER methods consist of three parts: symbol segmentation,
classification and structure analysis. These methods firstly segment inputs
into independent symbols, and then classify them using HMM [17, 22, 39],
Elastic Matching [6], or Support Vector Machines [19]. As for structure
analysis, formal grammars are carefully designed to model the 2D and syntactic
structures of formulas, where [25] proposes graph grammar to recognize
mathematical expression and [7] designs correction mechanism into a parser
based on definite clause grammar. However, limited by explicit segmentation
and manual rules, performance of traditional methods is still far from real
applications.

Recently, deep learning has made huge success in HMER. Deng _et al_. [11]
firstly showed the ability of image-to-markup generation using neural
networks, and Zhang _et al_. also proposed WAP [50] and DWAP [47] that apply
an attention based encoder-decoder to generate LaTeX strings for HMER in an AR
manner. After DWAP, AR-based encoder-decoders with image-to-markups become the
fundamental approach of HMER, and further developments can be divided into two
directions: decoding target design and decoding architecture design.

In terms of decoding target, Zhang _et al_. proposed TreeDecoder [49] to use
tree structure sequence as targets instead of LaTeX strings. Wu _et al_.
futher simplify TreeDecoder and improve generalization capability as TD-V2
[40], while SSD [41] propose a new breadth-first markup named structural
string that enhance both language modeling and hierarchical structural
modeling. In addition, TSDNet [54] design a transformer-based tree decoder
that can better capture complicated correlations. More recently, Yuan _et al_.
proposed Syntax-Aware Network (SAN), in which syntactic information is
effectively merged into the AR decoder while maintaining efficiency. The
improvement of decoding target design allow HMER models to simultaneously
focus on both HME structure and local symbols, bringing a robuster recognition
ability.

In terms of decoding architecture, PAL [42] and PAL-V2 [43] design a
discriminator along with an adversarial learning method to improve the
robustness to writing styles. BTTR [53] firstly use transformer decoders for
HMER and put forward a bi-directionally training framework to enhance language
modeling in HMER, then ABM [4] improve it with bi-directionally mutual
learning and multi-scale coverage attention. Recently, CAN [27] designs extra
heads for symbol counting, which helps correct the prediction errors of
original decoders. Moreover, CoMER [52] proposed general attention refinement
module and different coverage attention fusion modules to better model
attentions in transformer-based HMER decoders, outperforming existing methods
by a large margin.

###  2.2 NAR for HMER

At present, there are no NAR-based HMER methods available. Here, we conduct a
survey of NAR sequence modeling works in other areas.

The most common algorithm for NAR sequence recognition is CTC [14], which is
widely used in scene text recognition [33, 38] and speech recognition [14,
20]. However, due to the 2D layouts and relations in HMEs, there is no
efficient way to adopt CTC or 2D-CTC [38] in HMER currently.

Another approach involves NAR transformers, previously explored in neural
machine translation [15] and scene text recognition [2, 32, 45]. In [15], a
direct implementation of the NAR transformer decoder enables the simultaneous
translation of all tokens, whereas PARSeq [2] adopts a query-based parallel
decoding paradigm, and [32, 45] improve upon this paradigm with iterative
refinements. Nevertheless, due to the structural layout and the presence of
imaginary tokens in HME, these NAR transformer-based methods are unable to
handle HMER.

The most related work to NAMER is Graph-to-Graph (G2G) [44], which is designed
for online HMER. G2G firstly treat each stroke trajectory as a input graph
node, then use graph neural networks to encode graph features and decode
Symbol Label Tree (SLT) based graph in an AR mode. However, G2G need
trajectory-symbol level extra annotations for training and online sequential
trajectory positions for testing, hence G2G cannot be applied to general HMER.

##  3 Methodology

###  3.1 Overview

Figure 2: Overall framework of the proposed method. NAMER consists of an
encoder, a visual aware tokenizer, and a parallel graph decoder. For an input
image, VAT firstly predicts all visible symbols and local relations at a
coarse level, then PGD will revise these tokens and establish the connectivity
between them. After PGD, a DAG is used for converting results to other format
like LaTeX. All modules are in NAR manners.

The overall framework of NAMER is illustrated in Fig. 2. Given an image
$\mathbf{I}\in\mathbb{R}^{3\times H\times W}$ and its LaTeX label
$\mathbf{y}=\\{y_{1},y_{2},\ldots,y_{L}\\}$, a backbone encoder first encodes
it into a downsampled visual feature
$\mathbf{F}\in\mathbb{R}^{C\times\frac{H}{s}\times\frac{W}{s}}$. Subsequently,
the VAT module predicts all visible symbols and local relation tokens along
with their locations in parallel. After adding the corresponding imaginary
tokens (e.g., an ending token “}” for “^”), initial coarse results are
constructed without considering their relations. Refer to the bottom right of
Fig. 2 for an illustration.

Next, these initial tokens, along with their 2D position embeddings, are fed
into the PGD module. PGD comprises two main branches: one for revising input
tokens and the other for predicting the directional connectivity between each
token. Both branches are designed in a non-autoregressive manner.

In the final step, leveraging the node correction results obtained in the PGD
module and the directional connection scores assigned to each edge, a directed
acyclic graph (DAG) is meticulously built. This graph serves as the foundation
for tracing a path starting from the node “<sos>” and concluding at the node
“<eos>”, thereby yielding the ultimate recognition result.

###  3.2 Visual Aware Tokenizer

Given a 2D spatial feature map, the VAT module is designed to tokenize all
visible elements, including visible symbols and local relation tokens. Visible
symbols refer to handwritten characters, while local relation tokens encompass
handwritten structural elements (e.g., “\frac”, “\sqrt”) and imaginary
relationship symbols (e.g., “_”, “^”). As depicted in Fig. 3(a), similar to
other handwritten characters, the “^” and “_” tokens exhibit distinct visual
patterns, with relationships in the upper or lower right corner. Therefore,
these local relation tokens together with visible symbols are all feasible to
predict, and VAT is trained to address them.

(a)

(b)

Figure 3: Feasibility analysis of VAT. (a) Visual patterns of local relation
tokens. “^” is in red color rect and “_” is in blue. (b) Coarse visual tokens
on an HME image. Though all locations are imprecise, the three cases can all
lead to a correct recognition finally.

Mathematically, given encoded feature maps
$\mathbf{F}_{8\mathrm{x}}\in\mathbb{R}^{C_{1}\times\frac{H}{8}\times\frac{W}{8}}$
and
$\mathbf{F}_{16\mathrm{x}}\in\mathbb{R}^{C_{2}\times\frac{H}{16}\times\frac{W}{16}}$,
we firstly use two CNN layers to merge an 8x downsampled feature map. This
step is taken because a higher resolution feature can effectively capture
small symbols:

$\mathbf{F}_{8\mathrm{x}}^{\prime}=\mathrm{Conv_{1x1}}({\mathrm{Upsample_{2x}}(\mathbf{F}_{16\mathrm{x}})}),~{}\mathbf{F}_{out}=\mathrm{Conv_{3x3}}(\mathrm{Concat}(\mathbf{F}_{8\mathrm{x}},\mathbf{F}_{8\mathrm{x}}^{\prime}))$
(1)

Here, $\mathrm{Conv_{ixi}}(\cdot)$ denotes a standard $i\times i$ Conv-BN-Relu
block [16], $\mathrm{Upsample_{2x}(\cdot)}$ denotes a bilinear 2x upsample
layer, and $\mathrm{Concat(\cdot)}$ denotes a feature concatenation operation.
Then, after a classification head, we can obtain the predicted probabilities
at each location of $\frac{H}{8}\times\frac{W}{8}$:

$\mathbf{P}=\mathrm{Softmax}(\mathrm{Conv_{3x3}}(\mathbf{F}_{out}))$ (2)

where $\mathbf{P}\in\mathbb{R}^{(K+1)\times\frac{H}{8}\times\frac{W}{8}}$, the
number $K$ stands for the total number of all visible elements and the class
id of $K+1$ denotes none token $\varnothing$.

For the HMER task, the aforementioned visual tokenization process does not
require predicting precise boxes or masks for each token; instead,
tokenization categories and approximate positions are usually sufficient to
recover the structure of mathematical expressions. As illustrated in Fig.
3(b), even if the tokenized positions differ, as long as the symbol instances
are distinguishable and their positions are roughly correct, it does not
affect subsequent structural recovery. Since character-level annotations are
hard to obtain, we propose extracting this information from a fixed HMER model
$\mathrm{M^{*}}$, and a pretrained DWAP [47] is used as $\mathrm{M^{*}}$.
However, most HMER models suffer from the attention shift problem, making it
impractical to directly obtain symbols’ positions from their attention maps.
Therefore, we design a bipartite matching-based [24, 5] dynamic assignment
approach to guide VAT training (see Fig. 4), finding it useful and necessary:

  1. 1.

Forward $\mathrm{M^{*}}\mathbf{(I,y)}$ under teacher forcing mode and obtain
the attention scores
$\mathbf{A}\in\mathbb{R}^{L\times\frac{H}{8}\times\frac{W}{8}}$ for the
decoding step of $\\{y_{1},y_{2},\ldots,y_{L}\\}$;

  2. 2.

Utilize the highest attention scores of each $y_{i}$ to estimate its spatial
position $\mathbf{T}\in\mathbb{Z}^{L\times 2}$, where
$\mathbf{T}_{l}=\mathrm{argmax}(\mathbf{A}_{l})$;

  3. 3.

Conduct a bipartite matching process between the prediction set $\mathbf{P}$
and the estimate set $\mathbf{T}$. The cost matrix
$\mathbf{D}\in\mathbb{R}^{L\times HW}$ is designed to assign the highest
predicted score of label $y_{i}$ within a $k_{m}\times k_{m}$ kernel to each
estimated $y_{i}$. Utilizing the Hungarian algorithm for optimal assignment,
these results are converted to the final training target
$\mathbf{P}^{*}\in\mathbb{Z}^{\frac{H}{8}\times\frac{W}{8}}$. Refer to the
Supplementary Materials for matching details.

Figure 4: Detailed structure of the proposed VAT module.

After assigning targets, the loss function of VAT can be written as:

$\mathcal{L}_{\mathrm{VAT}}=\mathrm{CrossEntropy}(\mathbf{P},\mathbf{P}^{*})$
(3)

During testing, VAT simply choose symbols that have highest classification
scores on each position, and filter the none token $\varnothing$ out:

$\mathbf{y}_{\mathrm{vat}}=\\{\mathrm{y_{i}}|\mathrm{y_{i}}\in\arg\max\mathbf{P}\cap\mathrm{y_{i}}\neq\varnothing\\}$
(4)

###  3.3 Parallel Graph Decoder

The detailed structure of PGD is shown in Fig. 5. After the VAT module, we
obtain all visible tokens $\mathbf{y}_{\mathrm{vat}}$. Then, for the
structural symbols in $\mathbf{y}_{\mathrm{vat}}$, we add their respective
attached imaginary tokens, updating $\mathbf{y}_{\mathrm{vat}}$ as
$\mathbf{y}_{\mathrm{vat}}^{\prime}$. The imaginary symbols for each
structural symbol are different. For example, “^” has one “}” ending imaginary
token, while “\frac” has two ending imaginary tokens for both numerator and
denominator (see Supplementary Materials for details).

Figure 5: Detailed structure of the proposed PGD module.

PGD consists of three parallel heads: Self-node Correction Head (SCH), Left
Connectivity prediction Head (LCH), and Right Connectivity prediction Head
(RCH). Each head has the same architecture but different training targets. All
heads adopt a transformer-based non-autoregressive decoder with cross-
attention in it. Mathematically, for each layer $j$:

$\displaystyle\mathbf{Q}_{j}^{\prime}=\mathbf{Q}_{j}+\mathrm{Attention}(\mathbf{Q}_{j},\mathbf{F}_{16\mathrm{x}}^{p},\mathbf{F}_{16\mathrm{x}}^{p}),$
(5)
$\displaystyle\mathbf{Q}_{j+1}=\mathrm{Transformer}(\mathbf{Q}_{j}^{\prime})$

where $\mathbf{F}_{16\mathrm{x}}^{p}$ stands for $\mathbf{F}_{16\mathrm{x}}$
with 2d position encoding,
$\mathrm{Attention}(\mathrm{query},\mathrm{key},\mathrm{value})$ refers to a
standard attention layer and $\mathrm{Transformer}(\cdot)$ stands for a simple
transformer layer [37]. $\mathbf{Q}_{j}$ denotes the query input for every
layer $j$, and $\mathbf{Q}_{0}$ is initialized as the sum of VAT tokens’
visual feature, position encoding and word embedding, as depicted in Fig. 5.

After parallel decoding across three heads, we obtain the final features of
each head: $\mathbf{Q}_{J}^{self}$, $\mathbf{Q}_{J}^{left}$, and
$\mathbf{Q}_{J}^{right}$.

The goal of self-node correction head is to learn an implicit vision-language
model to correct misclassified nodes, using the inputs of the visual features,
the predicted language tokens, and their approximate locations. The correction
process includes revising a symbol or deleting a symbol. So the training of
$\mathbf{Q}_{J}^{self}$ can still convert to a classification problem:

$\displaystyle\mathbf{p}_{y}^{self}=\mathrm{Softmax}(\mathbf{W}_{self}^{T}\cdot\mathbf{Q}_{J}^{self}$
$\displaystyle),~{}\mathbf{y}_{\mathrm{pgd}}=\arg\max(\mathbf{p}_{y}^{self}),$
(6)
$\displaystyle\mathcal{L}_{\mathrm{PGD}}^{\mathrm{self}}=\mathrm{CE}(\mathbf{p}$
$\displaystyle{}_{y}^{self},\mathbf{y}_{\mathrm{assign}}^{\mathrm{self}})$

here $\mathbf{W}_{self}^{T}$ is a trainable FC layer,
$\mathbf{y}_{\mathrm{assign}}^{\mathrm{self}}$ is the node’s GT target
according to the VAT’s bipartite matching results.

Moreover, the goal of the two connectivity heads are to learn an explicit
layout model in parallel to dynamically choose the left and right token for
each node. This task is straightforward: for each token in VAT’s predicted
$\mathbf{y}_{\mathrm{vat}}^{\prime}$, choose another token from
$\mathbf{y}_{\mathrm{vat}}^{\prime}$ as its left side or right side token, so
that PGD can adopt this connectivity as an edge and build a overall directed
graph. The selection processes are modeled as attentions from the initial
query $\mathbf{Q}_{0}$ over the computed layout features:

$\displaystyle\mathcal{L}_{\mathrm{PGD}}^{\mathrm{left}}$
$\displaystyle=\mathrm{CE}(\mathrm{Softmax}(\mathbf{Q}_{0}^{T}\cdot\mathbf{Q}_{J}^{left}),\mathbf{y}_{\mathrm{assign}}^{\mathrm{left}}),$
(7) $\displaystyle\mathcal{L}_{\mathrm{PGD}}^{\mathrm{right}}$
$\displaystyle=\mathrm{CE}(\mathrm{Softmax}(\mathbf{Q}_{0}^{T}\cdot\mathbf{Q}_{J}^{right}),\mathbf{y}_{\mathrm{assign}}^{\mathrm{right}})$

Similar to $\mathbf{y}_{\mathrm{assign}}^{\mathrm{self}}$, once the bipartite
matching in VAT is done, all nodes’ GT connectivity target
$\mathbf{y}_{\mathrm{assign}}^{\mathrm{left}}$,
$\mathbf{y}_{\mathrm{assign}}^{\mathrm{right}}$ can be obtained. The loss of
whole PGD is:

$\mathcal{L}_{\mathrm{PGD}}=\mathcal{L}_{\mathrm{PGD}}^{\mathrm{self}}+\mathcal{L}_{\mathrm{PGD}}^{\mathrm{left}}+\mathcal{L}_{\mathrm{PGD}}^{\mathrm{right}}$
(8)

And the overall training loss of NAMER is:

$\mathcal{L}_{all}=\mathcal{L}_{\mathrm{VAT}}+\lambda\cdot\mathcal{L}_{\mathrm{PGD}}$
(9)

where $\lambda$ is a hyper-parameter for balancing the training of VAT and
PGD.

###  3.4 Path Selection

After PGD, we can obtain all revised nodes
$\mathrm{V}=\arg\max(\mathbf{p}_{y}^{self})$, and connectivity scores from
node $\mathrm{V}_{i}$ to node $\mathrm{V}_{j}$:

$\mathrm{E}_{ij}=\mathrm{Softmax}(\mathbf{Q}_{0}(j)^{T}\cdot\mathbf{Q}_{J}^{left})_{i}+\mathrm{Softmax}(\mathbf{Q}_{0}(i)^{T}\cdot\mathbf{Q}_{J}^{right})_{j}$
(10)

The directed edge score $\mathrm{E}_{ij}$ is the sum of the right connectivity
score from node $\mathrm{V}_{i}$ to node $\mathrm{V}_{j}$ and the left
connectivity score from node $\mathrm{V}_{j}$ to node $\mathrm{V}_{i}$. To
minimize unnecessary computations, we delete edges with scores below a hyper-
parameter $\epsilon$ (if it fails to partition the graph into two disconnected
subgraphs).

So far, we have built a directed acyclic graph
$\textless\mathrm{V},\mathrm{E}\textgreater$ in parallel, with all positive
edge weights. Our path selection strategy is to find the longest path from the
node <sos>to the node <eos>. Finding the longest path in a DAG [3, 12] with
positive edges can be done using a dynamic programming approach based on a
topological ordering of the vertices, under the time complexity of $O(V+E)$.
It’s extremely fast because the edges of our graph are sparse, so the number
of E is nearly equal to the number of V, and V is a small value in HMER task.

##  4 Experiments

###  4.1 Datasets

We use two benchmark datasets, CROHME and HME100K, to evaluate our proposed
method.

CROHME Dataset [30, 31, 29] is the most widely-used public dataset in the
field of HMER, sourced from the Competition on Recognition of Online
Handwritten Mathematical Expressions. The CROHME training set contains 8,836
HMEs within 111 symbol classes. Additionally, there are three testing sets:
CROHME 2014, 2016, and 2019, comprising 986, 1,147, and 1,199 HMEs,
respectively. Each HME in the CROHME dataset is stored in InkML format,
capturing the trajectory coordinates of handwritten strokes. We convert this
trajectory information from the InkML files into image format for both
training and testing purposes.

HME100K [46] is a real scene HME dataset, encompassing 74,502 images for
training and 24,607 images for testing. With a total of 249 symbol classes,
these images originate from tens of thousands of writers and are primarily
captured using cameras. As a result, HME100K exhibits a higher degree of
authenticity and realism, showcasing variations in color, blur, and complex
backgrounds.

###  4.2 Implementation Details

Following most previous works [47, 49, 53, 54, 28, 4, 27, 40, 52], we keep
using the same DenseNet [18] architecture as our encoder. We use a two layer
Transformer in PGD, with feed-forward dimension ratio set to 2. And both the
$\mathrm{Conv}$ dimension in VAT and Transformer dimension in PGD are set to
256. The hyper-parameter $\lambda$ in Eq. 9 is set to 0.5 by default, and
$\epsilon$ is set to 0.5 for path selection. To improve the training speed of
NAMER, the estimate set $\mathbf{T}$ for VAT is obtained using a pretrained
DWAP [47] before training.

Our model is implemented on PyTorch. All experiments are conducted on four
32GB Nvidia Tesla V100 GPUs with batch size 32. The learning rate starts from
0 and monotonously increases to 2e-4 at the end of the first epoch and decays
to 2e-7 following the cosine schedules, with an Adam [21] optimizer. The
training epoch is set to 240 for the CROHME dataset, and 40 for the HME100K
dataset. For fair comparisons, we present results with and without data
augmentations. Only simple random scales and rotations are used for
augmentations.

###  4.3 Evaluation Metrics

We adopt the widely used Expression recognition rate (ExpRate) as the
evaluation metrics. ExpRate is defined as the percentage of correctly
recognized expressions. And ExpRates under $\leq 1$ and $\leq 2$ are also used
as a supplementary reference, indicating that the expression recognition rate
is tolerable at most one or two symbol-level errors. Besides, GPU memory usage
(MEM) and frames per sencond (FPS) are used to measure the inference costs.

Table 1: Performance of our model and other state-of-the-art methods on three CROHME datasets. ∗ indicates using other informations, e.g., online stoke trajectory coordinates or extra category of symbols. † indicates our reproduced result on the same training settings with NAMER. All results are reported as a percentage (%). Our model achieves the best performance on all CROHME datasets. Methods | Aug | CROHME 2014 | CROHME 2016 | CROHME 2019
---|---|---|---|---
ExpRate $\uparrow$ |  $\leq 1$ $\uparrow$ |  $\leq 2$ $\uparrow$ |  ExpRate $\uparrow$ |  $\leq 1$ $\uparrow$ |  $\leq 2$ $\uparrow$ |  ExpRate $\uparrow$ |  $\leq 1$ $\uparrow$ |  $\leq 2$ $\uparrow$
UPV[30] | No | 37.22 | 44.22 | 47.26 | - | - | - | - | - | -
TOKYO[30] | No | - | - | - | 43.94 | 50.91 | 53.70 | - | - | -
WAP[50] | No | 46.55 | 61.16 | 65.21 | 44.55 | 57.10 | 61.55 | - | - | -
PAL[42] | No | 39.66 | 56.80 | 65.11 | - | - | - | - | - | -
TAP∗[48] | No | 48.47 | 63.28 | 67.34 | 44.81 | 59.72 | 62.77 | - | - | -
DWAP[47] | No | 50.10 | - | - | 47.50 | - | - | - | - | -
DWAP-MSA[47] | No | 52.80 | 68.10 | 72.00 | 50.10 | 63.80 | 67.40 | 47.70 | 59.50 | 63.30
DLA[26] | No | 49.85 | - | - | 47.34 | - | - | - | - | -
PAL-V2[43] | No | 48.88 | 64.50 | 69.78 | 49.61 | 64.08 | 70.27 | - | - | -
WS-WAP[35] | No | 53.65 | - | - | 51.96 | 64.34 | 70.10 | - | - | -
TreeDecoder[49] | No | 49.10 | 64.20 | 67.80 | 48.50 | 62.30 | 65.30 | 51.40 | 66.10 | 69.10
BTTR[53] | No | 53.96 | 66.02 | 70.28 | 52.31 | 63.90 | 68.61 | 52.96 | 65.97 | 69.14
SSD[41] | No | 53.10 | 66.00 | 71.30 | 51.70 | 64.70 | 70.10 | 53.20 | 67.60 | 73.60
TSDNet[54] | No | 54.70 | 68.85 | 74.48 | 52.48 | 68.26 | 73.41 | 56.34 | 72.97 | 77.84
SAN[46] | No | 56.20 | 72.60 | 79.20 | 53.60 | 69.60 | 76.80 | 53.50 | 69.30 | 70.1
ABM[4] | No | 56.85 | 73.73 | 81.24 | 52.92 | 69.66 | 78.73 | 53.96 | 71.06 | 78.65
CAN-ABM[27] | No | 57.26 | 74.52 | 82.03 | 56.15 | 72.71 | 80.30 | 55.96 | 72.73 | 80.57
TD-V2[40] | No | 53.62 | - | - | 55.18 | - | - | 58.72 | - | -
CoMER[52]† | No | 58.57 | - | - | 57.89 | - | - | 59.71 | - | -
SAM-CAN[28] | No | 58.01 | - | - | 56.67 | - | - | 57.96 | - | -
GCN[51]∗ | No | 60.00 | - | - | 58.94 | - | - | 61.63 | - | -
NAMER | No | 60.51 | 75.03 | 82.25 | 60.24 | 73.50 | 80.21 | $\mathbf{61.72}$ | 75.31 | 82.07
BTTR[53]† | Yes | 55.17 | 67.85 | 72.11 | 56.58 | 68.88 | 74.19 | 59.55 | 72.23 | 76.06
Ding _et al_. [13] | Yes | 58.72 | - | - | 57.72 | 70.01 | 76.37 | 61.38 | 75.15 | 80.23
CAN-ABM[27]† | Yes | 62.47 | - | - | 59.81 | - | - | 61.72 | - | -
CoMER[52] | Yes | 59.33 | 71.70 | 75.66 | 59.81 | 74.37 | 80.30 | 62.97 | 77.40 | 81.40
CoMER[52]† | Yes | 63.35 | - | - | 60.06 | - | - | 62.21 | - | -
GCN[51]∗ | Yes | 64.47 | - | - | 62.51 | - | - | 64.05 | - | -
NAMER | Yes | 64.77 | 78.38 | 85.08 | 64.17 | 76.11 | 83.26 | 66.64 | 80.32 | 85.15

###  4.4 Comparison with State-of-the-Art

To demonstrate the superiority of our method, we compare it with previous SOTA
methods. Since most of the previous methods do not use data augmentation, so
we mainly focus on the results produced without data augmentation.

Tab. 1 displays results on the CROHME datasets. Our method attains SOTA
performance across all CROHME datasets. Presently, CoMER [52] stands as the
most stable SOTA HMER method. Notably, NAMER outperforms CoMER by 1.94%,
2.35%, and 1.49% on CROHME 2014, 2016, and 2019, respectively, without
augmentations. With augmentations, NAMER exhibits even greater superiority,
surpassing CoMER by 1.42%, 4.11%, and 3.67% on CROHME 2014, 2016, and 2019,
respectively. These results underscore the effectiveness of the proposed
method. NAMER also achieves comparable performance with the very recent
published method GCN [51], which uses a manually defined category of symbols
for supervision. Note that NAMER can be easily extended to this kind of extra
supervisions, with an extra head in PGD.

Table 2: Results on the HME100K dataset. † indicates our reproduced result using the same training settings with NAMER. Following previous methods’ experimental settings on HME100K, no augmentation is added during training. All results are reported as a percentage (%). NAMER achieves the best performance. Methods | HME100K
---|---
ExpRate $\uparrow$ |  $\leq 1$ $\uparrow$ |  $\leq 2$ $\uparrow$
DWAP [47] | 61.85 | 70.63 | 77.14
TreeDecoder [49] | 62.60 | 79.05 | 85.67
BTTR [53] | 64.10 | - | -
TD-V2 [40]† | 66.44 | - | -
SAN [46] | 67.10 | - | -
CAN [27] ] | 67.31 | 82.93 | 89.17
CoMER[52]† | 67.90 | 79.84 | 88.03
NAMER | 68.52 | 83.10 | 89.30

Tab. 2 shows the ExpRate on the HME100K datasets. We compare our proposed
method with several milestone works. Specifically, NAMER outperforms BTTR
[53], SAN [46], CAN [27], CoMER[52] by 4.42%, 1.42%, 1.21% and 0.62%,
respectively, showing the effectiveness in handling challenging real scene
photos.

###  4.5 Inference Cost

To assess the efficiency of NAMER, we measured its inference costs on the
CROHME dataset using a single Nvidia Tesla V100. We compared it with three
representative methods: RNN-based DWAP [47], tree structure-based TreeDecoder
[49], transformer-based CoMER [52]. For each method, we calculate the average
inference time and maximum GPU memory usage across all test sets, providing
the overall FPS, MEM, FLOPs, and encoder & decoder forward times. For NAMER,
the decoder forward time is the sum of VAT, PGD, and Path Selection modules.
As shown in Tab. 3, NAMER achieves significant speedups: it is 13.7$\times$
and 6.7$\times$ faster than CoMER in decoding time and overall inference time,
respectively. Even when compared with the simplest RNN-based DWAP baseline,
NAMER still achieves a speedup of 2.36$\times$ and 1.61$\times$ in decoding
time and overall FPS. Moreover, NAMER achieves significant lower memory usage,
while AR models require a beam search decoding that use more memory. These
improvements indicate the efficiency of the proposed non-autoregressive
design.

Table 3: Comparison of different methods in inference costs. While keeping the same encoder, NAMER achieves a 13.7x speedup in decoding and a 6.7x speedup in overall FPS compared to the previous SOTA method CoMER. (Details of NAMER’s decoding time 15.02ms: VAT=3.0ms, PGD=9.62ms, Path=2.4ms) Methods |  Time (ms) $\downarrow$ |  FLOPs (G) $\downarrow$ |  FPS $\uparrow$ |  Memory (M) $\downarrow$
---|---|---|---|---
Encoder |  Decoder
DWAP [47] | 18.31 | 35.41 | 30.30 | 18.62 | 4945
TreeDecocder [49] | 18.31 | 115.81 | 55.39 | 7.46 | 6290
CoMER [52] | 18.31 | 206.12 | 174.33 | 4.46 | 13441
NAMER | 18.31 | 15.02 | 11.39 | 30.00 | 2569

###  4.6 Ablation Study

Training targets of VAT. VAT is trained under the bipartite matching results
between a pretrained HMER model $\mathrm{M}^{*}$ and the predicted visible
probability $\mathbf{P}$. Here we carefully explore the influence of different
pretrained HMER models and different bipartite matching area for each token.
As shown in Tab. 4, using different methods for generating bipartite matching
targets is not sensitive for our model training, the performance is almost the
same. According to Tab. 1, performance of these for methods range across ~10%
of ExpRate, proving that the key goal of VAT is to instantiate visible tokens
and the approximate location is enough for HMER. In fact, bipartite matching
training allows NAMER to learn optimal solutions within a tolerance window,
enabling VAT to robustly learn from various coarse targets. Consequently,
methods with lower ExpRate may outperform CoMER as a VAT target slightly.

In Tab. 5, we investigate the effective matching area for each token during
bipartite matching, varying it from 3 to 7. The results indicate that a kernel
of $5\times 5$ is the best for bipartite matching, it is make sense because a
character symbol is usually ~40 pixel. (Remind that bipartite matching is
performed at the feature map of $\frac{H}{8}\times\frac{W}{8}$)

Table 4: Ablation study on different bipartite matching targets of VAT. All results are obtained on CROHME 2014 using the proposed NAMER without augmentations. VAT Target |  DWAP [47] |  BTTR [53] |  CAN [27] |  CoMER [52]
---|---|---|---|---
ExpRate | 60.51 | 60.65 | 60.55 | 60.35
Table 5: Ablation study on kernel sizes for computing VAT’s matching cost matrix. $k_{m}\times k_{m}$ |  3$\times$ 3 |  5$\times$ 5 |  7$\times$ 7
---|---|---|---
ExpRate | 59.74 | 60.51 | 60.35
Table 6: Ablation study on different heads of PGD. All results are obtained on CROHME dataset without data augmentations. LCH |  RCH |  SCH |  ExpRate $\uparrow$
---|---|---|---
2014 |  2016 |  2019
✓ |  |  | 55.03 | 54.84 | 57.13
| ✓ |  | 54.62 | 55.01 | 56.38
✓ | ✓ |  | 55.94 | 56.32 | 58.38
✓ |  | ✓ | 58.88 | 58.15 | 60.22
| ✓ | ✓ | 57.26 | 57.98 | 58.97
✓ | ✓ | ✓ | 60.51 | 60.24 | 61.72

Effectiveness of modules in PGD. To study the contributions of each head in
PGD, we conduct detail ablation experiments on Tab. 6, where we alternately
control the use of three heads: LCH, RCH, and SCH. The experimental results
illustrate three key conclusions:

  1. 1.

With only one head LCH or RCH, it can achieve comparable baselines, proving
the effectiveness of both the VAT and connectivity prediction heads;

  2. 2.

The merging of left and right connectivity scores can help enhance the
accuracy of the graph structure (~2.5% with SCH and ~1% without SCH);

  3. 3.

Based on VAT’s language feature, position feature, and image feature, SCH can
indeed do a vision-language modeling for correcting misclassified tokens in
VAT, upgrading the ExpRate by ~3.5% in average.

###  4.7 Case Study

We select four typical examples in Fig. 6 to assess NAMER’s capabilities and
make comparsions. The first two images exhibit ambiguities between “2”-“z”,
“y”-“j”, and “5”-“s”. In NAMER, VAT makes initial mistakes on these symbols,
yet the PGD effectively rectifies them by leveraging comprehensive visual and
linguistic contexts. Conversely, CoMER tends to misclassify these ambiguous
symbols. In the third case, VAT misidentifies an additional symbol “X”, which
PGD successfully eliminates as a redundant token. Notably, NAMER demonstrates
a more consistent ability to recognize local relations, correctly identifying
the two adjacent “_” symbols, whereas CoMER fails to do so accurately. The
final case highlights the error accumulation issue observed in AR-based
methods; CoMER misidentifies “$|$” as “1” and subsequently fails to recognize
the subsequent “$|$” tokens correctly. In contrast, NAMER exhibits superior
robustness and capability in handling such situations. Additionally, we have
observed that when PGD rectifies symbols, the cross and self-attention maps in
Eq. 5 usually focus on the corresponding image context areas and the correct
VAT tokens. This demonstrates the capability and interpretability of PGD.

###  4.8 Limitation

Due to the incapability of PGD to insert new tokens, limitations of NAMER lies
in handling extreme cases where VAT fails to recall relational tokens (e.g.,
‘^’, ‘_’). These cases are rare, typically arising from serious handwriting
errors that necessitate human understanding of context for correction (e.g.,
writing x^2+y^2=R^2 as x^2+y2=R^2). Solving such cases could be a future work,
specifically by empowering PGD with insertion capability.

Figure 6: Visualization and comparison of NAMER results at different stages:
the first three image rows display visualizations from NAMER, while the last
three text rows present comparisons with CoMER. Characters in red color on
line PGD denote that PGD has made a correction on VAT’s results.

##  5 Conclusion

This paper presents NAMER, a novel NAR Modeling paradigm designed for HMER.
Unlike prevailing AR approaches, NAMER adopts a bottom-up, coarse-to-fine
methodology. NAMER firstly predicts visible symbols and local relations, then
refines them and establishes connectivities in a NAR manner. To achieve this,
a Visual Aware Tokenizer, a Parallel Graph Decoder, and a Bipartite Matching
Training strategy are elaborately designed. To the best of our knowledge,
NAMER is the first work to explore NAR in HMER, leveraging comprehensive
visual and linguistic contexts. Experiments have demonstrated the
effectiveness and efficiency of our method. In future work, we intend to
extend the capabilities of NAMER to tackle more intricate structural
recognition tasks, such as interpreting chemical structure images and complex
CAD images.

## References

  * [1] Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
  * [2] Bautista, D., Atienza, R.: Scene text recognition with permuted autoregressive sequence models. In: European Conference on Computer Vision. pp. 178–196. Springer (2022)
  * [3] Bellman, R.: On a routing problem. Quarterly of applied mathematics 16(1), 87–90 (1958)
  * [4] Bian, X., Qin, B., Xin, X., Li, J., Su, X., Wang, Y.: Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 113–121 (2022)
  * [5] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)
  * [6] Chan, K.F., Yeung, D.Y.: Elastic structural matching for online handwritten alphanumeric character recognition. In: Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No. 98EX170). vol. 2, pp. 1508–1511. IEEE (1998)
  * [7] Chan, K.F., Yeung, D.Y.: Error detection, error correction and performance evaluation in on-line mathematical expression recognition. Pattern Recognition 34(8), 1671–1684 (2001)
  * [8] Chan, W., Jaitly, N., Le, Q., Vinyals, O.: Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 4960–4964. IEEE (2016)
  * [9] Chen, X., Jin, L., Zhu, Y., Luo, C., Wang, T.: Text recognition in the wild: A survey. ACM Computing Surveys (CSUR) 54(2), 1–35 (2021)
  * [10] Cheng, H., Liu, C., Hu, P., Zhang, Z., Ma, J., Du, J.: Bidirectional trained tree-structured decoder for handwritten mathematical expression recognition. arXiv preprint arXiv:2401.00435 (2024)
  * [11] Deng, Y., Kanervisto, A., Ling, J., Rush, A.M.: Image-to-markup generation with coarse-to-fine attention. In: International Conference on Machine Learning. pp. 980–989. PMLR (2017)
  * [12] Dijkstra, E.: A note on two problems in connexion with graphs. Numerische Mathematik 1(1), 269–271 (1959)
  * [13] Ding, H., Chen, K., Huo, Q.: An encoder-decoder approach to handwritten mathematical expression recognition with multi-head attention and stacked decoder. In: Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part II 16. pp. 602–616. Springer (2021)
  * [14] Graves, A., Fetrnández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on Machine learning. pp. 369–376 (2006)
  * [15] Gu, J., Bradbury, J., Xiong, C., Li, V., Socher, R.: Non-autoregressive neural machine translation. In: International Conference on Learning Representations (ICLR) (2018)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [17] Hu, L., Zanibbi, R.: Hmm-based recognition of online handwritten mathematical symbols using segmental k-means initialization and a modified pen-up/down feature. In: 2011 International Conference on Document Analysis and Recognition. pp. 457–462. IEEE (2011)
  * [18] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
  * [19] Keshari, B., Watt, S.: Hybrid mathematical symbol recognition using support vector machines. In: Ninth International Conference on Document Analysis and Recognition (ICDAR 2007). vol. 2, pp. 859–863. IEEE (2007)
  * [20] Kim, S., Hori, T., Watanabe, S.: Joint ctc-attention based end-to-end speech recognition using multi-task learning. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 4835–4839. IEEE (2017)
  * [21] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [22] Kosmala, A., Rigoll, G., Lavirotte, S., Pottier, L.: On-line handwritten formula recognition using hidden markov models and context dependent graph grammars. In: Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR’99 (Cat. No. PR00318). pp. 107–110. IEEE (1999)
  * [23] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012)
  * [24] Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly 2(1-2), 83–97 (1955)
  * [25] Lavirotte, S., Pottier, L.: Mathematical formula recognition using graph grammar. In: Document Recognition V. vol. 3305, pp. 44–52. SPIE (1998)
  * [26] Le, A.D.: Recognizing handwritten mathematical expressions via paired dual loss attention network and printed mathematical expressions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 566–567 (2020)
  * [27] Li, B., Yuan, Y., Liang, D., Liu, X., Ji, Z., Bai, J., Liu, W., Bai, X.: When counting meets hmer: counting-aware network for handwritten mathematical expression recognition. In: European Conference on Computer Vision. pp. 197–214. Springer (2022)
  * [28] Liu, Z., Yuan, Y., Ji, Z., Bai, J., Bai, X.: Semantic graph representation learning for handwritten mathematical expression recognition. In: International Conference on Document Analysis and Recognition. pp. 152–166. Springer (2023)
  * [29] Mahdavi, M., Zanibbi, R., Mouchere, H., Viard-Gaudin, C., Garain, U.: Icdar 2019 crohme+ tfd: Competition on recognition of handwritten mathematical expressions and typeset formula detection. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1533–1538. IEEE (2019)
  * [30] Mouchere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: Icfhr 2014 competition on recognition of on-line handwritten mathematical expressions (crohme 2014). In: 2014 14th International Conference on Frontiers in Handwriting Recognition. pp. 791–796. IEEE (2014)
  * [31] Mouchère, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: Icfhr2016 crohme: Competition on recognition of online handwritten mathematical expressions. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). pp. 607–612. IEEE (2016)
  * [32] Qiao, Z., Zhou, Y., Wei, J., Wang, W., Zhang, Y., Jiang, N., Wang, H., Wang, W.: Pimnet: a parallel, iterative and mimicking network for scene text recognition. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 2046–2055 (2021)
  * [33] Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence 39(11), 2298–2304 (2016)
  * [34] Shi, B., Yang, M., Wang, X., Lyu, P., Yao, C., Bai, X.: Aster: An attentional scene text recognizer with flexible rectification. IEEE transactions on pattern analysis and machine intelligence 41(9), 2035–2048 (2018)
  * [35] Truong, T.N., Nguyen, C.T., Phan, K.M., Nakagawa, M.: Improvement of end-to-end offline handwritten mathematical expression recognition by weakly supervised learning. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). pp. 181–186. IEEE (2020)
  * [36] Tu, Z., Lu, Z., Liu, Y., Liu, X., Li, H.: Modeling coverage for neural machine translation. arXiv preprint arXiv:1601.04811 (2016)
  * [37] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [38] Wan, Z., Xie, F., Liu, Y., Bai, X., Yao, C.: 2d-ctc for scene text recognition. arXiv preprint arXiv:1907.09705 (2019)
  * [39] Winkler, H.J.: Hmm-based handwritten symbol recognition using on-line and off-line features. In: 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings. vol. 6, pp. 3438–3441. IEEE (1996)
  * [40] Wu, C., Du, J., Li, Y., Zhang, J., Yang, C., Ren, B., Hu, Y.: Tdv2: A novel tree-structured decoder for offline mathematical expression recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 2694–2702 (2022)
  * [41] Wu, J., Hu, J., Chen, M., Dai, L., Niu, X., Wang, N.: Structural string decoder for handwritten mathematical expression recognition. In: 2022 26th International Conference on Pattern Recognition (ICPR). pp. 3246–3251. IEEE (2022)
  * [42] Wu, J.W., Yin, F., Zhang, Y.M., Zhang, X.Y., Liu, C.L.: Image-to-markup generation via paired adversarial learning. In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10–14, 2018, Proceedings, Part I 18. pp. 18–34. Springer (2019)
  * [43] Wu, J.W., Yin, F., Zhang, Y.M., Zhang, X.Y., Liu, C.L.: Handwritten mathematical expression recognition via paired adversarial learning. International Journal of Computer Vision 128, 2386–2401 (2020)
  * [44] Wu, J.W., Yin, F., Zhang, Y.M., Zhang, X.Y., Liu, C.L.: Graph-to-graph: towards accurate and interpretable online handwritten mathematical expression recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 2925–2933 (2021)
  * [45] Yang, X., Qiao, Z., Zhou, Y., Wang, W.: Ipad: Iterative, parallel, and diffusion-based network for scene text recognition. arXiv preprint arXiv:2312.11923 (2023)
  * [46] Yuan, Y., Liu, X., Dikubab, W., Liu, H., Ji, Z., Wu, Z., Bai, X.: Syntax-aware network for handwritten mathematical expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4553–4562 (2022)
  * [47] Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 2018 24th international conference on pattern recognition (ICPR). pp. 2245–2250. IEEE (2018)
  * [48] Zhang, J., Du, J., Dai, L.: Track, attend, and parse (tap): An end-to-end framework for online handwritten mathematical expression recognition. IEEE Transactions on Multimedia 21(1), 221–233 (2018)
  * [49] Zhang, J., Du, J., Yang, Y., Song, Y.Z., Wei, S., Dai, L.: A tree-structured decoder for image-to-markup generation. In: International Conference on Machine Learning. pp. 11076–11085. PMLR (2020)
  * [50] Zhang, J., Du, J., Zhang, S., Liu, D., Hu, Y., Hu, J., Wei, S., Dai, L.: Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognition 71, 196–206 (2017)
  * [51] Zhang, X., Ying, H., Tao, Y., Xing, Y., Feng, G.: General category network: Handwritten mathematical expression recognition with coarse-grained recognition task. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2023)
  * [52] Zhao, W., Gao, L.: Comer: Modeling coverage for transformer-based handwritten mathematical expression recognition. In: European Conference on Computer Vision. pp. 392–408. Springer (2022)
  * [53] Zhao, W., Gao, L., Yan, Z., Peng, S., Du, L., Zhang, Z.: Handwritten mathematical expression recognition with bidirectionally trained transformer. In: Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part II 16. pp. 570–584. Springer (2021)
  * [54] Zhong, S., Song, S., Li, G., Chan, S.H.G.: A tree-based structure-aware transformer decoder for image-to-markup generation. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 5751–5760 (2022)

Supplementary Materials of
NAMER: Non-Autoregressive Modeling for Handwritten Mathematical Expression
Recognition

Chenyu Liu Jia Pan Jinshui Hu Baocai Yin Bing Yin
Mingjun ChenCong LiuJun Du† Qingfeng Liu

22footnotetext: † Corresponding author.

##  6 NAMER Label Token Details

In this section, we provide details of labels in NAMER, including all local
relation tokens and the attached imaginary tokens for them.

Table 7: Local Relation Tokens in VAT: Our defined imaginary relationship symbols are common to both the CROHME [30, 31, 29] dataset and the HME100K [46] dataset. Due to HME100K being a more complex dataset with a larger set of symbol classes, it contains a larger number of handwritten structural elements compared to CROHME. | Dataset | Label Symbol
---|---|---
IRS | All | ^
_
\limits
HSE |  CROHME [30, 31, 29] | \frac
\sqrt
HSE | HME100K [46] | \frac
\sqrt
\dot
\ddot
\boxed
\widehat
\overline
\xlongequal
\textcircled
\xrightarrow
\overrightarrow

###  6.1 Local Relation Tokens in VAT

As mentioned in Section 3.2, local relation tokens in the Visual Aware
Tokenizer (VAT) consists of Handwritten Structural Elements (HSE) and
Imaginary Relationship Symbols (IRS). Details of HSE and IRS in different
datasets are shown in Tab. 7.

###  6.2 Attached Imaginary Tokens in PGD

As outlined in Section 3.3, after obtaining all visible tokens
$\mathbf{y}_{\mathrm{vat}}$ from VAT, the respective attached imaginary tokens
related to the local relation tokens in $\mathbf{y}_{\mathrm{vat}}$ are added.
For instance, in a mathematical expression $x^{yz}+1$ with VAT results {“x”,
“^”, “y”, “z”, “+”, “1”, }, the ending token “}” related to “^” must be added
to fully recover the upper corner relation structure.

The definition of an attached imaginary token is straightforward: it
represents the ending token for the local relation token, as illustrated in
Tab. 8. It’s important to note that most local relation tokens have a single
“}” ending imaginary token, whereas “\frac” and “\sqrt” have two ending
imaginary tokens. We have defined all attached tokens as “}”, ensuring
uniformity with our label tokens in LaTeX.

Note that all the attached imaginary tokens are automatically generated by
scanning all label strings using a simple Python script, not manually
maintained. This script is written based on LaTeX grammar of special tokens.

Table 8: Details of all attached imaginary tokens. It is defined as the ending tokens for each local relation token. Local Relation Token |  Attached Imaginary Token
---|---
^ | one ’}’ for ending
_ | one ’}’ for ending
\limits | one ’}’ for ending
\dot | one ’}’ for ending
\ddot | one ’}’ for ending
\boxed | one ’}’ for ending
\widehat | one ’}’ for ending
\overline | one ’}’ for ending
\xlongequal | one ’}’ for ending
\textcircled | one ’}’ for ending
\xrightarrow | one ’}’ for ending
\overrightarrow | one ’}’ for ending
\frac | one ’}’ for numerator ending,
one ’}’ for denominator ending
\sqrt | one ’}’ for radicand ending,
one ’}’ for index ending

⬇

1# Python code for dynamic assignment in VAT. It’s an example for one training
sample.

2# And it’s easy to extend this code to support batch training.

3import torch

4from scipy.optimize import linear_sum_assignment

5

6def vat_matching(P_vat, y_label, T_dwap, vocab_size, assign_kernel=(5, 5)):

7 ’’’

8 Parameters

9 \----------------

10 P_vat: FloatTensor, [K+1, H/8, W/8]

11 predicted probabilities of VAT, which is the Eqn (2)’s \mathbf{P}

12 y_label: LongTensor, [L, 1]

13 all visible tokens, y_1, y_2, ..., y_L

14 T_dwap: LongTensor, [L, 2]

15 estimated spatial positions of y_label using a pretrained DWAP

16 vocab_size: int

17 vocabulary size

18 assign_kernel: tuple (int, int)

19 effective local matching window k_m * k_m for P_vat and T_vat

20

21 Returns

22 \----------------

23 P_target: LongTensor, [H/8, W/8]

24 The bipartite matching result of training target \mathbf{P}^* for VAT

25 ’’’

26 _, H, W = P_vat.shape # get sizes of feature map

27 L, _ = y_label.shape # get length of visible tokens

28

29 assign_pad = (assign_kernel[0]//2, assign_kernel[1]//2) # compute paddings

30 y_indices = torch.arange(L).cuda() # a tensor for indexing

31

32 # convert T_vat to a 3D matrix

33 T_vat_mat = torch.zeros([L, H, W]).cuda()

34 T_vat_mat[y_indices, T_dwap[y_indices, 0], T_dwap[y_indices, 1]] = 1.0

35

36 # set effective matching positions using maxpooling

37 T_vat_mat = torch.nn.functional.max_pool2d(T_vat_mat.unsqueeze(0),
kernel_size=assign_kernel, stride=(1,1), padding=assign_pad)[0]

38

39 # computing overall distance matrix for assigning H*W predictions to T
tokens

40 dist_mat = torch.zeros([L, H, W]).cuda()

41 dist_mat[y_indices] = (P_vat[y_label[y_indices]] -
T_vat_mat[y_indices]).abs()

42 # limiting matching within a k_m x k_m window for each token

43 dist_mat = dist_mat * T_vat_mat + (1 - T_vat_mat) * 1e6

44

45 # hungarian algorithm for optimal assignment

46 row_ind, col_ind = linear_sum_assignment(dist_mat.view(L,
-1).cpu().numpy())

47 h_ind = col_ind // W

48 w_ind = col_ind % W

49

50 # generating P_target for training VAT module

51 P_target = (torch.ones([H, W]) * vocab_size).long() # initialized as all
\varnothing.

52 P_target[h_ind, w_ind] = y_label[row_ind]

53 return P_target’

Listing 1: Example Python Code for Dynamic Assignment Approach in VAT

##  7 Details of Dynamic Assignment Process

In this section, we present detailed code for the proposed bipartite matching-
based dynamic assignment approach, which is initially introduced in Section
3.2 to generate training targets for VAT.

The code snippet in LABEL:python_code outlines the matching and assignment
process. Specifically, given the estimated spatial positions
$\mathbf{T}\in\mathbb{Z}^{L\times 2}$, for the parallel matching of all
batches and tokens, we firstly convert $\mathbf{T}$ to a 3D matrix resembling
an indicator function (line 34 in LABEL:python_code). Subsequently, a max-
pooling operation is applied to achieve local $k_{m}\times k_{m}$ window-based
matching (line 37). Next, the overall distance matrix is computed between
VAT’s predicted probabilities $\mathbf{P}$ and the converted
$\mathbf{T}_{mat}$. Finally, the Hungarian algorithm is employed for optimal
assignment, resulting in the training target $\mathbf{P}^{*}$ for VAT.

##  8 Ablation Study on Path Selection

Path selection involves directed acyclic graph (DAG) construction and the
longest path selection. We performed ablations on the edge $\mathrm{E}_{ij}$
of Eq. (10), finding a balanced 1:1 ratio of left-to-right (L2R) and right-to-
left (R2L) score yields the optimal results, as shown in Tab. 9.

Table 9: Ablation study on edge value $\mathrm{E}_{ij}$ of DAG construction for Path Selection. L2R:R2L |  1:0 |  1:0.5 |  1:1 |  0.5:1 |  0:1
---|---|---|---|---|---
ExpRate | 58.88 | 59.43 | 60.51 | 59.13 | 57.26