# New constraints on the minimum mass for thermonuclear lithium burning in
brown dwarfs ††thanks: Based on observations made with the Gran Telescopio
Canarias (GTC), installed in the Spanish Observatorio del Roque de los
Muchachos of the Instituto de Astrofísica de Canarias, in the island of La
Palma (programmes GTC100-20B and GTC51-21B).

E. L. Martín,1,2,3 N. Lodieu,1,2 C. del Burgo,4
1Instituto de Astrofísica de Canarias (IAC), Calle Vía Láctea s/n, E-38200 La
Laguna, Tenerife, Spain
2Departamento de Astrofísica, Universidad de La Laguna (ULL), E-38206 La
Laguna, Tenerife, Spain
3Consejo Superior de Investigaciones Científicas, E-28006 Madrid, Spain
4Instituto Nacional de Astrofísica, Óptica y Electrónica, Luis Enrique Erro 1,
Sta. Ma. Tonantzintla, Puebla, Mexico
E-mail<EMAIL_ADDRESS>(EM)

(Accepted 2021 10 08. Received 2021 06 04; in original form )

###### Abstract

The theory of substellar evolution predicts that there is a sharp mass
boundary between lithium and non-lithium brown dwarfs, not far below the
substellar-mass limit. The imprint of thermonuclear burning is carved on the
surface lithium abundance of substellar-mass objects during the first few
hundred million years of their evolution, leading to a sharp boundary between
lithium and non-lithium brown dwarfs, so-called, the lithium test. The
theoretical predictions can be tested by comparing with observations of
lithium in the individual components of binaries with dynamical masses
measured from orbital motions. New optical spectroscopic observations of the
binaries DENIS J063001.4$-$184014AB and DENIS J225210.7$-$173013AB obtained
using the 10.4-m Gran Telescopio de Canarias are reported here. They allow us
to re-determine their combined optical spectral types (M9.5 and L6.5,
respectively) and to search for the presence of the Li I resonance doublet.
The non detection of the Li I feature in the combined spectrum of DENIS
J063001.4$-$184014AB is converted into estimates for the depletion of lithium
in the individual components of this binary system. In DENIS
J225210.7$-$173013AB we report the detection of a weak Li I feature which we
tentatively ascribe as arising from the contribution of the T3.5-type
secondary. Combining our results with data for seven other brown dwarf
binaries in the literature treated in a self-consistent way, we confirm that
there is indeed a sharp transition in mass for lithium depletion in brown
dwarfs, as expected from theoretical calculations. We estimate such mass
boundary is observationally located at 51.48${}^{+0.22}_{-4.00}$
$M_{\mathrm{Jup}}$, which is lower than the theoretical determinations.

###### keywords:

brown dwarfs – spectroscopy – binaries: visual – abundances – stars:
individual: Sahlmann 2 AB

††pubyear: 2021††pagerange: New constraints on the minimum mass for
thermonuclear lithium burning in brown dwarfs ††thanks: Based on observations
made with the Gran Telescopio Canarias (GTC), installed in the Spanish
Observatorio del Roque de los Muchachos of the Instituto de Astrofísica de
Canarias, in the island of La Palma (programmes GTC100-20B and
GTC51-21B).–References

##  1 Introduction

Brown dwarfs are objects that do not have enough mass to settle on the
H-burning main-sequence because they develop cores dominated by degenerate
electrons as they keep contracting and cooling forever. For solar chemical
composition, the limiting mass of a star was found to be located at 73
$M_{\mathrm{Jup}}$ for completely convective numerical models (Kumar, 1963),
which is close to the modern theoretical value of 78.5 $M_{\mathrm{Jup}}$
(e.g., Baraffe et al., 2015).

Since lithium is destroyed by proton bombardment at temperatures around
2$\times$106 K, just below those required for H-fusion, there is a minimum
mass for thermonuclear lithium burning (MMLB), which is slightly below the
stellar/substellar boundary. The MMLB was estimated to lie at around 63
$M_{\mathrm{Jup}}$(Pozio, 1991; Nelson et al., 1993), and the search for the
Li I resonance doublet in optical spectra of brown dwarf candidates became
known as the lithium test (Magazzu et al., 1993). Objects cooler than
effective temperature 2700 K, corresponding to spectral type M7 and later
(Rajpurohit et al., 2013), and with lithium detected, are considered as
certified bona fide brown dwarfs with masses below 63 $M_{\mathrm{Jup}}$
(e.g., Tinney, 1998; Kirkpatrick et al., 2000).

Not long after the first bona fide brown dwarf was discovered in the Pleiades
cluster (Rebolo et al., 1995), the first brown dwarf binary was revealed
(Basri & Martín, 1999). Careful astrometric and spectroscopic monitoring of
the gravitational interactions between the components of ultracool binary
systems are able to yield dynamical masses that lie below the minimum stellar
mass, supporting the brown dwarf status of one or both components (e.g.,
Zapatero Osorio et al., 2004; Burgasser & Blake, 2009; Dupuy et al., 2009). In
some cases the brown dwarf status is confirmed both by dynamical mass and by
the detection of lithium (e.g. Sahlmann et al., 2015b; Dupuy & Liu, 2017).

A long-term astrometric monitoring campaign using the FORS2 optical camera
mounted on the Very Large Telescope (VLT) focused on 20 ultracool dwarfs
(Sahlmann et al., 2014) that had been discovered in a photometric and proper
motion study of 4800 squared degrees at low galactic latitude covered by the
DENIS survey (Phan-Bao et al., 2008). Two of those objects were found to be
binaries. Here we focus on DENIS J063001.4$-$184014AB (hereafter Sahlmann 2
AB, inspired by the Washington Double Star catalog convention of naming
objects after the discoverer of the binary nature), which was classified as an
M8.5 dwarf from optical low-resolution reconnaissance spectroscopy and it was
found to be an astrometric binary with an orbital period of $3.1\,$years
(Sahlmann et al., 2015a). The dynamical mass of each component in the binary
has been obtained from combining the astrometric data with spatially resolved
near-infrared imaging, and radial velocities for the primary (Sahlmann et al.,
2021). These authors noted that there was no sign of strong Li I resonance
doublet in a high-resolution optical spectrum of the system, despite the low
masses of $M_{1}=54^{+10}_{-9}$ $M_{\mathrm{Jup}}$ and $M_{2}=54^{+6}_{-5}$
$M_{\mathrm{Jup}}$ of the components, but they could only place a upper limit
of 1 Å on the measurable pseudo-equivalent width (pEW) because of low signal
to noise ratio.

The second focus of this work is on the binary DENIS J225210.7$-$173013AB
(hereafter Reid 1 AB). This object was identified as an L7.5 nearby ultracool
dwarf from near-infrared spectroscopic follow-up of candidates detected in the
DENIS survey using CGS4 on UKIRT (Kendall et al., 2004). High resolution
images obtained with the Near-Infrared Camera and Multi-Object Spectrometer
(NICMOS) on the Hubble Space Telescope (HST) resolved the object into two
components with an angular separation of 0.13 arcsec (Reid et al., 2006).
These authors also obtained low-resolution near-infrared spectra with Spex on
the NASA IRTF telescope and estimated individual spectral types of L6$\pm$1
and T2$\pm$1 for Reid 1 A and Reid 1 B, respectively. Monitoring of the
orbital motion of the Reid 1 AB system was carried out with the Canada-France-
Hawaii Telescope, the Keck telescopes and HST (Dupuy & Liu, 2017). These
authors derived an orbital period of $8.8\,$years, and estimated a spectral
type of L4$\pm$1 and a mass of $M_{1}=59^{+5}_{-5}$ $M_{\mathrm{Jup}}$ for the
primary, and a spectral type of T3.5$\pm$0.5 and a mass of
$M_{2}=41^{+4}_{-4}$ $M_{\mathrm{Jup}}$ for the secondary. No detection of the
Li I resonance doublet was reported by Reid et al. (2008) in the optical
spectrum of the integrated light from the binary.

This paper is structured as follows: In Section 2, we present new optical
spectra specifically aimed at searching for lithium in these two binaries. It
turns out that we do not detect lithium in Sahlmann 2 AB, which allows us to
infer that both components have experienced Li depletion, and we do detect
weak lithium in Reid 1 AB that we interpret as likely coming from the
secondary. In Section 3, our results are put in context with other very low-
mass (VLM) binaries for which dynamical masses and lithium data are available
in the literature. The consistency between ages and metallicities obtained
from stars and brown dwarfs for three VLM binaries that are members in
hierarchical multiple systems is discussed in Section 4. In Section 5, we
compare our results with calculations of lithium burning, and we confirm that
there is a sharp transition in dynamical mass between non lithium bearing and
lithium bearing brown dwarfs, as expected from the theory of substellar
evolution. However, the location of the MMLB from the observational side is
not completely consistent with theoretical calculations. Section 6 provides a
summary of our conclusions and a discussion of future prospects that are
opened by this research.

##  2 Observations, data reduction and spectroscopic measurements

Optical spectra of the integrated light of the Sahlmann 2 AB and the Reid 1 AB
systems were collected in service mode with the Optical System for Imaging and
low Resolution Integrated Spectroscopy (OSIRIS; Cepa et al., 2000) instrument
on the 10.4-m Gran Telescopio de Canarias (GTC). Sahlmann 2 AB was observed on
2021 February 18th (UT) for programme GTC100-20B, and Reid 1 AB was observed
on 2021 August 4th (UT) for programme GTC51-21A. In the later programme, a
spectrum of the known (Cruz et al., 2003) L1-type lithium dwarf 2MASS
J20575409$-$0252302 (hereafter 2M2057-02) was taken with the same instrumental
configuration. The Principal Investigator (PI) for both programmes was the
lead author of this paper.

The R1000R grating covering the 5100–10000 Å wavelength range and a slit width
of 1.0 arcsec, yielding a spectral resolving power of 672 at the central
wavelength, was used. The sky conditions were clear and with subarcsecond
seeing. Four on-source integrations with individual exposure times of 630 s
were obtained for Sahlmann 2 AB, six exposures of 1500 s for Reid 1 AB and
also six exposures of 1260 s each for 2M2057+0252. The targets were nodded by
15 arcsec on the slit to allow us for sky subtraction. Parallactic angle was
selected to minimize atmospheric refraction. Flat field lamp, arc lamp, and
dark frame exposures were used for standard calibrations within the IRAF
environment (Tody, 1993).

The four individual spectra of Sahlmann 2 AB and the mean combined one are
shown in Fig. 1. The mean combined spectrum of Sahlmann 2 AB, Reid 1 AB and
the L1-type lithium dwarf 2M2057-02 are displayed in Fig. 2.

Following the same procedure as in our previous work in the nearby open
clusters the Hyades (Martín et al., 2018) and Coma Berenices (Martín et al.,
2020), where we used similar instrumental setup, spectral types were derived
by comparison with field late-M and L dwarfs from the Sloan database
(Bochanski et al., 2010; Schmidt et al., 2010). The GTC/OSIRIS mean spectrum
of the Sahlmann 2 AB system was found to be intermediate between the M9 and
the L0 standards, and hence it is classified as M9.5$\pm$0.5 type, i.e., one
spectral subclass later than the classification of M8.5 reported by Phan-Bao
et al. (2008). The GTC/OSIRIS mean spectrum of the Reid 1 AB system was found
to be intermediate between the L6 and the L7 standards, and hence it is
classified as L6.5$\pm$0.5 type, i.e., one spectral subclass earlier than it
was estimated in Knapp et al. (2004).

The following atomic spectral features were clearly detected in the spectra:
Hα in emission, the Rb I resonance doublet at 780.03 and 794.76 nm, the Na I
subordinate doublet at 819.0 nm and the Cs I resonance doublet at 852.11 and
894.30 nm. Pseudo equivalent widths (pEW) were measured by direct integration
of the line profiles with respect to the local pseudo-continuum using the task
splot in the IRAF environment. Gaussian fitting was applied to each spectral
line to help guiding the choice of the integration boundaries for pEW
determination.

The Li I feature was not detected in any of our GTC/OSIRIS spectra of Sahlmann
2 AB. A zoom on the Li I spectral region of each individual spectrum is shown
in the lower panel of Fig. 1. Conversely, a weak Li I feature was detected in
our mean combined GTC/OSIRIS spectrum of Reid 1 AB and a strong feature was
detected in the L1 standard. These spectra are displayed in Fig. 3.

Signal-to-noise ratios (SNR) in a spectral region relatively free from strong
absorption features in the GTC spectra of our targets (between 680.5 and 683.5
nm) were measured with key ”m” in the splot task to estimate the sensitivity
of the data to detection of the Li I resonance doublet at 670.8 nm.

We used a spectral region slightly different than that adopted by Martín et
al. (2018) because we could see an obvious absorption bandhead in the spectrum
of Sahlmann 2 AB in the region between 675.0 and 680.0 nm. Our SNR estimates
are conservative because there is no region completely free from absorption
features in the spectra of ultracool dwarfs.

The spectroscopic measurements and their uncertainties are summarized in Table
LABEL:tab_Phan2:pew. The adopted 1$\sigma$ upper limits and the uncertainties
on the pEW measurements come from a conservative assessment of the errors
obtained from the SNR of the combined spectra using the equation discussed in
Martín et al. (2018), and the errors obtained manually using the task splot
with the options for direct integration and gaussian fitting. The weak Hα in
emission and the line strength of the Cs I and Na I measured in Sahlmann 2 AB
are normal for an old field M9.5 dwarf of solar-metallicity. There is no
spectroscopic sign of youth in the Na I doublet of the Sahlmann 2 AB system
which is an indicator of low surface gravity if it is weaker than the average
for field dwarfs (Martín et al., 2010). The upper limit on the presence of the
Li I in Sahlmann 2 AB is rather conservative because in fact we could not see
any feature whose pseudo-equivalent width is pEW $\geq$ 0.1 Å around 670.8 nm
in any of the spectra. Our adopted upper limit of 0.17 Å on the pEW of the Li
I resonance doublet in Sahlmann 2 AB improves the sensitivity to lithium
detection in Sahlmann 2 AB by a factor of almost six times over the upper
limit of 1 Å reported in Sahlmann et al. (2021) from a noisy (SNR $<$ 3) high-
resolution UVES spectrum obtained with the European Southern Observatory Very
Large Telescope.

For Reid 1 AB, we also detect weak Hα in emission, which is unusual for a
late-L dwarf because this chromospheric indicator is detected in less than
about 20% of field L5–L8 dwarfs (Pineda et al., 2016). The strength of the Cs
I and Rb I features in Reid 1 AB are similar to those reported in the
literature (Martín et al., 1999; Kirkpatrick et al., 2000; Lodieu et al.,
2015a; Pineda et al., 2016). The detection of a weak Li I feature is highly
unusual for a late-L dwarf.

Table 1: Pseudo EWs of atomic lines in Å estimated from the combined GTC/OSIRIS spectra of each target.  Name | Hα | Li I | Rb I | Rb I | Na I | Cs I | Cs I | SNR
---|---|---|---|---|---|---|---|---
| 656.28 nm | 670.78 nm | 780.03 nm | 794.76 nm | 819.0 nm | 852.11 nm | 8943.0 nm | 680.5–683.5 nm
Reid 1 AB |  $-$0.9$\pm$0.6 | 0.5$\pm$0.1 | 4.9$\pm$0.1 | 5.6$\pm$0.1 | 4.9$\pm$0.6 | 5.1$\pm$0.1 | 4.5$\pm$0.1 | 26
Sahlmann 2 AB |  $-$1.1$\pm$0.4 |  $<$0.17 | 1.0$\pm$0.2 | 2.3$\pm$0.1 | 8.3$\pm$0.5 | 1.4$\pm$0.1 | 1.5$\pm$0.1 | 53
2MASS2057-02 |  $-$12.6$\pm$0.2 | 4.5$\pm$0.1 | 2.4$\pm$0.1 | 2.5$\pm$0.1 | 5.6$\pm$0.5 | 1.6$\pm$0.1 | 1.9$\pm$0.1 | 35

Figure 1: Full OSIRIS spectra (top panel) of Sahlmann 2 AB compared with
standard ultracool dwarfs of spectral type M9 (blue line) and L0 (red line)
from the Sloan archive, and zoom around the Li I resonance doublet (bottom
panel). The position of the Li I feature at 670.8 nm is marked with a green
horizontal dashed line. Figure 2: Full OSIRIS spectra of DENIS0603 (Sahlmann 2
AB), 2MASS2057 and DENIS2252 (Reid 1 AB) compared with standard ultracool
dwarfs of spectral types between dL0 and dL7 from the Sloan archive. Figure 3:
Zoom around Hα and the Li I resonance doublet (bottom panel) for the three
targets observed in this work. The position of Hα and the Li I feature at
670.8 nm is marked with a red horizontal dashed line.

##  3 Dynamical mass and lithium in the individual components of VLM binaries

###  3.1 Lithium depletion in each component of Sahlmann 2 AB

To convert the adopted upper limit pEW Li I of $<$ 0.17 Å for the combined
light of the Sahlmann 2 AB system into lithium detection limits for each
component, the individual contributions to the flux around the spectral region
of the Li I resonance doublet are needed. Sahlmann et al. (2021) estimated
$\Delta I_{\mathrm{FORS2}}$ = 2.8$\pm$0.5 mag from a Keck adaptive optics
assisted spatially resolved imaging observation that gave $\Delta
K_{\mathrm{NIRC2}}$ = 1.74$\pm$0.06 mag. These authors also carried out binary
fitting of the low-resolution near-infrared spectral data of Sahlmann 2 AB
from Bardalez Gagliuffi et al. (2019) and concluded that the best-fit was
M8.5$\pm$0.5 for Sahlmann 2 A and L4.5$\pm$2.5 for Sahlmann 2 B. In this work
those individual spectral types are adopted. Using as photometric reference
the bright ($m_{R}$ = 15.051$\pm$0.014 mag; Costa et al. (2005)) and nearby (d
= $4.04\pm 0.01$ pc) M8.5 dwarf DENIS J104814.6$-$395606, a color of $R-K_{S}$
= 6.60$\pm$0.02 mag is assumed for Sahlmann 2 A. On the other hand, we take an
average color of $R-K_{S}$ = 7.6$\pm$0.2 mag for the two L5 standards of
Liebert & Gizis (2006) as representative of Sahlmann 2 B. From those colors,
we obtain $\Delta R$ = 2.9$\pm$0.2 mag or flux ratio of 14.6$\pm$1.0 around
the spectral region of the Li I resonance doublet.

The second step is to estimate the upper limits on the pEW Li I for each
component. The contribution to the flux of each component was accounted for by
multiplying the pEW by a factor $1+r$, where $r\,=\,Fratio\,\times\,Aratio$,
where $Fratio$ is the ratio of the fluxes, and $Aratio$ the ratio of the areas
(Martin & Rebolo, 1993). Assuming that both components in the Sahlmann 2 AB
system have the same radii, the upper limit for Sahlmann 2 A is pEW Li I $<$
0.18 Å and for Sahlmann 2 B is pEW Li I $<$ 2.7 Å.

The third step is to compare the upper limit on the pEW Li I for each
component to the upper envelope of pEW Li I in ultracool dwarfs as a function
of spectral class. The pEW Li I depends strongly on the spectral type as
illustrated in Fig. 4. Both components in the Sahlmann 2 AB system have about
a factor of at least four weaker pEW Li I than the maximum values for their Li
bearing counterparts of the same spectral type implying more than a factor of
10 Li depletion according to the calculations provided in Martín et al.
(2018), i.e., they have destroyed more than 90% of their initial Li content.
Their Li depletion factors are estimated in more detail in Section 5.

Figure 4: Pseudo-equivalent widths (pEW) of Li I versus spectral type (L0 =
10; L5 = 15; T0 = 20; T5 = 25 in the abscissa axis) for ultracool dwarfs in
the field and in open clusters. The following data sources have been used:
Burgasser et al. (2015) for field late-M and L dwarfs; Cruz et al. (2009) for
field L dwarfs; Kirkpatrick et al. (2000) and Kirkpatrick et al. (2008) for
field L dwarfs; Lodieu et al. (2015b) for the components of Luhman16AB; Martín
et al. (1999) for field late-M and L dwarfs; Lodieu et al. (2018) and Martín
et al. (2018) for L dwarfs in the Hyades; Pineda et al. (2016) for the T2
dwarf SIMP0136; Rebolo et al. (1996) for late-M brown dwarfs in the Pleiades;
Tinney (1998) for the M9 lithium brown dwarf LP 944-20; and Zapatero Osorio et
al. (2004) for field L dwarfs. Note that the range of spectral types of brown
dwarfs with Li I detections is between M8.5$\pm$0.5 and T3.5$\pm$0.5, which is
the domain that we adopt for this study.

###  3.2 Lithium in each component of Reid 1 AB

We follow the same approach to disentangle the contributions to the lithium
detection in the components of Reid 1 AB. Following Dupuy & Liu (2012), we
consider that the primary has a spectral type of L4.5$\pm$1.5, the secondary
has a spectral type of T3.5$\pm$0.5, and the magnitude difference between them
is 0.98$\pm$0.03 mag in the HST F110W-band. Using the flux ratio at $R$-band
between L4.5 and L8.5 derived from the $R-J$ data of Liebert & Gizis (2006), a
flux ratio of 3.1$\pm$0.6 in the $R$-band is obtained. This ratio is
multiplied by a factor of 6.25 obtained from the flux ratio between the L8 and
the T3 dwarfs illustrated in Figure 16 of Burgasser et al. (2003) from Keck
LRIS optical spectra to yield a flux ratio of 19.4$\pm$0.8. No lithium
detection was reported by Reid et al. (2008) from optical spectra obtained
with GMOS with resolving power of R = 1220. They did note lithium detections
with GMOS in eight L dwarfs and provided pEW Li I measurements between 3.5 and
12.2 Å for them. Our GTC spectra improve the sensitivity to Li I detection by
almost an order of magnitude given that we are able to measure a pEW Li I $=$
0.5 $\pm$ 0.1 Å. If this Li I feature comes from the primary, it would mean a
very large Li depletion because it is much weaker than those measured in
late-L dwarfs. On the other hand, if the Li I feature comes from the
secondary, it implies an upper limit of pEW Li I $<$ 0.51 Å for Reid 1 A, and
pEW Li I $=$ 10.2$\pm$2.5 Å for Reid 1 B. Such a value of pEW Li I for Reid 1
B is similar to those reported for the T0.5 dwarf Luhman16B (Lodieu et al.,
2015b) and the T2 dwarf SIMP0136+0933 (Pineda et al., 2016), which supports
our interpretation that the Li I feature in our integrated light GTC spectrum
most likely comes from the secondary.

###  3.3 Lithium in the individual components of VLM binaries from the
literature

We made a search for published results to put our results for Sahlmann 2 AB
and Reid 1 AB in context. The criteria employed to select the VLM binaries for
this study are the following: total mass $<$ 150 $M_{\mathrm{Jup}}$, error in
total mass $<$ 10%, spectral type of the components between M8.5$\pm$0.5 and
T3.5$\pm$0.5, and individual dynamical mass of each component determined from
orbital motion. The early spectral type limit at M8.5$\pm$0.5 corresponds to
the lithium depletion boundary (LDB) at an age of about 150 Myr. At an age of
112$\pm$5 Myr the LDB has been determined at M7 in the Pleiades cluster by
Dahm (2015). The late spectral type limit corresponds to the latest dwarf with
lithium detection, beyond which there might be significant chemical depletion
of pure neutral lithium atoms due to the formation of Li-bearing molecules
(Lodders, 1999; Gharib-Nezhad et al., 2021). Even though it is predicted that
the Li I resonance doublet could remain strong down to Teff = 500 K (Burrows
et al., 2002), this feature has not yet been observed at spectral type later
than T3.5$\pm$0.5 (this work), and thus we limit our study to objects not
later than T3$\pm$1.0, corresponding to Teff about 1100 K. The relevant
information available in the literature for the sample of seven VLM binaries
that meet our criteria, together with that for Sahlmann 2 AB and Reid 1 AB,
are summarized in Table LABEL:tab_Phan2:LiMass.

Two of the binaries, namely Epsilon Indi Ba,b, and Luhman 16 AB, are the two
closest substellar systems to the Sun, they have dynamic mass measurements
(Dupuy et al., 2019; Lazorenko & Sahlmann, 2018), and they have spatially
resolved optical spectra of the individual components with sufficient quality
to detect the Li I resonance doublet. It is indeed detected in both components
of Luhman 16 AB (Faherty et al., 2014; Lodieu et al., 2015b), but not in the
components of Epsilon Indi B (King et al., 2010). Epsilon Indi Ba has a
spectral type of T1$\pm$1, which is just at the edge of our range of study
(considering the uncertainty), and we include it in the analysis. Epsilon Indi
Bb has a spectral type of T6$\pm$1, outside our range, and we exclude it from
the analysis.

The other five binaries from the literature have published optical spectra for
the spatially unresolved light of the components. In the previous section we
estimated upper limits to the pEW of the Li I resonance doublet in each
individual component of Sahlmann 2 AB. Here, we apply the same method to the
four other binaries in our sample for which upper limits on the pEW of the Li
I resonance doublet of the integrated light are available in the literature.

The first one that we consider is GJ 569 B, which is composed of two M9
components with an observed magnitude difference of 0.7$\pm$0.2 mag in the HST
$F814W$-band filter (Martín et al., 2006). Since the two components have
similar optical spectral type, we assume that that they also have similar
$R-I$ color and that the flux ratio in the spectral region of the Li I feature
is 1.9$\pm$0.4, which implies that the upper limit of pEW Li I = 0.05 Å from a
Keck HIRES spectrum (Zapatero Osorio et al., 2005) translates into an upper
limit of 0.08 Å for the primary and 0.17 Å for the secondary.

The second system is 2MASS J07003664$+$3157266AB which is composed of an
L3$\pm$1 primary and an L6.5$\pm$1.5 secondary with a magnitude difference of
1.49$\pm$0.02 mag in the $J$-band (Dupuy & Liu, 2012). Using the $R-J$ =
5.91$\pm$0.05 color of the L3 standard, and the average color $R-J$ =
6.2$\pm$0.2 mag of the L6 and L7 standards from Liebert & Gizis (2006), we
estimate a magnitude difference of 2.3$\pm$0.3 mag in the $R$-band, and a flux
ratio of 8.3$\pm$2.7. The upper limit of pEW Li I of 0.3 Å in the combined
light is from a spectrum obtained by Thorstensen & Kirkpatrick (2003). From
these considerations, we obtain pEW Li I $<$ 0.3 Å for 2MASS J0700$+$3157 A
and pEW Li I $<$ 3.6 Å for 2MASS J0700$+$3157 B.

The third binary is 2MASS J2132114$+$134148 AB. The A component has a spectral
type of L4.5$\pm$1.5, the B component has a spectral type of L8.5$\pm$1.5, and
the magnitude difference is 0.85$\pm$0.04 mag in the J-band (Dupuy & Liu,
2012). Using again the $R-J$ data of Liebert & Gizis (2006), a flux ratio of
3.1$\pm$0.6 in the $R$-band is obtained. An optical spectrum of 2MASS
J2132$+$1341 AB was obtained by Cruz et al. (2007) using the Gemini Multi-
object spectrometer (GMOS) with a resolving power of R = 1220. These authors
do not note the presence of the Li I resonance doublet in their GMOS data. On
the other hand, they note the presence of the Li I feature in a spectrum of
the L5 dwarf 2MASS J0310140$-$275645, which has almost the same apparent near-
infrared magnitudes as 2MASS J2132$+$1341 AB, and it was observed with the
same GMOS intrumental configuration. They provide a measurement of pEW Li I
$=$ 10$\pm$1 Å for 2MASS J0310140$-$275645\. Plots of the spectra of both
2MASS J0310140$-$275645 and 2MASS J2132$+$1341 AB are available at
http://database.bdnyc.org/query. We zoomed the plots on the spectral region
from 667.0 nm to 687.0 nm, and we digitized them using the online tool
WebPlotDigitizer. Using the IRAF task splot, we obtain a pEW Li I consistent
with the published value for 2MASS J0310140$-$275645 and an upper limit for
2MASS J2132$+$1341 AB of pEW Li I $<$ 0.8 Å from which we derive pEW Li I $<$
1.1 Å for the A component and pEW Li I $<$ 3.8 Å for the B component.

Table 2: Lithium and dynamical mass data for VLM binaries (total binary mass $<150$ $M_{\mathrm{Jup}}$). Name | pEW(Li I)1 | pEW(Li I)2 | M1 | M2 | Age | SpT
---|---|---|---|---|---|---
| Å | Å | MJup | MJup | Myr |
GJ 569 Ba,Bb |  $<$ 0.08 |  $<$ 0.17 | $80^{+9}_{-8}$ | $58^{+7}_{-9}$ | 440$\pm$60 | M9,M9
Epsilon Indi Ba,Bb |  $<$ 0.10 |  $<$ 0.10 | $68^{+0.9}_{-0.9}$ | $53.1^{+0.3}_{-0.3}$ | 4000$\pm$500 | T1,T6
2MASS J0700$+$3157 AB |  $<$ 0.35 |  $<$ 3.6 | $68^{+2.6}_{-2.6}$ | $73^{+2.9}_{-3}$ | 2000$\pm$700 | L3,L6.5
2MASS J2132$+$1341AB |  $<$ 1.1 |  $<$ 3.8 | $68^{+4}_{-4}$ | $60^{+4}_{-4}$ | 1440$\pm$370 | L4.5,L8.5
Reid 1 AB |  $<$ 0.51 | 10.2$\pm$2.5 | $59^{+5}_{-5}$ | $41^{+4}_{-4}$ | 1110$\pm$220 | L4.5,T3.5
Sahlmann 2 AB |  $<$ 0.18 |  $<$ 2.7 | $54^{+10}_{-9}$ | $54^{+6}_{-5}$ | 200$\pm$50 | M8.5,L4.5
Gl 417 BC | 16.8$\pm$0.3 |  | $51.5^{+1.7}_{-1.8}$ | $47.7^{+1.9}_{-1.9}$ | 490$\pm$40 | L4.5,L6
SDSS J0423$-$414AB | 11.4$\pm$0.7 |  | $51.6^{+2.3}_{-2.5}$ | $31.8^{+1.5}_{-1.6}$ | 810$\pm$90 | L6.5,T2.5
Luhman 16 AB | 8.2$\pm$1.0 | 8.4$\pm$1.5 | $33.5^{+0.3}_{-0.3}$ | $28.6^{+0.3}_{-0.3}$ | 1000$\pm$800 | L7.5,T0.5

For the two brown dwarf binaries with lithium detections, namely Gl 417 BC and
SDSS J0423$-$0414AB we check whether or not it is reasonable that both
components have preserved their initial lithium content. Gl 417 BC is composed
of a L4.5$\pm$1 primary and a L6$\pm$1 secondary with a magnitude difference
of 0.347$\pm$0.025 mag in the $K$-band (Dupuy & Liu, 2012). According to the
same authors, SDSS J0423$-$0414 AB is composed of an L6.5 primary and a T2.5
secondary with a magnitude difference of 1.18$\pm$0.08 mag in the $K$-band.
Using the same procedures as outlined above for binaries with components of
similar spectral type, the flux ratios derived for the $R$-band are
1.9$\pm$0.1 and 27.8$\pm$0.9, respectively.

Kirkpatrick et al. (2001) measured a pEW Li I of 11.5 Å for the combined-light
spectrum Keck/LRIS of Gl 417 BC obtained with LRIS at Keck-I. They did not
give an error bar, but from their Figure 2, we estimate $\pm$0.2 Å. We derive
a pEW Li I of 16.8$\pm$0.3 Å for the A component. This is consistent with the
upper envelope of pEW Li I measured in field L4.5 dwarfs (see Fig. 4). There
could be a modest contribution of the B component to the Li I feature, but we
cannot quantify it precisely with the data in hand. High spectral resolution
spectroscopy might be able to resolve the two Li I features from the
components in this system.

Kirkpatrick et al. (2008) measured a pEW Li I of 11 Å for the spatially
unresolved spectrum of SDSS J0423$-$0414 AB obtained with LRIS at Keck-I. They
did not provide an error bar, but the spectrum shown in their Figure 16 is of
high quality, and hence we assign an error bar of 0.5 Å, which is the upper
limit they assign to their high quality objects without lithium detection.
Using the flux ratio we obtain pEW Li I of 11.4$\pm$0.7 Å for the A component,
which is also consistent with the upper envelope of Li measurements for L6.5
spectral class (see Fig. 4). The contribution of the B component is negligible
because of the high flux ratio of 27.8$\pm$0.9 that we estimated previously.

###  3.4 Unresolved pairs of brown dwarfs

It has been noted that some of the objects in our sample could be unresolved
pairs of brown dwarfs because the evolutionary models cannot provide a
consistent account of their age, luminosity and mass. Simon et al. (2006)
suggested that GJ 569Bb could be a pair of brown dwarfs with masses of about
40 $M_{\mathrm{Jup}}$ each. Dupuy & Liu (2017) considered that 2MASS
J0700$+$3157 B could be composed of two equal mass brown dwarfs with mass of
36.7${}^{+1.4}_{-1.5}$ MJup each. Sahlmann et al. (2021) proposed that
Sahlmann 2 B is made up of two unresolved brown dwarfs with mass of 25
$M_{\mathrm{Jup}}$ each.

All these possible unresolved candidate pairs have individual masses well
below those adopted in this work. Should it be demonstrated that any of them,
or all of them, are indeed composed of two lower-mass objects, the non-
detection of Li in them could be surprising. We strongly encourage additional
observations of these three objects in particular to try to resolve them into
their putative pairs, and try to get the lithium detections, spectral types,
and individual masses. One possible solution is that the individual masses of
the unresolved pairs are so low that their spectral types are cooler than T4,
and they might be affected by significant weakening of the Li I resonance
doublet due to Li locking into dust grains and molecules.

##  4 Ages and metallicities from stellar companions

Three of the VLM binaries considered in this study are wide companions of
stars and can be assumed they are members in hierarchical multiple systems
sharing a common origin. We estimate here the ages and metallicities of these
three stars, which can serve as reference in their respective systems.

We inferred the stellar parameters of the three stars using the Bayesian
inference code of del Burgo & Allende Prieto (2016, 2018) and incorporating
the data from Gaia DR2 (Gaia Collaboration et al., 2018). This statistical
method was applied to the PARSEC v1.2S library of stellar evolution models
(Bressan et al., 2012). It has been proven that its predicted masses for main-
sequence stars are, on average, within 4% of accuracy when compared with the
dynamical masses of detached eclipsing binaries (del Burgo & Allende Prieto,
2018). In this analysis, the code takes as input the absolute $G$ magnitude
(obtained from the apparent magnitude and the parallax), $G_{BP}-G_{RP}$
colour, [Fe/H], and their uncertainties, returning theoretical predictions for
a series of stellar parameters. We adopted the corrected Gaia photometry
passbands ($G$, $G_{BP}$, and $G_{RP}$) from Maíz Apellániz & Weiler (2018).
The values for the iron abundance [Fe/H] were averaged from those with given
uncertainties for these stars in the PASTEL catalogue, which is based on high-
resolution spectra (Soubiran et al., 2016).

Table 3:  Literature parameters of the primaries: spectral type ($SpT$)/luminosity class ($LC$), Gaia photometry ($G$, $G_{BP}$, $G_{RP}$), parallax $\Pi$, and iron abundance [Fe/H]. References for the literature values are given in the main text.  Name | Alternative name | $SpT/LC$ | $G$ | $G_{BP}$ | $G_{RP}$ | $\Pi$ | [Fe/H]
---|---|---|---|---|---|---|---
|  |  | ($mag$) | ($mag$) | ($mag$) | ($mas$) | (dex)
Gl 417 | HD 97334 | G2/? | 6.2410 $\pm$ 0.0006 | 6.5882 $\pm$ 0.0028 | 5.802 $\pm$0.005 | 44.14 $\pm$ 0.04 | 0.07 $\pm$ 0.04
GJ 569 A | BD$+$162708 | M3V | 9.1256 $\pm$ 0.0008 | 10.350 $\pm$ 0.003 | 8.059 $\pm$0.003 | 100.68 $\pm$ 0.05 | -0.15 $\pm$ 0.20
Epsilon Indi | HD 209100 | K5V | 4.273 $\pm$ 0.005 | 4.9221 $\pm$ 0.0008 | 3.6386 $\pm$0.0018 | 274.80 $\pm$ 0.25 | -0.13 $\pm$ 0.09
Table 4:  Inferred stellar parameters of the primaries: iron abundance [Fe/H], radius ($R$), effective temperature ($T_{\text{eff}}$), mass ($M$), surface gravity ($\log g$), luminosity ($\log L$), age ($\tau$), and evolution phase.  Name | $R$ | $T_{\text{eff}}$ | $M$ | $\log g$ | $\log L$ | $\tau$ | phase
---|---|---|---|---|---|---|---
| (R☉) | ($\,K$) | (M☉) | ($g$: cm s-2) | ($L$: L☉) | $Myr$ |
Gl 417 | 1.032 $\pm$ 0.009 | 5913 $\pm$ 18 | 1.063 $\pm$ 0.028 | 4.437 $\pm$ 0.017 | 0.070 $\pm$ 0.005 | 2600 $\pm$ 1180 | MS
GJ 569 A | 0.540 $\pm$ 0.010 | 3432 $\pm$ 42 | 0.44 $\pm$ 0.07 | 4.61 $\pm$ 0.10 | -1.438 $\pm$ 0.008 | 87${}^{+153}_{-87}$ | PMS
Epsilon Indi | 0.7068 $\pm$ 0.0028 | 4676 $\pm$ 8 | 0.730 $\pm$ 0.014 | 4.603 $\pm$ 0.010 | -0.667 $\pm$ 0.004 | 9266$\pm$2114 | MS

Gl 417 A: Our Bayesian analysis provides a most likely mean age of
2.60$\pm$1.18 Gyr, on the Main-Sequence (MS). Kirkpatrick et al. (2001)
estimated an age in the range from 80 to 300 Myr, based on several indicators
such as the X-ray luminosity (log(LX/Lbol)= 4.55), the CaII emission
(log=-4.40), and the rotational period (7.6 days). However, Dupuy et al.
(2019) derived a gyrochronology age of 750${}^{+150}_{-120}$ Myr and found the
estimate of Kirkpatrick et al. (2001) inconsistent. The TESS light curve also
yields a rotational period of about 7.5 days, which suggests an age similar to
the Praesepe open cluster (670 Myr) according to the calibration provided by
Curtis et al. (2019). The age inferred from the substellar evolution models is
0.49$\pm$0.04 Gyr (Dupuy & Liu, 2017), which is in good agreement with our
estimate (within 1.8$\sigma$).

GJ 569 A: Our analysis places it on the pre-MS phase, with an upper limit for
the mean age $<$240 Myr. This is somewhat lower, but not far from the value of
300 Myr from its kinematics, which places it as a member of the Ursa Major
moving group (Kenworthy et al., 2001). The age inferred from the substellar
evolution models is 440$\pm$60 Myr (Dupuy & Liu, 2017), which is somewhat
older than the PMS age that we obtain for the stellar primary, but within
1.75$\sigma$.

Epsilon Indi A: We infer a mean age of 9$\pm$2 Gyr (on the MS). This is
consistent with the value of $>$9.87 Gyr derived from the kinematics (Eker et
al., 2015), but greater than the range of 3.7$-$5.7 Gyr given by Feng et al.
(2019), which is based on the same rotation-age relationship of Eker et al.
(2015). The TESS light curve indicates a rotation period longer than 18 days,
which would imply an age older than 1 Gyr (Curtis et al., 2019). The age from
the substellar evolution models is 4.0$\pm$0.5 Gyr (Dieterich et al., 2018),
which a factor of two younger than our Bayesian estimate, comparing mean
values, but only within 2$\sigma$ of discrepancy in view of the uncertainties.

For all the three stellar companions, our Bayesian analysis and those obtained
by other authors for the brown dwarfs are in agreement within 2$\sigma$
uncertainty level. We note that our conclusions on the location of the MMLB
does not depend on the precise age adopted for the systems under study, as
long as they are not extremely young ($<$150 Myr). If any of the systems under
study would be younger than about 150 Myr, the MMLB could be shifted to higher
massess because the brown dwarfs may not have had enough time to complete
their Li burning.

The iron abundances of the three stars are similar to the Sun (within about
50%), providing support to our assumption of solar metallicity for the
comparison of the observed MMLB with the theoretical MMLB, which is discussed
in the next section.

##  5 The minimum mass for thermonuclear lithium burning in brown dwarfs

Figure 5: Individual dynamical mass versus pEW (Li I) for components of VLM
binary systems (total binary mass $<$ 150 MJup). Measurements are shown with
black squares while upper limits are displayed as triangles. Blue dashed lines
represent the calculations of Li burning by Nelson et al. (1993). Red dashed
lines represent the isomasses between 51 and 63 Jupiter masses from the models
by Phillips et al. (2020) for ages between 100 Myr and 5000 Myr. Theoretical
prediction of Li depletion factors have been converted into pEW Li I resonance
doublet using the prescription provided by Martín et al. (2018), assuming an
upper limit on the pEW of 20 Å. The green vertical band shows the sharp region
between Li-bearing and Li-free brown dwarfs.

Detailed theoretical calculations of the minimum mass for thermonuclear
lithium burning (MMLB) have been carried out by several groups using different
numerical codes. Pozio (1991) found that a 63 $M_{\mathrm{Jup}}$ model with
solar composition could destroy 90% of its initial Li content. Nelson et al.
(1993) obtained slightly less Li depletion, i.e. their model with 65
$M_{\mathrm{Jup}}$ destroyed 90% of lithium.Chabrier et al. (1996) found that
their model with $63$ $M_{\mathrm{Jup}}$ was able to destroy 99% of lithium.
The very recent models with updated equation of state and opacities (Phillips
et al., 2020) destroy 90% of lithium at a mass of $58$ $M_{\mathrm{Jup}}$. In
this work we adopt as the definition of the MMLB the mass at which the
permanent imprint of thermonuclear lithium fusion leaves an amount of 10% of
the original Li abundance in the photosphere of the brown dwarf. The
calculations of Nelson et al. (1993) and Phillips et al. (2020) bracket the
range of theoretical MMLB values found in the literature, which we summarize
as MMLBth = 63$\pm$5$\,M_{\mathrm{Jup}}$.

The availability of individual dynamical masses, effective temperatures,
bolometric luminosities and lithium data for the components of very low-mass
(VLM) binary systems makes it possible to test observationally the theoretical
predictions for the hydrogen and lithium burning minimum masses. An early
attempt to carry out such a test was performed by Zapatero Osorio et al.
(2005) using only GJ 569B, but the uncertainties were to high to draw strong
conclusions. Based on a sample of 13 VLM binaries, the minimum mass for
hydrogen burning (MMHB) was estimated at $70$ $M_{\mathrm{Jup}}$ and the MMLB
at $60$ $M_{\mathrm{Jup}}$ by Dupuy & Liu (2017). They noted that their
results were in agreement with the models of Baraffe et al. (2015), but not
with those by Burrows et al. (1997).

A comparison between the pEW versus dynamical mass data and two different sets
of evolutionary models is illustrated in Fig. 5. The data and the models
coincide on the fact that there is a sharp mass boundary between lithium
depletion versus preservation in brown dwarfs. However, the data suggest that
the boundary is displaced to lower mass than what the models predict, as can
be appreciated by eye from the location of the boundary between the objects
with and without lithium, depicted as a green band in Fig. 5, and the location
of the theoretical curves. Random uncertainties in the dynamical mass
measurements could make the objects scatter around the true MMLB, making it
appear as if some objects with lithium could have a mass higher than other
objects without lithium. Other factors such as a spread in age or metallicity
could also make the MMLB wider. However, that is not observed, suggesting that
mass is by far the single most dominant factor.

###  5.1 Lithium abundances, lithium depletion factors and the observed mass
boundary

We used the pEW values listed in Table LABEL:tab_Phan2:LiMass and the basic
parameters from the literature Lodieu et al. (2015b); Dupuy & Liu (2017);
Dupuy et al. (2019), to calculate the Li abundances or upper limits for 14
individual components of VLM binaries using the same procedures as in our
previous work on L-type members in the Coma Ber and Hyades open clusters
(Martín et al., 2018, 2020). The resulting abundances and depletion factors
are summarized in Table LABEL:tab_Phan2:LiAbundances. For each object, we also
estimated the Li depletion factor predicted by the models of Phillips et al.
(2020) using the age and mass estimates given in Table LABEL:tab_Phan2:LiMass.
We find a good agreement between the Li depletion factors estimated from the
observations and those predicted by the models, except for the two components
of Sahlmann 2 AB which are both more depleted than the theoretical
expectation. Therefore, we note that this binary system is particularly
critical to test the validity of the models.

Table 5: Lithium abundances and depletion factors for individual components of VLM binaries. Name | log (Lbol/L⊙) | Teff | log g | pEW(Li I)obs/ | logN(Li) | Li depletion | Li depletion
---|---|---|---|---|---|---|---
|  | (K) |  | pEW(Li I)max |  | (observed) | (predicted)
GJ 569 Ba | -3.44$\pm$0.04 | 2420$\pm$40 | 5.2 |  $<$0.04 |  $<$0.5 |  $>$99.5% | 100%
GJ 569 Bb | -3.67$\pm$0.03 | 2170$\pm$50 | 5.2 |  $<$0.11 |  $<$1.3 |  $>$95% | 1-100%
Epsilon Indi Ba | -4.70$\pm$0.02 | 1310$\pm$10 | 5.4 |  $<$0.01 |  $<$0.1 |  $>$99.9% | 100%
2MASS J0700$+$3157 A | -3.95$\pm$0.04 | 1860$\pm$40 | 5.4 |  $<$0.05 |  $<$0.6 |  $>$99.5% | 100%
2MASS J0700$+$3157 B | -4.45$\pm$0.04 | 1430$\pm$100 | 5.4 |  $<$0.24 |  $<$2.0 |  $>$90% | 100%
2MASS J2132$+$1341A | -4.22$\pm$0.05 | $1660^{+50}_{-40}$ | 5.3 |  $<$0.06 |  $<$0.8 |  $>$99.5% | 100%
2MASS J2132$+$1341B | $-4.50^{+0.05}_{-0.04}$ | $1400^{+30}_{-40}$ | 5.3 |  $<$0.42 |  $<$2.5 |  $>$80% | 35-100%
Reid 1 A | $-4.26^{+0.05}_{-0.04}$ | 1590$\pm$50 | 5.3 |  $<$0.05 |  $<$0.6 |  $>$99.5% | 85-100%
Reid 1 B | $-4.76^{+0.08}_{-0.07}$ | $1210^{+50}_{-40}$ | 5.2 | 1.0$\pm$0.1 | 3.3$\pm$0.1 | 0-10% | 0%
Sahlmann 2 A | -3.3$\pm$0.1 | 2400$\pm$100 | 5.1 |  $<$0.10 |  $<$1.3 |  $>$99% | 0-93%
Sahlmann 2 B | -4.1$\pm$0.1 | 1400$\pm$100 | 5.1 |  $<$0.16 |  $<$1.7 |  $>$97.5% | 0-36%
Gl 417 B | -4.13$\pm$0.03 | $1639^{+29}_{-31}$ | 5.1 | 1.0$\pm$0.1 | 3.3$\pm$0.1 | 0-10% | 1-11%
SDSS J0423$-$414A | -4.41$\pm$0.04 | $1430^{+30}_{-40}$ | 5.2 | 0.8$\pm$0.2 | 3.1$\pm$0.3 | 0-70% | 1-14%
Luhman 16 A | -4.68$\pm$0.08 | $1305^{+180}_{-135}$ | 5.0 | 0.8$\pm$0.2 | 3.1$\pm$0.3 | 0-70% | 0%
Luhman 16 B | -4.71$\pm$0.10 | $1190\pm 60$ | 4.9 | 1.0$\pm$0.1 | 3.3$\pm$0.1 | 0-10% | 0%

We performed curve fitting for our compilation of Li abundance versus
individual dynamical mass. To explore the constraints obtained from the data
on the location of the MMLB, we chose a natural exponential decay function
$a\times exp(-b(x-c))$ and excluded the two points corresponding to the binary
components of the system Luhman 16AB as well as Reid1 B because they are well
below the MMLB boundary. We neither employ Epsilon Indi Bb in the analysis.
The curve_fit function of the Open Source Python111https://docs.python.org
scientific library SciPy (Virtanen et al., 2020) was used to derive the free
parameters $a$, $b$, and $c$. A Monte Carlo simulation was carried out to
derive the uncertainties. We generated 10 000 distributions of points and
applied our fitting approach rejecting bad fits before extracting the
statistics. Assuming the original pairs and their uncertainties adjust to
Gaussian distributions, we employed the NumPy222https://numpy.org/ function
random.normal to generate suitable samples. For the upper limits of Li
abundance, we used the NumPy function random.random. The exponential that fits
the original distribution has a very steep decay. For $c$ we adopted the value
that better fits this distribution and increased its uncertainty in view of
the shift with respect to the most likely value of $c$ that emerges from the
Monte Carlo simulation. Our approach yields $c=51.48^{+0.22}_{-4.00}$ MJup.

An exponential law is a well-motivated approximation based on the estimate
that thermonuclear Li burning is a extremely strong function of the maximum
central temperature attained in the interior of brown dwarfs (Bildsten et al.,
1997). Besides, the observed data distribution supports it. In conclusion, we
obtain MMLBobs= 51.48${}^{+0.22}_{-4.00}$ $M_{\mathrm{Jup}}$. The result and
scaffolding of this fitting exercise are illustrated in Fig. 6.

Figure 6: Number of Li atoms in the atmosphere of brown dwarfs versus their
individual dynamical mass for our sample of components of VLM binary systems.
Measurements are shown with black squares and error bars, while upper limits
are displayed as down-pointed triangles with error bars on the masses. A
probabilistic cloud of simulated data with a Monte-Carlo code is associated to
each datapoint. The best fit using a natural exponential law is shown with a
red curve. From this fit, we obtain MMLBobs = 51.48${}^{+0.22}_{-4.00}$
$M_{\mathrm{Jup}}$. This figure was made with the Matplotlib package (Hunter,
2007)

.

##  6 Final remarks and future prospects

The main goal of this work is to set observational constraints on the MMLB for
solar metallicity and intermediate age brown dwarfs, and to test the validity
of evolutionary models. We have considered a total of 15 individual components
that belong to nine different binary systems, six of them are in common with
the study of Dupuy & Liu (2017). The new binaries considered in this work are
GJ 569 Ba,Bb, Epsilon Indi Ba,Bb, and Sahlmann 2 AB, and we have provided
additional data for the binary Reid 1 AB.

From the study of this sample of 15 objects with dynamical mass and lithium
data, we have determined a value of MMLBobs=51.48${}^{+0.22}_{-4.00}$
$M_{\mathrm{Jup}}$, which is lower than all of the available theoretical
determinations of the MMLB location by the following amount:
MMLB${}_{th}-MMLB_{obs}=\delta$MMLB= 12.0${}^{+9.0}_{-4.8}$
$M_{\mathrm{Jup}}$. This difference is minimized, but not cancelled for the
models computed by Phillips et al. (2020). There seems to be a historical
trend that when the evolutionary models get updated their predicted MMLB
diminishes slightly. Further updates may bring the observational and
theoretical MMLB to a complete agreement.

Uncertainties in the models of lithium depletion for very low mass stars and
brown dwarfs have been discussed in detail by Burke et al. (2004), although
they neither consider masses below $68$ $M_{\mathrm{Jup}}$, nor ages older
than 200 Myr. Sources of theoretical uncertainty considered by those authors
include the equation of state, the treatment of convective transport, the
opacity data, the outer (atmospheric) and inner boundary conditions, and the
effects of rotation. Another point of interest for the theoretical
determination of the MMLB are the effects of dark matter accretion in the
evolution of brown dwarfs, which have been suggested to be of potential
relevance for the location of the H-burning minimum mass (Zentner & Hearin,
2011). In view of the increasing precision on the determination of the MMLB
from BD binaries, it would be worthwhile to explore the effects of different
dark matter candidates in the lithium burning of fully convective stars and
degenerate brown dwarfs, particularly in environments where condensations of
dark matter may be expected, such as the Galactic center.

From the observational side, it is clear that more work is worthwhile to
increase the accuracy of dynamical masses, and to expand the sample of brown
dwarf binaries with dynamical masses and lithium abundance determination. With
a larger sample it should be possible to increase the statistical significance
of the observational constraints on the MMLB, and to check on the effects of
age, metallicity, rotation, and other factors that could play a role in
shaping the imprint of lithium burning in brown dwarf evolution. In future
work, we plan to present new observations of lithium in brown dwarf binaries
aimed at increasing the constraints on the determination of the MMLB as a
function of age, mass, metallicity, rotation, and possible accretion and
accumulation of dark matter in environments close to the Galactic center.

Besides testing substellar evolutionary models, the determination of the
observational MMLB can provide a useful benchmark to understand the properties
of transiting brown dwarfs for which dynamical mass determinations can also be
made. There is a growing number of these systems being discovered thanks to
several transit surveys (e.g., Acton et al. (2021)). Comparison of their
present mass, presence of lithium, stellar age, and the observational MMLB can
set interesting constraints on their mass loss rate.

## Acknowledgements

This research has been supported by the Spanish Ministry of Economy and
Competitiveness (MINECO) and the Fondo Europeo de Desarrollo Regional (FEDER)
under grants PID2019-109522GB-C53 and PID2019-107061GB-C66. This research made
use of the databases at the Centre de Données astronomiques de Strasbourg
(http://cds.u-strasbg.fr); NASA’s Astrophysics Data System Service
(http://adsabs.harvard.edu/abstract_service.html), and the paper repositories
at arXiv. This research has made use of the SIMBAD database, operated at CDS,
Strasbourg, France and NASA’s Astrophysics Data System. This research has made
use of the VizieR catalogue access tool, CDS, Strasbourg, France. The original
description of the VizieR service was published in A&AS 143, 23. This work has
made use of data from the European Space Agency (ESA) mission Gaia
(https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and
Analysis Consortium (DPAC,
https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has
been provided by national institutions, in particular the institutions
participating in the Gaia Multilateral Agreement. This research made use of
Matplotlib and Scipy, two community-developed core Python packages. We thank
Gilles Chabrier and Isabelle Baraffe for kindly providing a digitized version
of the calculations that are used in Fig. 5 and for comments on the
manuscript. We are grateful for the helpful referee review provided by John
Gizis, and we also thank Trent Dupuy, Yakiv Pavlenko, Rafael Rebolo, Victor
Sanchez Bejar and Maria Rosa Zapatero Osorio for reviewing the manuscript and
providing comments. Jun-Yan Zhang assisted with digitizing the spectra from
the BDNYC database.

## Data availability

The data underlying this article will be placed in the GTC archive.

## References

  * Acton et al. (2021) Acton J. S., et al., 2021, arXiv e-prints, p. arXiv:2105.08574
  * Baraffe et al. (2015) Baraffe I., Homeier D., Allard F., Chabrier G., 2015, A&A, 577, A42
  * Bardalez Gagliuffi et al. (2019) Bardalez Gagliuffi D. C., et al., 2019, ApJ, 883, 205
  * Basri & Martín (1999) Basri G., Martín E. L., 1999, AJ, 118, 2460
  * Bildsten et al. (1997) Bildsten L., Brown E. F., Matzner C. D., Ushomirsky G., 1997, ApJ, 482, 442
  * Bochanski et al. (2010) Bochanski J. J., Hawley S. L., Covey K. R., West A. A., Reid I. N., Golimowski D. A., Ivezić Ž., 2010, AJ, 139, 2679
  * Bressan et al. (2012) Bressan A., Marigo P., Girardi L., Salasnich B., Dal Cero C., Rubele S., Nanni A., 2012, MNRAS, 427, 127
  * Burgasser & Blake (2009) Burgasser A. J., Blake C. H., 2009, AJ, 137, 4621
  * Burgasser et al. (2003) Burgasser A. J., Kirkpatrick J. D., Liebert J., Burrows A., 2003, ApJ, 594, 510
  * Burgasser et al. (2015) Burgasser A. J., et al., 2015, AJ, 149, 104
  * Burke et al. (2004) Burke C. J., Pinsonneault M. H., Sills A., 2004, ApJ, 604, 272
  * Burrows et al. (1997) Burrows A., et al., 1997, ApJ, 491, 856
  * Burrows et al. (2002) Burrows A., Ram R. S., Bernath P., Sharp C. M., Milsom J. A., 2002, ApJ, 577, 986
  * Cepa et al. (2000) Cepa J., et al., 2000, in M. Iye & A. F. Moorwood ed., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series Vol. 4008, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series. pp 623–631
  * Chabrier et al. (1996) Chabrier G., Baraffe I., Plez B., 1996, ApJ, 459, L91
  * Costa et al. (2005) Costa E., Méndez R. A., Jao W. C., Henry T. J., Subasavage J. P., Brown M. A., Ianna P. A., Bartlett J., 2005, AJ, 130, 337
  * Cruz et al. (2003) Cruz K. L., Reid I. N., Liebert J., Kirkpatrick J. D., Lowrance P. J., 2003, AJ, 126, 2421
  * Cruz et al. (2007) Cruz K. L., et al., 2007, AJ, 133, 439
  * Cruz et al. (2009) Cruz K. L., Kirkpatrick J. D., Burgasser A. J., 2009, AJ, 137, 3345
  * Curtis et al. (2019) Curtis J. L., Agüeros M. A., Douglas S. T., Meibom S., 2019, ApJ, 879, 49
  * Dahm (2015) Dahm S. E., 2015, ApJ, 813, 108
  * Dieterich et al. (2018) Dieterich S. B., et al., 2018, ApJ, 865, 28
  * Dupuy & Liu (2012) Dupuy T. J., Liu M. C., 2012, ApJS, 201, 19
  * Dupuy & Liu (2017) Dupuy T. J., Liu M. C., 2017, ApJS, 231, 15
  * Dupuy et al. (2009) Dupuy T. J., Liu M. C., Ireland M. J., 2009, ApJ, 699, 168
  * Dupuy et al. (2019) Dupuy T. J., et al., 2019, AJ, 158, 174
  * Eker et al. (2015) Eker Z., et al., 2015, AJ, 149, 131
  * Faherty et al. (2014) Faherty J. K., Beletsky Y., Burgasser A. J., Tinney C., Osip D. J., Filippazzo J. C., Simcoe R. A., 2014, ApJ, 790, 90
  * Feng et al. (2019) Feng F., et al., 2019, ApJS, 242, 25
  * Gaia Collaboration et al. (2018) Gaia Collaboration Brown A. G. A., Vallenari A., Prusti T., de Bruijne J. H. J., Babusiaux C., Bailer-Jones C. A. L., 2018, preprint
  * Gharib-Nezhad et al. (2021) Gharib-Nezhad E., Marley M. S., Batalha N. E., Visscher C., Freedman R. S., Lupu R. E., 2021, arXiv e-prints, p. arXiv:2106.00781
  * Hunter (2007) Hunter J. D., 2007, Computing in Science and Engineering, 9
  * Kendall et al. (2004) Kendall T. R., Delfosse X., Martín E. L., Forveille T., 2004, A&A, 416, L17
  * Kenworthy et al. (2001) Kenworthy M., et al., 2001, ApJ, 554, L67
  * King et al. (2010) King R. R., McCaughrean M. J., Homeier D., Allard F., Scholz R. D., Lodieu N., 2010, A&A, 510, 99
  * Kirkpatrick et al. (2000) Kirkpatrick J. D., et al., 2000, AJ, 120, 447
  * Kirkpatrick et al. (2001) Kirkpatrick J. D., Dahn C. C., Monet D. G., Reid I. N., Gizis J. E., Liebert J., Burgasser A. J., 2001, AJ, 121, 3235
  * Kirkpatrick et al. (2008) Kirkpatrick J. D., et al., 2008, ApJ, 689, 1295
  * Knapp et al. (2004) Knapp G. R., et al., 2004, AJ, 127, 3553
  * Kumar (1963) Kumar S. S., 1963, ApJ, 137, 1121
  * Lazorenko & Sahlmann (2018) Lazorenko P. F., Sahlmann J., 2018, A&A, 618, A111
  * Liebert & Gizis (2006) Liebert J., Gizis J. E., 2006, PASP, 118, 659
  * Lodders (1999) Lodders K., 1999, ApJ, 519, 793
  * Lodieu et al. (2015a) Lodieu N., Zapatero Osorio M. R., Rebolo R., Béjar V. J. S., Pavlenko Y., Pérez-Garrido A., 2015a, A&A, 581, A73
  * Lodieu et al. (2015b) Lodieu N., Zapatero Osorio M. R., Rebolo R., Béjar V. J. S., Pavlenko Y., Pérez-Garrido A., 2015b, A&A, 581, A73
  * Lodieu et al. (2018) Lodieu N., Rebolo R., Pérez-Garrido A., 2018, A&A, 615, L12
  * Magazzu et al. (1993) Magazzu A., Martin E. L., Rebolo R., 1993, ApJ, 404, L17
  * Maíz Apellániz & Weiler (2018) Maíz Apellániz J., Weiler M., 2018, A&A, 619, A180
  * Martin & Rebolo (1993) Martin E. L., Rebolo R., 1993, A&A, 274, 274
  * Martín et al. (1999) Martín E. L., Delfosse X., Basri G., Goldman B., Forveille T., Zapatero Osorio M. R., 1999, AJ, 118, 2466
  * Martín et al. (2006) Martín E. L., Brandner W., Bouy H., Basri G., Davis J., Deshpande R., Montgomery M. M., 2006, A&A, 456, 253
  * Martín et al. (2010) Martín E. L., et al., 2010, A&A, 517, A53
  * Martín et al. (2018) Martín E. L., Lodieu N., Pavlenko Y., Béjar V. J. S., 2018, ApJ, 856, 40
  * Martín et al. (2020) Martín E. L., Lodieu N., Béjar V. J. S., 2020, A&A, 640, A9
  * Nelson et al. (1993) Nelson L. A., Rappaport S., Chiang E., 1993, ApJ, 413, 364
  * Phan-Bao et al. (2008) Phan-Bao N., et al., 2008, MNRAS, 383, 831
  * Phillips et al. (2020) Phillips M. W., et al., 2020, A&A, A38, 637
  * Pineda et al. (2016) Pineda J. S., Hallinan G., Kirkpatrick J. D., Cotter G., Kao M. M., Mooley K., 2016, ApJ, 826, 73
  * Pozio (1991) Pozio F., 1991, Mem. Soc. Astron. Italiana, 62, 171
  * Rajpurohit et al. (2013) Rajpurohit A. S., Reylé C., Allard F., Homeier D., Schultheis M., Bessell M. S., Robin A. C., 2013, preprint
  * Rebolo et al. (1995) Rebolo R., Zapatero Osorio M. R., Martín E. L., 1995, Nature, 377, 129
  * Rebolo et al. (1996) Rebolo R., Martin E. L., Basri G., Marcy G. W., Zapatero-Osorio M. R., 1996, ApJ, 469, L53
  * Reid et al. (2006) Reid I. N., Lewitus E., Burgasser A. J., Cruz K. L., 2006, ApJ, 639, 1114
  * Reid et al. (2008) Reid I. N., Cruz K. L., Kirkpatrick J. D., Allen P. R., Mungall F., Liebert J., Lowrance P., Sweet A., 2008, AJ, 136, 1290
  * Sahlmann et al. (2014) Sahlmann J., Lazorenko P. F., Ségransan D., Martín E. L., Mayor M., Queloz D., Udry S., 2014, A&A, 565, A20
  * Sahlmann et al. (2015a) Sahlmann J., Lazorenko P. F., Ségransan D., Martín E. L., Mayor M., Queloz D., Udry S., 2015a, A&A, 577, A15
  * Sahlmann et al. (2015b) Sahlmann J., et al., 2015b, A&A, 579, A61
  * Sahlmann et al. (2021) Sahlmann J., et al., 2021, MNRAS, 500, 5453
  * Schmidt et al. (2010) Schmidt S. J., West A. A., Hawley S. L., Pineda J. S., 2010, AJ, 139, 1808
  * Simon et al. (2006) Simon M., Bender C., Prato L., 2006, ApJ, 644, 1183
  * Soubiran et al. (2016) Soubiran C., Le Campion J.-F., Brouillet N., Chemin L., 2016, A&A, 591, A118
  * Thorstensen & Kirkpatrick (2003) Thorstensen J. R., Kirkpatrick J. D., 2003, PASP, 115, 1207
  * Tinney (1998) Tinney C. G., 1998, MNRAS, 296, L42
  * Tody (1993) Tody D., 1993, in Hanisch R. J., Brissenden R. J. V., Barnes J., eds, Astronomical Society of the Pacific Conference Series Vol. 52, Astronomical Data Analysis Software and Systems II. p. 173
  * Virtanen et al. (2020) Virtanen P., et al., 2020, Nature Methods, 17, 261
  * Zapatero Osorio et al. (2004) Zapatero Osorio M. R., Lane B. F., Pavlenko Y., Martín E. L., Britton M., Kulkarni S. R., 2004, ApJ, 615, 958
  * Zapatero Osorio et al. (2005) Zapatero Osorio M. R., Martín E. L., Lane B. F., Pavlenko Y., Bouy H., Baraffe I., Basri G., 2005, Astronomische Nachrichten, 326, 948
  * Zentner & Hearin (2011) Zentner A. R., Hearin A. P., 2011, Phys. Rev. D, 84, 101302
  * del Burgo & Allende Prieto (2016) del Burgo C., Allende Prieto C., 2016, MNRAS, 463, 1400
  * del Burgo & Allende Prieto (2018) del Burgo C., Allende Prieto C., 2018, MNRAS, 479, 1953


# SCENE TEXT RECOGNITION MODELS EXPLAINABILITY USING LOCAL FEATURES

###### Abstract

Explainable AI (XAI) is the study on how humans can be able to understand the
cause of a model’s prediction. In this work, the problem of interest is Scene
Text Recognition (STR) Explainability, using XAI to understand the cause of an
STR model’s prediction. Recent XAI literatures on STR only provide a simple
analysis and do not fully explore other XAI methods. In this study, we
specifically work on data explainability frameworks, called attribution-based
methods, that explain the important parts of an input data in deep learning
models. However, integrating them into STR produces inconsistent and
ineffective explanations, because they only explain the model in the global
context. To solve this problem, we propose a new method, STRExp, to take into
consideration the local explanations, i.e. the individual character prediction
explanations. This is then benchmarked across different attribution-based
methods on different STR datasets and evaluated across different STR models.

Index Terms—  Computer Vision, Scene Text Recognition, Explainable AI.

##  1 Introduction

Scene Text Recognition (STR) [1, 2, 3] refers to the act of reading text from
a natural scene setting. STR images are usually more difficult to predict than
optical character recognition (OCR) images from scanned documents. There is a
large research literature [3] addressing these present challenges to obtain
the best STR model predictor.

Many of these works, however, only focus on solving the main problem of
increasing model accuracy. They do not address the problem of explaining why
STR models work in general. Previous experiments focus on attention maps [4,
5, 6] that only provide a simple analysis and do not focus on providing
explainability to STR. The vast majority of XAI methods out there [7, 8, 9] in
computer vision are usually only evaluated on single-class image
classification tasks and are generally ineffective in explaining STR networks.
STR predictors are black-box models constituting various deep neural network
architectures and comprising of a multi-class output prediction, making them
more challenging to understand. Motivated by this, a study is presented
merging both XAI and STR, called Scene Text Recognition (STR) Explainability.
We ask the question related to XAI, ”Why do STR models work?” [10, 7, 11, 12].
This question focuses on trying to explain the input data of a deep learning
model and persuades us to learn more about the need to explain why STR models
work in general. Not only does explainability benefit AI engineers, but they
can also provide the explanations to convince non-AI experts that these models
are trustable and safe to use, even in high stake decisions.

Fig. 1: Examples of attribution-based method explanations, wherein greener
areas are more important and red areas are less important. When executed on
STR models, previous attribution-based methods produce inconsistent and
ineffective data explanations (first 11 images), that produce false positives
(green areas far away from the text), and false negatives (red areas near the
text). STRExp reduces this explanation inconsistency (12th image) by placing
more importance in the actual text areas of the image.

In this work, the problem of interest is in STR Data Explainability, which
focuses on providing explainability to the input data. To the best of our
knowledge, there is only little work on Explainable STR. Thus, we focus on
recent literatures, called interpretable multi-label classification, that have
some similar characteristics with our problem. However, integrating these to
previous attribution-based methods lead to inconsistent and ineffective
explanations in STR models (Fig. 1). This is because these works only try to
explain multi-label class models in the global context [13, 14, 15] (Fig. 2).
To solve this problem, we propose to execute the explanations to include the
local individual character explanations. Combining both the local and global
explanations produces stronger average explanations on the input. Data
explanation provides evidence to understand why an STR model works. This
increases model trustability and simplicity [9].

In summary, the contributions of this work are as follows: (1) To the best of
our knowledge, this work is the first in creating a new data explainability
framework specifically made for the task of STR. (2) The local explanations of
STR models are leveraged and combined with the global explanations to reduce
the inconsistency and ineffectiveness of previous attribution-based methods.
We call this STRExp. (3) Our method is then benchmarked across different STR
datasets, and show superior explanation performance when compared to previous
attribution-based methods across different STR models.

##  2 Related Works

The area relating to the task of explaining the input features of a host model
are typically associated with attribution-based explanations. In the current
literature, attribution-based explanation methods [9, 16, 15] make up the
majority of machine learning explanations. These methods normally interpret
which of the individual input features contribute most to the output’s
prediction. However, most of their explanation examples in computer vision are
applied to single-label image classification problems. Existing works
implementing them into multi-label classification systems show simple and
ineffective explanations. Thus, the task of integrating them into Scene Text
Recognition (STR), a subset of a multi-label classification problem, is a
major challenge in itself.

Attribution-based explanations leverage the model features extracted in order
to create an interpretable data representation. Popular methods such as LIME
[15] and SHAP [16] provide feature-based explanations in terms of highlighting
pixels/superpixels. However, in the context of a multi-label classification
problem like STR, these attribution-based methods suggest on transforming the
former into a single-label class output [13, 15]. Thus, this form of
explainability execution can only output the model’s attributions in the
global context. In our case, we also take into consideration the individual
characters of the STR model to reduce the ineffectiveness produced by these
previous method’s explanations.

##  3 Methodology

###  3.1 STR Attribution-based methods

An attribution-based method, $\mathbb{A}$, consists of an iterative algorithm
that uses either the forwardpass parameters, backwardpass parameters, or both,
to compute an explanation (Fig. 1) of a model. Given an STR model $M$, its
parameters $\theta$, and one of the attribution-based methods
$A_{i}\in\mathbb{A}$, the latter’s explanation is given by
$E(X)=p^{attr}=A_{i}(M_{\theta}(X))$. This is also called the attributions of
$A_{i}$. Thus, for each input feature $X=\\{X_{1},X_{2},...,X_{n}\\}$, its
attribution representation is also given by
$E(X)=\\{E(X_{1}),...,E(X_{n})\\}$.

Fig. 2: Previous attribution-based methods only execute their explanations in
the global context of the STR model. We further improve data explainability by
querying the best attribution-based methods, and then combining both the local
and global explanations. This new method is called STRExp.

The calculation of the attribution $E(X)$ is done using either the forwardpass
parameters $L$, backwardpass parameters $G$, or both, depending on the
attribution-based method. The forwardpass parameters refers to the layer maps
during model prediction, $L=\\{L_{1},L_{2},...,L_{n}\\}$, where $L_{1}$ refers
to the first layer, $L_{2}$ refers to the second layer, etc. The backwardpass
parameters refers to the gradients of each layer during backpropagation, set
to $G=\frac{\partial e(Y,\hat{Y})}{\partial\theta}=\\{\frac{\partial
e(Y,\hat{Y})}{\partial\theta_{L_{1}}},\frac{\partial
e(Y,\hat{Y})}{\partial\theta_{L_{2}}},...,\frac{\partial
e(Y,\hat{Y})}{\partial\theta_{L_{n}}}\\}$, where $e(Y,\hat{Y})$ is some error
function between the target value $Y$ and the predicted value $\hat{Y}$ with
respect to the model parameters at a specific layer $\theta_{L}$.

###  3.2 Global and Local Explanations

For each input features $X=\\{X_{1},X_{2},...,X_{n}\\}$, an STR model $M$ with
parameters $\theta$ must predict a sequence of label space
$Y=\\{Y_{1},Y_{2},...,Y_{n}\\}$. Its corresponding score is given by
$r=R(M_{\theta}(x))$, where $R$ is some function that calculates the mean
output to convert it into a single-label space. Each attribution-based method
is set as $A_{i}\in\mathbb{A}$. Thus, each attribution/explanation output is
then set to $p^{attr}_{global}=A_{i}(x,r)$, which indicates that this
explanation is in the global context of the model $M_{\theta}$, capturing the
STR model’s distribution
$Y=\\{Y_{1},Y_{2},...,Y_{n}\\}=P(\\{X_{1},X_{2},...,X_{n}\\}|\theta)$. After
evaluating each explanations, an explainability evaluator metric $\gamma$ is
used to query the best $A_{i}$. We used selectivity [17] for $\gamma$. For
each input feature segmentation, $X=\\{X_{1},X_{2},...,X_{n}\\}$, its
corresponding attribution, $s^{attr}_{x}=\mathbb{E}(p^{attr}_{x})$, is
acquired. Finally, the total performance for $A_{i}$ is set to
$z_{i}=\gamma([s^{attr}_{x_{1}},s^{attr}_{x_{2}},...,s^{attr}_{x_{m}}])$ for
all total segmentations $m$. After minimizing the Selectivity Area Under the
Curve (AUC), $min(E_{A_{1}}(X)),E_{A_{2}}(X)),...,E_{A_{t}}(X)))$, the best
attribution-based method $B=\mathbb{A}(z)$ is queried.

Fig. 3: From top to bottom: VITSTR[18](1st & 2nd figure), PARSeq[19](3rd & 4th
figure), TRBA[1](5th & 6th figure), SRN[2](7th & 8th figure), ABINET[20](9th &
10th figure) and MATRN[21](11th & 12th figure) quantitative results.

The problem with using the global context is that it provides incoherent
gradients to the attribution-based method. You can only impose a single target
value of $\frac{\partial e(Y=1,\hat{Y})}{\partial\theta}$ during
backpropagation for all STR outputs. This leads to the main problem of
producing those inconsistent and ineffective data explanations. To solve this
problem, we propose to involve the local explanations of each individual
character prediction. Thus, the value of its gradients will be
$G_{c}=\\{\frac{\partial e(Y=c_{1},\hat{Y})}{\partial\theta},\frac{\partial
e(Y=c_{2},\hat{Y})}{\partial\theta},...,\frac{\partial
e(Y=c_{n},\hat{Y})}{\partial\theta}\\}$ where $c_{i}$ are the individual
target character values of the STR model. This solves the gradient incoherency
produced by the global explanations. We build from the best attribution-based
method $B$ and execute it in the local context of the STR model. For each
image $I$ with input features $X=\\{X_{1},X_{2},...,X_{n}\\}$, there is a
fixed number of total characters $c\in C$. The local score of the individual
character is then computed by $r_{c_{k}}=R(M_{\theta}(x),k)$ and its
attribution is computed by $p^{attr}_{local}=B(x,r_{c_{k}})$. Here, $c_{k}$ is
the character $c$ at the $kth$ position. This attribution describes the STR
model’s single-character prediction distribution
$Y_{i}=P(\\{X_{1},X_{2},...,X_{n}\\}|\theta)$, without depending on the
outputs of the other characters. This process is executed for each other
characters $c$ until there are $k$ individual character attributions. As
described in Fig. 2, these local explanations are also combined with the
global explanation of the best attribution-based method,
$p^{attr}_{global}=B(x,r)$ to further improve the data explanation
performance. The final attribution is then set to
$p^{final}_{final}=\mathbb{E}(p^{attr}_{local},p^{attr}_{global})$, and then
its selectivity is again computed to compare it with previous attribution-
based methods. The code can be found in https://github.com/markytools/strexp.

##  4 Results and Discussion

###  4.1 Metric Evaluation

The metrics used are derived from the selectivity metric. Explanation
selectivity [17] measures how fast a function $f(x)$ goes down when removing
features with the highest relevance. The highest scored segmentation feature
from an attribution-based method explanation is removed and the STR
performance is plotted (y-axis). This process is repeated iteratively until
all input features have been removed (x-axis). The final selectivity value
will be the area under the curve (AUC). The best attribution-based method will
have the lowest selectivity (i.e. the lowest AUC), suggesting that removing
this method’s most important areas in the image first will coincide with a
larger drop in the STR model performance compared to other attribution-based
methods. Two STR performances are used to derive two new metrics from
selectivity. One is using the STR text accuracy called Selectivity Accuracy,
while the other is using the STR mean confidence called Selectivity
Confidence. For each dataset, the average selectivity is computed across all
images.

Fig. 4: From top to bottom: PARSeq[19](1st & 2nd figure), SRN[2](3rd & 4th
figure), and TRBA[1](5th & 6th figure) qualitative results.

###  4.2 STRExp Quantitative Results

Using the selectivity metric, the quantitative results are obtained to compare
STRExp with previous attribution-based methods across different STR real
datasets and across different STR models. For evaluation, the STR model
architectures used are: VITSTR [18], TRBA [1], PARSeq [19], SRN [2], ABINET
[20], and MATRN [21]. The attribution-based methods (and their abbreviations)
to be compared are [14]: Integrated Gradients (IntGrd), GradientSHAP (GSHAP),
DeepLift (DpLft), Saliency (Sal), Input X Gradient (IXG), Guided Backprop (Gd-
Bp), Deconvolution (Deconv), KernelSHAP (KSHAP), Feature Ablation (FtAblt),
LIME (LIME), and Shapley (Shapley). The evaluation is done on different real-
world STR test datasets [3]: CUTE80, SVT, IIIT5k_3000, SVTP, IC03_860,
IC03_867, IC13_857, IC13_1015, IC15_1811, IC15_2077.

Fig. 3 shows the benchmarks of STRExp. The y-axis in the figure represents
either the selectivity accuracy or selectivity confidence metric. The x-axis
represents the abbreviations of the attribution-based methods to be compared.
The line colors represent the different STR datasets. Thus, a single (x,y)
coordinate represents the selectivity of the attribution-based method when
evaluated on that dataset. The dot/point of each line signifies the
attribution-based method that has the lowest selectivity on that dataset.
Ours(GL) refers to STRExp having its output combined with both the global and
local explanations, while Ours(L) refers to STRExp only using the local
explanations. STRExp is executed to produce the explanations of the VITSTR STR
Model [18], PARSeq STR Model [19], TRBA STR Model[1], SRN STR Model[2], ABINET
STR Model [20], and MATRN STR Model [21]. The results show that our proposed
method, STRExp, generally has the lowest selectivity accuracy and lowest
selectivity confidence in the majority of cases compared to previous
attribution-based methods evaluated across different STR datasets.

###  4.3 STRExp Qualitative Results

The general traits/conditions for a good qualitative image are: (1) The best
attribution-based method will have greener areas inside and near the text
compared to previous methods, because hiding these areas of the image first
(according to the selectivity metric) will produce a greater accuracy drop
compared to other areas. (2) The best attribution-based method will have less
red areas inside and near the text compared to previous methods. It does not
make any sense to hide the text areas last when trying to produce a lower
selectivity.

In the top of Fig. 4, STRExp is evaluated on PARSeq[19] STR Model in some
image samples of the IC03_867 (first 5 images) and the IC13_857 (last 5
images) datasets, showing how a low selectivity accuracy from Fig. 3 impacts
the results visually. The first image column with the ”QUEEN” text has more
greener colors near and inside the text in STRExp compared to previous
attribution-based methods. In the second column ”BREAK”, STRExp has more
greener areas and less redder areas in the text compared to previous methods.
This trend follows for all other image columns in the figure.

In the middle of Fig. 4, another STR Model, SRN[2], is evaluated on CUTE80
(first 5 images) and SVT (last 5 images) datasets, showing how a low
selectivity accuracy from Fig. 3 impacts the results visually. In the bottom
of Fig. 4, the TRBA[1] STR Model is evaluated on the IC03_860 (first 5 images)
and IC15_2077 (last 5 images) datasets, showing how a low selectivity accuracy
from Fig. 3 impacts the results visually.

##  5 Conclusion

This work proposes STRExp that leverages the local individual character
explanations to produce better STR explanations compared to previous
attribution-based methods.

##  6 Acknowledgement

DOST ERDT scholarships and ERDT FRDG.

## References

  * [1] Jeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, and Hwalsuk Lee,  “What is wrong with scene text recognition model comparisons? dataset and model analysis,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4715–4723.
  * [2] Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, and Errui Ding,  “Towards accurate scene text recognition with semantic reasoning networks,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12113–12122.
  * [3] Xiaoxue Chen, Lianwen Jin, Yuanzhi Zhu, Canjie Luo, and Tianwei Wang,  “Text recognition in the wild: A survey,”  ACM Computing Surveys (CSUR), vol. 54, no. 2, pp. 1–35, 2021.
  * [4] Yi-Chao Wu, Fei Yin, Xu-Yao Zhang, Li Liu, and Cheng-Lin Liu,  “Scan: Sliding convolutional attention network for scene text recognition,”  arXiv preprint arXiv:1806.00578, 2018.
  * [5] Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai,  “Aster: An attentional scene text recognizer with flexible rectification,”  IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 9, pp. 2035–2048, 2018.
  * [6] Sahar Siddiqui, Elena Sizikova, Gemma Roig, Najib J Majaj, and Denis G Pelli,  “Using human psychophysics to evaluate generalization in scene text recognition models,”  arXiv preprint arXiv:2007.00083, 2020.
  * [7] Tim Miller,  “Explanation in artificial intelligence: Insights from the social sciences,”  Artificial intelligence, vol. 267, pp. 1–38, 2019.
  * [8] Nadia Burkart and Marco F Huber,  “A survey on the explainability of supervised machine learning,”  Journal of Artificial Intelligence Research, vol. 70, pp. 245–317, 2021.
  * [9] Jianlong Zhou, Amir H Gandomi, Fang Chen, and Andreas Holzinger,  “Evaluating the quality of machine learning explanations: A survey on methods and metrics,”  Electronics, vol. 10, no. 5, pp. 593, 2021.
  * [10] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal,  “Explaining explanations: An overview of interpretability of machine learning,”  in 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA). IEEE, 2018, pp. 80–89.
  * [11] Christoph Molnar,  “A guide for making black box models explainable,”  URL: https://christophm. github. io/interpretable-ml-book, 2018.
  * [12] Benjamin Shickel and Parisa Rashidi,  “Sequential interpretability: Methods, applications, and future direction for understanding deep learning models in the context of sequential data,”  arXiv preprint arXiv:2004.12524, 2020.
  * [13] Oscar Luaces, Jorge Díez, José Barranquero, Juan José del Coz, and Antonio Bahamonde,  “Binary relevance efficacy for multilabel classification,”  Progress in Artificial Intelligence, vol. 1, no. 4, pp. 303–313, 2012.
  * [14] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al.,  “Captum: A unified and generic model interpretability library for pytorch,”  arXiv preprint arXiv:2009.07896, 2020.
  * [15] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin,  “” why should i trust you?” explaining the predictions of any classifier,”  in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016, pp. 1135–1144.
  * [16] Scott M Lundberg and Su-In Lee,  “A unified approach to interpreting model predictions,”  Advances in neural information processing systems, vol. 30, 2017.
  * [17] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller,  “Methods for interpreting and understanding deep neural networks,”  Digital Signal Processing, vol. 73, pp. 1–15, 2018.
  * [18] Rowel Atienza,  “Vision transformer for fast and efficient scene text recognition,”  in International Conference on Document Analysis and Recognition. Springer, 2021, pp. 319–334.
  * [19] Darwin Bautista and Rowel Atienza,  “Scene text recognition with permuted autoregressive sequence models,”  in European Conference on Computer Vision. Springer, 2022, pp. 178–196.
  * [20] Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang,  “Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7098–7107.
  * [21] Byeonghu Na, Yoonsik Kim, and Sungrae Park,  “Multi-modal text recognition networks: Interactive enhancements between visual and semantic features,”  in European Conference on Computer Vision. Springer, 2022, pp. 446–463.