# The First High-Contrast Images of X-Ray Binaries: Detection of Candidate
Companions in the $\gamma$ Cas Analog RX J1744.7$-$2713

M. Prasow-Émond Département de Physique, Université de Montréal, C.P. 6128,
Succ. Centre-Ville, Montréal, QC H3C 3J7, Canada  Institute for Research on
Exoplanets, Université de Montréal, Département de Physique, C.P. 6128 Succ.
Centre-ville, Montréal, QC H3C 3J7, Canada  J. Hlavacek-Larrondo Département
de Physique, Université de Montréal, C.P. 6128, Succ. Centre-Ville, Montréal,
QC H3C 3J7, Canada  K. Fogarty Division of Physics, Math, and Astronomy,
California Institute of Technology, Pasadena, CA, USA  NASA Ames Research
Center, Moffett Field, CA 94035, USA  J. Rameau Institute for Research on
Exoplanets, Université de Montréal, Département de Physique, C.P. 6128 Succ.
Centre-ville, Montréal, QC H3C 3J7, Canada  Univ. Grenoble Alpes, CNRS, IPAG,
F-38000 Grenoble, France  L.-S. Guité Département de Physique, Université de
Montréal, C.P. 6128, Succ. Centre-Ville, Montréal, QC H3C 3J7, Canada  D.
Mawet Division of Physics, Math, and Astronomy, California Institute of
Technology, Pasadena, CA, USA  Jet Propulsion Laboratory, California Institute
of Technology, Pasadena, CA 91109, USA  P. Gandhi Department of Physics and
Astronomy, University of Southampton, Highfield, Southampton, SO17 1BJ, United
Kingdom  A. Rao  Department of Physics and Astronomy, University of
Southampton, Highfield, Southampton, SO17 1BJ, United Kingdom  J. F. Steiner
Harvard-Smithsonian Center for Astrophysics, Cambridge, MA 02138, United
States  É. Artigau Institute for Research on Exoplanets, Université de
Montréal, Département de Physique, C.P. 6128 Succ. Centre-ville, Montréal, QC
H3C 3J7, Canada  D. Lafrenière Département de Physique, Université de
Montréal, C.P. 6128, Succ. Centre-Ville, Montréal, QC H3C 3J7, Canada
Institute for Research on Exoplanets, Université de Montréal, Département de
Physique, C.P. 6128 Succ. Centre-ville, Montréal, QC H3C 3J7, Canada  A.
Fabian Institute of Astronomy, Cambridge University, Madingley Road, Cambridge
CB3 0HA, United Kingdom  D. J. Walton  Institute of Astronomy, Cambridge
University, Madingley Road, Cambridge CB3 0HA, United Kingdom  L. M. Weiss
Department of Physics, University of Notre Dame, Notre Dame, IN 46556, USA  R.
Doyon Département de Physique, Université de Montréal, C.P. 6128, Succ.
Centre-Ville, Montréal, QC H3C 3J7, Canada  Institute for Research on
Exoplanets, Université de Montréal, Département de Physique, C.P. 6128 Succ.
Centre-ville, Montréal, QC H3C 3J7, Canada  C. L. Rhea Département de
Physique, Université de Montréal, C.P. 6128, Succ. Centre-Ville, Montréal, QC
H3C 3J7, Canada  T. Bégin Département de Physique, Université de Montréal,
C.P. 6128, Succ. Centre-Ville, Montréal, QC H3C 3J7, Canada  B. Vigneron
Département de Physique, Université de Montréal, C.P. 6128, Succ. Centre-
Ville, Montréal, QC H3C 3J7, Canada  M.-E. Naud Institute for Research on
Exoplanets, Université de Montréal, Département de Physique, C.P. 6128 Succ.
Centre-ville, Montréal, QC H3C 3J7, Canada

###### Abstract

X-ray binaries provide exceptional laboratories for understanding the physics
of matter under the most extreme conditions. Until recently, there were few,
if any, observational constraints on the circumbinary environments of X-ray
binaries at $\sim$ 100–5000 AU scales; it remains unclear how the accretion
onto the compact objects or the explosions giving rise to the compact objects
interact with their immediate surroundings. Here, we present the first high-
contrast adaptive optics images of X-ray binaries. These observations target
all X-ray binaries within $\sim$ 3 kpc accessible with the Keck/NIRC2 vortex
coronagraph. This paper focuses on one of the first key results from this
campaign: our images reveal the presence of 21 sources potentially associated
with the $\gamma$ Cassiopeiae analog high-mass X-ray binary RX
J1744.7$-$2713\. By conducting different analyses – a preliminary proper
motion analysis, a color-magnitude diagram and a probability of chance
alignment calculation – we found that three of these 21 sources have a high
probability of being bound to the system. If confirmed, they would be in wide
orbits ($\sim$ 450 AU to 2500 AU). While follow-up astrometric observations
will be needed in $\sim$ 5–10 years to confirm further the bound nature of
these detections, these discoveries emphasize that such observations may
provide a major breakthrough in the field. In fact, they would be useful not
only for our understanding of stellar multiplicity, but also for our
understanding of how planets, brown dwarfs and stars can form even in the most
extreme environments.

stars: abundances — infrared: planetary systems — binaries: general

††journal: AJ

tablenum SIXtablenum

##  1 Introduction

Exoplanets can exist in extreme environments; in fact, Wolszczan & Frail
(1992) reported the discovery of a planetary system orbiting the millisecond
pulsar PSR B1257+12. These planets were both the first confirmed exoplanets,
as well as the first confirmed planets orbiting a compact object. In this
system, the planets are small (from a fraction to a few $M_{\oplus}$) and in
close orbits to the pulsar (up to a few astronomical units; hereafter AU).
Currently, there are just a few ($\sim$ 7) confirmed sub-stellar companions
orbiting pulsars (e.g., Bailes et al. 2011; Starovoit & Rodin 2017). These
planets are thought to originate from a fallback disk post-supernovae in which
they would have formed similarly as in protoplanetary disks (e.g., Yan et al.
2013), or simply from gravitational capture (e.g., Sigurdsson et al. 2003).

Since then, thousands of exoplanets and brown dwarfs have been discovered with
the advent of new, state-of-the-art instruments. These discoveries have
bettered our understanding of the Solar System and have shown that planets and
brown dwarfs can exist in a variety of environments: from hot Jupiters that
orbit exceedingly close to their host star (e.g., Seager & Sasselov 1998;
Evans et al. 2018) to planets found at orbital separations of up to several
thousand AU (e.g., Lafrenière et al. 2011; Naud et al. 2014). Overall, these
discoveries emphasize that we still do not fully understand the conditions
required for the formation of planets.

A recent study argued that sub-stellar companions could even exist in extreme
environments such as X-ray binaries (Imara & Di Stefano, 2018). These systems
consist of a compact stellar remnant (white dwarf, neutron star or black hole)
accreting material from a donor star, and their interaction releases strong
X-ray radiation (e.g., Tauris & van den Heuvel 2006). They have been
extensively studied and monitored for several decades enabling a variety of
major breakthroughs on accretion physics (e.g., Kara et al. 2019 for a recent
example), as well as on the formation of outflows under extreme magnetic field
conditions (see Fender et al. 2004 for a review). X-ray binaries are thus
unique laboratories for studying a variety of astronomical phenomena under
extreme conditions.

Although studies indicate that planets orbiting individual pulsars may be rare
(e.g., Loehmer et al. 2004), a planet in close orbit to an X-ray binary could
produce X-ray eclipses detectable with current instruments (Imara & Di
Stefano, 2018). However, binary systems, such as X-ray binaries, may be more
likely to host wide orbit companions (up to several hundred to thousand of
AUs) because of planet-planet or planet-star interactions (e.g., Bonavita et
al. 2016). Simulations also predict that giant planets and brown dwarfs can
survive stellar explosions such as those undergone by X-ray binaries, but that
these may tend to push out the companions to wider orbits (e.g., Welsh 2013).
In this case, detecting sub-stellar companions via transits or radial velocity
shifts may not be optimal. In addition, in the case of high-mass X-ray
binaries (HMXB), the companion star has not yet exploded, implying that the
system should be fairly young (i.e., less than a few dozen Myrs, e.g., Girardi
et al. 1996). Sub-stellar companions, if present, could therefore still be
bright and amenable for detection via direct imaging (e.g., Burrows et al.
2001).

As part of a pilot study aiming to explore the immediate environments of X-ray
binaries, we have obtained the first high-contrast adaptive optics images of
X-ray binaries ideally suited for detecting companions that have formed in
situ at large ($\sim$ 100-5000 AU) radii or have been pushed out to large
radii. These images consist of NIRC2 observations taken with Keck targeting
all X-ray binaries within $\sim$ 3 kpc.

Here, we present the first set of observations, targeting the $\gamma$
Cassiopeiae-like HMXB harbouring a Be donor star, RX J1744.7$-$2713\. In
Section 2, we present the observations and data reduction. In Section 3, we
show the high-contrast images of RX J1744.7$-$2713, the background/foreground
contaminant probability, the color-magnitude diagram, as well as how we
determined the nature of the detections. In Section 4, we discuss our results
and their implications, while in Section 5 we summarize and discuss future
perspectives.

Figure 1: Left: Keck/NIRC2 $L^{\prime}$ high-contrast image of RX
J1744.7$-$2713 acquired on July 11 2020 and treated and reduced with a PCA
annular ADI algorithm. The data were processed using VIP (Gomez Gonzalez et
al., 2017). The 21 sources detected with SNR $>$ 5 when computing the signal-
to-noise map are labeled. The white X symbol denotes the position of the X-ray
binary masked by the coronagraph. The inset at the bottom is a zoom on the
central region to focus on the closest candidate at $\sim$ 450 AU. Right:
Keck/NIRC2 $K_{s}$ high-contrast image of RX J1744.7$-$2713 acquired on July
12 2020. Labeled are the same objects detected from the $L^{\prime}$ image.
The red X symbol denotes the position of the X-ray binary masked by the
coronagraph.

##  2 Observations and Data Reduction

###  2.1 RX J1744.7-2713

RX J1744.7$-$2713 (RA: 17h 44m 45.7659s; DEC: $-$27d 13h 44.477s) is located
at a distance of 1.22 $\pm$ 0.04 kpc (Gaia Collaboration et al., 2021) and was
first identified as a HMXB by Israel et al. (1997). The distance above is
based upon geometric parallax inversion and includes a zero-point offset
correction of $-$0.0348 mas according to the algorithm described in Lindegren
et al. (2021). Its optical component – HD 161103 – has been spectroscopically
characterized in Motch et al. (1997) and Lopes de Oliveira et al. (2006), and
is classified as a Be star.

RX J1744.7$-$2713 has similar properties to $\gamma$ Cassiopeiae (also known
as 2S 0053+604; e.g., Raguzova & Popov 2005; Shrader et al. 2015) and has an
X-ray luminosity of (3.08 $\pm$ 0.49) $\times$ 1032 erg s-1 (Naze & Motch,
2018). Although the origin of X-ray emission in RX J1744.4$-$2713 is still
debated, it may point to accretion onto a white dwarf (WD; Lopes de Oliveira
et al. 2006).

It should be noted that in the case of Be stars, the star’s massive outflows
lead to a diffuse and gaseous circumstellar disk known as a decretion disk
(e.g., Kravtsov et al. 2020), however these disks have radii at sub-AU scales,
which for RX J1744.7$-$2713 is well within the inner working angle of the
NIRC2 vortex coronagraph.

In the case of RX J1744.4$-$2713, Coleiro & Chaty (2013) estimated the age of
the system to be $\sim 60$ Myr by investigating the expected offset between
the position of the Galaxy’s spiral arms and HMXBs. However, considering the
uncertainties (e.g., see their Fig. 13), the age of RX J1744.4$-$2713 is
likely to be anywhere between a couple Myrs to 80 Myr (95% confidence). Given
these uncertainties, we consider the range of 5–60 Myr throughout this study
and note that even if we were to consider an age outside this range, our
conclusions remain the same.

Furthermore, the system has proper motions of $-0.955\pm 0.100$ mas/yr and
$-2.062\pm 0.077$ mas/yr in RA and DEC respectively and is located in the
Galactic plane, as most of the HMXBs in our galaxy (e.g., Tauris & van den
Heuvel 2006).

###  2.2 Observations

The Keck/NIRC2 vortex coronagraph (Mawet et al., 2005; Serabyn et al., 2016)
was used to observe RX J1744.7$-$2713 in pupil-tracking mode – with the
narrow-field camera (9.971 $\pm$ 0.004 mas/pixel; Service et al. 2016) – in
$L^{\prime}$-band in 2017 ($\lambda$ = 3.776 $\mu$m, $\Delta\lambda$ = 0.700
$\mu$m; PI: Mawet) and in both $L^{\prime}$-band and $K_{s}$-band in 2020
($\lambda$ = 2.146 $\mu$m, $\Delta\lambda$ = 0.311 $\mu$m; PI: Fogarty).
Observations were adaptive-optics (AO)-assisted with the Keck II Shack-
Hartmann wavefront sensor in 2017, which performs wavefront sensing in $R$,
and the infrared pyramid wavefront sensor (PyWFS) in 2020, which performs
wavefront sensing in $H$ (Wizinowich et al., 2000; Bond et al., 2018). Given
the $R$ and $H$ magnitudes of RX J1744.7$-$2713 ($R\sim 9$; Cutri et al. 2003,
$H\sim 7$; Zacharias et al. 2012), the performances of the two wavefront
sensors are comparable; however, we chose to use the PyWFS for our 2020
observations since sensing in $H$ is advantageous for several extremely red
HMXBs in our 2020 survey sample. Tip-tilt adjustments were performed with
Quadrant Analysis of Coronagraphic Images for Tip-tilt sensing (QACITS; Huby
et al. 2017) in order to keep the star well-centered on the vortex focal plane
mask.

Individual exposure times, coadds, total number of frames and total
integration times obtained for each observation are summarized in Table 1.

In our 2017 observations, we only obtained $\sim 14^{\circ}$ of field
rotation, but in our 2020 observations we obtained $\sim 40^{\circ}$ of
rotation in both bands to enable reduction using angular differential imaging
(ADI; Marois et al. 2006). Additional frames taken with RX J1744.7$-$2713
moved off-axis were taken throughout each run to sample the evolution of the
point-spread function (PSF) throughout the run and to obtain photometric
measurements of RX J1744.7$-$2713.

Table 1: Keck/NIRC2 Observing Log for RX J1744.7$-$2713\. Abbreviations: Wave Front Sensor (WFS), Parallactic angle coverage (P. A. cov.) UT Date | 2017 Sep 8 | 2020 Jul 11 | 2020 Jul 12
---|---|---|---
Filter | $L^{\prime}$ | $L^{\prime}$ | $K_{s}$
WFS | SH | py | py
Int. Time (s) | 0.5 | 0.5 | 0.6
Coadds | 60 | 60 | 45
Total frames | 40 | 120 | 92
P. A. cov. (o) | 14.3 | 38.3 | 39.6

###  2.3 Data Reduction

Data reduction (flat-fielding, bad pixel masking, sky removal, image
registration and selection, and centering via speckles) was performed using
the Vortex Image Processing (VIP) package (Gomez Gonzalez et al., 2017). The
frames were then de-rotated and mean-combined, and a Principal Component
Analysis (PCA) annular ADI algorithm from VIP was applied to obtain the high-
contrast images (see Figure 1). Note that the number of principal components
($n_{\mathrm{comp}}$) for each individual source was chosen to reach an
optimal signal-to-noise ratio (SNR), by testing many values ranging from 1 to
50. We then generated the SNR and significance (signal in terms of $\sigma$;
Mawet et al. 2014) maps to identify sources with a SNR greater than 5 (which
results in a signal greater than 4$\sigma$ for all of our sources). This also
provided estimates of the sources’ coordinates.

###  2.4 Magnitude Calculation

To calculate the full width at half maximum (FWHM) of the PSF for both
filters, we fit a gaussian profile to the median-collapsed PSF data cube. We
obtained a FWHM of 83.2 milliarcseconds (hereafter mas) and 47.5 mas for
$L^{\prime}$ and $K_{s}$ respectively. Note that we also generated a 2D
synthetic PSF for $L^{\prime}$ flux fitting – using a gaussian model and the
corresponding FWHM – because of an artefact caused by thermal background noise
from the primary mirror that has been folded back on-axis by the vortex phase
mask. However, the use of a synthetic PSF does not affect the flux fitting
much, as we get consistent results in the error bars when we test our $K_{s}$
data with both real and synthetic PSFs.

Using the reduced data cube and the normalized and centered synthetic PSF, we
injected fake companions with an inverse flux into the science data cube to
estimate the photometric counts, coordinates, and errors (Lagrange et al.,
2010) of any detected source with a SNR $>5$ in the $L^{\prime}$ SNR map. This
process uses a Nelder-Mead optimization algorithm to find the coordinates and
the flux of a source in a given aperture (a radius of 3 FWHM). We then divided
the counts of the sources by the counts of the HMXB’s median-collapsed PSF
data cube – measured in the same aperture as the sources – in order to obtain
the flux of the candidate companion relative to the X-ray binary (i.e., the
contrast).

Knowing the apparent magnitudes of RX J1744.7$-$2713 ($L^{\prime}$ = 5.809
$\pm$ 0.077 mag and $K_{s}$ = 6.507 $\pm$ 0.023 mag; Cutri et al. 2003), the
contrast can be converted to apparent magnitudes and absolute magnitudes
(using the known distance in parsec), and then to mass using a model from MESA
Isochrones & Stellar Tracks (MIST; Paxton et al. 2011, 2013, 2018, 2015;
Dotter 2016; Faherty et al. 2016; Choi et al. 2016).

Servillat et al. (2012) reported that there is no significant variability in
the light curve of HD 161103 (Be star in RX J1744.7$-$2713). However, large
time scale ($\sim$ 50–100 yr) variability could still occur. We therefore
injected a 5% additional error to the magnitudes to occur for any large time
scale variability, and adding even larger errors does not change our results.
As for the line-of-sight interstellar reddening $E(B-V)$, Schlafly &
Finkbeiner (2011) estimated it to be $E(B-V)\sim 3.4$. Since the extinction
$A(V)$ is related to $E(B-V)$ via $A(V)=3.2E(B-V)$, we have $A(V)\sim 10.9$,
and assuming a typical extinction power law varying as
$A(V)\propto\lambda^{-1.75}$ (Draine 1989; and see Doherty et al. 1994;
Viehmann et al. 2005; Dong et al. 2017; Uyama et al. 2020 for examples of
papers that use that technique), we obtain $A(K_{s})\sim 0.85$ and
$A(L\textquoteright)\sim 0.32$ (via a simple cross product). Therefore,
$K_{s}$ and $L^{\prime}$ are not strongly impacted by dust reddening; such an
extinction results in a color shift of $\sim 0.5$ mag for $K_{s}-L^{\prime}$
and a magnitude shift of $\sim$ 0.3 for $L^{\prime}$. These corrections have
been applied to our results; by adding them to the $K_{s}$ and $L^{\prime}$
magnitudes.

###  2.5 Injection/Recovery

In order to estimate the errors on the parameters – the flux and the position
(i.e., the separation from the XRB in mas and the parallactic angle in
degrees) – obtained with the optimization process, we proceeded by injection
recovery. We first injected fake companions with known values for the
parameters, and then ran the optimization code to compare the outputs with the
known inputs. For each parameter, we tested this method for 90 different
values, with a range covering the values of the true candidate companions of
the system. We found that the errors were relatively stable while increasing
the values of the parameters, meaning they are more significant for smaller
values within the range. Averaging the results, we obtained errors of the
order of $\sim$ 0.5 mas for the separation and $\sim$ 0.01o for the position
angle. As for the flux, we considered only the errors explained in Section
2.4, as they are dominant and physically significant.

The injection/recovery method gives relatively small errors, but other types
of error must be included (e.g., the error when determining the center of the
vortex coronagraph within the different frames). As our 2020 $L^{\prime}$ and
$Ks$ observations are spaced out by only one day, the astrometry parameters
should be the same. Therefore, we also considered the difference between our
2020 $L^{\prime}$ and $Ks$ results as part of astrometry errors, which are
greater than the injection/recovery errors.

##  3 Companion candidates

###  3.1 High-Contrast Images

The left panel of Figure 1 presents the $L^{\prime}$ high-contrast image of RX
J1744.7$-$2713, while the right panel shows the $K_{s}$ image, both from the
2020 observation runs. The $L^{\prime}$ image from 2017 is shown in the
Appendix, in which only 3 sources are detected. With deeper observations in
2020 (unlike the poor parallactic coverage for angular differential imaging in
2017), it revealed a then-unresolved from the central X-ray binary $\sim$ 450
AU candidate companion. We therefore only consider the 2020 observations for
the remainer of the paper, except for a preliminary proper motion analysis in
Section 3.4.

Using the $L^{\prime}$ SNR map, we find a total of 21 sources that have a SNR
$>$ 5\. These sources are labeled from B to V in Fig. 1 and are also detected
in the $K_{s}$ image with a SNR $>$ 5\. Moreover, as the $K_{s}$ image is more
sensitive (more sources), note that we only considered the sources detected in
$L^{\prime}$ when measuring the magnitudes, since potential sub-stellar
companions are usually brighter in $L^{\prime}$ (Faherty et al., 2016). The
projected separations from RX J1744.7$-$2713, estimated using the distance and
the plate scale, range from $\sim$ 450 to $\sim$ 5500 AU.

In the next sections, we examine the nature of these sources via different
analyses.

###  3.2 Background Source Contamination

####  3.2.1 TRILEGAL

To assess the possibility that the field of view may be dominated by
background or foreground sources, we derived an estimation of the expected
number of sources for the field of view using TRILEGAL – a 3D model used for
simulating the stellar photometry of any Galaxy field (Girardi et al. 2005;
and see Chun et al. 2015; Dietrich & Ginski 2018; Jones et al. 2021; Williams
et al. 2021 for examples of papers that use this model). Constraining the
model to the faintest detected magnitude in $L^{\prime}$, we calculated the
expected number of sources for the entire field of view (i.e. 10.24” $\times$
10.24”). Fig. 5 presents the distribution of the $L^{\prime}$ magnitude for
the simulated sources and for RX J1744.7$-$2713.

We first conducted a Kolmogorov-Smirnov test to assess the possibility that
the two distributions follow the same statistical function. Regardless of the
horizontal binning size, we obtain a very small probability ($\ll 0.1)$ of
observing the same distribution. Therefore, we can reject the null hypothesis
and conclude that at least one of the sources is likely to be bound. However,
we note that for fainter magnitudes, the distributions are roughly similar, so
the corresponding sources are likely unrelated sources. On the contrary, there
is a larger proportion of brighter objects than predicted by TRILEGAL. We thus
conclude that the brighter sources in RX J1744.7$-$2713 (C, D, E, F, and G in
Fig. 1) are less likely to be contaminants, as their presence is not explained
by the model. However, this method is not precise and is only intended to
provide a first rough estimate.

Figure 2: $L^{\prime}$ distribution for RX J1744.7$-$2713 (red) and for the
sources simulated by TRILEGAL (yellow). The orange color is where the two
distributions are superposed.

####  3.2.2 Probability of Chance Alignment

In addition to TRILEGAL, we derived a probability of chance alignment – i.e.,
the likelihood that the candidates are not bound to the system – using the Two
Micron All-Sky Survey (2MASS) Point Source Catalog (PSC; see Correia et al.
2006; Lafrenière et al. 2008, 2014 for examples of that technique). More
specifically, we retrieved all the sources in a 15’ $\times$ 15’ field
surrounding RX J1744.7$-$2713, and we constructed a cumulative distribution of
the number of sources as a function of the apparent $K_{s}$ magnitude. We then
divided this result by the area (in arcsec2) to obtain a surface density
$\Sigma$ that depends on the limiting magnitude of the candidate, i.e.,
$\Sigma$ = $\Sigma(K_{s}<Ks_{\mathrm{candidate}})$. Assuming a random uniform
Poisson distribution of unrelated objects across the chosen area, the
probability that a candidate is not related to the system is given by:

$P_{\mathrm{unrelated}}(\Sigma,\Theta)=1-\exp(-\pi\Sigma\Theta^{2})$ (1)

where $\Theta$ is the projected separation from RX J1744.7$-$2713 (in arcsec).
Table 2 shows the $\Sigma$, $\Theta$, and
$1-P_{\mathrm{unrelated}}(\Sigma,\Theta)$ values for all sources (B to V).

Table 2: Probability of unrelation with RX J1744.7$-$2713 for all the sources (B to V in Fig. 1), as estimated from 2MASS PSC. Source | $\Sigma$ | $\Theta$ | $1-P_{\mathrm{unrelated}}(\Sigma,\Theta)$
---|---|---|---
| (arcsec-2) | (arcsec) | (%)
B | 3.24 $\times$ 10-2 | 0.37 | 98.6
C | 2.11 $\times$ 10-2 | 1.18 | 91.2
D | 7.14 $\times$ 10-3 | 1.25 | 96.6
E | 1.66 $\times$ 10-2 | 3.02 | 62.3
F | 2.47 $\times$ 10-2 | 3.19 | 45.5
G | 2.43 $\times$ 10-2 | 4.51 | 21.2
H | 3.29 $\times$ 10-2 | 4.23 | 15.8
I | 3.29 $\times$ 10-2 | 3.37 | 30.9
J | 3.29 $\times$ 10-2 | 1.96 | 67.3
K | 3.29 $\times$ 10-2 | 3.93 | 20.2
L | 3.29 $\times$ 10-2 | 2.85 | 43.2
M | 3.29 $\times$ 10-2 | 4.42 | 13.4
N | 3.29 $\times$ 10-2 | 2.80 | 44.4
O | 3.29 $\times$ 10-2 | 4.29 | 14.9
P | 3.29 $\times$ 10-2 | 1.83 | 70.6
Q | 3.29 $\times$ 10-2 | 2.18 | 61.3
R | 3.29 $\times$ 10-2 | 1.91 | 68.6
S | 3.29 $\times$ 10-2 | 3.74 | 23.6
T | 3.29 $\times$ 10-2 | 4.02 | 18.9
U | 3.29 $\times$ 10-2 | 3.69 | 24.5
V | 3.29 $\times$ 10-2 | 2.48 | 53.1

This table shows that sources B, C, D have a high probability of not being
unrelated to the system, respectively of 98.6%, 91.2%, and 96.6%, which is
consistent with Fig. 5. Moreover, sources E, J, P, Q, and R have high but less
convincing probabilities, while the remaining sources are likely to be
background stars since their probability is lower than 60%.

###  3.3 Color-Magnitude Diagram

Since both $L^{\prime}$ and $K_{s}$ observations were taken, we were able to
construct a color-magnitude diagram, as has been done in previous studies to
estimate the physical properties of companions (e.g., Rameau et al. 2013;
Salaris & Bedin 2018; Jones et al. 2021). Figure 3 presents the absolute
magnitude in $L^{\prime}$ versus $K_{s}-L^{\prime}$ for the 21 potential
companions to RX J1744.7$-$2713\. The track from MIST at 10 Myr is overlaid,
as well as the extinction vector (see Section 2.4).

From this diagram, the fainter sources (H to L and O to V) could be coherent
with low-mass stars (M- or K-type), while B and M could be G-type stars.
Moreover, B, C, E, F, and G have estimated masses potentially coherent with
massive A- or B-type stars. Due to their off-set $K_{s}-L^{\prime}$ color, the
other sources (F, G, M) seem to be inconsistent with being sub-stellar or
stellar companions, and may therefore be unrelated background objects such as
extincted background stars or artefacts.

Figure 3: Absolute magnitude in $L^{\prime}$ versus $K_{s}-L^{\prime}$ color
for the 21 companion candidates (colored circles) in RX J1744.7$-$2713\. The
colorbar indicates the projected separation (in AU) from the X-ray binary.
Overlaying is the MIST evolutionary track for stars at 10 Myr (Paxton et al.,
2011, 2013, 2018, 2015; Dotter, 2016; Faherty et al., 2016; Choi et al.,
2016). The inset at the top-right is a close-up of the fainter sources.

The inset at the bottom-right shows the extinction vector, i.e. the shift in
magnitude and color caused by extinction (corrections have already been
applied).

A Table with the position, the $L^{\prime}$ and $K_{s}$ magnitudes, the
estimated mass, and the projected separation of the 21 sources companion
candidates is presented in the Appendix. Note that the masses were estimated
from the 5–60 Myr stated in Section 2.1 and that we did not apply the
extinction correction for this Table.

Thus, RX J1744.7$-$2713 does not currently show evidence of exoplanet-mass
companions, as the candidate companions have masses from $\sim$ 0.2 to 6
$M_{\mathrm{\odot}}$. However, we reached excellent SNR and we detected
sources down to $L^{\prime}\sim 16.5$. Another way to gather more information
on the sources is via additional bands (e.g., $H$) or via spectroscopic
characterization.

###  3.4 Preliminary Proper Motion Analysis

To confirm more robustly that the sources are bound, a proper motion analysis
could be conducted (e.g., Marois et al. 2008). Given the distance ($\sim 1$
kpc) and proper motion of RX J1744.7$-$2713 (see Section 2.1), a $>$ 3$\sigma$
detection will only be possible in $\sim$ 5–10 years. However, considering
that there is a three-year interval between our 2017 and 2020 sets of
observations, a preliminary proper motion analysis was conducted for sources
C, D, and J. The plots of astrometric positions (position angle in degrees,
separation in mas, and relative separations in right ascension and
declination) for each of these sources relative to the X-ray binary can be
found in the Appendix. Accounting for the errors from Section 2.5, we conclude
that both scenarios – stationary background or comoving with RX J1744.7$-$2713
– are possible for sources D and J at this current epoch. We note that the
position of source C is offset from the tracks, which indicates that it could
be a non-stationary background object.

###  3.5 Stability of the Wide Orbits

The 21 sources detected in RX J1744.7$-$2713 at high SNR would have wide
projected separations (from $\sim$ 450 AU to $\sim$ 5500 AU) which raises
concerns as to whether potential companions could be gravitationally bound to
the system. However, very large separations have been observed in many known
systems including companions white dwarfs or other evolved systems (e.g., 2000
AU for GU Psc b; Naud et al. 2014, 2985 AU for WISE 2005+5424; Mace et al.
2013, 16 500–-26 500 AU for LSPM 1459+0857 AB; Day-Jones et al. 2011). Baron
et al. (2019) reported that the occurrence of sub-stellar companions at very
large orbits are rare. However, in RX J1744.7$-$2713 the mass ratios are small
(from $q\sim 0.003$ to $\sim$ 0.08), and such ratios have yet to be studied in
detail. Despite the large separations, the large mass of the central system
(i.e., the X-ray binary) implies a high binding energy. Assuming a total mass
$M_{\mathrm{XRB}}$ of $\sim$ 18 $M_{\odot}$ for RX J1744.7$-$2713 (typical
masses for WD and Be stars) and a mass $M_{c}$ of $\sim$ 80 $M_{\mathrm{Jup}}$
as a lower limit for the companions, we can estimate the binding energy of the
system via the following equation (assuming circular orbits):

$E_{\mathrm{bind}}\sim-\frac{GM_{\mathrm{XRB}}M_{c}}{1.27r}$ (2)

Here, 1.27 is the average projection factor between $r$ and the semi-major
axis, assuming a random viewing angle (e.g., Brandeker et al. 2006). We obtain
$E_{\mathrm{bind}}$ $\sim$ $-$4.3 $\times$ 1043 ergs and $E_{\mathrm{bind}}$
$\sim$ $-$3.2 $\times$ 1042 ergs for the closest (450 AU) and farthest (5500
AU) candidates respectively. In both cases, the binding energy is higher than
in many known bound systems (e.g., $\sim 10^{41}$ erg; Naud et al. 2014).

Therefore, it is not unreasonable that the multiple stellar companions
candidates, if confirmed with proper motion analysis, are gravitationally
bound to the HMXB despite their wide separations. However, it is important to
note that it is unlikely that all these sources would be bound due to dynamic
instability. In fact, although we do not have detailed information on their
potential orbital motion yet, candidate companions C and D have similar
separations from RX J1744.7$-$2713, and simulations show that this
configuration can be unstable (e.g., Kiseleva et al. 1994). On the contrary, a
scenario of a triple system – composed of the X-ray binary (considered as one
system here), candidate companion B, and candidate companion C, D, E or J – is
more plausible according to basic $N$-body simulations.

##  4 Discussion

These analyses allowed us to reject most of the 21 sources as candidate
companions. Until a more significant proper motion analysis can be undertaken,
we consider sources B, D and J to be strong candidates of stellar companions
bound to RX J1744.7$-$2713 and we discuss the implications of these results in
the following section.

###  4.1 Stellar Multiplicity

The discovery of multiple probable stellar candidate companions in RX
J1744.7$-$2713 implies that it would be a multiple-star system. In fact, high
order multiplicity increases with primary mass (e.g., Peter et al. 2012), and
although surveys are incomplete, they suggest that stellar multiplicity is
frequent in high-mass stars such as young B stars (see Duchêne & Kraus 2013
for a review on stellar multiplicity). More precisely, if we consider only the
confirmed companions, the multiplicity frequency would be $\geq$ 80% for stars
with a mass $\geq$ 16 $M_{\mathrm{\odot}}$ (e.g., Chini et al. 2012).
Regarding the companion frequency (CF; i.e., the average number of companion
per target in a population), it is $\approx$ 130 $\pm$ 20% for high-mass stars
(Duchêne & Kraus, 2013). Moreover, the frequency $N(n)$ of multiplicity $n$
roughly follows a geometric distribution $N(n)\propto 2.3^{-n}$ (up to $n=7$;
Eggleton & Tokovinin 2008) for solar-type stars; however there is currently no
derived relation for massive stars. As RX J1744.7$-$2713 is already a known
high-mass binary system, the complete sample of this project could therefore
enable us to better understand if X-ray binaries are likely to stellar
multiplicity, as well as probing stellar multiplicity at mass ratios below
$\sim$ 0.1.

###  4.2 Similarities to $\gamma$ Cassiopeiae

In addition, RX J1744.7$-$2713 is a $\gamma$ Cassiopeiae analog, and the
latter has two faint optical companions (e.g., Mason et al. 2001). Therefore,
it suggests that Be stars might be more favorable to stellar multiplicity. As
$\gamma$ Cassiopeiae is a HMXB candidate (e.g., Haberl 1995), it also means
that stellar companions could be common in X-ray binaries.

###  4.3 Formation Scenarios

In the case of X-ray binaries, companions are believed to form (1) from the
direct fragmentation of a collapsing pre-stellar core, i.e., roughly at the
same time as the stars that will eventually form the X-ray binary (e.g., Bate
2012), (2) from the circumbinary disk surrounding the initial binary system
(i.e., pre-explosion of the then-compact object; e.g., Kratter et al. 2010),
or (3) from the supernova fallback disk surrounding the X-ray binary (post-
explosion; e.g., Wolszczan & Frail 1992). Note that fallback disks could lead
to the formation of sub-stellar companions, but they are not massive enough to
allow star formation. However, this work does not show evidence of such disks,
so we can not conclude how the candidate companions have formed. Furthermore,
if companions are formed under the last scenario, then they must be younger
than the X-ray binary; their age could be lower than 5 Myr (the lower limit
for RX J1744.7$-$2713’s age stated in Section 2.1).

Overall, X-ray binaries could host multi-generation companions that form at
different epochs through the evolution of the binary. This work explores the
idea that the interactions between X-ray binaries and their surroundings might
be complex, leading to multiple companions and hierarchical orbits. Moreover,
it opens up the discussion on how sub-stellar and stellar formation could
occur even in the most extreme environments.

##  5 Summary and conclusions

In this work, we have shown the first high-contrast images of the $\gamma$ Cas
analog RX J1744.7$-$2713, using observations from the Keck/NIRC2 vortex
coronagraph. These images reveal the presence of 21 sources detected with a
SNR $>$5\. From a combination of various analyses, we found that three of
these source are strong candidates as stellar companions, and that they would
be in wide orbits ($\sim$ 450 AU to 2500 AU). Follow-up observations will be
needed in $\sim$ 5–10 years to conduct a more significant proper motion
analysis and confirm the nature of these detections. We also aim to obtain the
spectrum of the sources in order to assess the nature of the sources. If
confirmed, these results have direct implications for our understanding of how
multiple star systems form and how X-ray binaries may actually not be simple
binary systems. These observations are part of a pilot study aiming to provide
direct imaging for all X-ray binaries within $\sim$ 3 kpc accessible with
Keck/NIRC2. To date, 14 X-ray binaries have been observed, and the remaining
results will be presented in a future paper (Prasow-Émond et al. in prep.).

We want to thank V. Christiaens for his help with VIP. M.P.E. is supported by
the Institute for Data Valorisation (IVADO) through the M. Sc. Excellence
Scholarship, by the Department of Physics of the Université de Montréal and by
the Institute for Research on Exoplanets (iREx). J.H.L. is supported by the
Natural Sciences and Engineering Research Council of Canada (NSERC) through
the Canada Research Chair programs and wishes to acknowledge the support of an
NSERC Discovery Grant and the NSERC accelerator grant. J.R. acknowledges
support by the French National Research Agency in the framework of the
Investissements d’Avenir program (ANR-15-IDEX-02), through the funding of the
"Origin of Life" project of the Univ. Grenoble-Alpes. D.J.W. acknowledges
financial support from STFC in the form of an Ernest Rutherford fellowship.

## References

  * Bailes et al. (2011) Bailes, M., Bates, S. D., Bhalerao, V., et al. 2011, Science, 333, 1717, doi: 10.1126/science.1208890
  * Baron et al. (2019) Baron, F., Lafrenière, D., Artigau, É., et al. 2019, AJ, 158, 187, doi: 10.3847/1538-3881/ab4130
  * Bate (2012) Bate, M. R. 2012, MNRAS, 419, 3115, doi: 10.1111/j.1365-2966.2011.19955.x
  * Bonavita et al. (2016) Bonavita, M., Desidera, S., Thalmann, C., et al. 2016, arXiv:1605.03962 [astro-ph], doi: 10.1051/0004-6361/201628231
  * Bond et al. (2018) Bond, C. Z., Wizinowich, P., Chun, M., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10703, Adaptive Optics Systems VI, ed. L. M. Close, L. Schreiber, & D. Schmidt, 107031Z, doi: 10.1117/12.2314121
  * Brandeker et al. (2006) Brandeker, A., Jayawardhana, R., Khavari, P., Haisch, Karl E., J., & Mardones, D. 2006, ApJ, 652, 1572, doi: 10.1086/508483
  * Burrows et al. (2001) Burrows, A., Hubbard, W. B., Lunine, J. I., & Liebert, J. 2001, Reviews of Modern Physics, 73, 719, doi: 10.1103/RevModPhys.73.719
  * Chini et al. (2012) Chini, R., Hoffmeister, V. H., Nasseri, A., Stahl, O., & Zinnecker, H. 2012, MNRAS, 424, 1925, doi: 10.1111/j.1365-2966.2012.21317.x
  * Choi et al. (2016) Choi, J., Dotter, A., Conroy, C., et al. 2016, ApJ, 823, 102, doi: 10.3847/0004-637X/823/2/102
  * Chun et al. (2015) Chun, S.-H., Jung, M., Kang, M., Kim, J.-W., & Sohn, Y.-J. 2015, A&A, 578, A51, doi: 10.1051/0004-6361/201525849
  * Coleiro & Chaty (2013) Coleiro, A., & Chaty, S. 2013, ApJ, 764, 185, doi: 10.1088/0004-637X/764/2/185
  * Correia et al. (2006) Correia, S., Zinnecker, H., Ratzka, T., & Sterzik, M. F. 2006, A&A, 459, 909, doi: 10.1051/0004-6361:20065545
  * Cutri et al. (2003) Cutri, R. M., Skrutskie, M. F., van Dyk, S., et al. 2003, VizieR Online Data Catalog, II/246
  * Day-Jones et al. (2011) Day-Jones, A. C., Pinfield, D. J., Ruiz, M. T., et al. 2011, MNRAS, 410, 705, doi: 10.1111/j.1365-2966.2010.17469.x
  * Dietrich & Ginski (2018) Dietrich, J., & Ginski, C. 2018, A&A, 620, A102, doi: 10.1051/0004-6361/201731341
  * Doherty et al. (1994) Doherty, R. M., Puxley, P., Doyon, R., & Brand, P. W. J. L. 1994, MNRAS, 266, 497, doi: 10.1093/mnras/266.2.497
  * Dong et al. (2017) Dong, H., Schödel, R., Williams, B. F., et al. 2017, MNRAS, 471, 3617, doi: 10.1093/mnras/stx1836
  * Dotter (2016) Dotter, A. 2016, ApJS, 222, 8, doi: 10.3847/0067-0049/222/1/8
  * Draine (1989) Draine, B. T. 1989, in Infrared Spectroscopy in Astronomy, ed. E. Böhm-Vitense, 93
  * Duchêne & Kraus (2013) Duchêne, G., & Kraus, A. 2013, ARA&A, 51, 269, doi: 10.1146/annurev-astro-081710-102602
  * Eggleton & Tokovinin (2008) Eggleton, P. P., & Tokovinin, A. A. 2008, MNRAS, 389, 869, doi: 10.1111/j.1365-2966.2008.13596.x
  * Evans et al. (2018) Evans, T. M., Sing, D. K., Goyal, J. M., et al. 2018, AJ, 156, 283, doi: 10.3847/1538-3881/aaebff
  * Faherty et al. (2016) Faherty, J. K., Riedel, A. R., Cruz, K. L., et al. 2016, ApJS, 225, 10, doi: 10.3847/0067-0049/225/1/10
  * Fender et al. (2004) Fender, R. P., Belloni, T. M., & Gallo, E. 2004, Monthly Notices of the Royal Astronomical Society, 355, 1105, doi: 10.1111/j.1365-2966.2004.08384.x
  * Gaia Collaboration et al. (2021) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2021, A&A, 649, A1, doi: 10.1051/0004-6361/202039657
  * Girardi et al. (1996) Girardi, L., Bressan, A., Chiosi, C., Bertelli, G., & Nasi, E. 1996, A&AS, 117, 113
  * Girardi et al. (2005) Girardi, L., Groenewegen, M. A. T., Hatziminaoglou, E., & da Costa, L. 2005, A&A, 436, 895, doi: 10.1051/0004-6361:20042352
  * Gomez Gonzalez et al. (2017) Gomez Gonzalez, C. A., Wertz, O., Absil, O., et al. 2017, AJ, 154, 7, doi: 10.3847/1538-3881/aa73d7
  * Haberl (1995) Haberl, F. 1995, A&A, 296, 685
  * Huby et al. (2017) Huby, E., Bottom, M., Femenia, B., et al. 2017, A&A, 600, A46, doi: 10.1051/0004-6361/201630232
  * Imara & Di Stefano (2018) Imara, N., & Di Stefano, R. 2018, arXiv:1703.05762 [astro-ph], doi: 10.3847/1538-4357/aab903
  * Israel et al. (1997) Israel, G. L., Stella, L., Angelini, L., et al. 1997, ApJ, 474, L53, doi: 10.1086/310418
  * Jones et al. (2021) Jones, O. C., Nally, C., Sharp, M. J., et al. 2021, MNRAS, 504, 565, doi: 10.1093/mnras/stab923
  * Kara et al. (2019) Kara, E., Steiner, J. F., Fabian, A. C., et al. 2019, Nature, 565, 198, doi: 10.1038/s41586-018-0803-x
  * Kiseleva et al. (1994) Kiseleva, G., Eggleton, P. P., & Anosova, J. P. 1994, MNRAS, 267, 161, doi: 10.1093/mnras/267.1.161
  * Kratter et al. (2010) Kratter, K. M., Matzner, C. D., Krumholz, M. R., & Klein, R. I. 2010, ApJ, 708, 1585, doi: 10.1088/0004-637X/708/2/1585
  * Kravtsov et al. (2020) Kravtsov, V., Berdyugin, A. V., Piirola, V., et al. 2020, A&A, 643, A170, doi: 10.1051/0004-6361/202038745
  * Lafrenière et al. (2008) Lafrenière, D., Jayawardhana, R., Brandeker, A., Ahmic, M., & van Kerkwijk, M. H. 2008, ApJ, 683, 844, doi: 10.1086/590239
  * Lafrenière et al. (2014) Lafrenière, D., Jayawardhana, R., van Kerkwijk, M. H., Brandeker, A., & Janson, M. 2014, ApJ, 785, 47, doi: 10.1088/0004-637X/785/1/47
  * Lafrenière et al. (2011) Lafrenière, D., Jayawardhana, R., Janson, M., et al. 2011, ApJ, 730, 42, doi: 10.1088/0004-637X/730/1/42
  * Lagrange et al. (2010) Lagrange, A. M., Bonnefoy, M., Chauvin, G., et al. 2010, Science, 329, 57, doi: 10.1126/science.1187187
  * Lindegren et al. (2021) Lindegren, L., Bastian, U., Biermann, M., et al. 2021, A&A, 649, A4, doi: 10.1051/0004-6361/202039653
  * Loehmer et al. (2004) Loehmer, O., Wolszczan, A., & Wielebinski, R. 2004, A&A, 425, 763, doi: 10.1051/0004-6361:20041263
  * Lopes de Oliveira et al. (2006) Lopes de Oliveira, R., Motch, C., Haberl, F., Negueruela, I., & Janot-Pacheco, E. 2006, A&A, 454, 265, doi: 10.1051/0004-6361:20054589
  * Mace et al. (2013) Mace, G. N., Kirkpatrick, J. D., Cushing, M. C., et al. 2013, ApJ, 777, 36, doi: 10.1088/0004-637X/777/1/36
  * Marois et al. (2006) Marois, C., Lafrenière, D., Doyon, R., Macintosh, B., & Nadeau, D. 2006, ApJ, 641, 556, doi: 10.1086/500401
  * Marois et al. (2008) Marois, C., Macintosh, B., Barman, T., et al. 2008, Science, 322, 1348, doi: 10.1126/science.1166585
  * Mason et al. (2001) Mason, B. D., Wycoff, G. L., Hartkopf, W. I., Douglass, G. G., & Worley, C. E. 2001, AJ, 122, 3466, doi: 10.1086/323920
  * Mawet et al. (2005) Mawet, D., Riaud, P., Absil, O., & Surdej, J. 2005, ApJ, 633, 1191, doi: 10.1086/462409
  * Mawet et al. (2014) Mawet, D., Milli, J., Wahhaj, Z., et al. 2014, ApJ, 792, 97, doi: 10.1088/0004-637X/792/2/97
  * Motch et al. (1997) Motch, C., Haberl, F., Dennerl, K., Pakull, M., & Janot-Pacheco, E. 1997, A&A, 323, 853.  https://arxiv.org/abs/astro-ph/9611122
  * Naud et al. (2014) Naud, M.-E., Artigau, É., Malo, L., et al. 2014, ApJ, 787, 5, doi: 10.1088/0004-637X/787/1/5
  * Naze & Motch (2018) Naze, Y., & Motch, C. 2018, A&A, 619, A148, doi: 10.1051/0004-6361/201833842
  * Paxton et al. (2011) Paxton, B., Bildsten, L., Dotter, A., et al. 2011, ApJS, 192, 3, doi: 10.1088/0067-0049/192/1/3
  * Paxton et al. (2013) Paxton, B., Cantiello, M., Arras, P., et al. 2013, ApJS, 208, 4, doi: 10.1088/0067-0049/208/1/4
  * Paxton et al. (2015) Paxton, B., Marchant, P., Schwab, J., et al. 2015, ApJS, 220, 15, doi: 10.1088/0067-0049/220/1/15
  * Paxton et al. (2018) Paxton, B., Schwab, J., Bauer, E. B., et al. 2018, ApJS, 234, 34, doi: 10.3847/1538-4365/aaa5a8
  * Peter et al. (2012) Peter, D., Feldt, M., Henning, T., & Hormuth, F. 2012, A&A, 538, A74, doi: 10.1051/0004-6361/201015027
  * Raguzova & Popov (2005) Raguzova, N. V., & Popov, S. B. 2005, Astronomical and Astrophysical Transactions, 24, 151, doi: 10.1080/10556790500497311
  * Rameau et al. (2013) Rameau, J., Chauvin, G., Lagrange, A. M., et al. 2013, ApJ, 772, L15, doi: 10.1088/2041-8205/772/2/L15
  * Salaris & Bedin (2018) Salaris, M., & Bedin, L. R. 2018, MNRAS, 480, 3170, doi: 10.1093/mnras/sty2096
  * Schlafly & Finkbeiner (2011) Schlafly, E. F., & Finkbeiner, D. P. 2011, ApJ, 737, 103, doi: 10.1088/0004-637X/737/2/103
  * Seager & Sasselov (1998) Seager, S., & Sasselov, D. D. 1998, The Astrophysical Journal, 502, L157, doi: 10.1086/311498
  * Serabyn et al. (2016) Serabyn, E., Liewer, K., & Mawet, D. 2016, Optics Communications, 379, 64, doi: 10.1016/j.optcom.2016.05.042
  * Service et al. (2016) Service, M., Lu, J. R., Campbell, R., et al. 2016, PASP, 128, 095004, doi: 10.1088/1538-3873/128/967/095004
  * Servillat et al. (2012) Servillat, M., Tang, S., Grindlay, J. E., & Los, E. 2012, in Proceedings of “An INTEGRAL view of the high-energy sky (the first 10 years)” - 9th INTEGRAL Workshop and celebration of the 10th anniversary of the launch (INTEGRAL 2012). 15-19 October 2012. Bibliotheque Nationale de France, 23.  https://arxiv.org/abs/1303.1179
  * Shrader et al. (2015) Shrader, C. R., Hamaguchi, K., Sturner, S. J., et al. 2015, ApJ, 799, 84, doi: 10.1088/0004-637X/799/1/84
  * Sigurdsson et al. (2003) Sigurdsson, S., Richer, H. B., Hansen, B. M., Stairs, I. H., & Thorsett, S. E. 2003, Science, 301, 193, doi: 10.1126/science.1086326
  * Starovoit & Rodin (2017) Starovoit, E. D., & Rodin, A. E. 2017, Astronomy Reports, 61, 948, doi: 10.1134/S1063772917110063
  * Tauris & van den Heuvel (2006) Tauris, T. M., & van den Heuvel, E. P. J. 2006, Formation and evolution of compact stellar X-ray sources, Vol. 39, 623–665
  * Uyama et al. (2020) Uyama, T., Ren, B., Mawet, D., et al. 2020, AJ, 160, 283, doi: 10.3847/1538-3881/abc69a
  * Viehmann et al. (2005) Viehmann, T., Eckart, A., Schödel, R., et al. 2005, A&A, 433, 117, doi: 10.1051/0004-6361:20041748
  * Welsh (2013) Welsh, W. F. 2013, Nature, 499, 33, doi: 10.1038/nature12405
  * Williams et al. (2021) Williams, B. F., Durbin, M. J., Dalcanton, J. J., et al. 2021, ApJS, 253, 53, doi: 10.3847/1538-4365/abdf4e
  * Wizinowich et al. (2000) Wizinowich, P., Acton, D. S., Shelton, C., et al. 2000, PASP, 112, 315, doi: 10.1086/316543
  * Wolszczan & Frail (1992) Wolszczan, A., & Frail, D. A. 1992, Nature, 355, 145, doi: 10.1038/355145a0
  * Yan et al. (2013) Yan, Z., Shen, Z.-Q., Yuan, J.-P., et al. 2013, MNRAS, 433, 162, doi: 10.1093/mnras/stt712
  * Zacharias et al. (2012) Zacharias, N., Finch, C. T., Girard, T. M., et al. 2012, VizieR Online Data Catalog, I/322A

Figure 4: Keck/NIRC2 $L^{\prime}$ high-contrast image of RX J1744.7$-$2713 acquired on September 08 2017 and treated and reduced with a PCA annular ADI algorithm (Gomez Gonzalez et al., 2017). Labeled are the sources detected with a SNR $>$ 5, using the same labels from Fig. 1. The white X symbol denotes the position of the X-ray binary masked by the coronagraph. Table 3: Physical properties of the 21 sources in RX J1744.7$-$2713, where $r$ is the radial separation from the X-ray binary in mas, P.A. is the position angle in the image in degrees, and $L^{\prime}$ and $Ks$ are the apparent magnitudes (see Section 2.5 for errors). No extinction correction was applied (see Section 2.4). The second column is the optimal number of principal components used for the fitting. Source | $n_{\mathrm{comp}}$ | $r$ | P.A. | $L^{\prime}$ | $Ks$ | Estimated mass | Projected separation
---|---|---|---|---|---|---|---
|  | (mas) | (deg) | (mag) | (mag) | ($M_{\mathrm{\odot}}$) | (AU)
B | 14 | 368 $\pm$ 4 | 340.5 $\pm$ 1.4 | 13.5 $\pm$ 0.7 | 13.2 $\pm$ 0.7 | 0.9–1.2 | 449
C | 4 | 1180 $\pm$ 6 | 154.2 $\pm$ 0.4 | 12.2 $\pm$ 0.6 | 12.2 $\pm$ 0.6 | 1.7–2.2 | 1440
D | 2 | 1246 $\pm$ 5 | 324.7 $\pm$ 0.5 | 10.7 $\pm$ 0.5 | 10.7 $\pm$ 0.5 | 4.1–4.8 | 1520
E | 5 | 3018 $\pm$ 2 | 140.6 $\pm$ 0.2 | 11.9 $\pm$ 0.6 | 11.8 $\pm$ 0.6 | 2.5–2.6 | 3681
F | 15 | 3189 $\pm$ 7 | 341.8 $\pm$ 0.2 | 12.2 $\pm$ 0.6 | 12.4 $\pm$ 0.6 | 1.8–2.2 | 3890
G | 29 | 4506 $\pm$ 12 | 359.1 $\pm$ 0.1 | 12.0 $\pm$ 0.6 | 12.4 $\pm$ 0.6 | 1.9–2.4 | 5497
H | 25 | 4228 $\pm$ 9 | 201.34 $\pm$ 0.09 | 15.5 $\pm$ 0.8 | 15.2 $\pm$ 0.8 | 0.2–0.6 | 5158
I | 16 | 3369 $\pm$ 4 | 148.4 $\pm$ 0.1 | 15.9 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.1–0.5 | 4110
J | 16 | 1956 $\pm$ 12 | 10.0 $\pm$ 0.1 | 14.8 $\pm$ 0.7 | 14.8 $\pm$ 0.7 | 0.3–0.8 | 2386
K | 28 | 3933 $\pm$ 10 | 291.9 $\pm$ 0.3 | 15.5 $\pm$ 0.8 | 15.4 $\pm$ 0.8 | 0.2–0.6 | 4798
L | 8 | 2851 $\pm$ 1 | 320.1 $\pm$ 0.2 | 16.0 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.1–0.5 | 3478
M | 33 | 4416 $\pm$ 5 | 350.3 $\pm$ 0.1 | 13.7 $\pm$ 0.7 | 14.0 $\pm$ 0.7 | 0.8–1.1 | 5387
N | 7 | 2803 $\pm$ 5 | 257.5 $\pm$ 0.2 | 15.8 $\pm$ 0.8 | 15.5 $\pm$ 0.8 | 0.2–0.5 | 3419
O | 25 | 4291 $\pm$ 8 | 287.0 $\pm$ 0.3 | 16.0 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.1–0.5 | 5235
P | 21 | 1835 $\pm$ 8 | 319.0 $\pm$ 0.2 | 15.8 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.2–0.5 | 2238
Q | 20 | 2176 $\pm$ 16 | 316.5 $\pm$ 0.6 | 15.5 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.2–0.6 | 2655
R | 26 | 1910 $\pm$ 17 | 354.18 $\pm$ 0.09 | 15.7 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.2–0.5 | 2331
S | 22 | 3737 $\pm$ 15 | 133.95 $\pm$ 0.05 | 16.0 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.1–0.5 | 4559
T | 32 | 4016 $\pm$ 9 | 149.5 $\pm$ 0.1 | 15.8 $\pm$ 0.8 | 15.5 $\pm$ 0.8 | 0.2–0.5 | 4899
U | 17 | 3689.3 $\pm$ 0.2 | 177.87 $\pm$ 0.01 | 15.7 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.2–0.5 | 4501
V | 8 | 2475 $\pm$ 18 | 186.6 $\pm$ 0.1 | 15.5 $\pm$ 0.8 | 15.6 $\pm$ 0.8 | 0.2–0.6 | 3020

Figure 5: Top row: Position angle in degrees of the sources C (left), D
(middle), and J (right) relative to its host, as measured by Keck/NIRC2 on
2017 September 08 (blue data point). The red data point is the position angle
measured by Keck/NIRC2 on 2020 July 11. An infinitely distant background
object with zero proper motion would be following the orange track, while a
comoving companion would lie within the blue area. Middle row: Same as the top
row, but for the separation from the X-ray binary in mas. Bottom row: Relative
separations between the source and the X-ray binary in right ascension
($\alpha$) and declination ($\delta$). The first epoch astrometric point is
plotted in blue (2017 September 08), while the second epoch astrometric point
is plotted in red (2020 July 11). The excepted position for a background
object is plotted in yellow, along with a proper and parallactic motion track.


# ALTERNATING WEAK TRIPHONE/BPE ALIGNMENT SUPERVISION FROM HYBRID MODEL
IMPROVES END-TO-END ASR

###### Abstract

In this paper, alternating weak triphone/BPE alignment supervision is proposed
to improve end-to-end model training. Towards this end, triphone and BPE
alignments are extracted using a pre-existing hybrid ASR system. Then,
regularization effect is obtained by cross-entropy based intermediate
auxiliary losses computed on such alignments at a mid-layer representation of
the encoder for triphone alignments and at the encoder for BPE alignments.
Weak supervision is achieved through strong label smoothing with parameter of
0.5. Experimental results on TED-LIUM 2 indicate that either triphone or BPE
alignment based weak supervision improves ASR performance over standard CTC
auxiliary loss. Moreover, their combination lowers the word error rate
further. We also investigate the alternation of the two auxiliary tasks during
model training, and additional performance gain is observed. Overall, the
proposed techniques result in over 10% relative error rate reduction over a
CTC-regularized baseline system.

Index Terms—  weak alignment supervision, end-to-end, hybrid ASR, label
smoothing, alternating

##  1 Introduction

End-to-end (E2E) automatic speech recognition (ASR) modeling has achieved
incredible milestones in recent years [1, 2, 3, 4]. The E2E modeling has been
democratizing machine learning in ASR as researchers do not need to be
exceptionally proficient at speaking, or writing a language. At the same time,
E2E has also been making multilingual ASR feasible and shortening the model
development time as demonstrated by Whisper [5]. For these reasons and beyond,
researchers have been moving away from traditional hybrid ASR modeling [6].
Nevertheless, E2E approaches tend to suffer from spiky and/or inaccurate
alignment problems [7, 8]. Thus, the more accurate hybrid alignments still
represent valuable resources these days.

In our previous study [9], we demonstrated that an auxiliary loss using
triphone alignment extracted by a hybrid system can indeed aid Attention-based
Encoder Decoder (AED) E2E modeling. Especially, if it is combined with weak
supervision achieved through label smoothing [10], and when the auxiliary loss
is placed around the middle, to an intermediate layer representation, of the
encoder. Standard AED training usually includes joint CTC-attention training
using the encoder output and the target sequence of the main task [11]. Our
previous study also confirmed that such auxiliary loss should be placed at the
higher layers of the encoder, and that it is less efficient than the hybrid
alignment based approach.

Inspired by these findings, we extend our previous study, and (1) combine
multiple auxiliary losses using different target sequences (triphone and BPE)
and different encoder layer outputs; (2) also replace the CTC loss with weak
BPE alignment supervision at the encoder output; (3) we experiment with
alternating the two losses during training.

##  2 Related Work

The major E2E architectures for speech recognition are Attention-based Encoder
Decoder (AED) [12, 13], Connectionist Temporal Classification (CTC) [14, 15],
and Recurrent Neural Networks-Transducer (RNN-T) [16, 17, 18, 19]; our study
focuses on improving BPE based AED model [20]. Multi-task learning has been
widely exploited for E2E modeling [11, 21, 22, 23, 24, 25]. In this paper, we
aim at utilizing pre-existing hybrid ASR models to aid the E2E modeling
similar to [26, 27], or [28] which used state alignment supervision in a
multi-task training approach with an AED model.

The initial exploration of using label smoothing and weak triphone alignment
supervision was proposed in our previous study [9]. Similar to our previous
study, we use the recipe from [13] which is built on RETURNN, a TensorFlow-
based deep learning platform [29]. Nevertheless, we improve upon the original
recipe to produce a new and stronger baseline.

##  3 Methodology

In the Figure 1, we show a diagram of the proposed AED modeling. The basic
components are the six-layer BLSTM encoder [30, 31], an RNN-based decoder with
a LSTM layer inside, and an auxiliary BPE based CTC loss. SpecAugment is used
to augment the input data [32]. The encoder output (enc) is projected to
encoder value (value), inverse-fertility (inv-fertility), and context (ctx)
that are the input to the decoder.

Fig. 1: Diagram of the network.

The dotted red lines/boxes represent the proposed triphone and BPE alignment
supervision where a label smoothing of 0.5 is used to construct a weak
supervision. The cross-entropy (CE) based triphone alignment supervision (Tri-
CE) is placed at the fourth BLSTM layer (BLSTM4) of the encoder. The rule of
thumb is to place the label-smoothed Tri-CE around the the middle layer of the
encoder [9, 26]. The BPE alignment supervision (BPE-CE) can be placed either
at the encoder output (enc) or at the encoder context (ctx). Our previous
study [9] shows that it is beneficial to place the BPE-related losses at the
high layers of encoder. The inv-fertility layer is not considered here as it
is usually optional in the AED networks [13]. In initial experiments, we found
that the training did not converge when putting both the BPE-CE and BPE-CTC at
the encoder output (enc) without additional tuning of the learning rate
scheduler. Given that there are three types of auxiliary tasks (triphone
alignment supervision, BPE alignment supervision, and a standard BPE CTC
loss), we construct 8 different architectures and losses ($\mathcal{L}$) as
defined with the following equations:

$\begin{split}&\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{Primary}}+\qquad\qquad\qquad\qquad\quad\mathcal{L}_{\text{BPE-
CTC-enc}}\\\
&\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{Primary}}+\mathcal{L}_{\text{Tri-
CE}}+\qquad\qquad\quad\mathcal{L}_{\text{BPE-CTC-enc}}\\\
&\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{Primary}}+\qquad\qquad\mathcal{L}_{\text{BPE-
CE-enc}}\\\
&\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{Primary}}+\qquad\qquad\mathcal{L}_{\text{BPE-
CE-ctx}}+\mathcal{L}_{\text{BPE-CTC-enc}}\\\
&\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{Primary}}+\qquad\qquad\mathcal{L}_{\text{BPE-
CE-enc}}+\mathcal{L}_{\text{BPE-CTC-ctx}}\\\
&\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{Primary}}+\mathcal{L}_{\text{Tri-
CE}}+\mathcal{L}_{\text{BPE-CE-enc}}\\\
&\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{Primary}}+\mathcal{L}_{\text{Tri-
CE}}+\mathcal{L}_{\text{BPE-CE-ctx}}+\mathcal{L}_{\text{BPE-CTC-enc}}\\\
&\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{Primary}}+\mathcal{L}_{\text{Tri-
CE}}+\mathcal{L}_{\text{BPE-CE-enc}}+\mathcal{L}_{\text{BPE-CTC-
ctx}}\end{split}$ (1)

Where enc and ctx indicate the location for placing the auxiliary losses, and
Primary corresponds to the loss of the main task, the CE loss measured at the
decoder output. The primary loss is a CE loss with BPEs as targets at the
output of the RNN decoder.

In the literature, auxiliary losses are usually scaled to regularize the total
loss [25, 26]. As discussed in our previous study [9], here we apply a label
smoothing parameter of 0.5 to create weak supervision, which also leads to
better training stability. In addition to these designs, we also propose
alternating the Tri-CE and BPE-CE when both are present. The intuition behind
the idea is that when there are multiple auxiliary losses running at the same
time, there might be conflicting/cancelling effects among them, or they cause
learning instability. The alternating is achieved through the pre-training
mechanism in RETURNN [29]. Pre-training is also used in the baseline case to
improve convergence stability.

##  4 Experimental Results

###  4.1 Setup

The experiments are performed on the TED-LIUM release 2 [33] using RETURNN
[29]. We follow a recipe from [13, 9]. In the repository, a full setup is
provided. For the TED-LIUM release 2 dataset, we use the same train/dev/test
division as in [13]. Note that in this study, language models are not applied.

Triphone alignments are obtained using an existing hybrid BLSTM ASR model that
uses 5000 triphones, 80-dimension MFCC as acoustic features, and four
1024-dimension hidden BLSTM layers. The hybrid model is trained using about
11,000 hours of training data that includes 9893-hour news, 157-hour movies,
961-hour LibriSpeech, and 81-hour WSJ. Traditional Gaussian Mixture Models
(GMM) is trained to obtain the triphone CART tree and then the BLSTM model is
trained with SpecAugment using RETURNN [29]. The RASR toolkit [34] is used to
perform Viterbi alignments to produces both frame-level triphone and word
indices. Here we only use the frame-level triphone indices.

BPE alignments are computed in the following way. First, each sentence is
parsed into BPE tokens. Then grapheme-to-phoneme (G2P) conversion is applied
to derive their pronunciations with three variants allowed. The @@ signs are
removed before applying G2P. The G2P model is trained with the Sequitur tool
[35]. After constructing a pronunciation lexicon for BPE tokens, we obtain the
BPE alignments the same way as we do for triphone alignments. The aligning
process produces both frame-level triphone and BPE indices. Here we only use
the frame-level BPE indices.

The structure of the AED model is shown in Figure 1. The BLSTM layers have a
hidden dimension of 1024. We use dropout and Adam optimizer [36] with a
variant of Newbob learning rate scheduling. In the RNN decoder, the CE loss is
applied with a label smoothing of 0.1. A beam size of 12 is used for decoding.
Word Error Rate (WER) is computed on the decoder output using NIST SCLITE
Scoring Toolkit [37].

###  4.2 New Baseline

The baseline in [29] is quite outdated. Therefore, we develop a new baseline
system and expect the proposed approach can improve on a stronger baseline.

The new baseline is built making the following changes:

  * •

Large memory GPU (RTX A6000) is used, and sequence length limitation (maximum
BPE length of 75) is removed. Previously, 30 hours (about 14%) of the total
audio of 213 hours were removed.

  * •

For the training data, we apply a segment boundary non-overlapping extension
111https://github.com/rwth-i6/i6_core/blob/main/datasets/tedlium2.py [38].
This will help the AED model for better endpointing.

  * •

We increase the number of training sub-epochs from 150 to 200. That equals to
50 full epochs.

  * •

We update the implementation of SpecAugment [38].

  * •

We change the input features from 40-dimensional MFCC to 80-dimensional log-
mel-filterbank. Segment-wise mean and variance normalization is applied.

  * •

In the original recipe, the learning rate decay rate of 0.7 is too aggressive.
Here, we customize learning rate contour and make it decay slowly.

  * •

We apply speed perturbation in 20% of the training time, with randomly chosen
factor of 1.1 or 0.9. We also add additive noises at raw audio level (20% of
the time and SNR above 0dB) and additive spectral noise to the mel-filterbank
(20% of the time and SNR above 0dB). The noise samples are obtained from Audio
Set [39].

  * •

Final evaluation model is obtained by averaging the models from the last four
sub-epochs (197, 198, 199, and 200) [40].

  * •

Other changes include using batch-size of 15000 (frames) instead of 18000 and
increasing the sequence ordering buckets.

Systems | WER [%]
---|---
dev | test
Hybrid ASR with n-gram LM | 10.4 | 9.8
Baseline in [9] (37.5 epochs) | 13.7 | 10.7
New-Baseline (50 epochs) | 9.4 | 8.4
Table 1: WER improvements from the new baseline.

After these steps, we produce a relatively competitive baseline as shown in
Table 1. We also run the ASR task on the test/dev dataset with the hybrid ASR
system (with n-gram LM) used in the triphone/BPE alignment and results are
listed in the table. Note that the number of epochs is different in the table.
However, this does not change the conclusion that the new baseline is
significantly improved as compared to [13].

###  4.3 Applying weak triphone and BPE alignment losses

We run a number of experiments according to Equation 1. Line 1 in Equation 1
represents the baseline with the original primary loss and BPE CTC loss. Line
2 represents the addition of triphone alignment CE loss (Tri-CE) to the
baseline, similar to that in our previous study [9]. Line 3 indicates using
BPE alignment CE loss (BPE-CE) to replace the existing BPE CTC loss (BPE-CTC).
Lines 4 and 5 indicate using both BPE-CE and BPE-CTC at the same time, without
Tri-CE. In these cases, BPE-CE and BPE-CTC losses are either put on enc and
ctx representations, respectively, or vice versa. Lines 6, 7, and 8 are
similar to Lines 3, 4, and 5 correspondingly with the addition of the
intermediate Tri-CE loss. For Lines 6-8, we also experiment with alternating
the Tri-CE and BPE-CE losses by setting the loss weighting of one to 1.0 and
the other to 0.0. Each pattern is held for five sub-epochs for a total 150
sub-epochs (75% of training time). This alternating is achieved through the
pre-training mechanism in RETURNN [29]. Note that during decoding, all the
loss-related network layers (Tri-CE, BPE-CE, and BPE-CTC) can be removed, and
thus all have the same amount of parameters (about 160M).

Following [13], the learning rates are kept at 0.0008 for 40 sub-epochs and
then decay after that (lr1), while for the training of alternating losses, the
learning rates are kept at 0.0008 for 150 sub-epochs and then decay after that
(lr2). For a comparison purpose, we also apply lr2 to the other experiments.

Experimental results are shown in Table 2. First, adding triphone alignment CE
loss or BPE alignment loss separately improves the ASR results over the
baseline. Based on the dev set, it shows that the BPE CE loss produces higher
improvement. This can be explained by the fact that the BPE CE loss is
consistent with the primary loss as both using BPE tokens. Nevertheless,
combination of the triphone CE and BPE CE loss leads to further improvements.
When alternating the two CE losses, we see a small degradation on the dev set,
while getting the best result on the test set (7.4%). When applying BPE CE
loss, we have choices of putting it on the encoder output (enc), the context
output (ctx), or replacing the BPE CTC loss. Based on the dev set, it is
favorable to put the BPE CE loss on the encoder output (enc), while moving BPE
CTC loss to the context output (ctx). In all cases, removing standard BPE CTC
from the pool of auxiliary losses hurts the recognition quality. Overall, the
system trained with Tri-CE, BPE-CE on enc, and BPE-CTC on ctx achieves
significant performance gain over the baseline (12.8% and 11.9% relative
improvement for the dev and test sets). The effect of alternating weak
triphone/BPE alignment supervision improved test set result slightly further
(0.1% absolute WER gain). The strong learning rates (lr2) work better for the
Tri-CE only and BPE-CE only experiments, while the weak learning rates (lr1)
work better for the baseline and the Tri-CE and BPE-CE experiments.

System |  Tri CE |  BPE CE |  BPE CTC | WER (lr1) | WER (lr2)
---|---|---|---|---|---
dev | test | dev | test
Baseline | - | - | enc | 9.4 | 8.4 | 9.8 | 8.7
Tri-CE | + | - | enc | 9.0 | 7.7 | 8.6 | 7.7
BPE-CE | - | enc | - | 8.7 | 8.0 | 8.5 | 7.7
- | ctx | enc | 8.8 | 7.9 | 8.5 | 7.6
- | enc | ctx | 8.4 | 8.0 | 8.5 | 7.8
Tri-CE and BPE-CE | + | enc | - | 8.2 | 7.6 | 8.6 | 7.7
+ | ctx | enc | 8.1 | 7.5 | 8.3 | 7.4
+ | enc | ctx | 8.1 | 7.8 | 8.5 | 7.7
Alternating Tri-CE and BPE-CE | + | enc | - | - | - | 8.4 | 7.6
+ | ctx | enc | - | - | 8.3 | 7.5
+ | enc | ctx | - | - | 8.2 | 7.4
Table 2: WERs [%] for different auxiliary loss combinations.

System | Enc. | Dec. |  Cross |  Loss |  Params. | WER [%]
---|---|---|---|---|---|---
att. | dev | test
This work | LSTM | RNN | yes | CE | 160M | 8.2 | 7.4
ESPnet | Conf | Transf | yes | CE | 31M | 7.8 | 7.6
EBranch |  Transf | yes | CE | 35M | 7.3 | 7.1
Citrinet | Conv | - | no | CTC | 142M | 9.8 | 8.9

Table 3: Overall WER results on TED-LIUM 2 vs. literature.

We also compare the best results here with those from the literature. All
results are obtained without language model integration and on TED-LIUM 2
dataset. Specifically, we compare with the ESPnet results, where the encoder
is either a 12-layer conformer or ebranchformer
222https://github.com/espnet/espnet/tree/master/egs2/tedlium2/asr1 [41, 42].
We also compare with results from Citrinet [43] which is a convolutional CTC
model that is trained from scratch. Comparison results are shown in Table 3
and it shows that our BLSTM-based AED model produces comparable results with
other Conformer-based AED models. However, by no means do we promote BLSTM
over Conformer here as usually Conformer produces competitive results, is fast
in decoding, and has a smaller memory footprint. From our experience,
Conformer based modeling is more sensitive to the learning rate control and
this is more the case when heterogeneous auxiliary losses are involved.
Nevertheless, our future goal is to transfer the improvement we observed here
to the state-of-the-art Conformer/Ebranchformer modeling.

##  5 Conclusion and Discussion

In this work, we proposed to use weak triphone alignment supervision and weak
BPE alignment supervision to aid the E2E ASR modeling. We demonstrated that
such alignments extracted by traditional hybrid system complement the standard
CTC auxiliary loss and improve cross-attention based encoder-decoder models.
As has been experimentally shown, the proposed weak alignment supervisions
resulted in significant gain in speech recognition accuracy, reducing the word
error rate by over 10% relative compared to a standard CTC auxiliary loss
based baseline. Nevertheless, we realize that although the proposed approach
works well with the BLSTM encoder, we still need to explore whether the
improvement can transfer to transformer-like encoder based AED systems.

## References

  * [1] R. Prabhavalkar, T. Hori _et al._ , “End-to-End speech recognition: A survey,” _arXiv preprint arXiv:2303.03329_ , 2023.
  * [2] J. Li, “Recent advances in end-to-end automatic speech recognition,” _APSIPA Trans. on Signal and Information Processing_ , vol. 11, no. 1, 2022\.
  * [3] D. Bahdanau, J. Chorowski _et al._ , “End-to-end attention-based large vocabulary speech recognition,” in _ICASSP_ , 2016.
  * [4] L. Lu, X. Zhang, and S. Renais, “On training the recurrent neural network encoder-decoder for large vocabulary end-to-end speech recognition,” in _ICASSP_ , 2016.
  * [5] A. Radford, J. Kim _et al._ , “Robust speech recognition via large-scale weak supervision,” _arXiv preprint arXiv:2212.04356_ , 2022.
  * [6] H. Bourlard and N. Morgan, _Connectionist speech recognition: A hybrid approach_. Norwell, MA, USA: Kluwer Academic Publishers, 1993.
  * [7] G. Kurata and K. Audhkhasi, “Guiding CTC posterior spike timings for improved posterior fusion and knowledge distillation,” in _Interspeech_ , 2019, pp. 1616–1620.
  * [8] J. Mahadeokar, S. Yuan _et al._ , “Alignment restricted streaming recurrent neural network transducer,” in _SLT_ , 2021, pp. 52–59.
  * [9] J. Jiang, Y. Gao, and Z. Tuske, “Weak alignment supervision from hybrid model improves End-to-end ASR,” _arXiv preprint arXiv:2311.14835_ , 2023.
  * [10] C. Szegedy, V. Vanhoucke _et al._ , “Rethinking the inception architecture for computer vision,” in _CVPR_ , 2016.
  * [11] S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning,” in _ICASSP_ , 2017.
  * [12] W. Chan, N. Jaitly _et al._ , “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in _ICASSP_ , 2016\.
  * [13] A. Zeyer, P. Bahar _et al._ , “A comparison of transformer and LSTM encoder decoder models for ASR,” in _ASRU_ , 2019.
  * [14] A. Graves, S. Fernández _et al._ , “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in _ICML_ , 2006.
  * [15] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in _ICML_ , 2014.
  * [16] A. Graves, “Sequence transduction with recurrent neural networks,” _arXiv preprint arXiv:1211.3711_ , 2012.
  * [17] K. Rao, H. Sak, and R. Prabhavalkar, “Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer,” in _ASRU_ , 2017.
  * [18] Y. He, T. Sainath _et al._ , “Streaming end-to-end speech recognition for mobile devices,” in _ICASSP_ , 2019.
  * [19] M. Jain, K. Schubert _et al._ , “RNN-T for latency controlled ASR with improved beam search,” _arXiv preprint arXiv:1911.01629_ , 2019.
  * [20] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” _arXiv preprint arXiv:1508.07909_ , 2015.
  * [21] F. Boyer, Y. Shinohara _et al._ , “A study of transducer based end-to-end asr with ESPnet: Architecture, auxiliary loss and decoding strategies,” in _ASRU_ , 2021.
  * [22] M. Luong, Q. Le _et al._ , “Multi-task sequence to sequence learning,” _arXiv preprint arXiv:1511.06114_ , 2015.
  * [23] M. Crawshaw, “Multi-task learning with deep neural networks: A survey,” _arXiv preprint arXiv:2009.09796_ , 2020.
  * [24] Y. Zhang and Q. Yang, “A survey on multi-task learning,” _IEEE Trans. on Knowledge and Data Engineering_ , vol. 34, no. 12, pp. 5586–5609, 2021.
  * [25] J. Lee and S. Watanabe, “Intermediate loss regularization for CTC-based speech recognition,” in _ICASSP_ , 2021.
  * [26] C. Liu, F. Zhang _et al._ , “Improving RNN transducer based ASR with auxiliary tasks,” in _SLT_ , 2021.
  * [27] A. Søgaard and Y. Goldberg, “Deep multi-task learning with low level tasks supervised at lower layers,” in _ACL_ , 2016.
  * [28] S. Toshniwal, H. Tang _et al._ , “Multitask learning with low-level auxiliary tasks for encoder-decoder based speech recognition,” _Interspeech_ , 2017.
  * [29] A. Zeyer, T. Alkhouli, and H. Ney, “RETURNN as a generic flexible neural toolkit with application to translation and speech recognition,” in _ACL_ , 2018.
  * [30] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [31] A. Graves, S. Fernández, and J. Schmidhuber, “Bidirectional LSTM networks for improved phoneme classification and recognition,” in _ICANN_ , 2005.
  * [32] D. Park, W. Chan _et al._ , “SpecAugment: A simple data augmentation method for automatic speech recognition,” _Interspeech_ , 2019.
  * [33] A. Rousseau, P. Deléglise _et al._ , “Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks,” in _LREC_ , 2014.
  * [34] D. Rybach, S. Hahn _et al._ , “RASR-The RWTH Aachen University open source speech recognition toolkit,” in _ASRU_ , 2011.
  * [35] M. Bisani and H. Ney, “Joint-sequence models for grapheme-to-phoneme conversion,” _Speech communication_ , vol. 50, no. 5, pp. 434–451, 2008\.
  * [36] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [37] J. Fiscus, “SCTK, the NIST scoring toolkit,” _https://github.com/usnistgov/SCTK_ , 2021.
  * [38] M. Zeineldeen, A. Zeyer _et al._ , “Chunked attention-based encoder-decoder model for streaming speech recognition,” _arXiv preprint arXiv:2309.08436_ , 2023.
  * [39] J. Gemmeke, D. Ellis _et al._ , “Audio set: An ontology and human-labeled dataset for audio events,” in _ICASSP_ , 2017.
  * [40] Y. Gao, C. Herold _et al._ , “Revisiting checkpoint averaging for neural machine translation,” _arXiv preprint arXiv:2210.11803_ , 2022.
  * [41] A. Gulati, J. Qin _et al._ , “Conformer: Convolution-augmented transformer for speech recognition,” in _Interspeech_ , 2020, pp. 5036–5040.
  * [42] K. Kim, F. Wu _et al._ , “E-Branchformer: Branchformer with enhanced merging for speech recognition,” in _SLT_ , 2023, pp. 84–91.
  * [43] S. Majumdar, J. Balam _et al._ , “Citrinet: Closing the gap between non-autoregressive and autoregressive end-to-end models for automatic speech recognition,” _arXiv preprint arXiv:2104.01721_ , 2021.