# The Low-redshift Lyman-continuum Survey: [S ii]-deficiency and the leakage
of ionizing radiation

Bingjie Wang (王冰洁) Department of Physics & Astronomy, Johns Hopkins
University, Baltimore, MD 21218, USA  Timothy M. Heckman  Department of
Physics & Astronomy, Johns Hopkins University, Baltimore, MD 21218, USA
Ricardo Amorín  Instituto de Investigacíon Multidisciplinar en Ciencia y
Tecnología, Universidad de La Serena, Raúl Bitrán 1305, La Serena, Chile
Departamento de Astronomía, Universidad de La Serena, La Serena, Chile
Sanchayeeta Borthakur  School of Earth & Space Exploration, Arizona State
University, Tempe, AZ 85287, USA  John Chisholm  Department of Astronomy,
University of Texas at Austin, Austin, TX 78712, USA  Harry Ferguson  Space
Telescope Science Institute, Baltimore, MD 21218, USA  Sophia Flury  Astronomy
Department, University of Massachusetts, Amherst, MA, 01003, USA  Mauro
Giavalisco  Astronomy Department, University of Massachusetts, Amherst, MA,
01003, USA  Andrea Grazian  INAF-Osservatorio Astronomico di Padova, Vicolo
dell’Osservatorio 5, I-35122, Padova, Italy  Matthew Hayes  Department of
Astronomy, University of Stockholm, AlbaNova, Stockholm, Sweden  Alaina Henry
Space Telescope Science Institute, Baltimore, MD 21218, USA  Anne Jaskot
Astronomy Department, Williams College, Williamstown, MA 01267, USA  Zhiyuan
Ji  Astronomy Department, University of Massachusetts, Amherst, MA, 01003, USA
Kirill Makan  Institut für Physik und Astronomie, Universität Potsdam, D-14476
Potsdam, Germany  Stephan McCandliss  Department of Physics & Astronomy, Johns
Hopkins University, Baltimore, MD 21218, USA  M. S. Oey  Department of
Astronomy, University of Michigan, Ann Arbor, MI 48109, USA  Göran Östlin
Department of Astronomy, University of Stockholm, AlbaNova, Stockholm, Sweden
Alberto Saldana-Lopez  Observatoire de Genève, Université de Genève, 1290
Versoix, Switzerland  Daniel Schaerer  Observatoire de Genève, Université de
Genève, 1290 Versoix, Switzerland  Trinh Thuan  Astronomy Department,
University of Virginia, Charlottesville, VA 22904, USA  Gábor Worseck
Institut für Physik und Astronomie, Universität Potsdam, D-14476 Potsdam,
Germany  Xinfeng Xu  Department of Physics & Astronomy, Johns Hopkins
University, Baltimore, MD 21218, USA

###### Abstract

The relationship between galaxy characteristics and the reionization of the
universe remains elusive, mainly due to the observational difficulty in
accessing the Lyman continuum (LyC) at these redshifts. It is thus important
to identify low-redshift LyC-leaking galaxies that can be used as laboratories
to investigate the physical processes that allow LyC photons to escape. The
weakness of the [S ii] nebular emission lines relative to typical star-forming
galaxies has been proposed as a LyC predictor. In this paper, we show that the
[S ii]-deficiency is an effective method to select LyC-leaking candidates
using data from the Low-redshift LyC Survey, which has detected flux below the
Lyman edge in 35 out of 66 star-forming galaxies with the Cosmic Origins
Spectrograph onboard the Hubble Space Telescope. We show that LyC leakers tend
to be more [S ii]-deficient and that the fraction of their detections
increases as [S ii]-deficiency becomes more prominent. Correlational studies
suggest that [S ii]-deficiency complements other LyC diagnostics (such as
strong Lyman-$\alpha$ emission and high [O iii]/[O ii]). Our results verify an
additional technique by which reionization-era galaxies could be studied.

circumgalactic medium – extragalactic astronomy – reionization – starburst
galaxies – interstellar medium

††facilities: HST (COS), Sloan††software: Astropy v. 3.2.2 (Astropy
Collaboration et al., 2013, 2018), Matplotlib v. 3.1.1 (Hunter, 2007), NumPy
v. 1.17.2 (Harris et al., 2020), Pymccorrelation v 0.2.3 (Curran, 2014; Privon
et al., 2020), SciPy v. 1.3.1 (Virtanen et al., 2020)

##  1 Introduction

The epoch of reionization (EoR), the phase during which the universe
transitions from fully neutral to ionized, remains largely unexplored
observationally. At the center is the question regarding the sources
responsible for the EoR. Deep imaging with the Hubble Space Telescope (HST)
indicates that the ultraviolet (UV) luminosity density of early star-forming
galaxies is high enough that they are the best candidates to provide the
ionizing photons necessary for reionizing the universe (e.g., Bouwens et al.
2016). Unfortunately, since the universe during the EoR is opaque to ionizing
photons, direct observations that access the LyC at these redshifts are
impossible.

Identifying LyC emitters (LCEs) at low redshifts thus becomes an important
step in the investigation of how galaxies could reionize the universe. Over
the past decade, the community has invested in a huge effort to identify small
samples of LCEs. Several proxies for LyC escape have been suggested. Strong
Ly$\alpha$ emission is perhaps the most known one, and has been shown to
correlate with LyC emission both in individual galaxies at low-$z$ (Verhamme
et al., 2017) and in stacked samples at $z\sim 3$ (Marchi et al., 2018;
Steidel et al., 2018). However, the absorption due to the neutral
intergalactic medium limits its utility at $z\gtrsim 6$. High [O iii]/[O ii]
flux ratios, which indicate high ionization state, have been used to select
LyC-emitting Green Pea galaxies (Izotov et al., 2016a; Jaskot et al., 2019).
This class of galaxies constitutes the majority of LCEs in the literature so
far. LyC predictors based on UV absorption lines and Mg ii emission have also
been proposed recently (Chisholm et al., 2018, 2020).

Among those efforts, Wang et al. (2019) (hereafter W19) tested a new
diagnostic for LyC leakage, the relative weakness of [2] nebular emission
lines ($\Delta$[2]), in a pilot HST program. Significant emerging flux below
the Lyman edge was detected in two out of three [2]-deficiency-selected star-
forming galaxies at $z\sim 0.3$.

In this paper, we explore the connection between LyC leakage and
[2]-deficiency with the expanded data set from the Low-redshift LyC Survey
(LzLCS)—a large HST program aiming for a first statistical sample at $z\sim
0.3$. The full sample consists of 66 star-forming galaxies, and is described
in Flury et al. (2021a).

The structure of this paper is as follows. In Section 2, we begin by reviewing
the physical basis for the [2]-deficiency diagnostic and its definition. In
Section 3, we summarize the galaxy samples and the relevant analyses. In
Section 4, we assess the robustness of the [2]-deficiency test, and compare it
to other proposed LyC diagnostics. In Section 5, we discuss the implications
for the physical properties of LCEs at low and high-$z$. Finally, we summarize
our conclusions in Section 6.

##  2 [2]-deficiency

Figure 1: (Left) This BPT/VO diagram drawn from the SDSS DR12 + BOSS DR8 star-
forming galaxy sample is used for defining $\Delta$[2]. The colorbar shows the
number of galaxies in each hexbin. The black dotted line is fitted to the
locus of the peak density of this distribution. Its uncertainty (gray shade)
is negligible expect for log([O iii]/H$\beta$) $>0.8$, where data becomes
sparse. The [2]-deficiency is defined as a galaxy’s displacement along the
$x$-axis from this ridge line. Also plotted are the galaxy samples considered
in this paper: LzLCS, Green Pea galaxies (GP; Izotov et al. 2016a, b, 2018a,
2018b), and [2]-deficiency-selected galaxies (W19; Wang et al. 2019). (Right)
A zoom-in on the upper part of the left figure. Solid colored dots represent
strong LCEs, circles represent weak LCEs, and gray dots are non-detections.

The relative weakness of the [2] 6717,6731 emission lines was first proposed
as a signpost to identify galaxies that likely allow the escape of LyC
radiation by Alexandroff et al. (2015). This was motivated by a simple
physical argument: the ionization potential for producing [2] is only 10.4eV,
which is less than that for ionizing neutral hydrogen. Therefore much of the
[2] emission arises in the warm partially ionized region near and just beyond
the outer edge of the Strömgren sphere in a classical H ii region. This region
is weak or even absent when the medium is optically thin to the LyC, resulting
in a significant drop in the relative intensity of [2] emission lines
(Pellegrini et al., 2012).

We measure the [2]-deficiency in a differential sense: as a quantity relative
to the majority of star-forming galaxies in the Sloan Digital Sky Survey
(SDSS). In practice, a BPT/VO diagram (Baldwin et al., 1981; Veilleux &
Osterbrock, 1987) of [2] 6717,6731/H$\alpha$  _vs._ [O iii] 5007/H$\beta$ is
used. We follow the procedure outlined in W19 in this paper, but with the
slight modification of including the star-forming galaxies in the Baryon
Oscillation Spectroscopic Survey (BOSS) to improve the sampling of high-
excitation galaxies. All fluxes are taken from the value added catalog
provided by the Portsmouth group (Thomas et al., 2013), and a signal-to-noise
(S/N) cut of five is imposed. Both emission-line ratios are dust-extinction
corrected based on the observed Balmer lines. Measured values of
log([2]/H$\alpha$) are binned in log([O iii]/H$\beta$), and the Gaussian mean
(or skewed Gaussian mean when more suitable) of each bin is then calculated.
The uncertainties in the mean values are estimated via bootstrap, and are
negligible except for the cases where data is sparse for log([O iii]/H$\beta$)
$>\,\sim$0.8. As seen in Figure 1, only a small fraction of galaxies in LzLCS
reside in this region of very high [O iii]/H$\beta$. We define the
[2]-deficiency as a galaxy’s displacement in log([2]/H$\alpha$) from a
polynomial fit to Gaussian means. The fitting formula is

$y=-0.475-0.051\xi-0.589\xi^{2}-0.360\xi^{3},$ (1)

where $\xi$ is log([O iii]/H$\beta$), and $y$ is log([2]/H$\alpha$). In
addition, we note that this curve only differs slightly from the one used in
W19 or that in Ramambason et al. (2020). The difference is within $1\sigma$
even for log([O iii]/H$\beta$) > 0.8. The samples of galaxies shown as filled
and unfilled circles in the Figure 1 and their relevant analyses are presented
in the section below.

##  3 Data

###  3.1 Galaxy samples

LzLCS (HST-GO-15626; PI A. Jaskot) consists of a sample of 66 star-forming
galaxies at $z\sim 0.3$, selected to meet at least one of the following three
criteria: [O iii]/[O ii] flux ratio $>3$, UV spectral slope $\beta<-2$,
$\Sigma_{\rm SFR}>0.1\,M_{\odot}{\rm yr}^{-1}{\rm kpc}^{-2}$.

We additionally consider the following two samples from the literature. First,
three star-forming galaxies were selected based on [2]-deficiency in a pilot
program (HST-GO-15341; PI T. Heckman), two of which have been observed with
significant LyC flux in W19. Second, 11 Green Pea galaxies (GPs) are included,
which constitute the majority of confirmed low-$z$ LCEs in the existing
literature before LzLCS (Izotov et al., 2016a, b, 2018a, 2018b).

###  3.2 Data analysis

Processing of the COS spectra in LzLCS is presented in detail in Flury et al.
(2021a). UV continuum fitting is discussed in Saldana-Lopez et al. (2021).
Here we describe the additional analyses performed for the purpose of this
paper.

The fluxes of [2] emission lines in SDSS spectra of are re-measured for all
sources in LzLCS. This is done out of caution as some LzLCS spectra are found
to have low S/N in the [2] lines. However, this does not imply that the fluxes
of the SDSS reference sample used to draw the curve of $\Delta$[2] = 0 are
subject to a similar error, since only emission lines with S/N $\geq$ 5 are
selected.

For SDSS spectra of the LzLCS sample, we categorize them into three subgroups
based on S/N and each is subjected to a different treatment: 1) both lines of
the [2] doublet, when detected, are simultaneously fitted with Gaussians, and
$1\sigma$ uncertainties as provided in SDSS spectra are propagated; 2) when
only one of the [2] lines is detected with significance, the undetected line
is inferred from the detection; that is, its Gaussian fit is constrained by
the line-center shift, amplitude (after taking the typical ratio of the
doublets into account), and the full width at half maximum of the detected
line; 3) when neither of the [2] lines are available, $3\sigma$ upper limits
are inferred from uncertainties. We list the measured flux along with
[2]-deficiency in the Appendix.

###  3.3 Escape fractions

Given the complications in estimating escape fractions ($f_{\rm esc}$) of the
LyC photons, two approaches are considered. First, the ratio between the flux
of LyC and of the stellar continuum at $\sim 1100$ Å rest-frame ($f_{\rm
LyC}/f_{1100}$) is used as a proxy of $f_{\rm esc}$. It has the advantage of
being less model-dependent. Second, it is common to estimate $f_{\rm esc}$ by
comparing the observed ratio of flux density at $\sim 900$ Å to that at $\sim
1500$ Å ($F_{900}/F_{1500}$) with the intrinsic ratio (Steidel et al., 2001).
After accounting for the dust attenuation, we obtain the absolute $f_{\rm
esc}$ ($f_{\rm esc,UV}$). Specifically, we derive $f_{\rm esc,UV}$ by finding
an intrinsic UV spectrum following the fitting methods of Chisholm et al.
(2019) and presented in Saldana-Lopez et al. (2021). We fit the observed
stellar continuum as a linear combination of multiple single-age and single
stellar metallicity bursts. We use 50 possible theoretical Starburst99 models
(Leitherer et al., 1999, 2010) that span a range of ages (1 – 40 Myr) and
metallicities (0.05 – 2 $Z_{\odot}$) that are relevant to the young starbursts
in the LzLCS. We assume a standard Kroupa initial mass function (Kroupa, 2002)
and use the Geneva stellar evolution tracks with high-mass loss rates (Meynet
et al., 1994). Finally, we account for dust attenuation using the Reddy et al.
(2016) law, which is observationally defined down to 950Å. We fit to the
stellar continuum redward of 950Å by masking the strong ISM absorption
features and fitting the best-fit linear coefficients and dust attenuation
parameter. We then extend the stellar continuum fit blueward to LyC and take
the ratio of the fit to the observations to determine $f_{\rm esc,UV}$.

##  4 Results

###  4.1 Statistical tests of the [S II]-deficiency diagnostic

Figure 2: (a) Histograms and (b) Gaussian KDEs showing distributions of
$\Delta$[2] among the whole sample. The two dotted lines represent $\Delta$[2]
= -0.2 and -0.1 respectively. (c) Fractions of strong and all LCE detections
in bins of $\Delta$[2]. Results from different samples are offset in
$\Delta$[2] for clarity. Correlations are exhibited despite of substantial
Poisson uncertainties driven by small number counts (see Table 1).

The larger sample size of LzLCS allows for a first statistical study on the
reliability of the [2]-deficiency diagnostic. Before proceeding to present the
results, we would first like to reiterate definitions of LCEs in the survey
(Flury et al., 2021a). The classification is determined based on two criteria:
the probability ($P(>N|B)$) that the observed or gross counts within the
extraction window of the LyC are due to background fluctuations (Worseck et
al., 2016), and escape fraction ($f_{\rm esc}$) or its proxy $f_{\rm
LyC}/f_{1100}$. First, we define LCEs as having $P(>N|B)<0.02275$. Second, a
subset of LCEs having $P(>N|B)<2.867\times 10^{-7}$ and $f_{\rm esc}$ or
$f_{\rm LyC}/f_{1100}>0.05$ is defined as “strong” detections. The rest is
considered non-LCEs. We note that using different definitions of $f_{\rm esc}$
yields qualitatively similar results.

We address the robustness of the [2]-deficiency diagnostic from two
perspectives. First, distributions of $\Delta$[2] among the different groups
are compared. Figure 2(a)-(b) show the histograms and Gaussian kernel density
estimates (KDEs) of $\Delta$[2], respectively. A preference for the class of
LCEs to have more negative $\Delta$[2] than that of the non-LCEs is present.

To quantify the above, we calculate the Anderson-Darling statistic for the
LCEs (and strong LCEs): 3.0 (2.3), suggesting that the null hypothesis that
the two samples come from the same distribution can be rejected at about
$98.0\%$ ($96.3\%$) level.

Second, we calculate fractions of LCE detections in bins of $\Delta$[2], as
shown in Figure 2(c). Significant correlations are exhibited, despite
substantial Poisson uncertainties on fractions which are driven by the small
number counts. We list the correlation coefficients in Table 1.

Taken together, those results indicate that a candidate’s likelihood of being
a LCE increases as [2]-deficiency becomes more prominent.

Table 1: Correlations Between $\Delta$[2] and $f_{\rm esc}$/ LCE Fractions | Kendall’s $\tau$ |  $p$-value
---|---|---
$f_{\rm LyC}/f_{1100}$aaCorrelations are calculated from the whole sample (see Figure 4). | $-0.256$ | $0.001$
$f_{\rm esc,UV}$aaCorrelations are calculated from the whole sample (see Figure 4). | $-0.229$ | $0.003$
$\mathit{F}$(s)bbFractions of strong LCEs in bins of $\Delta$[2] (see Figure 2c). | $-0.828$ | $0.022$
$\mathit{F}$(all)ccFractions of all LCEs in bins of $\Delta$[2]. | $-0.867$ | $0.017$
Table 2: LCE Fractions in [2]-weak Galaxies vs. Their Twins | LCE fractionaaPairs of [2]-weak ($\Delta$[2] $\leq-0.2$) and non-[2]-weak galaxies sharing similar values of each parameter listed in column 1 are selected, and this column lists the LCE fractions in non-[2]-weak samples. These can be compared to the LCE fraction in the [2]-weak sample of $0.86\pm 0.07$, and to the strong LCE of $0.59\pm 0.10$. | sig.bbSignificance in the difference between a LCE fraction in the [2]-weak sample and that in the non-[2]-weak sample. | Strong LCE fraction | sig.
---|---|---|---|---
$f_{\rm esc}({\rm Ly}{\alpha})$ | 0.59 $\pm$ 0.09 | 2.34 | 0.18 $\pm$ 0.09 | 2.97
$[$O iii$]$/$[$O ii$]$ | 0.62 $\pm$ 0.10 | 2.00 | 0.15 $\pm$ 0.10 | 3.02
$[$O i$]$/H$\beta$ | 0.62 $\pm$ 0.12 | 1.74 | 0.31 $\pm$ 0.12 | 1.78
${\rm EW}({\rm H}{\beta})$ | 0.50 $\pm$ 0.11 | 2.76 | 0.21 $\pm$ 0.11 | 2.48
${\rm EW}({\rm Ly}{\alpha})$ | 0.71 $\pm$ 0.12 | 1.06 | 0.29 $\pm$ 0.12 | 1.91
EW(C ii 1334) | 0.47 $\pm$ 0.10 | 3.14 | 0.20 $\pm$ 0.10 | 2.66
EW(Si ii 1260) | 0.35 $\pm$ 0.09 | 4.33 | 0.18 $\pm$ 0.09 | 2.97
$M_{\rm FUV}$ | 0.54 $\pm$ 0.10 | 2.62 | 0.15 $\pm$ 0.10 | 3.02
$M_{\rm NUV}$ | 0.40 $\pm$ 0.09 | 4.06 | 0.13 $\pm$ 0.09 | 3.35
UV $\beta$ | 0.39 $\pm$ 0.10 | 3.88 | 0.22 $\pm$ 0.10 | 2.57
$M_{\star}$ | 0.53 $\pm$ 0.09 | 2.89 | 0.13 $\pm$ 0.09 | 3.35
$r_{50}$ | 0.56 $\pm$ 0.12 | 2.20 | 0.31 $\pm$ 0.12 | 1.78
SFRHβ/area | 0.47 $\pm$ 0.10 | 3.14 | 0.20 $\pm$ 0.10 | 2.66
SFRUV/area | 0.59 $\pm$ 0.09 | 2.34 | 0.18 $\pm$ 0.09 | 2.97
${\rm SFR_{\rm H\beta}}/M_{\star}$ | 0.50 $\pm$ 0.11 | 2.78 | 0.25 $\pm$ 0.11 | 2.26
${\rm SFR_{\rm UV}}/M_{\star}$ | 0.47 $\pm$ 0.09 | 3.47 | 0.13 $\pm$ 0.09 | 3.35
Table 3: Uncertainties and statistical significances are estimated assuming
binomial distributions.

###  4.2 Comparison to other LyC diagnostics

Figure 3: Scatter plots showing correlations between $\Delta$[2] and other
galaxy characteristics. Colors indicate $f_{\rm esc}$. Triangles represent
that the associated $f_{\rm esc}$ are upper limits. $\tau$ and $p$ are
Kendall’s correlation coefficient and $p$-value respectively. All show
substantial scatter; the rest can be found in the Appendix.

A number of galaxy characteristics have been identified as potential signposts
of LyC leakage (e.g., Verhamme et al. 2015; Izotov et al. 2016a; Chisholm et
al. 2018; for results from LzLCS, see Flury et al. 2021b). Here we evaluate
the [2]-deficiency diagnostic in context of its relationship to other
signposts. We note that the velocity separation ($v_{\rm sep}$) of the two
Ly$\alpha$ peaks and $f_{\rm esc}$ of the LyC has been shown to be tightly
correlated (Izotov et al., 2018b). However, LzLCS lacks the necessary
observations using the low resolution COS/G140L grating to measure $v_{\rm
sep}$. We hence neglect a discussion regarding $v_{\rm sep}$ in what follows.

Figure 3 shows the galaxy distributions in the plane of $\Delta$[2]  _vs._
other signposts, colored-coded in $f_{\rm esc}$ (see Sec. 5.2 for details on
$f_{\rm esc}$). The corresponding Kendall’s $\tau$ correlation coefficients
and their significance ($p$-values) are also shown in the figures. Their
uncertainties are estimated via bootstrap. Most of the correlations are rather
weak. Among the strongest ones are between $\Delta$[2] and the half-light
radii ($r_{50}$) measured from COS NUV ACQ images and the star-formation rate
per unit area ($\Sigma_{\rm SFR,H\beta}$). This is consistent with the
frequent identification of LCEs as being highly compact. It is then reasonable
to speculate that LyC escape can be made possible by the extreme feedback
effects produced by a “dominant central object.” These objects are defined to
be compact, very massive, and young objects located at or near the galactic
nucleus (Heckman et al., 2011; Borthakur et al., 2014). Similar findings have
likewise been reported in Wang et al. (2019); Hogarth et al. (2020).

Although adopting the standard discriminant of $p=0.05$ means that most of the
correlations are not significant, we do see significant additional
correlations with EW($H\beta$), $M_{\rm FUV}$, $M_{\rm NUV}$, [O iii]/[O ii],
and SFRUV/$M_{\star}$ In all cases, the scatter is substantial. This suggests
that [2]-deficiency is providing information on LyC leakage that is largely
independent of the other signposts.

To further demonstrate this, we construct pairs of [2]-weak ($\Delta$[2]
$\leq-0.2$) and non-[2]-weak galaxies (“twins”) that share the closest values
for each of the other parameters shown in Figures 3 and in the Appendix. For
example, in the case of the EW(Ly$\alpha$) parameter, for each [2]-weak
galaxy, we find a non-[2]-weak galaxy that is the closest match in
EW(Ly$\alpha$). When one galaxy has multiple twins, we remove the duplicates
from calculations. We then compare LCE fractions between the two sub-samples
and test the statistical significance of the difference. The results are
listed in Table 3. We note that LCE fractions in the [2]-weak sample is
greater than those in the twin samples in all cases, although for some
parameters, the large uncertainties due to the relatively small sample size
lead to statistically insignificant results.

##  5 Discussion

###  5.1 Implications for photoionization models

Fundamentally, there are two different mechanisms by which LyC leakage can
occur from a star-forming region—a density-bounded nebula and a radiation-
bounded nebula with holes (Zackrisson et al., 2013). The former refers to a
scenario in which regions undergoing intense stellar formation fully ionize
their surroundings and have ionizing photons leftover that can escape, while
the latter refers to a “picket-fence” scenario in which supernovae or stellar
winds clear out low-density channels in the neutral ISM, through which LyC
photons escape (Bergvall et al., 2006).

The density-bounded case is the simplest picture explaining the correlation
between weak [2] and LyC escape, as the absence of H i near the edge of a
Strömgren sphere leads to significant decrease in [2] emissions. However, a
simple density-bounded model cannot account for the majority of LyC detections
in UV absorption-line studies (Chisholm et al., 2018; Gazagnes et al., 2018),
and for the observed optical emission-line ratios (Ramambason et al., 2020).

The “picket-fence” model therefore seems to be favored by observations. In
this case, [2]-deficiency indicates that a significant fraction of the total
solid angle as seen from the star-forming region is optically thin in the LyC.
We also note that a reduced column density of the neutral gas in the fence is
require to explain the largest observed $f_{\rm esc}$ (Gazagnes et al., 2020;
Ramambason et al., 2020).

Instead of the above two classical one-zone models, Ramambason et al. (2020)
propose two-zone photoionization models, in which ionization parameters and
covering fractions are varied along different lines of sight. The two-zone
models successfully reproduce the observed emission-line properties in low-$z$
LCEs, including [2]-deficiency, and are compatible to results from UV
absorption-line studies.

In Figure 5 we show schematic diagrams of such a two-zone model for a LCE and
a non-LCE in the context of a simple wind-blown shell. In the former case,
only the optically thick clouds contribute to [2] emission while both the
clouds and the optically thin regions between the clouds contribute to
emission from Balmer and high ionization metal lines. In the latter case the
entire shell contributes to the [2] emission.

###  5.2 Implications for escape fractions

Figure 4: Escape fractions are plotted as functions of [2]-deficiency. Linear
scale is used for $f_{\rm esc}$ $<0.1$. Colored dots represent strong LCEs,
circles represent weak LCEs, and gray dots represent upper limits. Most of the
[2]-deficit galaxies ($\Delta$[2] $\lesssim-0.2$) are LCEs, but only weak
(albeit statistically significant) correlations are found with $f_{\rm esc}$
(see Table 1).

The relationship between $\Delta$[2] and $f_{\rm esc}$ shows substantial
scatter, as evident in Figure 4. Generalized Kendall’s $\tau$ correlation
coefficients which account for upper limits in $f_{\rm esc}$ are listed in
Table 1 (Isobe et al., 1986). Our finding is in agreement with Ramambason et
al. (2020), who find that although [2]-deficiency can select LCE candidates,
it is not obvious how to infer a numerical value for $f_{\rm esc}$ directly
from $\Delta$[2].

Several factors could contribute to the observed complexity. The simplest
explanation is line-of-sight variations caused by porous H ii regions.
Observations from the Keck Lyman Continuum Spectroscopic Survey (Steidel et
al., 2018) and another sample of galaxies at $z\sim 3$ having high [O iii]/[O
ii] but low $f_{\rm esc}$ (Nakajima et al., 2020) are both consistent with
this picture.

A complication, though, arises from the possibility of anisotropically
escaping LyC photons (e.g., Zastrow et al. 2011; Cen & Kimm 2015). This
directly leads to ambiguity in the interpretation of non-detections—it is
unclear if a galaxy not observed to have LyC escaping is truly a non-detection
or is due to a particular orientation toward the observer. With a larger
sample, we would be able to infer the fraction of LCEs with its proportional
relationship to the H i covering fraction.

In short, an intricate interplay among factors dictates $f_{\rm esc}$.
Although unified models for describing different samples of LCEs have been
proposed (Cen, 2020), it is likely that a combination of other properties
(e.g., those shown in Figure 4 and with Mg ii studied in Henry et al. 2018;
Chisholm et al. 2020) is needed for accurately inferring $f_{\rm esc}$.

In the face of this complexity, it seems that the determination of leakiness
based on indirect signposts can only be done on a statistical basis rather
than for individual objects (cf. Runnholm et al. 2020 for predicting
Ly$\alpha$ radiation using multivariate regression). This reinforces the need
for large samples, which was the main driver for the LzLCS. Even larger
samples will ultimately be needed in the future.

Figure 5: Schematic diagrams for a LCE (left panel) and a non-LCE (right
panel) in the context of a simple wind-blown shell. In the case of a LCE, only
the optically thick clouds contribute to [2] emission while both the clouds
and the optically thin regions between the clouds contribute to emission from
Balmer and high ionization metal lines. In the case of a non-LCE, the entire
shell contributes to the [2] emission. We note that they only represent the
simplest cases where isotropy is assumed.

###  5.3 Analogs to high-$z$ galaxies

Having discussed what can be learned from low-$z$ LCEs, this final section
addresses whether they can truly represent the galaxies during EoR. One
question is the degree to which our definition of [2]-deficiency is based on
galaxies at low $z$, which differ significantly from EoR galaxies. This can be
addressed through spectroscopy using the James Webb Space Telescope (JWST) to
construct BPT/VO diagrams like Figure 1 for EoR galaxies. It is promising that
galaxies at $\langle z\rangle\sim$ 2.3 do follow the same locus as low-$z$
galaxies in Figure 1 (Strom et al., 2018; Wang et al., 2019).

In a recent analysis of simulated Lyman-break galaxies, Katz et al. (2020)
claim that the sample of $z\sim 3$ LCEs (Nakajima et al., 2020) are good
analogs of EoR galaxies, while the [2]-deficient galaxies in W19 may not
necessarily be so. The argument is largely based on whether the analogs
populate the same regions in respective BPT/VO diagnostic diagrams. Katz et
al. (2020) find that the LCEs in their simulations have deficits both in [2]
and [O iii] indicating that the dominant effect is metallicity or mass rather
than a property of the ISM.

We agree that metallicity or mass possibly play important roles (Jaskot et
al., 2019), especially given that many LCEs fall at the top (metal-poor) end
of the BPT/VO diagram as shown in Figure 1. It is also true that galaxies are
intrinsically complex systems, and thus it is difficult to disentangle primary
and secondary correlations. The issue of whether high-$z$ and local galaxies
can be analogs to one another is further complicated by the finding based on
the MOSDEF–LRIS Survey that a similarity in the location of high-$z$ and local
galaxies in the BPT/VO diagrams may not always be indicative of a similarity
in their physical properties (Topping et al., 2020).

In general, while simulations shed light on the study of EoR galaxies, the
multi-phase nature of the interstellar and circumgalactic media as well as the
high spatial resolution needed to explicitly simulate the LyC-escaping process
reinforce the need for robust observational evidence. Given the relatively
small pool of LCEs, the different physical properties exhibited by some of the
[2]-deficient LCEs nevertheless offer valuable insights into the possibly
different ways in which LyC can escape from galaxies.

##  6 Conclusions

We have reported on using the relative weakness of the [2] 6717,6731 nebular
emission lines defined with respect to normal star-forming galaxies as an
indicator for galaxies that are optically thin to ionizing radiation. This
method was proposed in Wang et al. (2019), and statistically tested in this
paper with new HST/COS observations of 66 star-forming galaxies in LzLCS
(Flury et al., 2021a). We find that [2]-deficiency is an effective way to
identify candidates for LyC-emitting galaxies, and can complement other
proposed LyC predictors.

More specifically, we have shown that the LyC-emitting galaxies are more
[2]-deficient than the other galaxies, that the detection fraction of them
increases strongly as a function of [2]-deficiency, and that the value of the
far-UV based escape fractions have statistically-significant (but weak)
correlations with [2]-deficiency. In addition, we have also shown that
[2]-deficiency does not show a significant correlation with most of the other
proposed indirect signposts of LyC leakage with the exception of the
compactness of the starburst. This implies that [2]-deficiency is an
independent indicator of LyC leakage.

We have discussed the photo-ionizing process in light of the scatter seen in
the relationship between [2]-deficiency and $f_{\rm esc}$. This likely
indicates line-of-sight variations in ionization parameters and covering
fractions, and/or anisotropically escaping LyC photons, which is in agreement
with serval other studies at low to intermediate redshifts (Steidel et al.,
2018; Ramambason et al., 2020).

To summarize, the increased sample size of low-$z$ LyC-emitting galaxies from
LzLCS allows us to statistically confirm that [2]-deficiency is a robust
technique for finding galaxies leaking a significant amount of the LyC
radiation. Although it is yet not obvious how accurately the numerical value
of escape fraction of LyC could be inferred from [2]-deficiency for individual
galaxies, it is very useful in the context of statistical estimates for
samples of galaxies. This gives us an additional technique to identify
potential LyC-emitting galaxies at low-$z$ and during the EoR with future
observations with the JWST.

B.W. thanks Yiwei Sun for helping with making the schematic diagrams. This
work is supported by HST-GO-15626, provided by NASA through a grant from the
Space Telescope Science Institute, which is operated by the Association of
Universities for Research in Astronomy, Inc., under NASA contract NAS5-26555.
This publication made use of the NASA Astrophysical Data System for
bibliographic information. This project also made use of SDSS data. Funding
for the Sloan Digital Sky Survey IV has been provided by the Alfred P. Sloan
Foundation, the U.S. Department of Energy Office of Science, and the
Participating Institutions. SDSS-IV acknowledges support and resources from
the Center for High-Performance Computing at the University of Utah. The SDSS
web site is www.sdss.org. SDSS-IV is managed by the Astrophysical Research
Consortium for the Participating Institutions of the SDSS Collaboration
including the Brazilian Participation Group, the Carnegie Institution for
Science, Carnegie Mellon University, the Chilean Participation Group, the
French Participation Group, Harvard-Smithsonian Center for Astrophysics,
Instituto de Astrofísica de Canarias, The Johns Hopkins University, Kavli
Institute for the Physics and Mathematics of the Universe (IPMU) / University
of Tokyo, the Korean Participation Group, Lawrence Berkeley National
Laboratory, Leibniz Institut für Astrophysik Potsdam (AIP), Max-Planck-
Institut für Astronomie (MPIA Heidelberg), Max-Planck-Institut für Astrophysik
(MPA Garching), Max-Planck-Institut für Extraterrestrische Physik (MPE),
National Astronomical Observatories of China, New Mexico State University, New
York University, University of Notre Dame, Observatário Nacional / MCTI, The
Ohio State University, Pennsylvania State University, Shanghai Astronomical
Observatory, United Kingdom Participation Group, Universidad Nacional Autónoma
de México, University of Arizona, University of Colorado Boulder, University
of Oxford, University of Portsmouth, University of Utah, University of
Virginia, University of Washington, University of Wisconsin, Vanderbilt
University, and Yale University.

Table 4: Flux Measurements of Galaxies in LzLCS.

Galaxy | LCE typeaaClasses of LCEs: $\bullet$ strong LCEs, $\circ$ weak LCEs, $\times$ non-LCEs. | [2] |  $\Delta$[2]
---|---|---|---
|  | ($\times 10^{-17}{\rm erg\,cm^{-2}\,s^{-1}\,\mbox{\AA}^{-1}}$) | (dex)
J0036 | $\times$ | 8.17 $\pm$ 4.44 | -0.60 $\pm$ 0.24
J0047 | $\circ$ | 51.06 $\pm$ 4.63 | 0.02 $\pm$ 0.04
J0113 | $\circ$ | 45.98 $\pm$ 10.24 | 0.23 $\pm$ 0.10
J0122 | $\circ$ | 11.81 $\pm$ 3.82 | -0.34 $\pm$ 0.14
J0129 | $\times$ | 98.92 $\pm$ 10.38 | -0.10 $\pm$ 0.05
J0723 | $\times$ | 54.75 $\pm$ 2.56 | 0.00 $\pm$ 0.02
J0804 | $\bullet$ | 4.33$\dagger$$\dagger$footnotemark:  | -0.67
J0811 | $\circ$ | 1.60$\dagger$$\dagger$footnotemark:  | -0.80
J0814 | $\times$ | 207.83 $\pm$ 7.99 | -0.03 $\pm$ 0.02
J0826 | $\times$ | 19.14 $\pm$ 5.77 | -0.25 $\pm$ 0.13
J0834 | $\times$ | 104.46 $\pm$ 12.70 | -0.01 $\pm$ 0.05
J0909 | $\bullet$ | 4.30 | -0.69
J0911 | $\circ$ | 149.60 $\pm$ 13.13 | -0.20 $\pm$ 0.04
J091207 | $\times$ | 124.31 $\pm$ 4.79 | -0.05 $\pm$ 0.02
J091208 | $\times$ | 44.18 $\pm$ 4.50 | -0.08 $\pm$ 0.04
J0917 | $\bullet$ | 79.66 $\pm$ 8.61 | -0.25 $\pm$ 0.05
J0925 | $\times$ | 20.05 $\pm$ 29.71 | -0.08 $\pm$ 0.64
J0933 | $\bullet$ | 11.55 $\pm$ 4.00 | -0.52 $\pm$ 0.15
J0940 | $\times$ | 49.59 $\pm$ 5.64 | 0.03 $\pm$ 0.05
J0952 | $\circ$ | 9.64 $\pm$ 7.02 | -0.40 $\pm$ 0.32
J0957 | $\times$ | 302.97 $\pm$ 14.10 | -0.19 $\pm$ 0.02
J0958 | $\circ$ | 13.43 $\pm$ 10.01 | -0.38 $\pm$ 0.32
J1014 | $\times$ | 23.52 $\pm$ 3.09 | -0.18 $\pm$ 0.06
J1026 | $\circ$ | 23.63 | 0.14
J1033 | $\bullet$ | 72.51 $\pm$ 11.64 | -0.05 $\pm$ 0.07
J1038 | $\circ$ | 158.05 $\pm$ 8.43 | -0.14 $\pm$ 0.02
J1051 | $\times$ | 45.32 $\pm$ 4.46 | 0.04 $\pm$ 0.04
J1053 | $\circ$ | 225.05 $\pm$ 15.12 | -0.08 $\pm$ 0.03
J1055 | $\times$ | 36.27 $\pm$ 4.04 | 0.14 $\pm$ 0.05
J1104 | $\times$ | 32.19 $\pm$ 4.98 | 0.11 $\pm$ 0.07
J1122 | $\circ$ | 11.44 $\pm$ 5.24 | 0.00 $\pm$ 0.20
J1128 | $\circ$ | 26.65 $\pm$ 3.34 | 0.11 $\pm$ 0.06
J1129 | $\times$ | 15.49 $\pm$ 3.67 | -0.04 $\pm$ 0.11
J1133 | $\circ$ | 41.98 $\pm$ 8.48 | -0.09 $\pm$ 0.09
J1158 | $\circ$ | 218.07 $\pm$ 12.62 | -0.10 $\pm$ 0.03
J1159 | $\circ$ | 8.38 $\pm$ 2.68 | -0.43 $\pm$ 0.14
J1209 | $\times$ | 164.70 $\pm$ 6.13 | -0.08 $\pm$ 0.02
J1219 | $\circ$ | 22.67 | -0.11
J1235 | $\bullet$ | 29.19 $\pm$ 11.95 | -0.19 $\pm$ 0.18
J1240 | $\times$ | 21.31 | -0.13
J1244 | $\times$ | 264.26 $\pm$ 11.60 | 0.10 $\pm$ 0.02
J1246 | $\circ$ | 87.90 $\pm$ 4.44 | 0.10 $\pm$ 0.02
J1248 | $\circ$ | 108.88 $\pm$ 9.37 | 0.09 $\pm$ 0.04
J1249 | $\times$ | 26.51 $\pm$ 4.13 | 0.05 $\pm$ 0.07
J1255 | $\times$ | 53.52 | 0.24
J1257 | $\times$ | 46.72 $\pm$ 4.23 | -0.02 $\pm$ 0.04
J1300 | $\times$ | 6.14 | -0.18
J1301 | $\circ$ | 55.34 $\pm$ 13.69 | -0.13 $\pm$ 0.11
J1305 | $\bullet$ | 10.05 | -0.20
J1310 | $\circ$ | 37.72 $\pm$ 8.34 | -0.33 $\pm$ 0.10
J1314 | $\times$ | 178.42 $\pm$ 11.23 | -0.02 $\pm$ 0.03
J1319 | $\times$ | 78.26 $\pm$ 7.72 | -0.16 $\pm$ 0.04
J1326 | $\circ$ | 72.90 $\pm$ 5.34 | 0.02 $\pm$ 0.03
J1329 | $\times$ | 145.64 $\pm$ 15.73 | -0.01 $\pm$ 0.05
J1345 | $\times$ | 208.06 $\pm$ 11.61 | 0.04 $\pm$ 0.02
J1350 | $\times$ | 39.36 | 0.33
J1403 | $\circ$ | 27.32 $\pm$ 23.46 | -0.03 $\pm$ 0.37
J1410 | $\bullet$ | 24.62 $\pm$ 2.44 | 0.00 $\pm$ 0.04
J1440 | $\circ$ | 199.28 $\pm$ 11.96 | -0.03 $\pm$ 0.03
J1517 | $\bullet$ | 78.45 $\pm$ 4.58 | -0.09 $\pm$ 0.03
J1540 | $\times$ | 189.29 $\pm$ 12.16 | -0.06 $\pm$ 0.03
J1559 | $\times$ | 90.14 $\pm$ 7.19 | 0.02 $\pm$ 0.03
J1604 | $\times$ | 34.26 $\pm$ 7.46 | -0.09 $\pm$ 0.09
J1646 | $\circ$ | 26.62 $\pm$ 11.65 | -0.28 $\pm$ 0.19
J1648 | $\circ$ | 11.68 $\pm$ 2.90 | 0.02 $\pm$ 0.11
J1720 | $\circ$ | 28.40 | -0.05

††footnotetext: Upper limit inferred from $3\sigma$ background.

Figure 6: Same as Figure 3, but with different galaxy characteristics. Colors
indicate $f_{\rm esc}$. Triangles represent that the associated $f_{\rm esc}$
are upper limits. $\tau$ and $p$ are Kendall’s correlation coefficient and
$p$-value respectively. Figure 6: (continued) Same as Figure 3, but with
different galaxy characteristics. Colors indicate $f_{\rm esc}$. Triangles
represent that the associated $f_{\rm esc}$ are upper limits. $\tau$ and $p$
are Kendall’s correlation coefficient and $p$-value respectively.

## References

  * Alexandroff et al. (2015) Alexandroff, R. M., Heckman, T. M., Borthakur, S., Overzier, R., & Leitherer, C. 2015, ApJ, 810, 104, doi: 10.1088/0004-637X/810/2/104
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Baldwin et al. (1981) Baldwin, J. A., Phillips, M. M., & Terlevich, R. 1981, PASP, 93, 5, doi: 10.1086/130766
  * Bergvall et al. (2006) Bergvall, N., Zackrisson, E., Andersson, B. G., et al. 2006, A&A, 448, 513, doi: 10.1051/0004-6361:20053788
  * Borthakur et al. (2014) Borthakur, S., Heckman, T. M., Leitherer, C., & Overzier, R. A. 2014, Science, 346, 216, doi: 10.1126/science.1254214
  * Bouwens et al. (2016) Bouwens, R. J., Smit, R., Labbé, I., et al. 2016, ApJ, 831, 176, doi: 10.3847/0004-637X/831/2/176
  * Cen (2020) Cen, R. 2020, ApJ, 889, L22, doi: 10.3847/2041-8213/ab6560
  * Cen & Kimm (2015) Cen, R., & Kimm, T. 2015, ApJ, 801, L25, doi: 10.1088/2041-8205/801/2/L25
  * Chisholm et al. (2020) Chisholm, J., Prochaska, J. X., Schaerer, D., Gazagnes, S., & Henry, A. 2020, MNRAS, 498, 2554, doi: 10.1093/mnras/staa2470
  * Chisholm et al. (2019) Chisholm, J., Rigby, J. R., Bayliss, M., et al. 2019, ApJ, 882, 182, doi: 10.3847/1538-4357/ab3104
  * Chisholm et al. (2018) Chisholm, J., Gazagnes, S., Schaerer, D., et al. 2018, A&A, 616, 30, doi: 10.1051/0004-6361/201832758
  * Curran (2014) Curran, P. A. 2014, arXiv e-prints, arXiv:1411.3816.  https://arxiv.org/abs/1411.3816
  * Flury et al. (2021a) Flury, S., Jaskot, A. E., et al. 2021a, in prep.
  * Flury et al. (2021b) —. 2021b, in prep.
  * Gazagnes et al. (2020) Gazagnes, S., Chisholm, J., Schaerer, D., Verhamme, A., & Izotov, Y. 2020, A&A, 639, A85, doi: 10.1051/0004-6361/202038096
  * Gazagnes et al. (2018) Gazagnes, S., Chisholm, J., Schaerer, D., et al. 2018, A&A, 616, A29, doi: 10.1051/0004-6361/201832759
  * Harris et al. (2020) Harris, C. R., Jarrod Millman, K., van der Walt, S. J., et al. 2020, Nature, 357–362, doi: 10.1038/s41586-020-2649-2
  * Heckman et al. (2011) Heckman, T. M., Borthakur, S., Overzier, R., et al. 2011, ApJ, 730, 5, doi: 10.1088/0004-637X/730/1/5
  * Henry et al. (2018) Henry, A., Berg, D. A., Scarlata, C., Verhamme, A., & Erb, D. 2018, ApJ, 855, 96, doi: 10.3847/1538-4357/aab099
  * Hogarth et al. (2020) Hogarth, L., Amorín, R., Vílchez, J. M., et al. 2020, MNRAS, 494, 3541, doi: 10.1093/mnras/staa851
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Isobe et al. (1986) Isobe, T., Feigelson, E. D., & Nelson, P. I. 1986, ApJ, 306, 490, doi: 10.1086/164359
  * Izotov et al. (2016a) Izotov, Y. I., Orlitová, I., Schaerer, D., et al. 2016a, Nature, 529, 178, doi: 10.1038/nature16456
  * Izotov et al. (2016b) Izotov, Y. I., Schaerer, D., Thuan, T. X., et al. 2016b, MNRAS, 461, 3683, doi: 10.1093/mnras/stw1205
  * Izotov et al. (2018a) Izotov, Y. I., Schaerer, D., Worseck, G., et al. 2018a, MNRAS, 474, 4514, doi: 10.1093/mnras/stx3115
  * Izotov et al. (2018b) Izotov, Y. I., Worseck, G., Schaerer, D., et al. 2018b, MNRAS, 478, 4851, doi: 10.1093/mnras/sty1378
  * Jaskot et al. (2019) Jaskot, A. E., Dowd, T., Oey, M. S., Scarlata, C., & McKinney, J. 2019, ApJ, 885, 96, doi: 10.3847/1538-4357/ab3d3b
  * Katz et al. (2020) Katz, H., Ďurovčíková, D., Kimm, T., et al. 2020, MNRAS, 498, 164, doi: 10.1093/mnras/staa2355
  * Kroupa (2002) Kroupa, P. 2002, Science, 295, 82, doi: 10.1126/science.1067524
  * Leitherer et al. (2010) Leitherer, C., Ortiz Otálvaro, P. A., Bresolin, F., et al. 2010, ApJS, 189, 309, doi: 10.1088/0067-0049/189/2/309
  * Leitherer et al. (1999) Leitherer, C., Schaerer, D., Goldader, J. D., et al. 1999, ApJS, 123, 3, doi: 10.1086/313233
  * Marchi et al. (2018) Marchi, F., Pentericci, L., Guaita, L., et al. 2018, A&A, 614, A11, doi: 10.1051/0004-6361/201732133
  * Meynet et al. (1994) Meynet, G., Maeder, A., Schaller, G., Schaerer, D., & Charbonnel, C. 1994, A&AS, 103, 97
  * Nakajima et al. (2020) Nakajima, K., Ellis, R. S., Robertson, B. E., Tang, M., & Stark, D. P. 2020, ApJ, 889, 161, doi: 10.3847/1538-4357/ab6604
  * Pellegrini et al. (2012) Pellegrini, E. W., Oey, M. S., Winkler, P. F., et al. 2012, ApJ, 755, 40, doi: 10.1088/0004-637X/755/1/40
  * Privon et al. (2020) Privon, G. C., Ricci, C., Aalto, S., et al. 2020, ApJ, 893, 149, doi: 10.3847/1538-4357/ab8015
  * Ramambason et al. (2020) Ramambason, L., Schaerer, D., Stasińska, G., et al. 2020, A&A, 644, A21, doi: 10.1051/0004-6361/202038634
  * Reddy et al. (2016) Reddy, N. A., Steidel, C. C., Pettini, M., & Bogosavljević, M. 2016, ApJ, 828, 107, doi: 10.3847/0004-637X/828/2/107
  * Runnholm et al. (2020) Runnholm, A., Hayes, M., Melinder, J., et al. 2020, ApJ, 892, 48, doi: 10.3847/1538-4357/ab7a91
  * Saldana-Lopez et al. (2021) Saldana-Lopez, A., et al. 2021, in prep.
  * Steidel et al. (2018) Steidel, C. C., Bogosavljević, M., Shapley, A. E., et al. 2018, ApJ, 869, 123, doi: 10.3847/1538-4357/aaed28
  * Steidel et al. (2001) Steidel, C. C., Pettini, M., & Adelberger, K. L. 2001, ApJ, 546, 665, doi: 10.1086/318323
  * Strom et al. (2018) Strom, A. L., Steidel, C. C., Rudie, G. C., Trainor, R. F., & Pettini, M. 2018, ApJ, 868, 117, doi: 10.3847/1538-4357/aae1a5
  * Thomas et al. (2013) Thomas, D., Steele, O., Maraston, C., et al. 2013, MNRAS, 431, 1383, doi: 10.1093/mnras/stt261
  * Topping et al. (2020) Topping, M. W., Shapley, A. E., Reddy, N. A., et al. 2020, MNRAS, 4444, 4430, doi: 10.1093/mnras/staa1410
  * Veilleux & Osterbrock (1987) Veilleux, S., & Osterbrock, D. E. 1987, ApJS, 63, 295, doi: 10.1086/191166
  * Verhamme et al. (2015) Verhamme, A., Orlitová, I., Schaerer, D., & Hayes, M. 2015, A&A, 578, A7, doi: 10.1051/0004-6361/201423978
  * Verhamme et al. (2017) Verhamme, A., Orlitová, I., Schaerer, D., et al. 2017, A&A, 597, A13, doi: 10.1051/0004-6361/201629264
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2
  * Wang et al. (2019) Wang, B., Heckman, T. M., Leitherer, C., et al. 2019, ApJ, 885, 57, doi: 10.3847/1538-4357/ab418f
  * Worseck et al. (2016) Worseck, G., Prochaska, J. X., Hennawi, J. F., & McQuinn, M. 2016, ApJ, 825, 144, doi: 10.3847/0004-637X/825/2/144
  * Zackrisson et al. (2013) Zackrisson, E., Inoue, A. K., & Jensen, H. 2013, ApJ, 777, 39, doi: 10.1088/0004-637X/777/1/39
  * Zastrow et al. (2011) Zastrow, J., Oey, M. S., Veilleux, S., McDonald, M., & Martin, C. L. 2011, ApJ, 741, L17, doi: 10.1088/2041-8205/741/1/L17


# Antonym vs Synonym Distinction using InterlaCed Encoder NETworks (ICE-NET)

Muhammad Asif Ali,1 Yan Hu,1 Jianbin Qin,2 Di Wang1
1 King Abdullah University of Science and Technology, KSA
2 Shenzhen University, China
{muhammadasif.ali, yan.hu<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Antonyms vs synonyms distinction is a core challenge in lexico-semantic
analysis and automated lexical resource construction. These pairs share a
similar distributional context which makes it harder to distinguish them.
Leading research in this regard attempts to capture the properties of the
relation pairs, i.e., symmetry, transitivity, and trans-transitivity. However,
the inability of existing research to appropriately model the relation-
specific properties limits their end performance. In this paper, we propose
InterlaCed Encoder NETworks (i.e., ICE-NET) for antonym vs synonym
distinction, that aim to capture and model the relation-specific properties of
the antonyms and synonyms pairs in order to perform the classification task in
a performance-enhanced manner. Experimental evaluation using the benchmark
datasets shows that ICE-NET outperforms the existing research by a relative
score of upto 1.8% in F1-measure. We release the codes for ICE-NET at
https://github.com/asif6827/ICENET.

##  1 Introduction

Antonyms vs synonyms distinction is a core challenge in natural language
processing applications, including but not limited to: sentiment analysis,
machine translation, named entity typing etc. Synonyms are defined as
semantically related words, whereas antonyms are defined as semantically
opposite words. For example “disperse" and “scatter" are synonyms, while
“disperse" and “garner" are antonyms (Ono et al., 2015).

Existing research on the antonym-synonym distinction is primarily categorized
into pattern-based and embedding-based approaches. Pattern-based approaches
attempt to curate distinguishing lexico-syntactic patterns for the word pairs
(Schwartz et al., 2015; Nguyen et al., 2017). A major limitation of the
pattern-based approaches is the sparsity of the feature space. Despite using
massive data sets, the generalization attempts result in highly overlapping
and noisy features, which further deteriorate the model’s performance.

Embedding based methods rely on the distributional hypothesis, i.e., _“words
that occur in the same contexts tend to have similar meanings"_ (Harris,
1954). These methods use widely available embedding resources to
capture/compute the semantic relatedness of synonym and antonym pairs (Nguyen
et al., 2016; Etcheverry and Wonsever, 2019). Ali et al. (2019) proposed
Distiller that uses non-linear projections to project the embedding vectors in
task-specific dense sub-spaces.

The key challenge faced by existing embedding-based approaches is their
inability to correctly model the inherent relation-specific properties among
different relation pairs. These models mix different lexico-semantic relations
and perform poorly when applied to a specific task (Ali et al., 2019).
Existing approaches, moreover, model each relation pair independently, which
is not adequate for antonym and synonym relation pairs as these relation pairs
exhibit unique properties that may be exploited by modeling the relation pair
in correlation with other instances (discussed in detail in Section 4).

Keeping in view the above-mentioned challenges, in this paper, we propose
InterlaCed Encoder NETworks (ICE-NET) for antonym vs synonym distinction. ICE-
NET uses multiple different encoders to capture relation-specific properties
of antonym and synonym pairs from pre-trained embeddings in order to augment
the end-performance of the antonyms vs synonyms distinction task.
Specifically, it uses: (i) an encoder (ENC-1) to capture the symmetry of
synonyms; (ii) an encoder (ENC-2) to model the symmetry for antonyms; and
(iii) an encoder (ENC-3) to preserve the transitivity of the synonyms and
trans-transitivity of antonym and synonym relation pairs by employing
attentive graph convolutions. These relation-specific properties of antonym
and synonym relation pairs are illustrated in Figure 1(a) and explained in
Section 3.1.

We are the first to make an attempt to use attentive graph convolutions for
modeling the underlying characteristics of antonym and synonym relation pairs.
Note, this work is different from existing works using graph convolutional
networks for relational data, e.g., Schlichtkrull et al. (2018), as antonyms
and synonyms possess unique properties which makes them different from
relation pairs in the Knowledge Graphs (KG), e.g., FB15K, WN18 Bordes et al.
(2013).

ICE-NET is a shift from the existing instance-based modeling approaches to
graph-based framing which allows effective information sharing across multiple
instances at a time to perform the end classification in a performance-
enhanced fashion. ICE-NET can be used with any available pre-trained embedding
resources, which makes it more flexible than the existing approaches relying
on huge text corpora. We summarize the major contributions of this paper as
follows:

  * •

We propose ICE-NET, i.e., a combination of interlaced encoder networks to
refine relation-specific information from the pre-trained embeddings.

  * •

ICE-NET is the first to use attentive graph convolutions for antonym vs
synonym distinction that provide a provision to analyze/classify a word pair
in correlation with multiple neighboring pairs/words, rather than independent
instant-level modeling.

  * •

We demonstrate the effectiveness of the proposed model using benchmark data
sets. ICE-NET outperforms the existing models by a margin of upto 1.8% in
terms of F1-measure.

##  2 Related Work

Earlier research on antonym synonym distinction attempts at capturing lexico-
syntactic patterns between the word pairs co-occurring within the same
sentence.

Lin et al. (2003) considered pharasal patterns: _“from X to Y"_ , and _“either
X or Y"_ to identify synonyms amongst distributionally similar words. Baroni
and Bisi (2004) used co-occurrence statistics to discover synonyms and
distinguish them from unrelated terms. Van der Plas and Tiedemann (2006) used
word alignment measures using parallel corpora from multiple different
languages to capture synonyms. Lobanova et al. (2010) used a set of seed pairs
to capture patterns in the data and later used these patterns to extract new
antonym pairs from text corpora. Roth and Im Walde (2014) proposed discourse
markers as features alternate to the lexico-syntactic patterns. Schwartz et
al. (2015) proposed automated routines to acquire a set of symmetric patterns
for word similarity prediction. Nguyen et al. (2017) proposed AntSynNET that
uses a set of lexico-syntactic patterns between the word pairs within the same
sentence captured over huge text corpora.

In the recent past, embedding models have received considerable research
attention for antonyms vs synonyms distinction. These models are based on
distributional hypotheses, i.e., words with similar meanings co-occur in a
similar context (Goldberg and Levy, 2014; Pennington et al., 2014; Grave et
al., 2018). A major advantage offered by the embedding-based approaches is the
freedom to curate and train embedding vectors for features extracted from text
corpora. Adel and Schütze (2014) used skip-gram modeling to train embedding
vectors using coreference chains. Nguyen et al. (2016) used lexical contrast
information in the skip-gram model for antonym and synonym distinction. Ono et
al. (2015) uses dictionaries along with distributional information to detect
probable antonyms. Ali et al. (2019) used a set of encoder functions to
project the word embeddings in constrained sub-spaces in order to capture the
relation-specific properties of the data. Xie and Zeng (2021) employed a
mixture-of-experts framework based on a divide-and-conquer strategy. They used
a number of localized experts focused on different subspaces and a gating
mechanism to formulate the expert mixture.

We observe some of the limitations of the existing work as follows. The
pattern-based approaches are limited owing to the noisy and overlapping nature
of the patterns. The embedding models are limited by the challenges posed by
the distributional nature of the word embeddings, e.g., in Glove embeddings
top similar words for the word _“small"_ yields a combination of synonyms,
antonyms, and irrelevant words Ali et al. (2019).

##  3 Background

###  3.1 Preliminaries

Antonyms and synonyms are a special kind of relation pairs (denoted by $r_{A}$
and $r_{S}$) with unique properties, i.e., (a) antonyms possess symmetry, (b)
synonyms exhibit symmetry and transitivity, (c) antonyms and synonyms when
analyzed in combination demonstrate trans-transitivity.

These properties are depicted in Figure 1 (a). For ease of interpretation, we
use $(h,r,t)$ to represent a relation tuple, where $h$ corresponds to the
“head" and $t$ is the “tail" of relation $r$. For word pair $(h,t)$ and
relation $r$, symmetry implies $(h,r,t)$ iff $(t,r,h)$. The transitivity
between the relation implies: _if $(h,r,t)$_ and _$(t,r,t^{{}^{\prime}})$_
hold then _$(h,r,t^{{}^{\prime}})$_ also holds, as shown by the words “nasty"
and “horrible" in Figure 1(a). Trans-transitivity implies: _if $(h,r_{A},t)$_
and _$(t,r_{S},t^{{}^{\prime}})$_ hold then  _$(h,r_{A},t^{{}^{\prime}})$_
also holds, also illustrated between the words “nasty" and “pleasing".

###  3.2 KG Embeddings Methods

Ali et al. (2019) pointed out a key limitation of the translational embedding
methods (commonly used for KG embeddings) in modeling symmetric relations. For
instance, for a symmetric relation $\rr$, it is not possible for translational
embeddings to preserve both vector operations: $\hh+\rr=\tt$ and $\tt+\rr=\hh$
at the same time. This is also illustrated in Figure 1(b), where we show
$\tt{}^{{}^{\prime}}\neq\tt$. For details refer to the original article by Ali
et al. (2019). Likewise, some of the key difference of our work from existing
work, i.e., R-GCN by Schlichtkrull et al. (2018) are explained in Appendix
A.2.

Figure 1: (a) Properties of the antonym and synonym relation pairs, i.e.,
symmetry, transitivity, and trans-transitivity; (b) Limitation of
translational embeddings in capturing the antonym and synonym relations Ali et
al. (2019).

##  4 Proposed Approach

Given that existing KG embeddings are not able to model the relation-specific
properties of the antonym and synonym pairs, we propose ICE-NET that takes
pre-trained word embeddings as inputs and projects them to low-dimensional
space. In order to ensure that low-dimensional space captures the relation-
specific properties of the data to the best possible extent ICE-NET uses three
different encoder networks. We call overall architecture as interlaced
structure, because these networks are interconnected, i.e., (a) loss function
of ENC-2 also depends upon ENC-1, (b) output of encoders (ENC-1, and ENC-2) is
used as input to the ENC-3. Details about each encoder are as follows:

###  4.1 ENC-1

The goal of this encoder is to capture the symmetry of the synonym relation
pairs. For this, we use a two-layered feed-forward function: $f_{1}(X)=\sigma
W_{12}*\sigma(W_{11}*X+b_{11})+b_{12}$ to project d-dimensional embeddings
($X\in\mathbf{R}^{d}$) to p-dimensions ($\mathbf{R}^{p}$). Here $W_{11}$ and
$W_{12}$ are the weight matrices; $b_{11}$ and $b_{12}$ are the bias terms.
For encoded word pairs to preserve symmetry among relation pairs, we employ
negative sampling techniques. Specifically, we use a margin-based loss (shown
in Equation 1) to project a word close to its true synonyms, while at the same
time push it from irrelevant words. This formulation preserves the symmetry of
the relation pair owing to the commutative nature of the inner product. It is
also justified by the fact: if $\xx_{h}$ is embedded close to $\xx_{t}$, then
$\xx_{t}$ is also embedded close to $\xx_{h}$.

$\displaystyle\text{L}_{1}$ $\displaystyle=\sum_{(h,t)\in
T_{1}}\max(0,\gamma_{1}-\tanh(\langle f_{1}(\xx_{h}),f_{1}(\xx_{t})\rangle))$
(1) $\displaystyle+\sum_{(h^{\prime},t^{\prime})\in
T^{\prime}_{1}}\max(0,\gamma_{1}+\tanh(\langle
f_{1}(\xx_{h}^{{}^{\prime}}),f_{1}(\xx_{t}^{{}^{\prime}})\rangle))$

Here $\gamma_{1}$ is the margin; $T_{1}$ corresponds to the synonym pairs;
$\xx_{h}$, $\xx_{t}$ are the embedding vectors for head and tail words.
$T^{\prime}_{1}$ is acquired by randomly replacing one of the words from the
pairs in $T_{1}$ and/or using antonyms as negative samples.

###  4.2 ENC-2

This encoder aims to capture the symmetry for the antonym relation pairs. For
this we use a two layered feed-forward function: $f_{2}(X)=\sigma
W_{22}*\sigma(W_{21}*X+b_{21})+b_{22}$ to project d-dimensional embeddings
($X\in\mathbf{R}^{d}$) to p-dimensions ($\mathbf{R}^{p}$). Here
$X\in\mathbf{R}^{d}$ corresponds to the pre-trained word embeddings; $W_{21}$
and $W_{22}$ are the weight matrices; $b_{21}$ and $b_{22}$ are the bias
terms. In order to preserve the symmetry of the antonym relations, we use
another margin-based loss function (shown in Equation 2) to project a word
close to its true antonyms, while at the same time push it from irrelevant
words.

$\displaystyle\text{L}_{2}$ $\displaystyle=\sum_{(h,t)\in
T_{2}}\max(0,\gamma_{2}-\tanh(\langle f_{2}(\xx_{h}),f_{1}(\xx_{t})\rangle))$
(2) $\displaystyle+\sum_{(h^{\prime},t^{\prime})\in
T^{\prime}_{2}}\max(0,\gamma_{2}+\tanh(\langle
f_{2}(\xx_{h}^{{}^{\prime}}),f_{1}(\xx_{t}^{{}^{\prime}})\rangle))$

Note, for $\text{L}_{2}$ we use both functions, i.e., $f_{1}(X),f_{2}(X)$,
that allows us to project $\xx_{h}$ close to its antonym $\xx_{t}$ as well as
synonyms of $\xx_{t}$. Here again the symmetry of the relation is preserved by
the commutative nature of the inner product. $\gamma_{2}$ is the margin term,
$T_{2}$ corresponds to the antonym pairs; $\xx_{h}$, $\xx_{t}$ are the
embedding vectors for head and tail words. $T^{\prime}_{2}$ is acquired by
randomly replacing one of the words from the pairs in $T_{2}$ and/or using
synonyms as negative samples.

Given that the encoders (ENC-1, ENC-2) use two different non-linear functions
to project the pre-trained embeddings, it allows us to learn two projections
for each word. Later, we use all possible projection scores as indicators for
the word pair to be probable antonym and/or synonym pair. This setting is
different from the previous research that embeds synonyms close to each other,
while antonyms are projected at an angle of 180∘ Ono et al. (2015) as it is
hard to preserve the relation-specific properties for the resultant
embeddings.

###  4.3 ENC-3

Finally, in order to preserve the transitivity of the synonym pairs and the
trans-transitivity of antonym and synonym relation pairs in combination we
propose an attentive graph convolutional encoder under transductive setting.
We exploit the fact that the a word may be represented as a node in the graph,
and each word may be surrounded by an arbitrary number of semantically related
words as neighbouring nodes in the graph. We argue that this setting is more
flexible in capturing the relation-specific properties involving arbitrary
number of words, as it allows modeling the relation pairs in complete
correlation with each other, which is more practical than modeling these pairs
independent of each other. It also provides the provision for effective
information sharing across the neighboring nodes using attention weights.
Similar ideas has already been applied to capture the semantic-relatedness for
embeddings trained for different languages Ali et al. (2023a, b).

In our case, we use two different graphs, namely: $\text{G}_{h}$, and
$\text{G}_{t}$, for preserving the relations amongst the head and tail words
respectively. We outline the graph construction process in Algorithm 1. It is
explained as follows:

Algorithm 1 Graph Construction

Inputs: Embedding; $D=D_{tr}+D_{dev}+D_{test}$ Outputs: Graphs:
$\text{G}_{h},\text{G}_{t}$

1:$\\{\text{Syn}_{h},\text{Ant}_{h}\\}_{h=1}^{V}$ $\leftarrow$ $\emptyset$;
$\text{G}_{t}\leftarrow\emptyset$

2:$\\{\text{Syn}_{t},\text{Ant}_{t}\\}_{t=1}^{V}$ $\leftarrow$ $\emptyset$;
$\text{G}_{h}\leftarrow\emptyset$

3:Train $\text{M}_{init}(D_{tr};\text{L}_{1},\text{L}_{2})$

4:for $\text{inst}(h,t)\leftarrow 1$ to $D$ do

5: $y^{*}$ = $\text{score}(\text{M}_{init},\text{inst})$

6: if $y^{*}\geq\text{ANT}_{thr}$ then

7: $\text{Update}\\{\text{Ant}_{h};\text{Ant}_{t}$}

8: else if $y^{*}\leq\text{SYN}_{thr}$ then

9: $\text{Update}\\{\text{Syn}_{h};\text{Syn}_{t}$}

10: end if

11:end for

12:for $pair\in\\{\text{Syn}_{t},\text{Ant}_{t}\\}$ do

13: $\text{G}_{h}$ $\leftarrow$ $\text{G}_{h}\cup\\{edge_{h}(pair)$}

14:end for

15:for $pair\in\\{\text{Syn}_{h},\text{Ant}_{h}\\}$ do

16: $\text{G}_{t}$ $\leftarrow$ $\text{G}_{t}\cup\\{edge_{t}(pair)$}

17:end for

18:return $\text{G}_{h}$; $\text{G}_{t}$

#### Graph Construction.

The graph construction process uses data set $D$ and 300-d pre-trained
Fasttext embeddings Grave et al. (2018) as inputs and returns two graphs
$\text{G}_{h}$ and $\text{G}_{t}$ as output. The details are as follows.

Firstly, we initialize dictionaries {$\text{Syn}_{h}$, $\text{Ant}_{h}$} and
{$\text{Syn}_{t}$, $\text{Syn}_{t}$} to store probable synonym and antonym
pairs with head word $h$ and tail word $t$ respectively (lines 1-2). We train
a basic model ($\text{M}_{init}$) using the encoders (ENC-1 and ENC-2) and
available training data (line 3) in an end-to-end fashion. Later,
$\text{M}_{init}$ is used to assign a score ($y^{*}$) to each pair in the data
$D$ (line 6). We use $y^{*}$ compared against the thresholds
{$\text{ANT}_{thr}$,$\text{SYN}_{thr}$,} to update the data structures
{$\text{Ant}_{h}$, $\text{Ant}_{t}$}, and {$\text{Syn}_{h}$, $\text{Syn}_{t}$}
respectively (lines 6-9). The core logic is: we add $\text{inst}(h,t)$ to
$\text{Ant}_{h}$, if (a) head word ($h$) corresponds to a key in
$\text{Ant}_{h}$, (b) it is a probable antonym pair with
($y^{*}\geq\text{ANT}_{thr}$). Later, we use the information in the
dictionaries to construct the graphs (lines 12-16).

We explain the construction of $\text{G}_{h}$ using the information in
$\text{Syn}_{t}$, $\text{Ant}_{t}$, as follows. Given that $\text{Syn}_{t}$
contains the information about the list of probable synonym pairs with the
tail word _“t"_. In order to preserve the transitivity for the synonym pairs
with tail _“t"_ , we formulate pairwise edges between the head terms in
$\text{Syn}_{t}$. It is based on the assumption that head words of the
relation pairs with the same tail, i.e., _“t"_ are likely to be synonyms of
each other. Likewise, $\text{Ant}_{t}$ contains the information about the list
of probable antonym pairs with the tail word _“t"_. In order to preserve the
trans-transitivity of relation pairs with tail _“t"_ , we formulate pairwise
edges between the head terms in $\text{Ant}_{t}$. It is based on the
assumption that the head words of the antonym relation pairs with same tail
_“t"_ are likely to be synonyms of each other. Eventually, we combine these
edges to formulate the graph $\text{G}_{h}$.

We follow a similar procedure to construct the graph $\text{G}_{t}$ using
information in $\text{Syn}_{h}$, $\text{Ant}_{h}$. Finally, we return graphs
$\text{G}_{h}$ and $\text{G}_{t}$ as the output of the graph construction
process.

#### Attentive Aggregation.

The graph construction process surrounds each word in the graphs $G_{h}$ and
$G_{t}$ by a set of probable synonyms. Later, it recomputes the representation
of each word as an attentive aggregation of the neighbors. For this, it uses
the following layer-wise information propagation mechanism:

$\vspace{-0.7ex}L^{(i+1)}=\rho(\tilde{\xi_{G}}L^{(i)}W_{i})$ (3)

where $\tilde{\xi_{G}}=\bar{D}^{-1/2}(\xi_{G}+I)\bar{D}^{-1/2}$ is the
normalized symmetric marix, $\bar{D}$ is the degree matrix of $\xi_{G}$,
$\xi_{G}$ is the weighted adjacency matrix containing attention weights for
$G$, $L^{(i)}$ is the input representation from the previous layer, $W_{i}$ is
the learn-able weight matrix. We also add identity matrix $I$ to $\xi_{G}$ in
order to allow self-connections for each word in the graphs. It allows the
encoder to analyze each word as a weighted combination of itself and its
semantic neighbors. Our formulation for attentive graph convolutions is
inspired by Ali et al. (2020), and its non-euclidean variant Ali et al.
(2021). Intuitive explanations in this regard are provided in Appendix A.1.

For ICE-NET, we use a two-layered attentive graph convolution encoder with
ReLU non-linearity to generate the final representations of each word.
Specifically, for the relation tuples $(h,r,t)$ in data $D$, the output of the
encoders (ENC-1 and ENC-2) is separately processed by the attentive graph
convolution networks to generate the final representations, as follows:

$\displaystyle\XX_{hh}=\tilde{\xi_{G_{h}}}(\text{ReLU}(\tilde{\xi_{G_{h}}}\;f_{1}(X_{h})W_{hh_{1}})W_{hh_{2}}$
(4)
$\displaystyle\XX_{ht}=\tilde{\xi_{G_{t}}}(\text{ReLU}(\tilde{\xi_{G_{t}}}\;f_{1}(X_{t})W_{ht_{1}})W_{ht_{2}}$
$\displaystyle\XX_{th}=\tilde{\xi_{G_{h}}}(\text{ReLU}(\tilde{\xi_{G_{h}}}\;f_{2}(X_{h})W_{th_{1}})W_{th_{2}}$
$\displaystyle\XX_{tt}=\tilde{\xi_{G_{t}}}(\text{ReLU}(\tilde{\xi_{G_{t}}}\;f_{2}(X_{t})W_{tt_{1}})W_{tt_{2}}$

Here $f_{1}(X),f_{2}(X)\in\mathbf{R}^{p}$ are the outputs of the encoders
(ENC-1, and ENC-2) used as inputs for ENC-3. $W_{i}$ are learn-able weights,
$\XX_{i}\in\mathbf{R}^{q}$ are the outputs of attentive graph convolution. In
order to train the attentive graph convolution network (ENC-3), we compute the
score vectors {$\xx_{1},\xx_{2}$} and {$\xx_{3},\xx_{4}$} as indicative of
synonymy and antonymy respectively.

$\displaystyle\xx_{1}=\cos{(\XX_{th},\XX_{tt})};$
$\displaystyle\;\xx_{2}=\cos{(\XX_{hh},\XX_{ht})}$ (5)
$\displaystyle\xx_{3}=\cos{(\XX_{hh},\XX_{tt})};$
$\displaystyle\;\xx_{4}=\cos{(\XX_{ht},\XX_{th})}$

where $\cos(\XX,\YY)$ is the element-wise cosine of the vector pairs in $\XX$
and $\YY$. We concatenate these scores to get the feature matrix:
$X_{\text{F}}$ = [$\xx_{1};\xx_{2};\xx_{3};\xx_{4}$], and use cross-entropy
loss to train the encoder, shown in Equation 6:

$\text{L}_{3}$ = $-\frac{1}{N}$ $\sum_{i=1}^{N}\log({p}(y_{i}|h_{i},t_{i}))$

(6)

where ${p}(\yy\mid\xx_{h},\xx_{t})=softmax(\ww\xx_{F}+{\bb})$ with
${\hat{y}}=\operatorname*{argmax}_{y}{p}(\yy|\xx_{h},\xx_{t})$, $\ww$ is the
weight matrix and $\bb$ is the bias term.

###  4.4 The Complete Model.

Finally, we combine the loss functions of the individual encoder networks,
i.e., $\text{L}_{1}+\text{L}_{2}+\text{L}_{3}$ as the loss function of ICE-
NET. We train the model in an end-to-end fashion.

##  5 Experiments and Results

###  5.1 Datasets

We evaluate the proposed approach on two different data sets: (a) A benchmark
data set by Nguyen et al. (2017) manually curated from WordNet and
Wordnik111http://www.wordnik.com. It encompasses randomly split synonyms and
antonym pairs corresponding to three word classes (adjective, noun, and verb).
(b) A lexical split curated by Xie and Zeng (2021). For both data sets, the
ratio between the antonyms and synonym pairs within each word class is
approximately 1:1. The statistics of each data set are shown in Table 1.

word class | (a) Random | (b) Lexical
---|---|---
train | dev | test | train | dev | test
Adjective | 5562 | 398 | 1986 | 4227 | 303 | 1498
Noun | 2836 | 206 | 1020 | 2667 | 191 | 954
Verb | 2534 | 182 | 908 | 2034 | 146 | 712

Table 1: Antonym/Synonym distinction datasets

###  5.2 Experimental Settings

Similar to the baseline methods, for main experimentation we report the
results using random split and 300-d Fasttext embeddings (Grave et al., 2018)
trained on wiki-news corpus. Results using the dLCE embeddings and lexical
split of the data are discussed in Section 6. The embedding vectors for the
OOV tokens are randomly initialized. For model training, we use Adam optimizer
Kingma and Ba (2014) with learning rate=0.001. The values for
$\text{SYN}_{thr}$ and $\text{ANT}_{thr}$ are set to 0.15 and 0.10
respectively. For $\text{L}_{1}$ and $\text{L}_{2}$ the values for the margin
terms are: $\gamma_{1}=\gamma_{2}=0.9$. Output dimensionality of ENC-1 and
ENC-2 is 80d and for ENC-3 is 60d. We used TensorFlow toolkit (version 2.12)
to run the experiments. We report mean and standard deviation of the scores
computed over five runs of the experiments. All experiments were performed
using Intel Core-i9-10900X CPU, and Nvidia 3090Ti GPU. On this GPU, a single
run of the experiments takes approximately thirty minutes.

Methodology | Adjective | Verb | Noun
---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
Baseline-1 (Random vectors) | 0.657 | 0.665 | 0.661 | 0.782 | 0.819 | 0.800 | 0.783 | 0.751 | 0.767
Baseline-2 (w/o Graph conv.) | 0.828 | 0.909 | 0.867 | 0.837 | 0.915 | 0.879 | 0.818 | 0.818 | 0.818
AntSynNet Nguyen et al. (2017) | 0.750 | 0.798 | 0.773 | 0.717 | 0.826 | 0.768 | 0.807 | 0.827 | 0.817
Parasiam Etcheverry and Wonsever (2019) | 0.855 | 0.857 | 0.856 | 0.864 | 0.921 | 0.891 | 0.837 | 0.859 | 0.848
Distiller Ali et al. (2019) | 0.854 | 0.917 | 0.884 | 0.871 | 0.912 | 0.891 | 0.823 | 0.866 | 0.844
MoE-ASD Xie and Zeng (2021) | 0.878 | 0.907 | 0.892 | 0.895 | 0.920 | 0.908 | 0.841 | 0.900 | 0.869
ICE-NET |  0.896$\pm$ 0.0005 |  0.919$\pm$ 0.0005 |  0.908$\pm$ 0.0005 |  0.899$\pm$ 0.001 |  0.932$\pm$ 0.001 |  0.915$\pm$ 0.001 |  0.895$\pm$ 0.001 | 0.871$\pm$ 0.001 |  0.883$\pm$ 0.001

Table 2: ICE-NET performance comparison using random split

###  5.3 Baseline Models

In order to test the effectiveness of ICE-NET, we design two baseline models.
Baseline-1 aims to analyze the ability of ICE-NET to encode the information in
the pre-trained embeddings. For this, we use random vectors in place of pre-
trained embeddings. Baseline-2 aims to analyze the ability of graph
convolutions to preserve relation-specific properties. For this, we use a
basic variant of ICE-NET relying only on the ENC-1 and ENC-2.

We also compare ICE-NET with existing state-of-the-art research on the
antonym-synonym distinction task, i.e., (i) AntSynNET by Nguyen et al. (2017),
(ii) Parasiam by Etcheverry and Wonsever (2019), (iii) Distiller by Ali et al.
(2019), and (iv) MoE-ASD by Xie and Zeng (2021). For all these models, we
report the scores reported in the original papers, as they are computed using
the same data settings as that of ours.

###  5.4 Main Results

The performance comparison of ICE-NET is reported in Table 2. For these
results, we use the random split of the data and 300-d Fasttext embeddings. We
boldface overall best scores with previous state-of-the-art underlined. A low
variance of the results shows that ICE-NET yields a stable performance across
multiple runs.

Comparing the performance of ICE-NET against the previous state-of-the-art, we
observe, for the adjective data sets, the ICE-NET outperforms existing best by
2.1%, 0.2% and 1.8% for precision, recall, and F1 scores respectively. For the
verbs data set, it outweighs the precision, recall and F1 score by 0.45%,
1.19%, and 0.77% respectively. For the nouns data set the improvement in
performance for the precision and F1-scores is 6.42%, and 1.61%.

Analyzing the performance of ICE-NET against the baseline models, a
significant decline in the performance for the baseline-1 shows that pre-
trained embeddings carry a significant amount of relation-specific information
which is refined by ICE-NET in a performance-enhanced fashion. Likewise, the
performance comparison against the baseline-2 shows that attentive graph
convolutions help the ICE-NET in capturing probable relation pairs by using
the relation-specific properties, i.e., symmetry, transitivity, and trans-
transitivity to the best possible extent, which in turn boosts the end
performance of the model.

These results show the impact of using attentive graph convolutions for the
distinction task. It affirms our hypothesis that graph convolutions offer an
optimal setting to model the relation-specific data because it provides the
provision for information sharing across semantically related words, rather
than modeling data instances completely independently of each other.

##  6 Analyses

In this section, we perform a detailed analyses of the ICE-NET under different
settings, namely: (i) dLCE embeddings Nguyen et al. (2016), (ii) Lexical
split, (iii) Ablation analysis, and (iv) Error analyses.

Methodology | Adjective | Verb | Noun
---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
AntSynNet Nguyen et al. (2017) | 0.763 | 0.807 | 0.784 | 0.743 | 0.815 | 0.777 | 0.816 | 0.898 | 0.855
Parasiam Etcheverry and Wonsever (2019) | 0.874 | 0.950 | 0.910 | 0.837 | 0.953 | 0.891 | 0.847 | 0.939 | 0.891
Distiller Ali et al. (2019) | 0.912 | 0.944 | 0.928 | 0.899 | 0.944 | 0.921 | 0.905 | 0.918 | 0.911
MoE-ASD Xie and Zeng (2021) | 0.935 | 0.941 | 0.938 | 0.914 | 0.944 | 0.929 | 0.920 | 0.950 | 0.935
ICE-NET |  0.936$\pm$0.0002 | 0.945$\pm$0.0002 |  0.940$\pm$0.0002 | 0.913$\pm$0.001 |  0.953$\pm$0.001 |  0.933$\pm$0.001 |  0.925$\pm$0.001 |  0.953$\pm$0.001 |  0.939$\pm$0.001

Table 3: ICE-NET performance comparison using random split and dLCE Embeddings

###  6.1 dLCE Embeddings

Results for ICE-NET using random split and dLCE embeddings are shown in Table
3. We also report the scores for the previous research using the same test
settings (i.e., data split and embeddings). These results show ICE-NET
outperforms the existing research yielding a higher value of F1-score across
all three data categories (adjective, verb and noun). These results compared
to the results in Table 2 (using fasttext embeddings) show that dLCE
embeddings being trained on lexical contrast information carry more
distinctive information for the distinction task compared to generalized pre-
trained word embeddings.

###  6.2 Lexical Split

In this subsection, we analyze the results of ICE-NET corresponding to the
lexical split of the antonym synonym distinction task Xie and Zeng (2021).
Note, the lexical split assumes no overlap across train, dev, and test splits
in order to avoid lexical memorization Shwartz et al. (2016). Generally, the
lexical split is considered a much tough evaluation setting compared to the
random split, as it doesn’t allow information sharing across different data
splits based on overlapping vocabulary.

For the lexical split, the results for both dLCE embeddings and Fasttext
embeddings are shown in Table 5. Comparing the performance of our model
against existing research, it is evident for both embeddings, i.e., Fasttext
and dLCE, ICE-NET yields a higher F1 measure compared to the existing models.

Adjacency | Adjective | Verb | Noun
---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
ICE-NET (A1 = Random) | 0.862 | 0.863 | 0.863 | 0.799 | 0.894 | 0.844 | 0.816 | 0.863 | 0.839
ICE-NET (A2 = Identity) | 0.849 | 0.886 | 0.867 | 0.761 | 0.896 | 0.823 | 0.830 | 0.861 | 0.845
ICE-NET (A3 = Fasttext) | 0.880 | 0.873 | 0.877 | 0.867 | 0.930 | 0.897 | 0.851 | 0.873 | 0.862
ICE-NET (A4 = $\text{M}_{init}$) | 0.881 | 0.909 | 0.895 | 0.899 | 0.925 | 0.912 | 0.874 | 0.867 | 0.870
ICE-NET (A5 = Weighted-$\text{M}_{init}$) |  0.896$\pm$0.0005 |  0.919$\pm$0.0005 |  0.908$\pm$0.0005 | 0.898$\pm$0.001 |  0.932$\pm$0.001 |  0.915$\pm$0.001 |  0.895$\pm$0.001 | 0.871$\pm$0.001 |  0.883$\pm$0.001

Table 4: ICE-NET performance comparison using different adjacency matrices and
random data split

###  6.3 Ablation Analyses

The core focus of ICE-NET is to employ attentive graph convolutions in order
to capture the relation-specific properties of antonym and synonym pairs in
order to perform the distinction task in a robust way. In order to simplify
things, we deliberately don’t include any hand-crafted features, e.g.,
negation prefixes etc., as a part of ICE-NET.

For the ablation analyses of ICE-NET, we: (a) compare the performance of ICE-
NET with and without attentive graph convolutions, (b) analyze the impact of
different attention weights.

#### (a) Impact of attentive convolutions.

In order to analyze the impact of attentive graph convolutions, we train a
variant of ICE-NET encompassing only the encoder networks. Note, we also used
a similar model in Section 5.4 (shown as baseline-2 in Table 2), however, the
end goal of this analysis is to dig out a few example pairs that benefited
especially from the attentive graph convolutions.

Some of the synonym and antonym word pairs that were corrected by attentive
convolutions include: {(lecture, reprimand), (single,retire)} and
{(tender,demand), (file, rank)} respectively. These word pairs were not easy
to categorize otherwise by the variant of ICE-NET without graph convolutions.
This shows the significance of the attentive convolutions in acquiring
relation-specific information from semantically related neighbors that was
helpful to reinforce the classification decision.

Embedding | Model | Adjective | Verb | Noun
---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
FastText | Parasiam Etcheverry and Wonsever (2019) | 0.694 | 0.866 | 0.769 | 0.642 | 0.824 | 0.719 | 0.740 | 0.759 | 0.748
MoE-ASD Xie and Zeng (2021) | 0.808 | 0.810 | 0.809 | 0.830 | 0.693 | 0.753 | 0.846 | 0.722 | 0.776
ICE-NET | 0.760$\pm$0.0005 |  0.870$\pm$0.0005 |  0.815$\pm$0.0005 | 0.740$\pm$0.001 | 0.777$\pm$0.001 |  0.758$\pm$0.001 | 0.763$\pm$0.002 |  0.826$\pm$0.002 |  0.793$\pm$0.002
dLCE | Parasiam Etcheverry and Wonsever (2019) | 0.768 | 0.952 | 0.850 | 0.769 | 0.877 | 0.819 | 0.843 | 0.914 | 0.876
MoE-ASD Xie and Zeng (2021) | 0.877 | 0.908 | 0.892 | 0.860 | 0.835 | 0.847 | 0.912 | 0.869 | 0.890
ICE-NET | 0.835$\pm$0.0004 |  0.971$\pm$0.0004 |  0.898$\pm$0.0004 | 0.793$\pm$0.002 |  0.938$\pm$0.002 |  0.859$\pm$0.002 | 0.886$\pm$0.001 |  0.915$\pm$0.001 |  0.900$\pm$0.001

Table 5: Antonym/Synonym distinction performance for the lexical split

#### (b) Varying attention weights.

We also analyze the impact of different attention weights on the end
performance of the model. Corresponding results are shown in Table 4. For
these experiments, we use five different types of attention weights, yielding
adjacency matrices: A1, A2, A3, A4, and A5 in Table 4. We use hard attention
weights that are not fine-tuned during the model training. The graphs
($\text{G}_{h}$ and $\text{G}_{t}$) used in these experiments correspond to
the best performing variant of ICE-NET.

For A1, we use random values as attention weights, i.e., we randomly assign a
value to each word pair from the range (0.1 $\sim$ 0.9). For A2, we use the
identity as the adjacency matrix for the word pairs in the graphs, i.e., we
completely ignore the effect of graph convolutions. For A3, we use the
embedding similarity scores of the fasttext embeddings as the attention
scores. This setting is based on the distributional hypothesis, i.e.,
distributionally similar words get higher scores. For A4, we use the embedding
similarity scores from the output of ENC-1 network for the model
$\text{M}_{init}$, trained entirely using two encoder networks. The motivation
for using these scores as attention weights is the fact that ENC-1 is
responsible for capturing the synonym pairs, so it will assign a higher score
to probable synonyms, and a relatively lower score to probable antonyms.

For A5, we use attention weights similar to the setting of A4 with the
difference that we down-scale the weights for probably erroneous edges in the
graph. For less confident relation pairs with scores closer to the thresholds,
i.e., $\text{ANT}_{thr}$, $\text{SYN}_{thr}$, we simply downscale the
attention weight by half. This setting in turn limits the error propagation in
the end-model caused by the erroneous edges in the graphs.

Results in Table 4 show that ICE-NET (A5), outperforms other variants of
attention weights. A similar performance is observed by the model ICE-NET
(A4). Relatively lower scores for the models using the random values and
identity matrices as attention weights show the significance of sharing
information amongst semantically related neighbors in an appropriate
proportion in order to perform the end task in a performance-enhanced way.
Likewise, the score for ICE-NET (A3) show that by default the distributional
scores of the pre-trained embeddings are not suitable for the end task. These
analyses clearly indicate that the choice of attention weight plays a vital
role in capturing the properties of the data.

###  6.4 Error Analyses

For the variant of ICE-NET using random split and Fasttext embeddings, we
collect a sample of approximately fifty error cases for each word class
(adjectives, verbs, and nouns) to analyze the most probable reasons for the
errors. We broadly categorize the errors into the following different
categories: (a) the inability of input embeddings to cater to multiple senses,
(b) the distributional embeddings for out-of-vocabulary (OOV) and/or rare
words, and (c) other cases, e.g., negation prefix, errors with unknown reasons
etc.

We separately report the number of erroneous edges/neighbours in the graphs:
$\text{G}_{h}$ and $\text{G}_{t}$. Information propagation over these
erroneous edges may also lead to the classification errors, however, it is
hard to quantify such errors.

For adjectives, almost 25% errors correspond to the sense category, 20% errors
are caused by rare words and/or OOV tokens, and the rest errors are attributed
to negation prefixes and unknown reasons. For nouns, 30% errors belong to the
sense category, 12.5% errors result due to rare words and/or OOV tokens, with
the rest of the errors assigned to the negation prefixes and unknown reasons.
For verbs, 13% errors correspond to the sense category, 15% errors are caused
by rare words/OOV tokens and the rest of the errors may be attributed to
negation prefixes and unexplained reasons. Regarding erroneous neighborhoods
in the graphs, almost 11%, 12% and 5% neighbors of the graphs for adjectives,
nouns and verbs respectively are erroneous, which deteriorate the end-
performance of ICE-NET by error propagation through attentive convolutions.

Considering the impact of different error categories on the end performance of
ICE-NET. For multi-sense tokens the distributional embedding vectors are
primarily oriented in the direction of the most prevalent sense of the
underlying training corpora, which may be different from the sense in the word
pair resulting in misclassifications. For example, “clean" and “blue" are two
synonym words in the adjective dataset. Looking at the most similar words in
the fasttext embeddings, we can see that the embedding vector for the word
“blue" is more related to the colors, which makes it sense-wise different from
the word “clean" which is more related to cleanliness. If we use these words
to explain the properties of water, then these words are synonyms, however, it
is not evident unless we explicitly consider the context along with word pair.
Note, the phenomenon of multiple senses of a given word is more dominant among
nouns compared with that of verbs and adjectives. This is also evident by a
relatively lower performance of nouns relative to other word classes. For rare
and OOV words, the embedding vectors are not adequately trained and their role
in the end model is no better than the random vectors. This in turn limits the
encoder networks of ICE-NET to encode relation-specific information.

##  7 Conclusion & Future Work

In this work we propose ICE-NET, which uses a set of interlaced encoder
networks to capture the relation-specific properties of antonym and synonym
pairs, i.e., symmetry, transitivity, and trans-transitivity, in order to
perform antonyms vs synonyms distinction task. Results show that ICE-NET
outperforms the existing research by a relative score of up to 1.8% for
F1-measure. Some promising future directions include: (i) using domain-
specific text corpora along with training seeds, (ii) strategy to cut down the
attention weights for the erroneous edges.

##  8 Limitations

Some of the core limitations of the ICE-NET are as follows:

  1. 1.

Nouns and adjectives exhibit multiple different senses, which requires the
need for the contextual information along with the word pair in order to model
them. However, owing to unavailability of multi-sense data sets for the
antonym vs synonym distinction task, current formulation of ICE-NET does not
support multi-sense settings.

  2. 2.

Erroneous edges in the adjacency graphs produced by $M_{init}$ lead to error
propagation. There is a need for an appropriate attention mechanism based on
the semantics of the data.

  3. 3.

The embeddings corresponding to the rare words and OOV tokens need to be
initialized as a weighted average of semantically related tokens rather than
random initialization.

## References

  * Adel and Schütze (2014) Heike Adel and Hinrich Schütze. 2014.  Using mined coreference chains as a resource for a semantic task.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1447–1452.
  * Ali et al. (2023a) Muhammad Ali, Maha Alshmrani, Jianbin Qin, Yan Hu, and Di Wang. 2023a.  Gari: Graph attention for relative isomorphism of arabic word embeddings.  In _Proceedings of ArabicNLP 2023_ , pages 181–190.
  * Ali et al. (2023b) Muhammad Ali, Yan Hu, Jianbin Qin, and Di Wang. 2023b.  Gri: Graph-based relative isomorphism of word embedding spaces.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 11304–11313.
  * Ali et al. (2020) Muhammad Asif Ali, Yifang Sun, Bing Li, and Wei Wang. 2020.  Fine-grained named entity typing over distantly supervised data based on refined representations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7391–7398.
  * Ali et al. (2021) Muhammad Asif Ali, Yifang Sun, Bing Li, and Wei Wang. 2021.  Fine-grained named entity typing over distantly supervised data via refinement in hyperbolic space.  _arXiv preprint arXiv:2101.11212_.
  * Ali et al. (2019) Muhammad Asif Ali, Yifang Sun, Xiaoling Zhou, Wei Wang, and Xiang Zhao. 2019.  Antonym-synonym classification based on new sub-space embeddings.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6204–6211.
  * Baroni and Bisi (2004) Marco Baroni and Sabrina Bisi. 2004.  Using cooccurrence statistics and the web to discover synonyms in a technical language.  In _LREC_.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  _Advances in neural information processing systems_ , 26.
  * Etcheverry and Wonsever (2019) Mathias Etcheverry and Dina Wonsever. 2019.  Unraveling antonym’s word vectors through a siamese-like network.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3297–3307.
  * Goldberg and Levy (2014) Yoav Goldberg and Omer Levy. 2014.  word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method.  _arXiv preprint arXiv:1402.3722_.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Harris (1954) Zellig S Harris. 1954.  Distributional structure.  _Word_ , 10(2-3):146–162.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_.
  * Lin et al. (2003) Dekang Lin, Shaojun Zhao, Lijuan Qin, and Ming Zhou. 2003.  Identifying synonyms among distributionally similar words.  In _IJCAI_ , volume 3, pages 1492–1493.
  * Lobanova et al. (2010) Anna Lobanova, Tom Van der Kleij, and Jennifer Spenader. 2010.  Defining antonymy: A corpus-based study of opposites by lexico-syntactic patterns.  _International Journal of Lexicography_ , 23(1):19–53.
  * Nguyen et al. (2016) Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016.  Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction.  _arXiv preprint arXiv:1605.07766_.
  * Nguyen et al. (2017) Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017.  Distinguishing antonyms and synonyms in a pattern-based neural network.  _arXiv preprint arXiv:1701.02962_.
  * Ono et al. (2015) Masataka Ono, Makoto Miwa, and Yutaka Sasaki. 2015.  Word embedding-based antonym detection using thesauri and distributional information.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 984–989.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Roth and Im Walde (2014) Michael Roth and Sabine Schulte Im Walde. 2014.  Combining word patterns and discourse markers for paradigmatic relation classification.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 524–530.
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks.  In _The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15_ , pages 593–607. Springer.
  * Schwartz et al. (2015) Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015.  Symmetric pattern based word embeddings for improved word similarity prediction.  In _CoNLL_ , volume 2015, pages 258–267.
  * Shwartz et al. (2016) Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016.  Improving hypernymy detection with an integrated path-based and distributional method.  _arXiv preprint arXiv:1603.06076_.
  * Van der Plas and Tiedemann (2006) Lonneke Van der Plas and Jörg Tiedemann. 2006.  Finding synonyms using automatic word alignment and measures of distributional similarity.  In _Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions_ , pages 866–873.
  * Xie and Zeng (2021) Zhipeng Xie and Nan Zeng. 2021.  A mixture-of-experts model for antonym-synonym discrimination.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 558–564.
  * Yang et al. (2014) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014.  Embedding entities and relations for learning and inference in knowledge bases.  _arXiv preprint arXiv:1412.6575_.

##  Appendix A Appendix

###  A.1 Justification for Attentive Convolutions

In this section, we provide intuitive explanations for: (a) the limitations
posed by the distributional pre-trained word embeddings, and (b) why attentive
graph convolutions are a better choice for capturing the relation-specific
properties of the data, (c) computational efficency.

#### (a) Word Embeddings.

We observe the nearest neighbours in the pre-trained word embeddings yield a
blend of multiple different lexico-semantic relations and perform poorly on a
specific task.

Underlying reason is the fact that the pre-trained word embeddings primarily
rely on the distributional hypotheses, i.e., words sharing a similar context
have similar meanings. From linguistic perspective, multiple words with
varying relations (i.e., the antonyms and synonyms, hypernyms etc.,) may be
used interchangeably within a fixed context. This in turn results these
contextually similar words to be embedded close to each other. For example,
nearest neighbours for the word “large" in the Glove embeddings are a
combination of synonyms {“larger", “huge"}, antonyms {“small", “smaller"} and
irrelevant words {“sized"} Ali et al. (2019).

We argue that in order to refine information from the pre-trained embeddings
for a specific task, the graphs provide a better alternative to analyze the
words in combination with semantically related neighbours rather than instant-
level modeling, as explained below.

#### (b) Attentive Graph Convolutions.

The intuitive explanation for the attentive graph convolution network is to
re-commute the representation of the word via attentive aggregation over the
neighbouring words. The core idea is to surround each word by a set of
semantically related neighbours during the graph construction process in order
to smoothen the representation of the word.

It is based on the assumption that within the graphs, i.e., $\text{G}_{t}$ and
$\text{G}_{h}$, the neighbourhood of each word is dominated by its
semantically relevant words compared to the antonyms and/or irrelevant words.
And, recomputing the representation of each word by aggregating information
from the neighbours will result in the final representation to more
semantically coherent compared to the distributional embeddings, as the
contribution of antonyms and other irrelevant words will be down-weighted.
This is illustrated in Figure 2, where the representation of the centre word
is recomputed using a combination of itself and its nearest neighbours
(including synonyms, antonyms and irrelevant words). We use $\xi_{i}$ as the
attentive weight to control its degree of association for the i-th neighbor.

Figure 2: Illustration of attentive Graph Convolution Networks

The final representation of the word, i.e., the output of the attentive graph
convolution network is later used for end-task, i.e., antonyms vs synonyms
distinction.

#### (c) Computational Efficiency.

Another noteworthy aspect is the computational efficiency of the attentive
graph convolutions. Theoretically, for each layer the convolutions need to be
computed between every word pair in the graphs which poses the following
limitations: (a) it is time consuming and computationally inefficient, (b)
accumulating information between all possible word pairs may incorporate noise
in the model training and deteriorate the performance.

To circumvent that we use appropriate thresholds, i.e., $\text{ANT}_{thr}$ and
$\text{SYN}_{thr}$, to select only highly confident candidates for the graph
construction. The values for these thresholds are computed empirically.

These thresholds are helpful in cutting down the un-necessary computations
over the graphs ($\text{G}_{h}$ and $\text{G}_{t}$) by limiting them to the
neighbourhood $\mathcal{N}_{i}$ of each word $i$. Likewise the attention
weights between word pairs ($\xi_{i}$) help in cutting down the noise by
appropriately defining the contribution of the neighboring words. This setting
is different from the graph convolution by Kipf and Welling (2016) that
equally consider the contribution of the neighboring nodes in the graph.

###  A.2 Difference from R-GCN Schlichtkrull et al. (2018)

Schlichtkrull et al. (2018) is proposed R-GCN, i.e. modeling the Relational
data using the Graph Convolutional Networks, and used it for entity
classification and the link prediction task. Although, their problem settings
for the link prediction task looks similar to ICE-NET, however, we emphasize
some key differences as follows:

  1. 1.

R-GCN uses GCN as an encoder to learn the representations followed by DistMult
decoder Yang et al. (2014) for link prediction. Note, this problem setting is
different from ours, as R-GCN primarily deals with asymmetric relations which
can be modeled by linear and/or bilinear transformations. On the other hand,
ICE-NET deals with symmetric relations that cannot be modeled by the existing
KG embedding methods, also shown in Figure 1(b).

  2. 2.

Another justification in favour of the above-mentioned argument is the fact
that currently the performance of the R-GCN is evaluated on KG embedding data
sets, i.e., WN18, FB15k, and these data sets do not include symmetric relation
pairs similar to antonym, synonym pairs etc.

  3. 3.

R-GCN proposes relation-specific feature aggregation for the neighbouring
nodes via a normalization sum. In contrast, we use attention weights to
incorporate the impact of the degree of association of neighboring
words/nodes.

  4. 4.

ICE-NET is the first work that uses multiple encoders to capture the relation-
specific properties of the antonym and synonym pairs (i.e., symmetry,
transitivity and trans-transitivity), to eventually perform the distinction
task in a performance-enhanced way.

###  A.3 Additional Data Sets

We also test the performance of ICE-NET on data sets other than the English.
For this, we used antonym synonym pairs for the Urdu language also used by Ali
et al. (2019). We acquired this data set from the authors of the Distiller Ali
et al. (2019). It is a relatively smaller data set encompassing approximately
750 instances, priorly splitted into 70% train, 25% test and 5% validation
sets. For this data set, we used Fasttext embeddings Grave et al. (2018) for
Urdu as the pre-trained embeddings.

The experimental results in Table 6 show that ICE-NET outperforms the baseline
models and Distiller by Ali et al. (2019) by significant margin. Specifically,
it improve the F1-score by approximately 3.2% compared to the existing state-
of-the art. These results also showcase the language-agnostic nature of ICE-
NET. The same settings can be applied to the antonyms vs synonyms distinction
task for multiple different languages provided with the availability of
distributional embeddings and supervised training data.

Model | P | R | F1
---|---|---|---
Baseline-1 (Random Vectors) | 0.687 | 0.653 | 0.670
Baseline-1 (w/o Graph conv.) | 0.825 | 0.795 | 0.810
Distiller Ali et al. (2019) | 0.897 | 0.867 | 0.881
ICE-NET | 0.905 | 0.915 | 0.910

Table 6: ICE-NET performance evaluation using